| Hi Luca,
No problem at all Since we've preferred to write GeoMesa in Scala, I do want to be extra supportive of Java-Scala interop issues. We have written additional Java classes to sweep some of the details under the rug (such exposing certain constants and having Java versions of our MapReduce I/O formats). That said, we probably miss some of those opportunities For routine direct access, it shouldn't matter; the GeoTools API should get most things done. If something can be cleaned up, we're definitely happy to help.
Language issues aside, for your case, it sounds like Spark 1.3.1 may work out. I missed the details, but some folks who are using GeoMesa internally also ran into issues with Spark 1.3.0.
Since Spark is a compiled dependency and we don't package it, I think you'll be able to update the Spark versions in your pom and see your tests work. (That's the very best case scenario.)
If you find that you need to rebuild GeoMesa locally, you can compile against different versions of Spark by changing the version mentioned in the root pom. As a further note/warning about transitive dependencies, we've recently decided to change our Kryo serialization dependency from version 3.0.0 to 2.2.1 (in the geomesa-feature/pom.xml).*
If upgrading Spark doesn't help, you may have to try compiling with Java 6 or 7 (assuming that you don't have too many lambdas to rewrite), or compile your functional Java code with Java 8 and try to compile the rest with Java 7.
* Our Kryo issues boil down to use with Kafka. The Kafka versions we are using has Kryo 2.21. We had been shading Kryo 3.0.0 for SimpleFeature serialization. Anyhow, if you run into anything Kryo/serialization errors, I'd love to hear about them.
Thanks,
Jim
----- Original Message -----
From: lmorandini@xxxxxxxx "Geomesa User discussions" <geomesa-users@xxxxxxxxxxxxxxxx>
To: "Geomesa User discussions" <geomesa-users@xxxxxxxxxxxxxxxx>
Cc:
Sent: Sat, 09 May 2015 19:35:12 +1000
Subject: Re: [geomesa-users] Compilaiton error in Java when applying "map" to an RDD generated via GeoMesaSpark.rdd
On 09/05/15 07:30, Jim Hughes wrote:
> Hi Luca,
>
> Great question. Sadly, I haven't used Spark from Java, nor have I used Java 8.
...
> Defining the scala.Function1 directly should work though:
Indeed it did :) Thanks for taking the time to help me, navigating between Java
and Scala can be rather difficult.
However, I tripped on another Java/Scala issue when running an integration test :(
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.513 sec <<<
FAILURE!
testTransaction(auorg.aurin.tweetcommons.JavaRDDTweetIT) Time elapsed: 16.734
sec <<< ERROR!
java.lang.IllegalArgumentException: null
at
com.esotericsoftware.reflectasm.shaded.orgobjectweb.asm.ClassReader.<init>(Unknown Source)
at
com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.<init>(Unknown Source)
at
comesotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.<init>(Unknown Source)
at
org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$getClassReader(ClosureCleaner.scala:41)
at
org.apache.spark.util.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:84)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:107)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1622)
at org.apache.spark.rdd.RDD.map(RDD.scala:286)
I googled around and found out it is an issue with a library used by Spark:
https://issues.apache.org/jira/browse/SPARK-6152
https://github.com/EsotericSoftware/reflectasm/issues/35
I am currently using 1.0.0-rc.5 and Spark 1.3.0, shall I start using newer versions ?
Regards,
Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
|
