Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Geomesa Spark build : ClassNotFoundException: org.apache.commons.io.Charsets
Hi,
I just modified some versions on the master pom to match my environnement. I may try your solution later. I finally decided to use maven on java to get the builded jars instead. However, now I'm not sure how to use the geomesa-spark integration in java, as I haven't found any example code for java in the documentation. I am trying to "translate" this simple scala code into java :

// Datastore params
val dsParams = Map("bigtable.table.name" -> "Geoloc_Praxedo_catalog")

// set SparkContext
val conf = new SparkConf().setMaster("local[*]").setAppName("testSpark")
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.kryo.registrator", classOf[GeoMesaSparkKryoRegistrator].getName)
val sc = SparkContext.getOrCreate(conf)

// create RDD with a geospatial query using Geomesa functions
val spatialRDDProvider = GeoMesaSpark(dsParams)
val filter = ECQL.toFilter("BBOX(coords, 48.815215, 2.249294, 48.904295, 2.419337)")
val query = new Query("history_feature_nodate", filter)
val resultRDD = spatialRDDProvider.rdd(new Configuration, sc, dsParams, query)

resultRDD.count

Is there any useful link or documentation to understand how the geomesa-spark java api works ?

Thanks a lot,
José

2017-03-06 15:21 GMT+01:00 Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>:
Hi,

Did you modify the pom? The module does build fine in master. Commons io looks like it should be pulled in from geomesa-convert-common:

https://github.com/locationtech/geomesa/blob/master/geomesa-convert/geomesa-convert-common/pom.xml#L34-L37

At any rate, you can add that snippet to the pom.xml for the failing module to explicitly include it. That should fix the error.

Thanks,

Emilio


On 03/06/2017 05:54 AM, Jose Bujalance wrote:
Hi,

I am trying to build the geomesa-spark module from source but the build fails when building the GeoMesa Spark Converter RDD Provider module. Some of the tests fail, giving the next error:

Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 4.779 sec <<< FAILURE! - in org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest
The ConverterSpatialRDDProvider should::read from local files(org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest)  Time elapsed: 1.686 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
        at org.apache.hadoop.security.Credentials.<clinit>(Credentials.java:222)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334)
        at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:184)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProvider.rdd(ConverterSpatialRDDProvider.scala:64)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$7.apply(ConverterSpatialRDDProviderTest.scala:50)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$7.apply(ConverterSpatialRDDProviderTest.scala:49)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.Charsets
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at org.apache.hadoop.security.Credentials.<clinit>(Credentials.java:222)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334)
        at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:184)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProvider.rdd(ConverterSpatialRDDProvider.scala:64)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$7.apply(ConverterSpatialRDDProviderTest.scala:50)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$7.apply(ConverterSpatialRDDProviderTest.scala:49)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

The ConverterSpatialRDDProvider should::read from local files with filtering(org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest)  Time elapsed: 0.319 sec  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.Credentials
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334)
        at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:184)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProvider.rdd(ConverterSpatialRDDProvider.scala:64)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$11.apply(ConverterSpatialRDDProviderTest.scala:58)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$11.apply(ConverterSpatialRDDProviderTest.scala:56)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

The ConverterSpatialRDDProvider should::read from a local file using Converter Name lookup(org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest)  Time elapsed: 0.083 sec  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.Credentials
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:334)
        at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:184)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProvider.rdd(ConverterSpatialRDDProvider.scala:64)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$15.apply(ConverterSpatialRDDProviderTest.scala:70)
        at org.locationtech.geomesa.spark.converter.ConverterSpatialRDDProviderTest$$anonfun$2$$anonfun$apply$15.apply(ConverterSpatialRDDProviderTest.scala:64)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

I don't see how can I install the Commons IO library and let the test access it. Does anybody have succesfully build Geomesa Spark and have an idea on how to solve this error ?

Thanks, 
José


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top