Re: [geomesa-users] NoSuchMethodError: thrift.TabletClientService$Client

[Diethard Steiner <diethard.steiner@xxxxxxxxx>]

Ok, I resolved the issue. The tag name is different (includes the Scala version):

git checkout -b geomesa-1.3.1 geomesa_2.11-1.3.1

Please update the docu.

On Wed, May 10, 2017 at 10:18 PM, Diethard Steiner <diethard.steiner@xxxxxxxxx> wrote:

Thanks a lot Jim for clarifying this! I did a search on the installation doc and it is indeed mentioned there as well (so apologies for my oversight), however, only at the very end. If I may make one suggestion, please move this section to the beginning.

I tried to build from source as per docu, which mentions this command:

$ git checkout tags/geomesa-1.3.1 -b geomesa-1.3.1

I can find the tag mentioned here, but not the branch:

$ git branch -a
* master
  remotes/origin/1.1.x
  remotes/origin/1.2
  remotes/origin/1.2.6.x
  remotes/origin/1.2.7.4
  remotes/origin/1.2.7.x
  remotes/origin/HEAD -> origin/master
  remotes/origin/_old/master
  remotes/origin/_old/wip_hadoop_2_0_accumulo_1_5_0
  remotes/origin/accumulo1.4/1.x
  remotes/origin/adf_fcr_arrow_reads
  remotes/origin/codecov
  remotes/origin/converter_array
  remotes/origin/dynamodb
  remotes/origin/faster_classpath
  remotes/origin/fcr_acc18_thrift
  remotes/origin/fcr_aws_geodocker_docs
  remotes/origin/fcr_hbase_pushdown
  remotes/origin/fcr_hbase_pushdownz3
  remotes/origin/fcr_spark_packaging
  remotes/origin/fcr_wps_stats
  remotes/origin/hbase-mrrf
  remotes/origin/hbase_and_arrow
  remotes/origin/hbase_filter_start
  remotes/origin/master
  remotes/origin/maxconcurrentqueries
  remotes/origin/scala-parsers-1.0.5
  remotes/origin/sparkexperiments
  remotes/origin/sparksql
  remotes/origin/travis2
  remotes/origin/travis_test
  remotes/origin/travismultithreads
  remotes/origin/wip-geomesa-1.2.1.x
  remotes/origin/wip_ssi
  remotes/origin/xmlperf

Am I missing something here?

Best regards,

Diethard

On Wed, May 10, 2017 at 4:27 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:

Hi Diethard,

It is a mismatch in the Accumulo version.  GeoMesa 1.3.x depends on Accumulo 1.7.x.  To get around this, you'll need to build GeoMesa locally using the Accumulo 1.8 profile (mvn clean install -Paccumulo-1.8) and you'll need to make sure that sbt picks up the artifacts which you have built locally. 

As SBT uses the pom's, it may pull the version of Accumulo mentioned in them.  With that, the fat jar created can easily find a wrong version of Accumulo. 

As a bold alternative, jar files are just zips, so one can always try out jar surgery to fix up a fat jar before fighting long battles with a build tool;).

Cheers,

Jim

On 05/10/2017 10:20 AM, Diethard Steiner wrote:

Hi,

I am trying to follow the Spark Examples (http://www.geomesa.org/documentation/tutorials/spark.html), and while I managed to create the fat jar successfully, I get this error when running the spark job:

```
NoSuchMethodError: org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.sendBaseOneway
```

(I did some research online which indicated that there might be some version problem, however, I seem to be using the correct versions ... )

Local Dev Env:

Linux Fedora 25

Java 8

Hadoop 2.8.0 (pseudo distributed setup)

Accumulo 1.8.1
geomesa-accumulo-dist_2.11-1.3.1
spark-2.1.0-bin-hadoop2.7

I am running Spark in standalone mode (so not via Yarn).

I am not using maven, but sbt. Here is my build.sbt:

```
name := "GeoMesaSparkExample"

version := "0.1"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.0.2" % "provided"
  , "org.apache.spark" %% "spark-catalyst" % "2.0.2" % "provided"
  , "org.apache.spark" %% "spark-sql" % "2.0.2" % "provided"
/**  , "org.apache.spark" %% "spark-yarn" % "2.0.2" % "provided" **/
  , "org.locationtech.geomesa" %% "geomesa-accumulo-datastore" % "1.3.1"
  , "org.locationtech.geomesa" %% "geomesa-accumulo-spark" % "1.3.1"
  , "ch.qos.logback" % "logback-classic" % "1.1.7"
  , "com.typesafe.scala-logging" %% "scala-logging" % "3.5.0"
  , "org.apache.accumulo" % "accumulo-core" % "1.8.1"
)

assemblyMergeStrategy in assembly := {
  case path => {
    val strategy = (assemblyMergeStrategy in assembly).value(path)
    if (strategy == MergeStrategy.deduplicate) {
      MergeStrategy.first
    } else {
      strategy
    }
  }
}
```

CountByDay.scala

```
package examples

import java.text.SimpleDateFormat

import org.apache.hadoop.conf.Configuration
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
import org.geotools.data.{DataStoreFinder, Query}
import org.geotools.factory.CommonFactoryFinder
import org.geotools.filter.text.ecql.ECQL
import org.locationtech.geomesa.accumulo.data.AccumuloDataStore
import org.locationtech.geomesa.spark.GeoMesaSpark
import org.opengis.feature.simple.SimpleFeature

import scala.collection.JavaConversions._

object CountByDay {

  val params = Map(
    "instanceId" -> "MY_INSTANCE",
    // zookeeper info can be found in ACCUMULO_HOME/conf/accumulo-site.xml
    "zookeepers" -> "localhost:2181",
    "user"       -> "root",
    "password"   -> "password",
    // no authentication required for local dev env setup
    //"auths"      -> "USER,ADMIN",
    "tableName"  -> "myNamespace.gdelt"
    )

  // see geomesa-tools/conf/sfts/gdelt/reference.conf
  val typeName = "gdelt"
  val geom     = "geom"
  val date     = "dtg"

  val bbox   = "-80, 35, -79, 36"
  val during = "2014-01-01T00:00:00.000Z/2014-01-31T12:00:00.000Z"

  // val filter = s"bbox($geom, $bbox) AND $date during $during"
  val filter = s"bbox($geom, $bbox)"

  def main(args: Array[String]) {
    // Get a handle to the data store
    val ds = DataStoreFinder.getDataStore(params).asInstanceOf[AccumuloDataStore]

    // Construct a CQL query to filter by bounding box
    val q = new Query(typeName, ECQL.toFilter(filter))

    // Configure Spark
    val conf = new SparkConf().setAppName("testSpark")
    val sc = SparkContext.getOrCreate(conf)

    // Get the appropriate spatial RDD provider
    val spatialRDDProvider = GeoMesaSpark(params)

    // Get an RDD[SimpleFeature] from the spatial RDD provider
    val rdd = spatialRDDProvider.rdd(new Configuration, sc, params, q)

    // Collect the results and print
    countByDay(rdd).collect().foreach(println)
    println("\n")

    ds.dispose()
  }

  def countByDay(rdd: RDD[SimpleFeature], dateField: String = "dtg") = {
    val dayAndFeature = rdd.mapPartitions { iter =>
      val df = new SimpleDateFormat("yyyyMMdd")
      val ff = CommonFactoryFinder.getFilterFactory2
      val exp = ff.property(dateField)
      iter.map { f => (df.format(exp.evaluate(f).asInstanceOf[java.util.Date]), f) }
    }
    dayAndFeature.map( x => (x._1, 1)).reduceByKey(_ + _)
  }
}
```

I submit the jar like so:

```
spark-submit --master local[4] \
  --class examples.CountByDay \
  target/scala-2.11/GeoMesaSparkExample-assembly-0.1.jar

```

Can someone please point out what I am doing wrong here?

Thanks,

Diethard

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users