Re: [geomesa-users] Advise on Geomesa data streaming

[Jim Hughes <jnh5y@xxxxxxxx>]

HI José,

Are you looking to 'stream' the data from GeoMesa into GraphHopper?

If the goal is to read from GeoMesa and write to an external source, you don't necessarily need to sort out a streaming data solution.  Instead you could call rdd.mapPartitions and inside that block instantiate a connection to the external datasource, and write the data out.

Other than that, if you are moving data from one system to another, you could skip Spark and write Java/Scala code directly.  Additionally, you could use the GeoMesa tools to 'export' data to files and then load those up.  (Lots of options!)

Cheers,

Jim

On 03/24/2017 12:25 PM, Jose Bujalance wrote:

Hi,

I have a Geomesa-Accumulo datastore containing 80M rows, and I would like to use that data to update a routing graph (GraphHopper).

In order to do that, I have to get all the Geomesa data row by row. I am working on Java with the Geomesa-Spark API (version 1.3.1). Obviously, I can't instantiate a Java list with 80M elements using the RDD.collect() method, because of obvious memory limitations.

So I am looking for a way to stream my Geomesa data row by row in an efficient way.

Right now, I am thinking about some tools like Spark Streaming, Geomesa-Stream, or Storm, but I have never used non of them, so I don't know if that's what I need and which tool would be more adapted to my problem.

Any idea?

Thanks a lot,

José

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users