Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
Hmm, sounds interesting. I don't have any experience with DataFlow either. Would you have to write to HBase and then import the files?

For Accumulo we have some custom map/reduce output formats that delegate to the AccumuloOutputFormat. We haven't implemented it, but we could easily extend the AccumuloFileOutputFormat instead to write out files and then bulk import them into an instance. But as far as I know, HBase/BigTable doesn't have a output format like that. So instead we just use the normal client connections from our distributed mappers.

On 01/24/2017 04:02 PM, Damiano Albani wrote:
On Tue, Jan 24, 2017 at 4:35 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hmm, ok I'm not sure about the guava jar... but I believe that error means that you can't connect to the BigTable instance you're trying to. Do you have the google-cloud-sdk installed? Can you hit your bigtable instance from there?

Yes, I do have the Cloud SDK installed and configured.
And I have a Maven-based project of mine which can connect just fine to my BigTable instance.
So I don't understand why it doesn't via geomesa-bigtable...
I'll continue investigating.

By the way, looking at the Github project with examples provided by Google for BigTable, I saw this:
  • Import HBase Sequence files Import HBase sequence files directly to Cloud Bigtable using Dataflow.
I have absolutely not experience with Dataflow but that description made me think of what you said previously:

There is no way to e.g. write database files directly, so whatever ingest mechanism you use will end up using the same client writers.

Could this be an option, then? Create HBase files and inserting them directly into BigTable via Dataflow?

--
Damiano Albani
Geodan


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top