Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] duplicate data in geomesa 1.2.1--how? and why?
Hi Ben,

1. The key used to write in geomesa depends on the particular index, but it will always include the feature ID, so if the feature ID changes you will get a duplicate record.

2. If you're using our converter framework, we do have some methods to use an MD5 of the values as the feature ID, which will prevent duplicates. If not, you can do the same thing by generating the feature ID yourself and setting the PROVIDED_FID or USE_PROVIDED_FID hint. We also have a pluggable SPI interface for generating feature IDs when they aren't set. See http://www.geomesa.org/documentation/user/datastores/runtime_config.html#geomesa-feature-id-generator. By default we generate a UUID that includes parts of the Z3 index, so that features grouped in space-time will also be grouped in accumulo. Note that the feature ID is a string and has no inherent restrictions on form.

3. The Z3 index uses the default date attribute to index records, not the insertion time.

Let me know if anything isn't clear!

Thanks,

Emilio

On 02/12/2017 03:28 PM, Benjamin Weaver wrote:

Hi all,


If we ingest, say, the same line of text data twice (by mistake) in Geomesa 1.2.1 we end up with duplicate data in our Accumulo (1.7.2) database. We are ingesting using Gemesa-generated featureIDs (setting our featureBuilder.setFeatureID to NULL without the use of Hints).


A colleague asked me, why are duplicates generated in this case? I realized I did not know.


1. How, exactly, in our configuration of geomesa + Accumulo, is a geomesa row, or record made unique? I know the importance of Accumulo logical rows, but in this case of identical data we would want to insure insertation of only one geomesa record, namely, one instance of our geomesa SimpleFeature.


1a. Are duplicate geomesa rows added because the time at insertion differs? or because different featureIDs are randomly generated on each insertion?



Potentially related questions:


2. How are featureIDs generated by geomesa? I thought randomly, but I read a comment somewhere suggesting that FeatureIDs were created out of an md5 hash of all the values in the feature. But a colleague points out that even if this is so, a featureID does not resemble an md5 hash, so must be composed at least partially by other means


3. A potentially related question: can we create a z3 index by using a data-derived timestamp--not the insertion timestamp-- as the time dimension?


All comments and perspectives are appreciated and welcome!


Ben Weaver





This email (and any attachments) may contain confidential information and is intended solely for the recipient(s) to whom the email is addressed. If you received this email in error, please inform us immediately and delete the email and all attachments without further using, copying or disclosing the information. This email and any attachments are believed to be, but cannot be guaranteed to be, secure or virus-free. Satellite Applications Catapult Limited is registered in England & Wales. Company Number: 7964746. Registered office: Electron Building, Fermi Avenue, Harwell Oxford, Didcot, Oxfordshire OX11 0QR. 

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top