For sure I see that the key changes on the date (it is the date that I am using for the simple feature type to be part of that z3 index) but when I issue the WFS call (and provide date to try and trigger z3)
it still just returns one record and not the duplicates from what I can tell.
I was using the java api to add features so I was getting a FeatureStore off of the DataStore and so I am not sure that is what you mean about using an updating feature writer. Is there a way to do that when
accessing ingest via the java api? Or is there another method to do what you are suggesting? So where I was left was try to identify the modification records and then really remove and then add them again vs records that are new. So how does one use an
updating feature writer.
Thanks,
Diane
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx]
On Behalf Of Emilio Lahr-Vivaz
Sent: Friday, February 10, 2017 2:07 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] question about updates to data and duplicates
Hi Diane,
A 'key' in accumulo consists of a row, a column family, a column qualifer, a timestamp and a visibiility marking. In general, timestamps are 'hidden' by the versioning iterator, which always applied by default. Thus, inserting a key with the same row/cf/cq/vis
but a newer timestamp will overwrite an older timestamp. Technically the older value is still there, but it will be hidden and eventually compacted out by accumulo.
The key used to write in geomesa depends on the particular index, but it will always include the feature ID (FID). The Z3 index also includes the default date and geometry.
In general, to be safe, we recommend always using an updating feature writer if you are modifying an existing feature. That can be slow though, so sometimes you can get away with appending, and relying on the versioning iterator to clean up the duplicates.
But if any of the indexed values (date, geometry, indexed attributes) change, you will end up with a different key in some of the indices, and thus duplicate data.
I'm not sure why you would be seeing no change with the date modification - maybe it's not the date being indexed? I wrote a quick script to check the z3 row keys generated for a few different inputs. You can see the key is changing each time:
original
Sat Dec 31 19:00:00 EST 2011, POINT (45 45)
%00;%08;%8f;5`%90;H$%12;%09; fid0
lon +1
Sat Dec 31 19:00:00 EST 2011, POINT (46 45)
%00;%08;%8f;5`%90;Xl6%09;!fid0
lat +1
Sat Dec 31 19:00:00 EST 2011, POINT (45 46)
%00;%08;%8f;5`%91;L%a6;R%09;2fid0
dtg +1h
Sat Dec 31 20:00:00 EST 2011, POINT (45 45)
%00;%08;%8f;5`%92;@$%90;%09;$fid0
dtg +1d
Sun Jan 01 19:00:00 EST 2012, POINT (45 45)
%00;%08;%8f;<D%02;%01;%00;%80;@ fid0
On 02/10/2017 01:01 PM, Diane Griffith wrote:
We have data that will see updates to its data. These updates can be to any of the fields including the time field as well as the latitude and longitude fields used to create the default Point geometry.
What we have observed is if the datetime changed, even over a month difference, we still just got one record back via WFS regardless of how many were still in the z3 table and the record got updated the z2 and the records tables in accumulo.
Again if we updated the latitude value, even a slight change to it, and added the feature again (instead of modify the feature or remove then add), then we got 2 copies of the record that have the same id field that we hint to use for FID.
I found an old post that sort of talked to this around how a Versioning Iterator is configured to return 1 record for scan time and both minor and major compactions and that had to so specifically for the Accumulo side of the question.
Then the response went on to talk to the GeoMesa side of it and asked do the 2 copies of a SimpleFeature have the same Feature ID? If yes then GeoMesa will write the same Accumulo keys for the data. If not then different keys will be written. So what does
that mean, Feature ID…is that the field hinted to use for FID or is that the 3 pieces that make up the index key?
So why does time field differences not impact duplicates coming back from WFS requests (using z3 index) but changes in latitude and longitude do? So is the recommendation if updates to the latitude/longitude/point data will happen then
to identify the record is changed and either modify or remove/add?
We are seeing this in GeoMesa 1.2.7.2 against Accumulo 1.6.2.
Thanks,
Diane Griffith
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users