| Re: [geomesa-users] Geomesa is not returning Entire WW data. |
Sorry. This is the actual BBOX we were using:[hadoop@ip-10-0-22-146 ~]$ geomesa-hbase explain -c atlas -f OSMWays -q "BBOX(geometry,-126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= '2019-12-23 06:00:00' AND nextTimestamp > '2019-12-23 06:00:00'"
Planning 'OSMWays' (BBOX(geometry, -126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= 2019-12-23T06:00:00+00:00) AND nextTimestamp > 2019-12-23T06:00:00+00:00
Original filter: (BBOX(geometry, -126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= '2019-12-23 06:00:00') AND nextTimestamp > '2019-12-23 06:00:00'
Hints: bin[false] arrow[false] density[false] stats[false] sampling[none]
Sort: none
Transforms: none
Strategy selection:
Query processing took 16ms for 1 options
Filter plan: FilterPlan[XZ3Index(geometry,ingestionTimestamp)[BBOX(geometry, -126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= 2019-12-23T06:00:00+00:00][nextTimestamp > 2019-12-23T06:00:00+00:00]]
Strategy selection took 1ms for 1 options
Strategy 1 of 1: XZ3Index(geometry,ingestionTimestamp)
Strategy filter: XZ3Index(geometry,ingestionTimestamp)[BBOX(geometry, -126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= 2019-12-23T06:00:00+00:00][nextTimestamp > 2019-12-23T06:00:00+00:00]
Geometries: FilterValues(ArrayBuffer(POLYGON ((-126 27, -126 36, -117 36, -117 27, -126 27))),true,false)
Intervals: FilterValues(List((-∞,2019-12-23T06:00Z]),true,false)
Plan: ScanPlan
Tables: atlas_OSMWays_xz3_geometry_ingestionTimestamp_v2
Ranges (31860): [%00;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%00;%0a;/%00;%00;%00;%00;%00;%00;%00;%02;], [%01;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%01;%0a;/%00;%00;%00;%00;%00;%00;%00;%02;], [%02;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%02;%0a;/%00;%00;%00;%00;%00;%00;%00;%02;], [%03;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%03;%0a;/%00;%00;%00;%00;%00;%00;%00;%02;], [%04;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%04;%0a;/%00;%00;%00;%00;%00;%00;%00;%02;]
Scans (360): [%0d;%0a;/%00;%00;%00;%0d;%cb;I$%95;::%0d;%0a;/%00;%00;%00;%0d;%d9;%e4;%92;M], [(%0a;/%00;%00;%00;%04;%ae;%bc;%92;N::(%0a;/%00;%00;%00;%04;%b3;NI*], [%05;%0a;/%00;%00;%00;%05;%cf;%db;m%b9;::%05;%0a;/%00;%00;%00;%0d;%cb;%00;%00;%04;], [9%0a;/%00;%00;%00;%04;%b3;N%db;r::9%0a;/%00;%00;%00;%04;%d2;%bf;m%bc;], [%19;%0a;/%00;%00;%00;%04;%d2;%c0;%00;%04;::%19;%0a;/%00;%00;%00;%05;%cf;%92;I(]
Column families: d
Remote filters: MultiRowRangeFilter, CqlFilter[(BBOX(geometry, -126.0,27.0,-117.0,36.0) AND ingestionTimestamp <= 2019-12-23T06:00:00+00:00) AND nextTimestamp > 2019-12-23T06:00:00+00:00]
Plan creation took 167ms
Query planning took 454msOn Wed, Jan 15, 2020 at 3:14 PM Jun Cai <joncai2012@xxxxxxxxx> wrote:And here is the output from the explain query CLI:Planning 'OSMWays' ingestionTimestamp <= 2019-12-23T06:00:00+00:00 AND nextTimestamp > 2019-12-23T06:00:00+00:00
Original filter: (BBOX(geometry, -180.0,-90.0,180.0,90.0) AND ingestionTimestamp <= '2019-12-23 06:00:00') AND nextTimestamp > '2019-12-23 06:00:00'
Hints: bin[false] arrow[false] density[false] stats[false] sampling[none]
Sort: none
Transforms: none
Strategy selection:
Query processing took 17ms for 1 options
Filter plan: FilterPlan[XZ3Index(geometry,ingestionTimestamp)[ingestionTimestamp <= 2019-12-23T06:00:00+00:00][nextTimestamp > 2019-12-23T06:00:00+00:00]]
Strategy selection took 2ms for 1 options
Strategy 1 of 1: XZ3Index(geometry,ingestionTimestamp)
Strategy filter: XZ3Index(geometry,ingestionTimestamp)[ingestionTimestamp <= 2019-12-23T06:00:00+00:00][nextTimestamp > 2019-12-23T06:00:00+00:00]
Geometries: FilterValues(List(POLYGON ((-180 -90, 180 -90, 180 90, -180 90, -180 -90))),true,false)
Intervals: FilterValues(List((-∞,2019-12-23T06:00Z]),true,false)
Plan: ScanPlan
Tables: atlas_OSMWays_xz3_geometry_ingestionTimestamp_v2
Ranges (1020): [%00;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%00;%0a;/%00;%00;%00;%09;I$%92;L], [%01;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%01;%0a;/%00;%00;%00;%09;I$%92;L], [%02;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%02;%0a;/%00;%00;%00;%09;I$%92;L], [%03;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%03;%0a;/%00;%00;%00;%09;I$%92;L], [%04;%0a;/%00;%00;%00;%00;%00;%00;%00;%01;::%04;%0a;/%00;%00;%00;%09;I$%92;L]
Scans (60): [%0b;::%0b;%0a;/%00;%00;%00;%11;%00;%00;%00;%02;], [%1d;::%1d;%0a;/%00;%00;%00;%11;%00;%00;%00;%02;], [%0d;::%0d;%0a;/%00;%00;%00;%11;%00;%00;%00;%02;], [%04;::%04;%0a;/%00;%00;%00;%11;%00;%00;%00;%02;], [%00;::%00;%0a;/%00;%00;%00;%11;%00;%00;%00;%02;]
Column families: d
Remote filters: MultiRowRangeFilter, CqlFilter[ingestionTimestamp <= 2019-12-23T06:00:00+00:00 AND nextTimestamp > 2019-12-23T06:00:00+00:00]
Plan creation took 108ms
Query planning took 329msOn Wed, Jan 15, 2020 at 3:08 PM Jun Cai <joncai2012@xxxxxxxxx> wrote:Hi Emilio,This is Jun. I am working with Amit on this issue. Here is the query summary from our log:15 Jan 2020 22:10:40,583 org.locationtech.geomesa.utils.audit.AuditLogger$: {"storeType":"hbase","typeName":"OSMWays","date":1579126240583,"user":"unknown","filter":"(BBOX(geometry, 0.0,45.0,9.0,54.0) AND ingestionTimestamp \u003c\u003d \u00272019-12-23 06:00:00\u0027) AND nextTimestamp \u003e \u00272019-12-23 06:00:00\u0027","hints":"RETURN_SFT\u003d*geometry:LineString:srid\u003d4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String","planTime":31,"scanTime":1139426,"hits":82089533,"deleted":false}We are firing the query via GeoTools interface. The missing features are not consistent between runs. Later runs always tend to have more data than previous ones.Thanks,JunOn Wed, Jan 15, 2020 at 2:51 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:Sorry, I pointed you to the wrong mailing list. I've included the right one now.
The bug I am thinking of was related to partitioned tables, but as you aren't using partitioning then that wouldn't affect you. Can you provide the explain plan[1] for the query? Are you querying this through geoserver? Are the missing features consistent when you run the same filter?
Thanks,
Emilio
[1]: https://www.geomesa.org/documentation/user/datastores/query_planning.html#explaining-query-plans
On 1/15/20 5:42 PM, Amit Srivastava wrote:
Hi Emilio,
Can you also point me to the bug which got fixed in 2.3.2. and 2.4.0?
On Wed, Jan 15, 2020 at 2:37 PM Amit Srivastava <amit.bit96@xxxxxxxxx> wrote:
Thanks Emilio for quick response. Below are the required details. Regarding update from 2.3.2 to 2.4.0, we can upgrade it but it will require some effort and time which I want to avoid for now.
Exact filter which I am using: BBOX(geometry,-180.0,-90.0,180.0,90.0) AND ingestionTimestamp <= '2019-12-23 06:00:00' AND nextTimestamp > '2019-12-23 06:00:00'hbase(main):002:0> scan 'atlas' ROW COLUMN+CELL OSMNodes~attributes column=m:v, timestamp=1577234629875, value=*geometry:Point:srid=4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.z.splits='60',geomesa.indices='z3:6:3:geometry:ingestionTimestamp,id:4:3:' OSMNodes~stats-date column=m:v, timestamp=1577234629875, value=2019-12-25T00:43:49.836Z OSMNodes~table.id.v4 column=m:v, timestamp=1577234646266, value=atlas_OSMNodes_id_v4 OSMNodes~table.z3.geometry.ingestionTimestamp.v6 column=m:v, timestamp=1577234629897, value=atlas_OSMNodes_z3_geometry_ingestionTimestamp_v6 OSMRelationMembers~attributes column=m:v, timestamp=1577234747359, value=ingestionTimestamp:Timestamp,relationId:String,featureTypeId:String,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.indices='attr:8:3:relationId:ingestionTimestamp,attr:8:3:featureTypeId:ingestionTimestamp,id:4:3:' OSMRelationMembers~stats-date column=m:v, timestamp=1577234747359, value=2019-12-25T00:45:47.320Z OSMRelationMembers~table.attr.featureTypeId.ingestionTimestamp.v8 column=m:v, timestamp=1577234751575, value=atlas_OSMRelationMembers_attr_featureTypeId_ingestionTimestamp_v8 OSMRelationMembers~table.attr.relationId.ingestionTimestamp.v8 column=m:v, timestamp=1577234747380, value=atlas_OSMRelationMembers_attr_relationId_ingestionTimestamp_v8 OSMRelationMembers~table.id.v4 column=m:v, timestamp=1577234755743, value=atlas_OSMRelationMembers_id_v4 OSMRelations~attributes column=m:v, timestamp=1577234692949, value=*geometry:MultiPolygon:srid=4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.z.splits='60',geomesa.indices='xz3:2:3:geometry:ingestionTimestamp,id:4:3:' OSMRelations~stats-date column=m:v, timestamp=1577234692949, value=2019-12-25T00:44:52.909Z OSMRelations~table.id.v4 column=m:v, timestamp=1577234710295, value=atlas_OSMRelations_id_v4 OSMRelations~table.xz3.geometry.ingestionTimestamp.v2 column=m:v, timestamp=1577234692970, value=atlas_OSMRelations_xz3_geometry_ingestionTimestamp_v2 OSMTestNodes~attributes column=m:v, timestamp=1577143864743, value=*geometry:Point:srid=4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.z.splits='60',geomesa.indices='z3:6:3:geometry:ingestionTimestamp,id:4:3:' OSMTestNodes~stats-date column=m:v, timestamp=1577143864743, value=2019-12-23T23:30:56.200Z OSMTestNodes~table.id.v4 column=m:v, timestamp=1577143890005, value=atlas_OSMTestNodes_id_v4 OSMTestNodes~table.z3.geometry.ingestionTimestamp.v6 column=m:v, timestamp=1577143864809, value=atlas_OSMTestNodes_z3_geometry_ingestionTimestamp_v6 OSMWayNodes~attributes column=m:v, timestamp=1577234724952, value=ingestionTimestamp:Timestamp,wayId:String,nodeId:String,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.indices='attr:8:3:wayId:ingestionTimestamp,attr:8:3:nodeId:ingestionTimestamp,id:4:3:' OSMWayNodes~stats-date column=m:v, timestamp=1577234724952, value=2019-12-25T00:45:24.908Z OSMWayNodes~table.attr.nodeId.ingestionTimestamp.v8 column=m:v, timestamp=1577234729162, value=atlas_OSMWayNodes_attr_nodeId_ingestionTimestamp_v8 OSMWayNodes~table.attr.wayId.ingestionTimestamp.v8 column=m:v, timestamp=1577234724973, value=atlas_OSMWayNodes_attr_wayId_ingestionTimestamp_v8 OSMWayNodes~table.id.v4 column=m:v, timestamp=1577234733300, value=atlas_OSMWayNodes_id_v4 OSMWays~attributes column=m:v, timestamp=1577234660315, value=*geometry:LineString:srid=4326,ingestionTimestamp:Timestamp,nextTimestamp:Timestamp,serializerVersion:String,featurePayload:String;geomesa.index.dtg='ingestionTimestamp',geomesa.z.splits='60',geomesa.indices='xz3:2:3:geometry:ingestionTimestamp,id:4:3:' OSMWays~stats-date column=m:v, timestamp=1577234660315, value=2019-12-25T00:44:20.278Z OSMWays~table.id.v4 column=m:v, timestamp=1577234677610, value=atlas_OSMWays_id_v4 OSMWays~table.xz3.geometry.ingestionTimestamp.v2 column=m:v, timestamp=1577234660337, value=atlas_OSMWays_xz3_geometry_ingestionTimestamp_v2
On Wed, Jan 15, 2020 at 2:20 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hello,
If possible, can you upgrade to 2.3.2 or 2.4.0? There is at least one bug that may cause that behavior, that has been fixed in those versions. Aside from that, it would be helpful if you could provide the exact filters you're using, and the data from your catalog table in hbase, as output by scanning it through the hbase shell.
Thanks,
Emilio
On 1/15/20 2:58 PM, Amit Srivastava wrote:
Hi All,
I am using Geomesa v2.3.0 with HBase. While running BBOX export query on Geomesa the results are inconsistent. I am seeing few features are missing in export. Can someone help in debugging this?
What investigation I have done so far?
- Compare the data ingested vs exported. Found below diff for it.
- Checked the Application logs which is calling Geomesa via GeoTools interface. I am not seeing any error in the application log.
- Checked HBase related metrics and logs. I am not seeing any error during the time when we performed the execution.
- Re-Run the query on a smaller BBOX which has missing data, the reported missing data in 2nd export got returned by Geomesa in 3rd query. Hence I can't find repreoducable steps for this issue.
Missing Features In 2nd Export Run:
Missing Features in 1st Export Run:
--
--