To add one more piece of information, this is what gets logged when we programmatically query the data using the GeoTools API:
INFO org.apache.parquet.filter2.compat.FilterCompat - Filtering using predicate: and(and(and(and(and(gteq(geom.x, -180.0), gteq(geom.y, -90.0)), lteq(geom.x, 180.0)), lteq(geom.y, 90.0)), and(gteq(dtg,
1555198800000), lteq(dtg, 1555203600000))), eq(vin, Binary{"VINSYSTEMTEST1"}))
Hope that helps.
Hi Emilio,
thanks for your hint. However that doesn’t change the behaviour.
Programmatically querying our data using the following filter constraints
-
Bounding box: -180,90,180,90 (the whole world)
-
2019-04-13T23:40:00Z <= dtg <= 2019-04-14T01:00Z
-
vin = VINSYSTEMTEST1
and the code I’ve sent to you with my last message doesn’t return any results (although there should be exactly one).
From debugging the program code I’ve extracted the following CQL filter for the :
dtg BETWEEN 2019-04-13T23:40:00+00:00 AND 2019-04-14T01:00:00+00:00 AND BBOX(geom, -180.0,-90.0,180.0,90.0) AND vin = 'VINSYSTEMTEST1'
Using that very same cql-filter with the command line client “geomesa-fs export” on the test data does return the intended result (exactly one hit).
Could you please have a look at it? Or should we address this to the geotools maintainers?
Thanks,
Christian
Hello,
I notice that in your code, you are using an 'equals' query on VIN, while on the command-line you are using a 'like' with wildcards. Could you try using a consistent predicate between the two? I'm not entirely sure that 'like' is handled correctly in the FSDS,
tbh.
Thanks,
Emilio
Hi Emilio,
This is Christian, a colleague of Markus - the initial requester, who is currently out-of-office. Maybe I can answer your question.
We’re using the GeoTools API to query the data. We basically do it in the same way as described in the geomesa-tutorial
https://github.com/geomesa/geomesa-tutorials/blob/master/geomesa-tutorials-common/src/main/java/org/geomesa/example/transformations/GeoMesaQueryTutorial.java
First, we build an org.geotools.data.Query instance using org.opengis.filter.FilterFactory2. We’re applying temporal and spatial constraints as well as a constraint on a custom attribute called VIN:
Query query =
new Query(typeName);
List<Filter> filters = new ArrayList<>();
FilterFactory2 filterFactory = CommonFactoryFinder.getFilterFactory2();
// temporal filter:
filters.add(filterFactory.between(
filterFactory.property(EventSimpleFeature.DATE_TIME_GROUP),
filterFactory.literal(dateFrom),
filterFactory.literal(dateTo)));
// spatial filter:
filters.add(filterFactory
.bbox(filterFactory.property(EventSimpleFeature.GEOMETRY),
boundingBox.getMinX(), boundingBox.getMinY(),
boundingBox.getMaxX(), boundingBox.getMaxY(), "EPSG:4326"));
// vin filter:
filters.add(filterFactory
.equals(filterFactory.property(EventSimpleFeature.VIN), filterFactory.literal(vin)));
query.setFilter(filterFactory.and(filters));
query.setPropertyNames(fieldNames);
return query;
With that query we then instantiate an org.geotools.data.FeatureReader to read the filtered (according the given query) entries:
FeatureReader<SimpleFeatureType, SimpleFeature> reader =
dataStore.getFeatureReader(query, Transaction.AUTO_COMMIT);
while (reader.hasNext()) {
SimpleFeature feature = reader.next();
// do something with the feature ...
}
Is that sufficient for a further analysis on your side?
Thanks for your support!
Best regards
Christian
Hello,
Could you provide the code you're using to query programmatically?
Thanks,
Emilio
Hello,
In a current project we are using geomesa-fs-data-store for our network-measurements. We are collecting these network-measurements from driving vehicles.
The issues is that we are facing a mismatch with returned data between “geomesa-fs export” and querying geomesa programmatically.
This is the geomesa-fs export-command: ./geomesa-fs export -p ………./geomesa-data-store/ -f ModemState --cql "dtg>=2019-04-13T23:40:00Z AND dtg<=2019-04-14T01:00:00Z AND vin LIKE '%VINSYSTEMTEST1%'"
This is the query from a debug-statement when querying geomesa programmatically:
2019-05-08 16:16:16.073 [http-nio-8130-exec-2] INFO org.apache.parquet.filter2.compat.FilterCompat - Filtering using predicate: and(and(and(and(and(gteq(geom.x, -180.0), gteq(geom.y, -90.0)), lteq(geom.x, 180.0)), lteq(geom.y,
90.0)), and(gteq(dtg, 1555198800000), lteq(dtg, 1555203600000))), eq(vin, Binary{"VINSYSTEMTEST1"}))
While the export returns 1 event, the programmatic query returns 0 events.
The event is found in the programmatic case when extending the search-interval to an earlier time. E.g. from 2019-04-13T23:40:00Z to 2019-04-13T00:40:00Z
I am also attaching the test-data.
We are using:
Geomesa-fs-datastore_2.11:2.2.0
Geomesa-fs-storage-parquet_2.11:2.2.0
Partitioning-Scheme: daily, z2-2bit
The storage-type is parquet on AWS S3.
I hope you can help. Thanks,
Markus Thamm
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.
|
If you are not the addressee, please inform us immediately that you have received this e-mail by mistake, and delete it. We thank you for your support.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users