Re: [geomesa-users] Error when querying date/time range

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geomesa-users] Error when querying date/time range

From: Joel Folkerts <joel.folkerts@xxxxxxxxx>
Date: Mon, 21 Aug 2017 13:37:15 -0500
Delivered-to: geomesa-users@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-users>
List-help: <mailto:geomesa-users-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=unsubscribe>

Thank you - that did the trick!!

-Joel

On Mon, Aug 21, 2017 at 12:48 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:

Hi Joel,

Ok, those results seem fairly reasonable. The query you are executing is going against the default z3 index, which interleaves time, latitude and longitude. Since only time is specified, the interleaving is fairly coarse, causing the imprecise results. To get better results, there are a few things you can do:

1. Disable loose bounding box: http://www.geomesa.org/documentation/user/datastores/query_config.html#loose-bounding-box
This will cause the CQL filter to be applied to all results, ensuring accuracy at the cost of some additional processing time
2. Add an attribute index on the date field: http://www.geomesa.org/documentation/user/datastores/index_basics.html#attribute-index
This will index the date field by itself, so the coarseness of the interleaved geometries won't matter
3. Include a geometry when querying - this will improve the interleaving precision

hope that helps,

Thanks,

Emilio
On 08/21/2017 01:10 PM, Joel Folkerts wrote:
Hi Emilio-

Yes, I set up the co-processors and can see them being registered in the region servers when HBase starts. I've cleared the data from the last email so the dates are a little different but am still seeing the problem. Note I am only performing a temporal filter:

return CQL.toFilter("(When DURING 2014-12-30T00:00:00.000Z/2014-12-30T23:59:59.000Z)"); // line 175 of HBaseQuickStart.java

OUTPUT:

Creating feature-type (schema): QuickStart

Creating new features

Inserting new features

Submitting query

[2017-08-21 12:01:59,305] TRACE Explainer: Planning 'QuickStart' When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00

[2017-08-21 12:01:59,306] TRACE Explainer: Original filter: When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00

[2017-08-21 12:01:59,309] TRACE Explainer: Hints: bin[false] arrow[false] density[false] stats[false] map-aggregate[false] sampling[none]

[2017-08-21 12:01:59,311] TRACE Explainer: Sort: none

[2017-08-21 12:01:59,312] TRACE Explainer: Transforms: None

[2017-08-21 12:01:59,315] TRACE Explainer: Strategy selection:

[2017-08-21 12:01:59,396] TRACE Explainer: Query processing took 68ms and produced 1 options

[2017-08-21 12:01:59,400] TRACE Explainer: Filter plan: FilterPlan[HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]]

[2017-08-21 12:01:59,400] TRACE Explainer: Strategy selection took 3ms for 1 options

[2017-08-21 12:01:59,402] TRACE Explainer: Strategy 1 of 1: HBaseZ3Index

[2017-08-21 12:01:59,402] TRACE Explainer: Strategy filter: HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]

[2017-08-21 12:01:59,427] TRACE Explainer: Geometries: FilterValues(List(POLYGON ((-180 -90, 0 -90, 180 -90, 180 90, 0 90, -180 90, -180 -90))),false)

[2017-08-21 12:01:59,443] TRACE Explainer: Intervals: FilterValues(List([2014-12-30T00:00:01.000Z,2014-12-30T23:59:58.000Z]),false)

[2017-08-21 12:01:59,614] TRACE Explainer: Plan: org.locationtech.geomesa.hbase.data.ScanPlan

[2017-08-21 12:01:59,615] TRACE Explainer: Table: geomesa_QuickStart_z3

[2017-08-21 12:01:59,616] TRACE Explainer: Ranges (13): { [0943836000000,094326104], [094326108000000,094344-80], [094344-76000000,094362-8], [094362-4000000,19432640], [19432644000000,194344112] }

[2017-08-21 12:01:59,616] TRACE Explainer: Filter: HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]

[2017-08-21 12:01:59,617] TRACE Explainer: Plan creation took 210ms

[2017-08-21 12:01:59,617] TRACE Explainer: Query planning took 356ms

1. Bierce|38782|Mon Dec 29 15:06:42 CST 2014|POINT (-79.1341120369213 38.123362284877786)|null

2. Addams|69945|Mon Dec 29 16:13:19 CST 2014|POINT (-78.91343013821982 37.35509231261945)|null

3. Clemens|40670|Mon Dec 29 17:16:40 CST 2014|POINT (-78.91131901031984 37.43714158170664)|null

4. Bierce|94117|Mon Dec 29 17:55:19 CST 2014|POINT (-79.08357338783861 37.87162635613247)|null

5. Clemens|73757|Mon Dec 29 18:17:38 CST 2014|POINT (-79.24588872978585 37.482763349262775)|null

6. Addams|83556|Mon Dec 29 16:21:40 CST 2014|POINT (-79.06874376441775 38.21485507393958)|null

7. Addams|37206|Mon Dec 29 16:37:28 CST 2014|POINT (-78.98274149666337 38.44403421353682)|null

8. Addams|81066|Mon Dec 29 16:19:43 CST 2014|POINT (-79.40980568780253 38.897938151457055)|null

9. Bierce|18970|Mon Dec 29 18:06:55 CST 2014|POINT (-78.92545983348037 38.45330541840287)|null

10. Bierce|19597|Mon Dec 29 17:46:16 CST 2014|POINT (-78.76642524544258 38.97293392198045)|null

11. Addams|84432|Mon Dec 29 19:52:53 CST 2014|POINT (-78.84949257612135 38.62687651755251)|null

12. Bierce|81487|Mon Dec 29 21:31:59 CST 2014|POINT (-78.84201779048216 37.6195979334259)|null

13. Bierce|6892|Mon Dec 29 22:48:33 CST 2014|POINT (-79.23041357169569 37.914511293511964)|null

14. Addams|31344|Mon Dec 29 21:51:25 CST 2014|POINT (-78.80828107960899 38.28339974006503)|null

15. Clemens|92147|Mon Dec 29 13:32:30 CST 2014|POINT (-78.603118982262 37.338821484984)|null

16. Addams|81099|Mon Dec 29 13:47:16 CST 2014|POINT (-78.34873124129228 37.869845432416696)|null

17. Bierce|21997|Mon Dec 29 14:50:47 CST 2014|POINT (-78.21858633518944 37.60727614841389)|null

18. Addams|14172|Mon Dec 29 14:02:31 CST 2014|POINT (-77.51306908547258 38.66047648630169)|null

19. Bierce|96154|Mon Dec 29 14:56:44 CST 2014|POINT (-78.24796068682105 38.29747516622206)|null

20. Addams|18465|Mon Dec 29 14:54:13 CST 2014|POINT (-77.78081152620675 38.62463134443681)|null

21. Bierce|88666|Mon Dec 29 15:27:30 CST 2014|POINT (-78.52489764211398 38.821220700459364)|null

22. Addams|86223|Mon Dec 29 17:20:50 CST 2014|POINT (-78.58423279816995 37.898586954800514)|null

23. Addams|2718|Mon Dec 29 17:28:41 CST 2014|POINT (-77.60064101336147 37.86598231978402)|null

24. Clemens|94388|Mon Dec 29 17:16:06 CST 2014|POINT (-78.20963255855335 38.112068407498505)|null

25. Addams|78573|Mon Dec 29 16:47:26 CST 2014|POINT (-77.86294925027478 38.052799096517994)|null

26. Clemens|73388|Mon Dec 29 17:26:44 CST 2014|POINT (-78.28175432568574 38.18566898567634)|null

27. Clemens|3755|Mon Dec 29 17:55:06 CST 2014|POINT (-78.1146776142142 38.52977882654116)|null

28. Clemens|96374|Mon Dec 29 18:40:59 CST 2014|POINT (-78.2157387189368 38.70639028573238)|null

29. Bierce|41596|Mon Dec 29 18:46:51 CST 2014|POINT (-77.58787426740285 38.06434690726756)|null

30. Addams|60429|Mon Dec 29 19:17:06 CST 2014|POINT (-77.73007552782313 38.958459158952486)|null

31. Bierce|48652|Mon Dec 29 20:40:31 CST 2014|POINT (-78.3878356380678 38.33104619503754)|null

32. Clemens|62126|Mon Dec 29 23:33:03 CST 2014|POINT (-77.68651693444643 37.182529764756424)|null

33. Bierce|55858|Mon Dec 29 21:34:36 CST 2014|POINT (-77.84727805879886 38.43210576427531)|null

34. Bierce|88897|Mon Dec 29 21:43:35 CST 2014|POINT (-77.62517310497805 38.400072514444865)|null

35. Bierce|31807|Mon Dec 29 21:59:12 CST 2014|POINT (-77.63439426590955 38.464419570900006)|null

36. Bierce|36124|Mon Dec 29 22:41:06 CST 2014|POINT (-77.79562898260852 38.46646234067417)|null

37. Bierce|17185|Mon Dec 29 23:40:14 CST 2014|POINT (-77.57094565071165 38.98195919697627)|null

38. Clemens|8723|Tue Dec 30 00:28:20 CST 2014|POINT (-78.81849566065084 37.295363463347535)|null

39. Clemens|47474|Tue Dec 30 00:42:36 CST 2014|POINT (-79.30049536448941 38.08199353770485)|null

40. Clemens|6674|Tue Dec 30 02:52:18 CST 2014|POINT (-78.90372581054386 38.494848260874534)|null

41. Clemens|77255|Tue Dec 30 03:05:48 CST 2014|POINT (-78.82916341519943 38.50932215232661)|null

42. Addams|48300|Tue Dec 30 05:26:28 CST 2014|POINT (-79.32231454886619 37.104102839783714)|null

43. Addams|54156|Tue Dec 30 06:15:53 CST 2014|POINT (-79.14066347585238 37.44834239417541)|null

44. Clemens|84368|Tue Dec 30 07:15:05 CST 2014|POINT (-78.91494448378386 37.04860832137139)|null

45. Addams|13803|Tue Dec 30 08:05:13 CST 2014|POINT (-79.15303234479443 37.22041043545786)|null

46. Addams|32205|Tue Dec 30 07:58:03 CST 2014|POINT (-79.18522979491057 38.78977589785558)|null

47. Addams|71448|Tue Dec 30 01:13:32 CST 2014|POINT (-77.72562013792908 37.85453330488843)|null

48. Bierce|90121|Tue Dec 30 02:17:24 CST 2014|POINT (-78.5533220887543 37.22062089245632)|null

49. Addams|75621|Tue Dec 30 02:25:00 CST 2014|POINT (-77.98073713166738 37.47369428834121)|null

// TRUNCATED

I'd expect to see only values being returned that occurred on Dec 30; however, I am also seeing activity from 29 Dec. Please advise if this is expected. Thank you!

-Joel
On Mon, Aug 21, 2017 at 8:59 AM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Joel,

Did you follow the instructions on setting up HBase filters and coprocessors in the docs? I realize that the tutorial doesn't mention that:

http://www.geomesa.org/documentation/user/hbase/install.html

I'm not sure if that would even cause the issue you're seeing though... It might be a classpath issue with older versions of geomesa mixed in, if I had to guess. Generally, we bin data by discrete weeks, so there must be something going pretty wrong for you to get results over a month off.

Could you try enabling explain logging, and reporting back? There are instructions here [1], but the short of it is to copy this file into src/main/resources and uncomment the explain logger section, then run your query again: https://github.com/locationtech/geomesa/blob/master/geomesa-hbase/geomesa-hbase-datastore/src/test/resources/log4j.xml

Thanks,

Emilio

[1] http://www.geomesa.org/documentation/user/datastores/explaining.html

On 08/18/2017 11:34 PM, Joel Folkerts wrote:
Using the geomesa-quickstart-hbase (1.3.3.1-SNAPSHOT), I am receiving query results outside of the temporal (when) query parameters when I only specify the 'when' attribute.

If I construct the following filter, it works as expected:

String cql = cqlGeometry + " AND " + cqlDates + " AND " + cqlAttributes; // line 172

queryFeatures(simpleFeatureTypeName, dataStore,
"Where", -78.5, 37.5, -78.0, 38.0,
"When", "2014-09-01T06:00:00.000Z", "2014-09-01T07:00:00.000Z",
"(Who = 'Bierce')");

CONSOLE OUTPUT:
1. Bierce|233179|Tue Jul 29 01:29:23 UTC 2014|POINT (-78.1563914300412 37.55068370938787)|null

However, if I construct a filter with only the temporal ('when') parameter, I receive results outside of the specified temporal range:

String cql = cqlDates; // Changing line 172 to this - only specifying the date range

queryFeatures(simpleFeatureTypeName, dataStore,
"Where", -78.5, 37.5, -78.0, 38.0,
"When", "2014-09-01T06:00:00.000Z", "2014-09-01T07:00:00.000Z",
"(Who = 'Bierce')");

CONSOLE OUTPUT:

1. Clemens|376958|Tue Jul 29 01:24:52 UTC 2014|POINT (-79.32356935554027 37.2330933099611)|null

2. Bierce|242290|Tue Jul 29 01:04:21 UTC 2014|POINT (-79.26106474788082 37.35070155601285)|null

3. Bierce|26866|Tue Jul 29 01:06:28 UTC 2014|POINT (-79.24711788069924 37.6932623925761)|null

4. Bierce|953185|Tue Jul 29 01:34:06 UTC 2014|POINT (-79.04971454350397 37.358282818762085)|null

5. Addams|269454|Tue Jul 29 01:53:22 UTC 2014|POINT (-78.77024220619475 37.70942117347827)|null

6. Clemens|303761|Tue Jul 29 02:01:06 UTC 2014|POINT (-78.81748849058852 37.7419228550425)|null

7. Clemens|768161|Tue Jul 29 02:20:33 UTC 2014|POINT (-79.2399050863839 37.06486923666385)|null

8. Clemens|961637|Tue Jul 29 03:19:43 UTC 2014|POINT (-79.15194165432969 37.25217823548506)|null

9. Addams|797358|Tue Jul 29 03:15:58 UTC 2014|POINT (-78.92912065781368 37.150616043436834)|null

10. Addams|701901|Tue Jul 29 03:16:15 UTC 2014|POINT (-79.00045739977287 37.210848811939385)|null

The above output is clearly outside the specified date range (01 Sep 2014 0600hrs - 01 Sep 2014 0700hrs) but I am not sure why only specifying the temporal (when) component would be causing this.

Any assistance would be greatly appreciated. Thank you!

-Joel
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

References:
- [geomesa-users] Error when querying date/time range
  - From: Joel Folkerts
- Re: [geomesa-users] Error when querying date/time range
  - From: Emilio Lahr-Vivaz
- Re: [geomesa-users] Error when querying date/time range
  - From: Joel Folkerts
- Re: [geomesa-users] Error when querying date/time range
  - From: Emilio Lahr-Vivaz

Prev by Date: Re: [geomesa-users] Error when querying date/time range
Next by Date: [geomesa-users] number of threads constantly increasing
Previous by thread: Re: [geomesa-users] Error when querying date/time range
Next by thread: [geomesa-users] number of threads constantly increasing
Index(es):
- Date
- Thread

Breadcrumbs