Re: [geomesa-users] Error when querying date/time range

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geomesa-users] Error when querying date/time range

From: Joel Folkerts <joel.folkerts@xxxxxxxxx>
Date: Mon, 21 Aug 2017 12:10:21 -0500
Delivered-to: geomesa-users@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-users>
List-help: <mailto:geomesa-users-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=unsubscribe>

Hi Emilio-

Yes, I set up the co-processors and can see them being registered in the region servers when HBase starts. I've cleared the data from the last email so the dates are a little different but am still seeing the problem. Note I am only performing a temporal filter:

return CQL.toFilter("(When DURING 2014-12-30T00:00:00.000Z/2014-12-30T23:59:59.000Z)"); // line 175 of HBaseQuickStart.java

OUTPUT:

Creating feature-type (schema): QuickStart

Creating new features

Inserting new features

Submitting query

[2017-08-21 12:01:59,305] TRACE Explainer: Planning 'QuickStart' When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00

[2017-08-21 12:01:59,306] TRACE Explainer: Original filter: When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00

[2017-08-21 12:01:59,309] TRACE Explainer: Hints: bin[false] arrow[false] density[false] stats[false] map-aggregate[false] sampling[none]

[2017-08-21 12:01:59,311] TRACE Explainer: Sort: none

[2017-08-21 12:01:59,312] TRACE Explainer: Transforms: None

[2017-08-21 12:01:59,315] TRACE Explainer: Strategy selection:

[2017-08-21 12:01:59,396] TRACE Explainer: Query processing took 68ms and produced 1 options

[2017-08-21 12:01:59,400] TRACE Explainer: Filter plan: FilterPlan[HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]]

[2017-08-21 12:01:59,400] TRACE Explainer: Strategy selection took 3ms for 1 options

[2017-08-21 12:01:59,402] TRACE Explainer: Strategy 1 of 1: HBaseZ3Index

[2017-08-21 12:01:59,402] TRACE Explainer: Strategy filter: HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]

[2017-08-21 12:01:59,427] TRACE Explainer: Geometries: FilterValues(List(POLYGON ((-180 -90, 0 -90, 180 -90, 180 90, 0 90, -180 90, -180 -90))),false)

[2017-08-21 12:01:59,443] TRACE Explainer: Intervals: FilterValues(List([2014-12-30T00:00:01.000Z,2014-12-30T23:59:58.000Z]),false)

[2017-08-21 12:01:59,614] TRACE Explainer: Plan: org.locationtech.geomesa.hbase.data.ScanPlan

[2017-08-21 12:01:59,615] TRACE Explainer: Table: geomesa_QuickStart_z3

[2017-08-21 12:01:59,616] TRACE Explainer: Ranges (13): { [0943836000000,094326104], [094326108000000,094344-80], [094344-76000000,094362-8], [094362-4000000,19432640], [19432644000000,194344112] }

[2017-08-21 12:01:59,616] TRACE Explainer: Filter: HBaseZ3Index[When DURING 2014-12-30T00:00:00+00:00/2014-12-30T23:59:59+00:00][None]

[2017-08-21 12:01:59,617] TRACE Explainer: Plan creation took 210ms

[2017-08-21 12:01:59,617] TRACE Explainer: Query planning took 356ms

1. Bierce|38782|Mon Dec 29 15:06:42 CST 2014|POINT (-79.1341120369213 38.123362284877786)|null

2. Addams|69945|Mon Dec 29 16:13:19 CST 2014|POINT (-78.91343013821982 37.35509231261945)|null

3. Clemens|40670|Mon Dec 29 17:16:40 CST 2014|POINT (-78.91131901031984 37.43714158170664)|null

4. Bierce|94117|Mon Dec 29 17:55:19 CST 2014|POINT (-79.08357338783861 37.87162635613247)|null

5. Clemens|73757|Mon Dec 29 18:17:38 CST 2014|POINT (-79.24588872978585 37.482763349262775)|null

6. Addams|83556|Mon Dec 29 16:21:40 CST 2014|POINT (-79.06874376441775 38.21485507393958)|null

7. Addams|37206|Mon Dec 29 16:37:28 CST 2014|POINT (-78.98274149666337 38.44403421353682)|null

8. Addams|81066|Mon Dec 29 16:19:43 CST 2014|POINT (-79.40980568780253 38.897938151457055)|null

9. Bierce|18970|Mon Dec 29 18:06:55 CST 2014|POINT (-78.92545983348037 38.45330541840287)|null

10. Bierce|19597|Mon Dec 29 17:46:16 CST 2014|POINT (-78.76642524544258 38.97293392198045)|null

11. Addams|84432|Mon Dec 29 19:52:53 CST 2014|POINT (-78.84949257612135 38.62687651755251)|null

12. Bierce|81487|Mon Dec 29 21:31:59 CST 2014|POINT (-78.84201779048216 37.6195979334259)|null

13. Bierce|6892|Mon Dec 29 22:48:33 CST 2014|POINT (-79.23041357169569 37.914511293511964)|null

14. Addams|31344|Mon Dec 29 21:51:25 CST 2014|POINT (-78.80828107960899 38.28339974006503)|null

15. Clemens|92147|Mon Dec 29 13:32:30 CST 2014|POINT (-78.603118982262 37.338821484984)|null

16. Addams|81099|Mon Dec 29 13:47:16 CST 2014|POINT (-78.34873124129228 37.869845432416696)|null

17. Bierce|21997|Mon Dec 29 14:50:47 CST 2014|POINT (-78.21858633518944 37.60727614841389)|null

18. Addams|14172|Mon Dec 29 14:02:31 CST 2014|POINT (-77.51306908547258 38.66047648630169)|null

19. Bierce|96154|Mon Dec 29 14:56:44 CST 2014|POINT (-78.24796068682105 38.29747516622206)|null

20. Addams|18465|Mon Dec 29 14:54:13 CST 2014|POINT (-77.78081152620675 38.62463134443681)|null

21. Bierce|88666|Mon Dec 29 15:27:30 CST 2014|POINT (-78.52489764211398 38.821220700459364)|null

22. Addams|86223|Mon Dec 29 17:20:50 CST 2014|POINT (-78.58423279816995 37.898586954800514)|null

23. Addams|2718|Mon Dec 29 17:28:41 CST 2014|POINT (-77.60064101336147 37.86598231978402)|null

24. Clemens|94388|Mon Dec 29 17:16:06 CST 2014|POINT (-78.20963255855335 38.112068407498505)|null

25. Addams|78573|Mon Dec 29 16:47:26 CST 2014|POINT (-77.86294925027478 38.052799096517994)|null

26. Clemens|73388|Mon Dec 29 17:26:44 CST 2014|POINT (-78.28175432568574 38.18566898567634)|null

27. Clemens|3755|Mon Dec 29 17:55:06 CST 2014|POINT (-78.1146776142142 38.52977882654116)|null

28. Clemens|96374|Mon Dec 29 18:40:59 CST 2014|POINT (-78.2157387189368 38.70639028573238)|null

29. Bierce|41596|Mon Dec 29 18:46:51 CST 2014|POINT (-77.58787426740285 38.06434690726756)|null

30. Addams|60429|Mon Dec 29 19:17:06 CST 2014|POINT (-77.73007552782313 38.958459158952486)|null

31. Bierce|48652|Mon Dec 29 20:40:31 CST 2014|POINT (-78.3878356380678 38.33104619503754)|null

32. Clemens|62126|Mon Dec 29 23:33:03 CST 2014|POINT (-77.68651693444643 37.182529764756424)|null

33. Bierce|55858|Mon Dec 29 21:34:36 CST 2014|POINT (-77.84727805879886 38.43210576427531)|null

34. Bierce|88897|Mon Dec 29 21:43:35 CST 2014|POINT (-77.62517310497805 38.400072514444865)|null

35. Bierce|31807|Mon Dec 29 21:59:12 CST 2014|POINT (-77.63439426590955 38.464419570900006)|null

36. Bierce|36124|Mon Dec 29 22:41:06 CST 2014|POINT (-77.79562898260852 38.46646234067417)|null

37. Bierce|17185|Mon Dec 29 23:40:14 CST 2014|POINT (-77.57094565071165 38.98195919697627)|null

38. Clemens|8723|Tue Dec 30 00:28:20 CST 2014|POINT (-78.81849566065084 37.295363463347535)|null

39. Clemens|47474|Tue Dec 30 00:42:36 CST 2014|POINT (-79.30049536448941 38.08199353770485)|null

40. Clemens|6674|Tue Dec 30 02:52:18 CST 2014|POINT (-78.90372581054386 38.494848260874534)|null

41. Clemens|77255|Tue Dec 30 03:05:48 CST 2014|POINT (-78.82916341519943 38.50932215232661)|null

42. Addams|48300|Tue Dec 30 05:26:28 CST 2014|POINT (-79.32231454886619 37.104102839783714)|null

43. Addams|54156|Tue Dec 30 06:15:53 CST 2014|POINT (-79.14066347585238 37.44834239417541)|null

44. Clemens|84368|Tue Dec 30 07:15:05 CST 2014|POINT (-78.91494448378386 37.04860832137139)|null

45. Addams|13803|Tue Dec 30 08:05:13 CST 2014|POINT (-79.15303234479443 37.22041043545786)|null

46. Addams|32205|Tue Dec 30 07:58:03 CST 2014|POINT (-79.18522979491057 38.78977589785558)|null

47. Addams|71448|Tue Dec 30 01:13:32 CST 2014|POINT (-77.72562013792908 37.85453330488843)|null

48. Bierce|90121|Tue Dec 30 02:17:24 CST 2014|POINT (-78.5533220887543 37.22062089245632)|null

49. Addams|75621|Tue Dec 30 02:25:00 CST 2014|POINT (-77.98073713166738 37.47369428834121)|null

// TRUNCATED

I'd expect to see only values being returned that occurred on Dec 30; however, I am also seeing activity from 29 Dec. Please advise if this is expected. Thank you!

-Joel

On Mon, Aug 21, 2017 at 8:59 AM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:

Hi Joel,

Did you follow the instructions on setting up HBase filters and coprocessors in the docs? I realize that the tutorial doesn't mention that:

http://www.geomesa.org/documentation/user/hbase/install.html

I'm not sure if that would even cause the issue you're seeing though... It might be a classpath issue with older versions of geomesa mixed in, if I had to guess. Generally, we bin data by discrete weeks, so there must be something going pretty wrong for you to get results over a month off.

Could you try enabling explain logging, and reporting back? There are instructions here [1], but the short of it is to copy this file into src/main/resources and uncomment the explain logger section, then run your query again: https://github.com/locationtech/geomesa/blob/master/geomesa-hbase/geomesa-hbase-datastore/src/test/resources/log4j.xml

Thanks,

Emilio

[1] http://www.geomesa.org/documentation/user/datastores/explaining.html

On 08/18/2017 11:34 PM, Joel Folkerts wrote:
Using the geomesa-quickstart-hbase (1.3.3.1-SNAPSHOT), I am receiving query results outside of the temporal (when) query parameters when I only specify the 'when' attribute.

If I construct the following filter, it works as expected:

String cql = cqlGeometry + " AND " + cqlDates + " AND " + cqlAttributes; // line 172

queryFeatures(simpleFeatureTypeName, dataStore,
"Where", -78.5, 37.5, -78.0, 38.0,
"When", "2014-09-01T06:00:00.000Z", "2014-09-01T07:00:00.000Z",
"(Who = 'Bierce')");

CONSOLE OUTPUT:
1. Bierce|233179|Tue Jul 29 01:29:23 UTC 2014|POINT (-78.1563914300412 37.55068370938787)|null

However, if I construct a filter with only the temporal ('when') parameter, I receive results outside of the specified temporal range:

String cql = cqlDates; // Changing line 172 to this - only specifying the date range

queryFeatures(simpleFeatureTypeName, dataStore,
"Where", -78.5, 37.5, -78.0, 38.0,
"When", "2014-09-01T06:00:00.000Z", "2014-09-01T07:00:00.000Z",
"(Who = 'Bierce')");

CONSOLE OUTPUT:

1. Clemens|376958|Tue Jul 29 01:24:52 UTC 2014|POINT (-79.32356935554027 37.2330933099611)|null

2. Bierce|242290|Tue Jul 29 01:04:21 UTC 2014|POINT (-79.26106474788082 37.35070155601285)|null

3. Bierce|26866|Tue Jul 29 01:06:28 UTC 2014|POINT (-79.24711788069924 37.6932623925761)|null

4. Bierce|953185|Tue Jul 29 01:34:06 UTC 2014|POINT (-79.04971454350397 37.358282818762085)|null

5. Addams|269454|Tue Jul 29 01:53:22 UTC 2014|POINT (-78.77024220619475 37.70942117347827)|null

6. Clemens|303761|Tue Jul 29 02:01:06 UTC 2014|POINT (-78.81748849058852 37.7419228550425)|null

7. Clemens|768161|Tue Jul 29 02:20:33 UTC 2014|POINT (-79.2399050863839 37.06486923666385)|null

8. Clemens|961637|Tue Jul 29 03:19:43 UTC 2014|POINT (-79.15194165432969 37.25217823548506)|null

9. Addams|797358|Tue Jul 29 03:15:58 UTC 2014|POINT (-78.92912065781368 37.150616043436834)|null

10. Addams|701901|Tue Jul 29 03:16:15 UTC 2014|POINT (-79.00045739977287 37.210848811939385)|null

The above output is clearly outside the specified date range (01 Sep 2014 0600hrs - 01 Sep 2014 0700hrs) but I am not sure why only specifying the temporal (when) component would be causing this.

Any assistance would be greatly appreciated. Thank you!

-Joel
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Follow-Ups:
- Re: [geomesa-users] Error when querying date/time range
  - From: Emilio Lahr-Vivaz

References:
- [geomesa-users] Error when querying date/time range
  - From: Joel Folkerts
- Re: [geomesa-users] Error when querying date/time range
  - From: Emilio Lahr-Vivaz

Prev by Date: Re: [geomesa-users] Kafka Datastore Nifi Component:
Next by Date: Re: [geomesa-users] Error when querying date/time range
Previous by thread: Re: [geomesa-users] Error when querying date/time range
Next by thread: Re: [geomesa-users] Error when querying date/time range
Index(es):
- Date
- Thread

Breadcrumbs