[geomesa-dev] Σχετ: Σχετ: Σχετ: Σχετ: Disable spatial indexing

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

[geomesa-dev] Σχετ: Σχετ: Σχετ: Σχετ: Disable spatial indexing

From: Maria Krommyda <mariakr1989@xxxxxxxx>
Date: Mon, 12 Nov 2018 13:39:02 +0000 (UTC)
Delivered-to: geomesa-dev@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-dev>
List-help: <mailto:geomesa-dev-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-dev>, <mailto:geomesa-dev-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-dev>, <mailto:geomesa-dev-request@locationtech.org?subject=unsubscribe>

Thank you for your time and advice.

Updating did not fix my issue but restarting everything did, I suspect some caching of settings.

Best regards,

Maria.

Στις 6:23 μ.μ. Πέμπτη, 8 Νοεμβρίου 2018, ο/η Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> έγραψε:

I believe it will always create the stats table, although it shouldn't write anything to it when you have it disabled.

I looked back, and we've had support for schemas without geometries since 1.2.0, so upgrading may not fix your issues:

https://github.com/locationtech/geomesa/commit/2bc17c780633fc343a68caa96f206901c48aae74

However, we are fairly flexible on versions, and you should be able to upgrade geomesa in place without changing any of your infrastructure versions. There are more details in the upgrade guide here:

https://www.geomesa.org/documentation/user/upgrade.html#upgrade-guide

Generally, if you stick with the geotools DataStore API, then you shouldn't need any changes when upgrading. If you use internal GeoMesa functions or classes, then you may need some code changes.

Thanks,

Emilio

On 11/8/18 11:04 AM, Maria Krommyda wrote:

It creates the table, table_records_v2 and the table_stats.

Is there any other way to disable the stats?

Wouldn't upgrading to newer version affect the accumulo/hadoop/zookeeper versions required?

What is the easiest/safest way to do the upgrade?

Thank you once again for taking the time to get back to me.

Maria.

Στις 5:30 μ.μ. Πέμπτη, 8 Νοεμβρίου 2018, ο/η Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> έγραψε:

Hmm, is it creating the _records table then? I am not sure what else would be wrong... you should be getting some kind of error I would think.

Just to clarify, you can upgrade all the way to our latest release (2.0.2), and you wouldn't need to re-ingest your data, everything will still work.

Thanks,

Emilio

On 11/8/18 9:39 AM, Maria Krommyda wrote:

I would prefer not to upgrade, given that the other tables have quite some volume of data.

Setting

dsConf.put("generateStats", "false");

suppresses the error, but the _stats table is created and inserting data still doesn't work properly.

I do not get any error, but no data are stored to the table either.

Thank you very much for your time,

Maria.

Στις 3:37 μ.μ. Πέμπτη, 8 Νοεμβρίου 2018, ο/η Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> έγραψε:

Good to hear. The issue that you're hitting now is a stats-related bug. You could upgrade to 1.3.6, or just disable stats on your data store as described here:

https://www.geomesa.org/documentation/1.3.5/user/datastores/runtime_config.html#geomesa-stats-generate

If you upgrade, you may need to delete the _stats table and re-ingest your data. Alternatively, you can disable the StatsCombiner on the table, truncate the table, and the re-enable it, but if you have a small data set re-ingesting is easier.

Thanks,

Emilio

On 11/8/18 7:00 AM, Maria Krommyda wrote:

I am using geomesa- accumulo 2.11-1.3.5

I have an existing storage schema, where all my tables require spatial indexing, thus the need to use Geomesa.

To that I want to add a table that doesn't have any spatial information, but has a nullable Polygon, so as to access it the same way.

I understand that this is not a primary use case but it would made a lot of things easier for me, with the retrieval of the information.

I have used "records" as the index and it seems to be working.

I have however a different error when trying to store data.

ParserRuntimeException

IllegalArgumentException, requirement failed, Upper bound must be greater than lower bound

Below I have the complete stack.

Any ideas on that would be highly appreciated.

Best regards,

Maria.

org.parboiled.errors.ParserRuntimeException: Error while parsing action 'stat/org$locationtech$geomesa$utils$stats$StatParser$$stats/OneOrMore/ZeroOrMore/Sequence/org$locationtech$geomesa$utils$stats$StatParser$$singleStat/org$locationtech$geomesa$utils$stats$StatParser$$histogram/org$locationtech$geomesa$utils$stats$StatParser$$histogramAction1' at input position (line 1, pos 129):
                                                                                                                               ^

java.lang.IllegalArgumentException: requirement failed: Upper bound must be greater than lower bound: lower='POINT (180 90)'(4611686018427387903) upper='POINT (-180 -90)'(0)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:366)
    at org.parboiled.matchers.SequenceMatcher.match(SequenceMatcher.java:46)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.FirstOfMatcher.match(FirstOfMatcher.java:41)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.SequenceMatcher.match(SequenceMatcher.java:46)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.ZeroOrMoreMatcher.match(ZeroOrMoreMatcher.java:39)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.SequenceMatcher.match(SequenceMatcher.java:46)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.SequenceMatcher.match(SequenceMatcher.java:46)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.matchers.SequenceMatcher.match(SequenceMatcher.java:46)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    at org.parboiled.parserunners.BasicParseRunner.run(BasicParseRunner.java:72)
    at org.parboiled.parserunners.ReportingParseRunner.runBasicMatch(ReportingParseRunner.java:86)
    at org.parboiled.parserunners.ReportingParseRunner.run(ReportingParseRunner.java:66)
    at org.parboiled.scala.parserunners.ReportingParseRunner.run(ReportingParseRunner.scala:34)
    at org.locationtech.geomesa.utils.stats.StatParser$.parse(StatParser.scala:37)
    at org.locationtech.geomesa.utils.stats.Stat$.apply(Stat.scala:180)
    at org.locationtech.geomesa.accumulo.data.stats.AccumuloGeoMesaStats$$anonfun$statUpdater$1.apply(AccumuloGeoMesaStats.scala:116)
    at org.locationtech.geomesa.accumulo.data.stats.AccumuloGeoMesaStats$$anonfun$statUpdater$1.apply(AccumuloGeoMesaStats.scala:116)
    at org.locationtech.geomesa.index.stats.MetadataStatUpdater.<init>(MetadataBackedStats.scala:322)
    at org.locationtech.geomesa.accumulo.data.stats.AccumuloGeoMesaStats.statUpdater(AccumuloGeoMesaStats.scala:116)
    at org.locationtech.geomesa.index.geotools.GeoMesaFeatureWriter.<init>(GeoMesaFeatureWriter.scala:113)
    at org.locationtech.geomesa.accumulo.data.AccumuloAppendFeatureWriter.<init>(AccumuloFeatureWriter.scala:23)
    at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.createFeatureWriterAppend(AccumuloDataStore.scala:107)
    at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getIndexWriterAppend(GeoMesaDataStore.scala:444)
    at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getFeatureWriterAppend(GeoMesaDataStore.scala:435)
    at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getFeatureWriterAppend(GeoMesaDataStore.scala:53)
    at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore$$anonfun$addFeatures$1.apply(GeoMesaFeatureStore.scala:44)
    at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore$$anonfun$addFeatures$1.apply(GeoMesaFeatureStore.scala:44)
    at org.locationtech.geomesa.utils.io.WithClose$$anonfun$apply$1.apply(SafeClose.scala:53)
    at org.locationtech.geomesa.utils.io.WithClose$.apply(SafeClose.scala:51)
    at org.locationtech.geomesa.utils.io.WithClose$.apply(SafeClose.scala:53)
    at org.locationtech.geomesa.index.geotools.GeoMesaFeatureStore.addFeatures(GeoMesaFeatureStore.scala:44)
    at sensorthing.storage.schema.insertFeatures(schema.java:21)
    at sensorthing.storage.Datastream.addData(Datastream.java:52)
    at sensorthing.apis.Datastreams.Datastream(Datastreams.java:95)
    at sensorthing.apis.test.main(test.java:14)
Caused by: java.lang.IllegalArgumentException: requirement failed: Upper bound must be greater than lower bound: lower='POINT (180 90)'(4611686018427387903) upper='POINT (-180 -90)'(0)
    at scala.Predef$.require(Predef.scala:219)
    at org.locationtech.geomesa.utils.stats.WholeNumberBinnedArray.<init>(BinnedArray.scala:134)
    at org.locationtech.geomesa.utils.stats.BinnedGeometryArray.<init>(BinnedArray.scala:213)
    at org.locationtech.geomesa.utils.stats.BinnedArray$.apply(BinnedArray.scala:121)
    at org.locationtech.geomesa.utils.stats.Histogram.<init>(Histogram.scala:39)
    at org.locationtech.geomesa.utils.stats.StatParser$$anonfun$org$locationtech$geomesa$utils$stats$StatParser$$histogram$1$$anonfun$apply$10.apply(StatParser.scala:130)
    at org.locationtech.geomesa.utils.stats.StatParser$$anonfun$org$locationtech$geomesa$utils$stats$StatParser$$histogram$1$$anonfun$apply$10.apply(StatParser.scala:124)
    at org.parboiled.scala.rules.Rule$$anonfun$exec$4.apply(Rule.scala:221)
    at org.parboiled.scala.rules.Rule$$anonfun$exec$4.apply(Rule.scala:217)
    at org.parboiled.scala.rules.Rule$$anonfun$push$1.apply(Rule.scala:133)
    at org.parboiled.scala.rules.Rule$$anonfun$push$1.apply(Rule.scala:132)
    at org.parboiled.scala.rules.Rule$$anon$1.run(Rule.scala:129)
    at org.parboiled.matchers.ActionMatcher.match(ActionMatcher.java:96)
    at org.parboiled.parserunners.BasicParseRunner.match(BasicParseRunner.java:77)
    at org.parboiled.MatcherContext.runMatcher(MatcherContext.java:351)
    ... 47 more

Στις 10:36 μ.μ. Τετάρτη, 7 Νοεμβρίου 2018, ο/η Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> έγραψε:

What version of GeoMesa are you using? The 'id' index was called the 'record' index in earlier versions, so that might be the issue there.

The attribute index usually has a secondary z-curve as part of it, which is why you are still getting an error with just the attribute index. You can specify an older attribute index that doesn't include the secondary part, by appending the index version number. I.e. in your case "attr:3".

May I ask what you use case is? While it's possible to use GeoMesa without any spatial indices, it's not really a primary use case.

Thanks,

Emilio

On 11/7/18 2:52 PM, Maria Krommyda wrote:

Dear Emilio,

Thank you for your response and for your time.

Yes, I have other attributes that I would like to index.

However, I think that I am doing something totally wrong as I cannot make the command work.

If I add

sft.getUserData().put("geomesa.indices.enabled ","id"); //or id,attr

I get as an error that configured index id does not exist, I assume because I have a user defined feature id.

If I do

sft.getUserData().put("geomesa.indices.enabled ","attr");

I get as an error that configured index attr does not support the schema

Then I tried to do

sft.getDescriptor("never_null_attribute").getUserData().put("index", "true");

sft.getUserData().put("geomesa.indices.enabled ","attr");

Which worked, in the sense that the schema was initialized and that accumulo did not have any z3 tables.

When I try to store a null polygon though, I get the same exception. (Illegal argument error indexing feature with null geometry, index xz3)

Any ideas/suggestions would be more than welcomed.

Best regards,

Maria.

_______________________________________________
geomesa-dev mailing list
geomesa-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-dev

_______________________________________________
geomesa-dev mailing list
geomesa-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-dev

References:
- [geomesa-dev] Σχετ: Disable spatial indexing
  - From: Maria Krommyda
- Re: [geomesa-dev] Σχετ: Disable spatial indexing
  - From: Emilio Lahr-Vivaz
- [geomesa-dev] Σχετ: Σχετ: Disable spatial indexing
  - From: Maria Krommyda
- Re: [geomesa-dev] Σχετ: Σχετ: Disable spatial indexing
  - From: Emilio Lahr-Vivaz
- [geomesa-dev] Σχετ: Σχετ: Σχετ: Disable spatial indexing
  - From: Maria Krommyda
- Re: [geomesa-dev] Σχετ: Σχετ: Σχετ: Disable spatial indexing
  - From: Emilio Lahr-Vivaz

Prev by Date: [geomesa-dev] [JIRA] (GEOMESA-2464) Spark - FSDS spark runtime jar does not work with ORC encoded files
Next by Date: [geomesa-dev] [JIRA] (GEOMESA-2465) Build - intermittent failures in PlaybackIteratorTest
Previous by thread: Re: [geomesa-dev] Σχετ: Σχετ: Σχετ: Disable spatial indexing
Next by thread: [geomesa-dev] [JIRA] (GEOMESA-2459) Converters - ResourceLoadingCache requires guava 14+ (which isn't always available)
Index(es):
- Date
- Thread

Breadcrumbs