Hello,
Ah yes, HBase can get bogged down serving up large requests, which
can impact other requests. The WFS and preview limits do not apply
to WMS. You can 'time-enable' your layer, which can restrict the
layer preview to a default time filter and may be useful.
You may be able to do some configuration at the HBase level to help
with this, such as the number of read/write queues and threads. We
also have a few geomesa-specific system properties that you can use.
The easiest flag is to block 'full table scans', i.e. scans which
have to traverse your entire data set. This could be from a layer
preview, or just a query filter that doesn't have an appropriate
index to use. See here:
https://www.geomesa.org/documentation/user/datastores/runtime_config.html#geomesa-scan-block-full-table
Additionally, you can set up a 'query timeout', which will attempt
to kill any query that runs too long. This can be less reliable that
blocking the query up front (due to the nature of trying to kill a
distributed process), but it can catch queries that otherwise might
slip through. See here:
https://www.geomesa.org/documentation/user/datastores/runtime_config.html#geomesa-query-timeout
Thanks,
Emilio
On 12/10/18 6:02 AM, Roger Downing
wrote:
Hi there,
Thanks for you help so far on this, much
appreciated.
I have found that the geoserver geomesa
store seems to be part of the problem. If I try to get a layer
preview with geoserver then it never returns, and if I try to
query on command line subsequently then that too times out
until I stop geoserver. With geoserver stopped I can query on
the bounding box below and a time range and get thousands of
results in less than 10 seconds, which is excellent.
In geoserver I have the global WFS result
limit set to 1000 and I notice that the preview limit is set
to 50, would you know whether this applies when getting a WMS
preview of the vector layer? I also note that I can see the
preview WMS request go in via the geoserver log, and it is
followed up by what appears to be an unbounded query for the
whole layer extent.
Please disregard my queries about indexing
earlier, I was confusing myself. It is apparent that indexing
is functioning well in geomesa.
Thanks,
Roger
Do you mean
that you specified an attribute index? It should create a
spatial index if you have a geometry and you don't explicitly
define 'geomesa.indices.enabled' in your user data. More
information on the default indices here:
https://www.geomesa.org/documentation/user/datastores/index_basics.html
You can see what table and ranges are being scanned by using
the 'explain' command, or by enabling explain logging during
your query:
https://www.geomesa.org/documentation/user/datastores/query_planning.html#explaining-query-plans
What kind of timeout are you seeing, exactly?
Thanks,
Emilio
On 12/7/18 12:47 PM, Roger Downing wrote:
Hi there,
Yes I ingested using the geomesa-hbase ingest command from a
directory of avro format data stored in S3. It ingested with
no errors in 2.5 hours nearly 600 million records, which is
great. If I do not include specification of z curve on
ingest but have an index set in the SFT will it build a
spatial index? I'm wondering whether there is no index and
so it's defaulting to a full table scan.
Thanks,
Roger
Hello,
Were you able to ingest through the geomsa-hbase script,
or did you ingest in some other fashion? It is likely a
connection/configuration issue with the command line tools
- you can use ./geomesa-hbase classpath to ensure that
your hbase-site.xml is getting picked up (the classpath
should contain $HBASE_HOME/conf). If it's not, you can
copy it into the conf/ folder of the tools.
Thanks,
Emilio
On 12/7/18
11:12 AM, Roger Downing wrote:
Hi there,
I’ve ingested about 600
million points into Geomesa with 5 attributes. I
tried to do a geomesa-hbase export on a very small
bounding box and with a max result count of 100, but
it timed out. I then tried to do a geomesa-hbase
explain on the same CQL query “BBOX(gps,
-83.315506,42.292548,-83.212166,42.349537)”
and it timed out once again. Using hbase shell I can
see the geomesa tables. Are there any obvious things
I should try? All commands are being run from the
master node of an AWS EMR cluster with HBase
installed via bootstrap.
Thanks again,
Roger
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
|
