Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Tools/Utilities to analyze Geomesa query stats
Yeah, the spark stuff requires a lot of classpath mojo. There was a module to build the entire uber-jar, but it may be out of date...
But you shouldn't need the spark jar in order to read the query history table. Instead of using the path '/geoserver/geomesa/analytics/' (the spark endpoint) use '/geoserver/geomesa/stats' (the query history endpoint). Then register the data store as you were doing. 

So something like:
curl -XPOST http://bdpgeoserver:8181/geoserver/geomesa/stats/ds/CoalesceSearch -d 'instanceId=foo' ... curl http://bdpgeoserver:8181/geoserver/geomesa/stats/ds/CoalesceSearch # should return the instance you just registered curl http://bdpgeoserver:8181/geoserver/geomesa/stats/CoalesceSearch/queries -d 'typeName=bar' -d 'dates=2017-11-28T00:00:00.000Z/2017-11-29T00:00:00.000Z' # return queries from the last day
Let me know if you want to actually use the spark endpoint, and I'll look into it more. 

Thanks,

Emilio

On 11/28/2017 07:37 PM, David Boyd wrote:

Emilio:

   Thanks man.   This is cool stuff.
I got a good way through the process.  Now I am currently running 1.3.1 if it makes a difference. 

The analytics endpoint come up and I get the expected list of options.
I am able to set and retrieve the spark config map.

When I try to register my data store using:
curl -d 'instanceId=rdk' -d 'zookeepers=bdpnode4:bdpnode5:bdpnode6' -d 'tableName=CoalesceSearch' -d 'user=<myuser>' -d 'password=<mypass>' -v http://bdpgeoserver:8181/geoserver/geomesa/analytics/ds/CoalesceSearch 
I get back a 400 error - Bad request
However, in looking at my tomcat logs the spark library failed to load with this error:
Nov 28, 2017 11:42:26 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile INFO: validateJarFile(/opt/tomcat7/webapps/geoserver/WEB-INF/lib/spark_2.11-1.3.1-geomesa-assembly.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class 
So I tried a simple query:
Davids-MacBook-Pro:Documents BigDataDaddy$ curl --header 'Accept: text/plain' --get --data-urlencode 'q=select count(EventRecordset.GlobalEventId) from EventRecordset' http://bdpgeoserver:8181/geoserver/geomesa/analytics/sql 
java.lang.NoClassDefFoundError: org/apache/spark/sql/types/DataType
    at org.locationtech.geomesa.web.analytics.AnalyticEndpoint$$anonfun$12.apply(AnalyticEndpoint.scala:107)     at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)     at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265)     at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) 
Which I assume is due to the jar file not loading.
So I tried the end point to access the Queries table. But since the datastore did not register I am assuming it will not work. 

Any thoughts?
I assume where :alias shows up in the list of endpoints one should substitute the name of the datastore registered? 



On 11/28/17 8:47 AM, Emilio Lahr-Vivaz wrote:
Although it's not documented well (at all?), GeoMesa has a rest API for reading the queries table:
https://github.com/locationtech/geomesa/tree/master/geomesa-web/geomesa-web-data 
http://www.geomesa.org/documentation/user/web_data.html
You shouldn't need the spark jars from the documentation if you are just accessing the query table. The documentation focuses on the analytic endpoint, but there is another endpoint under 'stats' that will let you query the queries table:
https://github.com/locationtech/geomesa/blob/master/geomesa-web/geomesa-web-data/src/main/scala/org/locationtech/geomesa/web/analytics/StatsEndpoint.scala#L51
If you can't muddle through it, write back and I can provide better instructions. 

Thanks,

Emilio

On 11/27/2017 08:04 PM, David Boyd wrote:

All:
   Was browsing the queries table in my instance looking at the times.   I was curious if anyone had any short utils to browse or download the queries table for my catalog.   I see the new query-rasterstats in 1.3.4 but nothing to pull from that table.    I know I can do an scan from the accumulo shell and download the results to post process or write a small accumulo query application, but I was wondering if someone had thrown 
something together already.




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit 
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top