Re: [geomesa-users] Issue geomesa-hbase and geoserver plugin

[Gaetano Perrone <gaetano.perrone@xxxxxxxxxxxxxxx>]

Hi Emilio,

the list of libraries in geoserver WEB-INF lib is the following:

activation-1.1.jar
aopalliance-1.0.jar
batik-anim-1.7.jar
batik-awt-util-1.7.jar
batik-bridge-1.7.jar
batik-css-1.7.jar
batik-dom-1.7.jar
batik-ext-1.7.jar
batik-gvt-1.7.jar
batik-js-1.7.jar
batik-parser-1.7.jar
batik-script-1.7.jar
batik-svg-dom-1.7.jar
batik-svggen-1.7.jar
batik-transcoder-1.7.jar
batik-util-1.7.jar
batik-xml-1.7.jar
bcprov-jdk15on-1.54.jar
cas-client-core-3.3.3.jar
cglib-nodep-2.2.jar
commons-beanutils-1.9.2-noclassprop.jar
commons-cli-1.2.jar
commons-codec-1.9.jar
commons-collections-3.2.2.jar
commons-collections4-4.1.jar
commons-configuration-1.6.jar
commons-dbcp-1.4.jar
commons-fileupload-1.2.1.jar
commons-httpclient-3.1.jar
commons-io-2.5.jar
commons-jxpath-1.3.jar
commons-lang-2.6.jar
commons-logging-1.1.3.jar
commons-pool-1.5.4.jar
ehcache-2.10.3.jar
ejml-core-0.32.jar
ejml-ddense-0.32.jar
encoder-1.1.jar
ezmorph-1.0.6.jar
freemarker-2.3.18.jar
GeographicLib-Java-1.44.jar
geomesa-hbase_2.11-2.0.2-bin.tar.gz
geomesa-hbase-gs-plugin_2.11-2.0.2-shaded.jar
gs-gwc-2.13.1.jar
gs-kml-2.13.1.jar
gs-main-2.13.1.jar
gs-ows-2.13.1.jar
gs-platform-2.13.1.jar
gs-platform-2.13.1-tests.jar
gs-rest-2.13.1.jar
gs-restconfig-2.13.1.jar
gs-restconfig-wcs-2.13.1.jar
gs-restconfig-wfs-2.13.1.jar
gs-restconfig-wms-2.13.1.jar
gs-sec-cas-2.13.1.jar
gs-sec-jdbc-2.13.1.jar
gs-sec-ldap-2.13.1.jar
gs-security-tests-2.13.1-tests.jar
gs-wcs1_0-2.13.1.jar
gs-wcs1_1-2.13.1.jar
gs-wcs2_0-2.13.1.jar
gs-wcs-2.13.1.jar
gs-web-core-2.13.1.jar
gs-web-demo-2.13.1.jar
gs-web-gwc-2.13.1.jar
gs-web-rest-2.13.1.jar
gs-web-sec-cas-2.13.1.jar
gs-web-sec-core-2.13.1.jar
gs-web-sec-jdbc-2.13.1.jar
gs-web-sec-ldap-2.13.1.jar
gs-web-wcs-2.13.1.jar
gs-web-wfs-2.13.1.jar
gs-web-wms-2.13.1.jar
gs-web-wps-2.13.1.jar
gs-wfs-2.13.1.jar
gs-wms-2.13.1.jar
gs-wps-core-2.13.1.jar
gt-api-19.1.jar
gt-app-schema-resolver-19.1.jar
gt-arcgrid-19.1.jar
gt-complex-19.1.jar
gt-coverage-19.1.jar
gt-cql-19.1.jar
gt-data-19.1.jar
gt-epsg-hsql-19.1.jar
gt-geojson-19.1.jar
gt-geopkg-19.1.jar
gt-geotiff-19.1.jar
gt-grid-19.1.jar
gt-gtopo30-19.1.jar
gt-image-19.1.jar
gt-imagemosaic-19.1.jar
gt-jdbc-19.1.jar
gt-jdbc-postgis-19.1.jar
gt-main-19.1.jar
gt-metadata-19.1.jar
gt-opengis-19.1.jar
gt-process-19.1.jar
gt-process-feature-19.1.jar
gt-process-geometry-19.1.jar
gt-process-raster-19.1.jar
gt-property-19.1.jar
gt-referencing-19.1.jar
gt-render-19.1.jar
gt-shapefile-19.1.jar
gt-svg-19.1.jar
gt-swing-19.1.jar
gt-tile-client-19.1.jar
gt-transform-19.1.jar
gt-wfs-ng-19.1.jar
gt-wms-19.1.jar
gt-wmts-19.1.jar
gt-xml-19.1.jar
gt-xsd-core-19.1.jar
gt-xsd-fes-19.1.jar
gt-xsd-filter-19.1.jar
gt-xsd-gml2-19.1.jar
gt-xsd-gml3-19.1.jar
gt-xsd-ows-19.1.jar
gt-xsd-sld-19.1.jar
gt-xsd-wcs-19.1.jar
gt-xsd-wfs-19.1.jar
gt-xsd-wmts-19.1.jar
gt-xsd-wps-19.1.jar
guava-17.0.jar
gwc-core-1.13.1.jar
gwc-diskquota-core-1.13.1.jar
gwc-diskquota-jdbc-1.13.1.jar
gwc-georss-1.13.1.jar
gwc-gmaps-1.13.1.jar
gwc-kml-1.13.1.jar
gwc-rest-1.13.1.jar
gwc-tms-1.13.1.jar
gwc-ve-1.13.1.jar
gwc-wms-1.13.1.jar
gwc-wmts-1.13.1.jar
h2-1.1.119.jar
hadoop-auth-2.7.4.jar
hadoop-aws-2.7.4.jar
hadoop-aws-2.8.3-amzn-1.jar_bk
hadoop-client-2.7.4.jar
hadoop-common-2.7.4.jar
hadoop-hdfs-2.7.4.jar
hsqldb-2.3.0.jar
htrace-core-3.1.0-incubating.jar
imageio-ext-arcgrid-1.1.20.jar
imageio-ext-geocore-1.1.20.jar
imageio-ext-png-1.1.20.jar
imageio-ext-streams-1.1.20.jar
imageio-ext-tiff-1.1.20.jar
imageio-ext-utilities-1.1.20.jar
itext-2.1.5.jar
jackson-core-2.5.0.jar
jai_codec-1.1.3.jar
jai_core-1.1.3.jar
jai_imageio-1.1.jar
jasypt-1.8.jar
JavaAPIforKml-2.2.0.jar
jdom2-2.0.6.jar
jettison-1.0.1.jar
jgridshift-1.0.jar
json-lib-2.2.3-jdk15.jar
json-simple-1.1.jar
jsr-275-1.0-beta-2.jar
jsr305-2.0.3.jar
jt-affine-1.0.22.jar
jt-algebra-1.0.22.jar
jt-attributeop-1.4.0.jar
jt-bandcombine-1.0.22.jar
jt-bandmerge-1.0.22.jar
jt-bandselect-1.0.22.jar
jt-binarize-1.0.22.jar
jt-border-1.0.22.jar
jt-buffer-1.0.22.jar
jt-classifier-1.0.22.jar
jt-colorconvert-1.0.22.jar
jt-colorindexer-1.0.22.jar
jt-concurrent-tile-cache-1.0.22.jar
jt-contour-1.4.0.jar
jt-crop-1.0.22.jar
jt-errordiffusion-1.0.22.jar
jt-format-1.0.22.jar
jt-imagefunction-1.0.22.jar
jt-iterators-1.0.22.jar
jt-lookup-1.0.22.jar
jt-mosaic-1.0.22.jar
jt-nullop-1.0.22.jar
jt-orderdither-1.0.22.jar
jt-piecewise-1.0.22.jar
jt-rangelookup-1.4.0.jar
jt-rescale-1.0.22.jar
jt-rlookup-1.0.22.jar
jt-scale-1.0.22.jar
jts-core-1.14.0.jar
jts-example-1.14.0.jar
jts-io-1.14.0.jar
jt-stats-1.0.22.jar
jt-translate-1.0.22.jar
jt-utilities-1.0.22.jar
jt-utils-1.4.0.jar
jt-vectorbin-1.0.22.jar
jt-vectorbinarize-1.4.0.jar
jt-vectorize-1.4.0.jar
jt-warp-1.0.22.jar
jt-zonal-1.0.22.jar
jt-zonalstats-1.4.0.jar
log4j-1.2.17.jar
mail-1.4.jar
marlin-0.7.5-Unsafe.jar
metrics-core-2.2.0.jar
miglayout-3.7-swing.jar
net.opengis.fes-19.1.jar
net.opengis.ows-19.1.jar
net.opengis.wcs-19.1.jar
net.opengis.wfs-19.1.jar
net.opengis.wmts-19.1.jar
net.opengis.wps-19.1.jar
netty-3.6.2.Final.jar
netty-all-4.0.41.Final.jar
org.eclipse.emf.common-2.12.0.jar
org.eclipse.emf.ecore-2.12.0.jar
org.eclipse.emf.ecore.xmi-2.12.0.jar
org.eclipse.xsd-2.12.0.jar
org.json-2.0.jar
org.w3.xlink-19.1.jar
picocontainer-1.2.jar
pngj-2.0.1.jar
postgresql-42.1.1.jar
protobuf-java-2.5.0.jar
serializer-2.7.1.jar
slf4j-api-1.6.4.jar
slf4j-log4j12-1.6.4.jar
spring-aop-4.3.7.RELEASE.jar
spring-beans-4.3.7.RELEASE.jar
spring-context-4.3.7.RELEASE.jar
spring-context-support-4.3.7.RELEASE.jar
spring-core-4.3.7.RELEASE.jar
spring-_expression_-4.3.7.RELEASE.jar
spring-jdbc-4.3.7.RELEASE.jar
spring-ldap-core-2.0.2.RELEASE.jar
spring-security-cas-4.0.4.RELEASE.jar
spring-security-config-4.0.4.RELEASE.jar
spring-security-core-4.0.4.RELEASE.jar
spring-security-ldap-4.0.4.RELEASE.jar
spring-security-web-4.0.4.RELEASE.jar
spring-tx-4.3.7.RELEASE.jar
spring-web-4.3.7.RELEASE.jar
spring-webmvc-4.3.7.RELEASE.jar
sqlite-jdbc-3.20.0.jar
stax-1.2.0.jar
stax-api-1.0.1.jar
wicket-core-7.6.0.jar
wicket-dnd-0.7.3.jar
wicket-extensions-7.6.0.jar
wicket-request-7.6.0.jar
wicketstuff-select2-7.6.0.jar
wicket-util-7.6.0.jar
wicket-webjars-0.5.3.jar
xml-apis-1.4.01.jar
xml-apis-ext-1.3.04.jar
xml-commons-resolver-1.2.jar
xmlpull-1.1.3.1.jar
xpp3-1.1.3.4.O.jar
xpp3_min-1.1.4c.jar
xstream-1.4.10.jar
zookeeper-3.4.10.jar

The error stack trace is the following:

2018-06-29 11:45:07,643 ERROR [geoserver.ows] -
java.lang.RuntimeException: Error getting FeatureType, this should never happen!
        at org.geoserver.wms.map.GetMapKvpRequestReader.checkStyle(GetMapKvpRequestReader.java:1076)
        at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:496)
        at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:87)
        at org.geoserver.ows.Dispatcher.parseRequestKVP(Dispatcher.java:1531)
        at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:690)
        at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:260)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)
        at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)
        at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)
        at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
        at org.geoserver.catalog.ResourcePool.getDataStore(ResourcePool.java:690)
        at org.geoserver.catalog.ResourcePool.getCacheableFeatureType(ResourcePool.java:963)
        at org.geoserver.catalog.ResourcePool.tryGetFeatureType(ResourcePool.java:950)
        at org.geoserver.catalog.ResourcePool.getFeatureType(ResourcePool.java:944)
        at org.geoserver.catalog.ResourcePool.getFeatureType(ResourcePool.java:931)
        at org.geoserver.catalog.impl.FeatureTypeInfoImpl.getFeatureType(FeatureTypeInfoImpl.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:147)
        at com.sun.proxy.$Proxy27.getFeatureType(Unknown Source)
        at org.geoserver.wms.map.GetMapKvpRequestReader.checkStyle(GetMapKvpRequestReader.java:1072)
        ... 101 more
Caused by: java.util.concurrent.CompletionException: java.io.IOException: java.lang.reflect.InvocationTargetException
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:929)
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:235)
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:231)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:65)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$.getConnection(HBaseConnectionPool.scala:86)
        at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:44)
        at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:34)
        at org.vfny.geoserver.util.DataStoreUtils.getDataAccess(DataStoreUtils.java:90)
        at org.geoserver.catalog.ResourcePool.getDataStore(ResourcePool.java:650)
        ... 113 more
Caused by: java.util.concurrent.CompletionException: java.io.IOException: java.lang.reflect.InvocationTargetException
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:929)
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:235)
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:231)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:65)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$.getConnection(HBaseConnectionPool.scala:86)
        at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:44)
        at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:34)
        at org.vfny.geoserver.util.DataStoreUtils.getDataAccess(DataStoreUtils.java:90)
        at org.geoserver.catalog.ResourcePool.getDataStore(ResourcePool.java:650)
        ... 113 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
        at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3159)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2$$anon$4.run(HBaseConnectionPool.scala:57)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2$$anon$4.run(HBaseConnectionPool.scala:53)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2.load(HBaseConnectionPool.scala:64)
        at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2.load(HBaseConnectionPool.scala:50)
        at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:922)
        ... 123 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
        ... 131 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
        at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:907)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:691)
        ... 136 more

The architecture is:

• EMR Infrastructure with hadoop , hbase and spark (tried with last emr cluster and emr with hbase v 1.3.x)
  
• geoserver on another machine 

geoserver version:  2.13.1

Thanks in advance,

Gaetano

2018-06-29 18:23 GMT+02:00 Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>:

GeoServer doesn't have to run on any particular machine, although it needs to be able to access the hbase cluster.

The 's3 schema' error is probably related to not having the aws hadoop jars in geoserver - you should be able to fix that just with aws hadoop jars, without needing any hbase jars.

Thanks,

Emilio

On 06/29/2018 12:03 PM, Gaetano Perrone wrote:

Hi Emilio, I forgot an important thing : geoserver is on another machine different than geomachine, so I have uploaded suggested libraries here; is this a problem? Must geoserver run on the same machine of geomesa? (we should avoid this in order to scale the application server)... 

About the issue, could you try to see Catalina and geoserver data logs to see if there is a 'Unable to understand fs  s3 schema' error? If there is this error, geoserver seems to work, but the application is not able to use the coprocessor dynamic library... 

As I can I will send you all jars inside geoserver web inf lib directory in order to solve the issue... 

Thanks in advance, 

Gaetano 

Il ven 29 giu 2018, 17:13 Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> ha scritto:

You can definitely run against emr hbase 1.3, using the normal geomesa shaded jar. I've done that before, but I don't have a list of required jars handy. What additional jars are you copying into the lib folder?

I would guess that if you can get it working against 1.3, it would also work against hbase 1.4 with the same setup.

You might take a look at http://www.geomesa.org/documentation/tutorials/geomesa-hbase-s3-on-aws.html, which doesn't cover geoserver but might have some useful details.

Thanks,

Emilio

On 06/29/2018 06:19 AM, Gaetano Perrone wrote:

Hi Emilio, thanks for the answer;

I have followed all your steps (by using the built master git jar and the last release version 2.0.2) but unfortunately on emr environment I always have the same error:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
    at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:907)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:691)

I also have tried to change emr hbase version to 1.3 as you suggested (and the opposite, compile the plugin with 1.4 hbase version) ...

Do you have any other suggestion ?

Thanks in advance . 

Gaetano

2018-06-28 22:22 GMT+02:00 Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>:

Hello,

That jar list is out of date... I've opened a ticket to fix it here: https://geomesa.atlassian.net/browse/GEOMESA-2312.

I think that the error you're getting is coming from hbase 1.4 vs hbase 1.3 (which we compile against). I believe that if you use the geomesa shaded jar (which bundles hbase 1.3), and use the following additional jars, things should work:

commons-cli-1.2.jar
commons-configuration-1.6.jar
commons-io-2.5.jar
commons-logging-1.1.3.jar
hadoop-auth-2.7.4.jar
hadoop-client-2.7.4.jar
hadoop-common-2.7.4.jar
hadoop-hdfs-2.7.4.jar
htrace-core-3.1.0-incubating.jar
metrics-core-2.2.0.jar
netty-3.6.2.Final.jar
netty-all-4.0.41.Final.jar
servlet-api-2.4.jar
zookeeper-3.4.10.jar

I tested this out a little while ago, although not on aws. If you're using aws/s3, you'll need additional aws-specific hadoop jars (as you discovered).

If you get a list of required aws jars, please write back and I'll include them in the documentation.

Thanks,

Emilio

On 06/28/2018 01:48 PM, Gaetano Perrone wrote:

You must subscribe to this list before posting.  Please go to
https://locationtech.org/mailman/listinfo to subscribe to the list,
then try again.  Perhaps you are already subscribed using another
email address; if that is the case, please send email to this list
using the email address you've used to subscribe.



---------- Messaggio inoltrato ----------
From: Gaetano Perrone <gaetano.perrone@epsilonline.com>
To: geomesa-users@locationtech.org
Cc: 
Bcc: 
Date: Thu, 28 Jun 2018 15:09:44 +0200
Subject: Issue about geomesa-hbase and geoserver plugin (geomesa-hbase_2.11-2.0.2)

Hello,

I have an issue when trying to run geomesa-hbase plugin in geoserver:

I have inserted all the jars inside the geoserver/WEB-INF/lib directory as the guide says, and

I was able to ingest data through geomesa-hbase utility, but I had problems in geoserver gui:

the first problem that I resolved was that geomesa server was not able to read from s3 filesystem (used to read the hbase dynamic library path in order to use the distribuited runtime library for coprocessing).

I resolved by adding the following library:

hadoop-aws-2.8.4.jar

(no info about this in docs, so I don't know if it was the right way).

Now I am not able to resolve the following stack trace error:

java.lang.RuntimeException: Error getting FeatureType, this should never happen!
    at org.geoserver.wms.map.GetMapKvpRequestReader.checkStyle(GetMapKvpRequestReader.java:1076)
    at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:496)
    at org.geoserver.wms.map.GetMapKvpRequestReader.read(GetMapKvpRequestReader.java:87)
    at org.geoserver.ows.Dispatcher.parseRequestKVP(Dispatcher.java:1531)
    at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:690)
    at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:260)
...
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
    at org.geoserver.catalog.ResourcePool.getDataStore(ResourcePool.java:690)
    at org.geoserver.catalog.ResourcePool.getCacheableFeatureType(ResourcePool.java:963)
    at org.geoserver.catalog.ResourcePool.tryGetFeatureType(ResourcePool.java:950)
    at org.geoserver.catalog.ResourcePool.getFeatureType(ResourcePool.java:944)
    at org.geoserver.catalog.ResourcePool.getFeatureType(ResourcePool.java:931)
    at org.geoserver.catalog.impl.FeatureTypeInfoImpl.getFeatureType(FeatureTypeInfoImpl.java:121)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:147)
    at com.sun.proxy.$Proxy124.getFeatureType(Unknown Source)
    at org.geoserver.security.decorators.DecoratingFeatureTypeInfo.getFeatureType(DecoratingFeatureTypeInfo.java:97)
    at org.geoserver.security.decorators.SecuredFeatureTypeInfo.getFeatureType(SecuredFeatureTypeInfo.java:54)
    at org.geoserver.wms.map.GetMapKvpRequestReader.checkStyle(GetMapKvpRequestReader.java:1072)
    ... 101 more
Caused by: java.util.concurrent.CompletionException: java.io.IOException: java.lang.reflect.InvocationTargetException
    at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:929)
    at com.github.benmanes.caffeine.cache.UnboundedLocalCache.lambda$computeIfAbsent$2(UnboundedLocalCache.java:235)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
    at com.github.benmanes.caffeine.cache.UnboundedLocalCache.computeIfAbsent(UnboundedLocalCache.java:231)
    at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
    at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:65)
    at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$.getConnection(HBaseConnectionPool.scala:86)
    at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:44)
    at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:34)
    at org.vfny.geoserver.util.DataStoreUtils.getDataAccess(DataStoreUtils.java:90)
    at org.geoserver.catalog.ResourcePool.getDataStore(ResourcePool.java:650)
    ... 115 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3312)
    at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2$$anon$4.run(HBaseConnectionPool.scala:57)
    at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2$$anon$4.run(HBaseConnectionPool.scala:53)
    at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2.load(HBaseConnectionPool.scala:64)
    at org.locationtech.geomesa.hbase.data.HBaseConnectionPool$$anon$2.load(HBaseConnectionPool.scala:50)
    at com.github.benmanes.caffeine.cache.UnboundedLocalCache$UnboundedLocalLoadingCache.lambda$new$0(UnboundedLocalCache.java:922)
    ... 125 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
    ... 133 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
    at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:947)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:723)
    ... 138 more

This error was generated from geomesa-hbase-gs-plugin_2.11-2.0.2-shaded.jar, anyway the ProtobufUtil was present inside the jar and classpath was acquired by tomcat server .

I have already tried to:

• Compile the plugin by changin the hbase-client version
• Run the tomcat as root (no folder permissions issues)
  
• Use 1.x geomesa-hbase-gs-plugin : it seems to work but cannot read data ingested with newer geomesa-hbase utility version, and I would like to use newer version)
  

Any suggestion about this problem?

Info about the environment:

Hadoop	Hadoop 2.8.3-amzn-1
HBase	HBase 1.4.2
GeoMesa	Geomesa 2.0.2
GeoServer	v2.13.1
GeoMesa Plugin	gs-plugin_2.11-2.0.2-shaded.jar

Thanks very much,

Best Regards,

Gaetano

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users