Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS

From: "Ngo, James T." <James.Ngo@xxxxxxxxx>
Date: Wed, 29 Apr 2015 14:56:26 +0000
Accept-language: en-US
Delivered-to: geomesa-users@xxxxxxxxxxxxxxxx
List-archive: <https://www.locationtech.org/mhonarc/lists/geomesa-users>
List-help: <mailto:geomesa-users-request@locationtech.org?subject=help>
List-subscribe: <http://www.locationtech.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=subscribe>
List-unsubscribe: <http://www.locationtech.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=unsubscribe>
Thread-index: AQHQgQHMxdd6mFKsMUOGaOjKXh4eWZ1kFJSg
Thread-topic: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS

Hi,

I was able to create Kafka DS with org.locationtech.geomesa.utils.interop.SimpleFeatureTypes.createType API. The Kafka DS feature name shows up correctly on GeoServer UI. However, when I try to publish the layer associated with attributes, an exception is raised as follows. Not sure what is wrong with the attribute format and how to fix it. Has anyone seen this before?

Oops, something went wrong...

Sorry, something unexpected happened on the server.

Here's an error report you can include in a JIRA bug report about this issue:

java.lang.RuntimeException: Error occurred while building the resources for the configuration page

at org.geoserver.web.data.layer.NewLayerPage.buildLayerInfo(NewLayerPage.java:308)

at org.geoserver.web.data.layer.NewLayerPage$6.onClick(NewLayerPage.java:243)

at org.geoserver.web.wicket.SimpleAjaxLink$1.onClick(SimpleAjaxLink.java:46)

at org.apache.wicket.ajax.markup.html.AjaxLink$1.onEvent(AjaxLink.java:68)

at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:177)

at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:300)

at org.apache.wicket.request.target.component.listener.BehaviorRequestTarget.processEvents(BehaviorRequestTarget.java:119)

at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)

at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1250)

at org.apache.wicket.RequestCycle.step(RequestCycle.java:1329)

at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1436)

at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)

at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:484)

at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:138)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:159)

at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:28)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:75)

at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:71)

at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:71)

at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:46)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:50)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:54)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)

at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)

at org.geoserver.security.filter.GeoServerUserNamePasswordAuthenticationFilter.doFilter(GeoServerUserNamePasswordAuthenticationFilter.java:116)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)

at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:69)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:53)

at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:73)

at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:92)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)

at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:135)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:76)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:42)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:48)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:44)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)

at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)

at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Malformed attribute

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)

at com.google.common.cache.LocalCache.get(LocalCache.java:3934)

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)

at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)

at org.locationtech.geomesa.kafka.KafkaDataStore.createFeatureSource(KafkaDataStore.scala:107)

at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:387)

at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:352)

at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:663)

at org.geotools.data.store.ContentDataStore.getFeatureSource(ContentDataStore.java:104)

at org.geoserver.catalog.CatalogBuilder.buildFeatureType(CatalogBuilder.java:330)

at org.geoserver.web.data.layer.NewLayerPage.buildLayerInfo(NewLayerPage.java:301)

... 105 more

Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Malformed attribute

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)

at com.google.common.cache.LocalCache.get(LocalCache.java:3934)

at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)

at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)

at org.locationtech.geomesa.kafka.KafkaDataStore.org$locationtech$geomesa$kafka$KafkaDataStore$$createConsumerFeatureSource(KafkaDataStore.scala:121)

at org.locationtech.geomesa.kafka.KafkaDataStore$$anon$2.load(KafkaDataStore.scala:97)

at org.locationtech.geomesa.kafka.KafkaDataStore$$anon$2.load(KafkaDataStore.scala:96)

at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)

at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)

at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)

... 115 more

Caused by: java.lang.IllegalArgumentException: Malformed attribute

at org.locationtech.geomesa.utils.geotools.SimpleFeatureTypes$SpecParser.parse(SimpleFeatureTypes.scala:370)

at org.locationtech.geomesa.utils.geotools.SimpleFeatureTypes$.parse(SimpleFeatureTypes.scala:374)

at org.locationtech.geomesa.utils.geotools.SimpleFeatureTypes$.createType(SimpleFeatureTypes.scala:44)

at org.locationtech.geomesa.kafka.KafkaDataStore$$anonfun$resolveTopicSchema$1.apply(KafkaDataStore.scala:132)

at scala.Option.map(Option.scala:145)

at org.locationtech.geomesa.kafka.KafkaDataStore.resolveTopicSchema(KafkaDataStore.scala:132)

at org.locationtech.geomesa.kafka.KafkaDataStore$$anon$3.load(KafkaDataStore.scala:102)

at org.locationtech.geomesa.kafka.KafkaDataStore$$anon$3.load(KafkaDataStore.scala:100)

at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)

at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)

at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)

... 125 more

From: Emilio Lahr-Vivaz [mailto:elahrvivaz@xxxxxxxx]
Sent: Monday, April 27, 2015 11:50 AM
To: Geomesa User discussions; Ngo, James T.
Subject: Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS

Hi James,

Sorry for the lack of documentation - we'll try to get something better together. In the meantime, hopefully this will help:

To get a feature name to show up in the kafka data store in geoserver, you have to create a schema using a kafka producer data store. This would usually happen outside of geoserver, in some other process. The following code snippet shows how you get a kakfa producer:

Map params = new HashMap();
params.put("brokers ", "my-kafka-brokers");
params.put("zookeepers", "my-zookeepers");
params.put("isProducer", true); // important - this specifies that you will be writing features

DataStore dataStore = org.geotools.data.DataStoreFinder.getDataStore(params);

SimpleFeatureType myType = org.locationtech.geomesa.utils.interop.SimpleFeatureTypes.createType("myType", "name:String,*geom:Point:srid=4326");
dataStore.createSchema(myType);

You should now be able to see the feature name in geoserver, assuming you use the same connection properties. In order to view live features, you will need to write them using the normal GeoTools API using the above producer dataStore. Let me know if you have any questions about that, but there are lots of example on the getools/geoserver sites.

One note: the kafka data store is meant for ephemeral features - if you restart geoserver you will lose any features you have already written.

Thanks,

Emilio

On 04/27/2015 11:28 AM, Ngo, James T. wrote:

Good morning,

I’m newbie to Geomesa and I’m attempting to use geomesa-kafka tool to create “featureName” on “Kafka Data Source” on GeoServer.

I was able to build and deploy geomesa-kafka-geoserver-plugin-accumulo1.5.1.0-rc-5-SNAPSHOT-geoserver-plugin-jar into GeoServer successfully.

From GeoServer UI, I can add a new data store as “Kafka Data Store”. However, I don’t see any FeatureName is showed up to associate with Kafka DS to publish.

Since there is no documentation available on Geomesa-kafka, please provide us with more information how to publish FeatureName for each Kafka topic.

Thanks.

James Ngo

GD-MS

Senior Software Engineer
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

Follow-Ups:
- Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
  - From: Emilio Lahr-Vivaz

References:
- [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
  - From: Ngo, James T.
- Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
  - From: Emilio Lahr-Vivaz

Prev by Date: Re: [geomesa-users] geoserver - geomesa
Next by Date: Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
Previous by thread: Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
Next by thread: Re: [geomesa-users] geomesa-kafka: how to create topics/featureName for Kafka DS
Index(es):
- Date
- Thread

Breadcrumbs