Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] kafka streaming
You are right Emilio.
Thanks a lot,
Matteo.

Matteo Cusmai

On Thu, Dec 7, 2017 at 5:34 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
That's the KafkaDataStore (or just plain DataStore) that you get back from DataStoreFinder. It's part of the GeoTools API. There's also a 'remove-schema' command in the GeoMesa Kafka command line tools, part of the binary distribution, which does the same thing.

Thanks,

Emilio


On 12/07/2017 11:23 AM, Matteo Cusmai wrote:
Which is the class containing removeSchema method?

Matteo Cusmai

On Thu, Dec 7, 2017 at 2:43 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
The topics are created when you call .createSchema(), and deleted when you call .removeSchema(). However, most versions of Kafka do not actually enable topic deletion by default, so you have to modify your Kafka install to allow it. I believe the config is 'delete.topic.enable=true' in server.properties.

Alternatively, you can also specify a different 'namespace', i.e. the 'zkPath' data store parameter. Feature types are tracked per zkPath, and topic names have the zkPath prepended, so you can use the same feature type name with different zkPaths without any conflicts.

Thanks,

Emilio


On 12/07/2017 04:25 AM, Matteo Cusmai wrote:
Hi Emilio,
thanks for the suggestions.

I have another question about kafka, when I use the following instructions on producer side

final String sftName = "KafkaQuickStart7";
final String sftSchema = "name:String,age:Int,*geom:Point:srid=4326";
SimpleFeatureType sft = SimpleFeatureTypes.createType(sftName, sftSchema);
SimpleFeatureType streamingSFT = KafkaDataStoreHelper.createStreamingSFT(sft, zkPath);

It seems it creates a durable topic on kafka, and I cannot reuse the same sftName. There is a way to create temporary topic or reuse the same one?

Thanks a lot,
Matteo.


Matteo Cusmai

On Mon, Dec 4, 2017 at 5:15 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
You can register a feature event listener on a feature source, but there is no way to register a filter through the geotools API, so you'll have to check for events that interest you in your listener. The events that you will receive are outlined here:

https://github.com/locationtech/geomesa/blob/1.3.x/geomesa-kafka/geomesa-kafka-datastore/geomesa-kafka-datastore-common/src/main/scala/org/locationtech/geomesa/kafka/KafkaFeatureEvent.scala#L55-L71

You can see an example of using a listener here:

https://github.com/locationtech/geomesa/blob/1.3.x/geomesa-kafka/geomesa-kafka-datastore/geomesa-kafka-09-datastore/src/test/scala/org/locationtech/geomesa/kafka09/LiveKafkaConsumerFeatureSourceTest.scala#L146-L158

The 'TestLambdaFeatureListener' is just a helper class that accepts a lambda (closure) and executes it on events:

https://github.com/locationtech/geomesa/blob/1.3.x/geomesa-kafka/geomesa-kafka-datastore/geomesa-kafka-datastore-common/src/test/scala/org/locationtech/geomesa/kafka/TestLambdaFeatureListener.scala#L14

hope that helps,

Emilio


On 12/04/2017 10:53 AM, Matteo Cusmai wrote:
Hi Emilio,
thanks a lot for the answer.
I was wronging because I was using 1.4.0-SNAPSHOT. Now I am using 1.3.4 and I see the helper class.

By the way, I have another question: is it possible to use FeatureListener with a such of CQL Filter? I would like to subcribe to a topic by specifing a BBOX.

Thanks,
Matteo.


Matteo Cusmai

On Mon, Dec 4, 2017 at 2:48 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hello,

The KafkaDataStoreHelper is here:

https://github.com/locationtech/geomesa/blob/geomesa_2.11-1.3.5/geomesa-kafka/geomesa-kafka-datastore/geomesa-kafka-datastore-common/src/main/scala/org/locationtech/geomesa/kafka/KafkaDataStoreHelper.scala

However, this class has been removed in the current master branch (1.4.0-SNAPSHOT), so make sure you are using a 1.3.x version of GeoMesa. If you'd prefer to use the 1.4.x version, the docs for that are not yet hosted anywhere, but can be seen here:

https://github.com/locationtech/geomesa/pull/1777

To manually build the 1.4 docs, you can pull that branch and run:

mvn clean install -P docs -pl docs

that will generate the HTML files under docs/target/html/.

Thanks,

Emilio


On 12/02/2017 12:02 PM, Matteo Cusmai wrote:
Hi all,
I am trying to use kafka geomesa datastore capability, but I am not able to use code at http://www.geomesa.org/documentation/user/kafka/producers.html

I cannot resolve KafkaDataStoreHelper class. Which is the right dependency?

Thanks a lot.

Matteo Cusmai


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Back to the top