Re: [geomesa-users] Use geomesa processes with Spark

["GRISOT, REMI" <remi.grisot@xxxxxxxx>]

Hi Emilio,
Thank you for your answer! I'll try that.

Thank you,
Rémi

On 04/03/2019 17:09, Emilio Lahr-Vivaz wrote :
> Hello,
>
> A feature collection is kind of like a pointer to remote data, so it's
> generally not serializable. However, you can do an initial query through
> spark to load the features with speed = 0, then you can map each feature
> in your rdd/dataframe into a KNN query. You should be able to invoke the
> KNN process through code by instantiating the process, creating a local
> feature collection and invoking 'execute', similar to this unit test for
> the query process:
> https://github.com/locationtech/geomesa/blob/geomesa_2.11-2.2.1/geomesa-process/geomesa-process-vector/src/test/scala/org/locationtech/geomesa/process/query/QueryProcessTest.scala#L25
>
> Thanks,
>
> Emilio
>
> On 3/4/19 5:40 AM, GRISOT, REMI wrote:
>> Hello,
>>
>> I have position history for some cabs and I would like to find the
>> points of interests where they stopped.
>> To achieve this, I used geomesa processes (query and knn): first, I keep
>> only features when speed is 0 (query process) and then foreach feature,
>> I do a knn-search with k=1 and max radius = 500m.
>> This works fine, however I would like to parallelize this over a spark
>> cluster. So I converted my FeatureCollection into a spark RDD and tried
>> to apply filter and knn on each row. But I got an
>> NotSerializableException because AccumuloFeatureCollection is not
>> serializable.
>>
>> I saw that some geospatial functions are available in spark using jts
>> module, however I could not find KNN.
>> Is there a way to use geomesa processes with spark?
>>
>> Thank you!
>>
>> Rémi
>>
>> Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l’émetteur. L'internet ne garantissant pas l'intégrité de ce message lors de son acheminement, Atos (et ses filiales) décline(nt) toute responsabilité au titre de son contenu. Bien que ce message ait fait l’objet d’un traitement anti-virus lors de son envoi, l’émetteur ne peut garantir l’absence totale de logiciels malveillants dans son contenu et ne pourrait être tenu pour responsable des dommages engendrés par la transmission de l’un d’eux.
>>
>> This message and any attachments (the "message") are intended solely for the addressee(s). It contains confidential information, that may be privileged. If you receive this message in error, please notify the sender immediately and delete the message. Any use of the message in violation of its purpose, any dissemination or disclosure, either wholly or partially is strictly prohibited, unless it has been explicitly authorized by the sender. As its integrity cannot be secured on the internet, Atos and its subsidiaries decline any liability for the content of this message. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.locationtech.org/mailman/listinfo/geomesa-users


Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l’émetteur. L'internet ne garantissant pas l'intégrité de ce message lors de son acheminement, Atos (et ses filiales) décline(nt) toute responsabilité au titre de son contenu. Bien que ce message ait fait l’objet d’un traitement anti-virus lors de son envoi, l’émetteur ne peut garantir l’absence totale de logiciels malveillants dans son contenu et ne pourrait être tenu pour responsable des dommages engendrés par la transmission de l’un d’eux.

This message and any attachments (the "message") are intended solely for the addressee(s). It contains confidential information, that may be privileged. If you receive this message in error, please notify the sender immediately and delete the message. Any use of the message in violation of its purpose, any dissemination or disclosure, either wholly or partially is strictly prohibited, unless it has been explicitly authorized by the sender. As its integrity cannot be secured on the internet, Atos and its subsidiaries decline any liability for the content of this message. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.