Hi Rémi,
Thanks for checking. I was able to find the thread leak and put up a
fix[1]. It should go into our next release. For now, using a single
data store instance should mitigate the problem for you.
Thanks,
Emilio
[1] https://github.com/locationtech/geomesa/pull/1672
On 08/31/2017 05:11 AM, GRISOT, REMI
wrote:
Hi Emilio,
I checked the code, and the datastore is never used after the
call to the dispose method, so it comes maybe from the other
thread you mentioned.
Thank you for the information!
Rémi
I did find another thread that isn't being cleaned up
correctly. I'll put up a fix soon.
Thanks,
Emilio
On 08/30/2017 12:09 PM, Emilio
Lahr-Vivaz wrote:
Hi Rémi,
Is it possible that you were still using the data store
(or associated feature stores/writers) after calling
dispose()? I've noticed that if you call dispose() on a
datastore, then write features, it may open a connection
in the metadata class that is never closed. This may be
why your initial thread dump showed so many metadata
connections. Other than that, it appears things are being
cleaned up correctly, but I will keep looking into it.
Thanks,
Emilio
On 08/30/2017 11:34 AM,
GRISOT, REMI wrote:
I use geomesa 1.3.1 but
I tested other versions with the same problem : 1.3.0
and 1.3.3.
Thank you for the link, I will follow this.
Rémi
Ah, good to hear. I've created an issue here to
track the .dispose() issue:
https://geomesa.atlassian.net/browse/GEOMESA-2003
Which version of geomesa are you using?
Thanks,
Emilio
On 08/30/2017 11:18
AM, GRISOT, REMI wrote:
Hi Emilio,
Using a single data store instance shared
between all the threads, solved the problem:
the number of threads remains stable during
the ingest.
Thank you very much!
Rémi
Hi Rémi,
I gave you code a quick look and I
don't see any obvious resource leaks.
Generally if you use
getFeatureStore.addFeatures(), you
don't need to clean up any feature
writers. If you use getFeatureWriter*,
then you need to close it yourself.
Calling .dispose on the datastore
should clean up the metadata
connections, but won't close any
feature writers.
You might try keeping a single data
store instance around for the entire
ingest - generally a data store object
is fairly expensive to create, so only
one is created and shared across
threads. Possibly the
datastore.dispose() method is not
cleaning things up appropriately, but
that may not be apparent unless many
datastore instances are created (as
you're doing). I will check into that.
Thanks,
Emilio
On 08/30/2017
03:58 AM, GRISOT, REMI wrote:
Hi Emilio,
Thank you for your quick answer. I
tested an ingest based on the
Quickstart example, and the problem
definitely comes from my
implementation : in the test, the
number of threads stayed stable.
For my insertions, I do not use a
distributed setup: I have only one
machine, with a lot of RAM and CPUs.
In a later time, I will probably split
it into VMs, but for now, I would like
to make it work on a single node.
The ingest program works as follow:
- Datas are collected, and when a
limit is reached, a thread is created
(in a ThreadPoolExecutor). - That
thread create an instance of
DataStore, send datas and then destroy
the instance of Datastore (which I
thought should have closed feature
writers).
It is really nice to you to propose
check the code for resource cleanup!
The code is not public (I write it in
the field of my internship), but I
changed some variables name, and it
should be OK. I attached an archive
containing the relevant files of the
project.
I will have a look to the
documentation links and try to imrove
my conf with this.
Thank you,
Rémi Grisot
Hi Rémi,
What is your ingest setup like? If
you're seeing thread exhaustion,
my first guess would be that you
are not cleaning up resources
correctly (e.g. closing feature
writers). Are you distributing
your ingest? Usually for data sets
that large you would distribute
your ingestion across a cluster
using spark or map/reduce, so I'm
not sure we've tested billions of
points from a single JVM.
The metadata batch writer is
configured to use 2 threads, so
there may be others generated by
Accumulo but I wouldn't expect it
to swamp the JVM by itself. It
will also only write mutations
when you call 'flush' or 'close'
on a feature writer, so minimizing
those calls would reduce the
writes.
There are other batch writers
associated with each feature
writer you create; those are more
likely to be causing issues than
the metadata writer as there are
more of them, with more threads
and greater throughput.
You can control the underlying
batch writer configuration via
system properties as described
here [1], although the metadata
writer overrides max memory and
max threads [2].
[1]
http://www.geomesa.org/documentation/user/accumulo/configuratiissues.on.html#batch-writer-properties,
[2]
https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/data/AccumuloBackedMetadata.scala#L30
If you code is public, I'd be glad
to give it a quick check for
resource cleanup.
Thanks,
Emilio
On
08/29/2017 09:55 AM, GRISOT,
REMI wrote:
Ce message et toutes les pièces
jointes (ci-après le "message")
sont établis à l’intention
exclusive des destinataires
désignés. Il contient des
informations confidentielles et
pouvant être protégé par le
secret professionnel. Si vous
recevez ce message par erreur,
merci d'en avertir immédiatement
l'expéditeur et de détruire le
message. Toute utilisation de ce
message non conforme à sa
destination, toute diffusion ou
toute publication, totale ou
partielle, est interdite, sauf
autorisation expresse de
l’émetteur. L'internet ne
garantissant pas l'intégrité de
ce message lors de son
acheminement, Atos (et ses
filiales) décline(nt) toute
responsabilité au titre de son
contenu. Bien que ce message ait
fait l’objet d’un traitement
anti-virus lors de son envoi,
l’émetteur ne peut garantir
l’absence totale de logiciels
malveillants dans son contenu et
ne pourrait être tenu pour
responsable des dommages
engendrés par la transmission de
l’un d’eux.
This message and any attachments
(the "message") are intended
solely for the addressee(s). It
contains confidential
information, that may be
privileged. If you receive this
message in error, please notify
the sender immediately and
delete the message. Any use of
the message in violation of its
purpose, any dissemination or
disclosure, either wholly or
partially is strictly
prohibited, unless it has been
explicitly authorized by the
sender. As its integrity cannot
be secured on the internet, Atos
and its subsidiaries decline any
liability for the content of
this message. Although the
sender endeavors to maintain a
computer virus-free network, the
sender does not warrant that
this transmission is virus-free
and will not be liable for any
damages resulting from any virus
transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
Ce message et toutes les pièces jointes
(ci-après le "message") sont établis à
l’intention exclusive des destinataires
désignés. Il contient des informations
confidentielles et pouvant être protégé
par le secret professionnel. Si vous
recevez ce message par erreur, merci
d'en avertir immédiatement l'expéditeur
et de détruire le message. Toute
utilisation de ce message non conforme à
sa destination, toute diffusion ou toute
publication, totale ou partielle, est
interdite, sauf autorisation expresse de
l’émetteur. L'internet ne garantissant
pas l'intégrité de ce message lors de
son acheminement, Atos (et ses filiales)
décline(nt) toute responsabilité au
titre de son contenu. Bien que ce
message ait fait l’objet d’un traitement
anti-virus lors de son envoi, l’émetteur
ne peut garantir l’absence totale de
logiciels malveillants dans son contenu
et ne pourrait être tenu pour
responsable des dommages engendrés par
la transmission de l’un d’eux.
This message and any attachments (the
"message") are intended solely for the
addressee(s). It contains confidential
information, that may be privileged. If
you receive this message in error,
please notify the sender immediately and
delete the message. Any use of the
message in violation of its purpose, any
dissemination or disclosure, either
wholly or partially is strictly
prohibited, unless it has been
explicitly authorized by the sender. As
its integrity cannot be secured on the
internet, Atos and its subsidiaries
decline any liability for the content of
this message. Although the sender
endeavors to maintain a computer
virus-free network, the sender does not
warrant that this transmission is
virus-free and will not be liable for
any damages resulting from any virus
transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
Ce message et toutes les pièces jointes
(ci-après le "message") sont établis à
l’intention exclusive des destinataires
désignés. Il contient des informations
confidentielles et pouvant être protégé par le
secret professionnel. Si vous recevez ce message
par erreur, merci d'en avertir immédiatement
l'expéditeur et de détruire le message. Toute
utilisation de ce message non conforme à sa
destination, toute diffusion ou toute
publication, totale ou partielle, est interdite,
sauf autorisation expresse de l’émetteur.
L'internet ne garantissant pas l'intégrité de ce
message lors de son acheminement, Atos (et ses
filiales) décline(nt) toute responsabilité au
titre de son contenu. Bien que ce message ait
fait l’objet d’un traitement anti-virus lors de
son envoi, l’émetteur ne peut garantir l’absence
totale de logiciels malveillants dans son
contenu et ne pourrait être tenu pour
responsable des dommages engendrés par la
transmission de l’un d’eux.
This message and any attachments (the "message")
are intended solely for the addressee(s). It
contains confidential information, that may be
privileged. If you receive this message in
error, please notify the sender immediately and
delete the message. Any use of the message in
violation of its purpose, any dissemination or
disclosure, either wholly or partially is
strictly prohibited, unless it has been
explicitly authorized by the sender. As its
integrity cannot be secured on the internet,
Atos and its subsidiaries decline any liability
for the content of this message. Although the
sender endeavors to maintain a computer
virus-free network, the sender does not warrant
that this transmission is virus-free and will
not be liable for any damages resulting from any
virus transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
Ce message et toutes les pièces jointes (ci-après le
"message") sont établis à l’intention exclusive des
destinataires désignés. Il contient des informations
confidentielles et pouvant être protégé par le secret
professionnel. Si vous recevez ce message par erreur,
merci d'en avertir immédiatement l'expéditeur et de
détruire le message. Toute utilisation de ce message non
conforme à sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite, sauf
autorisation expresse de l’émetteur. L'internet ne
garantissant pas l'intégrité de ce message lors de son
acheminement, Atos (et ses filiales) décline(nt) toute
responsabilité au titre de son contenu. Bien que ce
message ait fait l’objet d’un traitement anti-virus lors
de son envoi, l’émetteur ne peut garantir l’absence
totale de logiciels malveillants dans son contenu et ne
pourrait être tenu pour responsable des dommages
engendrés par la transmission de l’un d’eux.
This message and any attachments (the "message") are
intended solely for the addressee(s). It contains
confidential information, that may be privileged. If you
receive this message in error, please notify the sender
immediately and delete the message. Any use of the
message in violation of its purpose, any dissemination
or disclosure, either wholly or partially is strictly
prohibited, unless it has been explicitly authorized by
the sender. As its integrity cannot be secured on the
internet, Atos and its subsidiaries decline any
liability for the content of this message. Although the
sender endeavors to maintain a computer virus-free
network, the sender does not warrant that this
transmission is virus-free and will not be liable for
any damages resulting from any virus transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
Ce message et toutes les pièces jointes (ci-après le "message")
sont établis à l’intention exclusive des destinataires désignés.
Il contient des informations confidentielles et pouvant être
protégé par le secret professionnel. Si vous recevez ce message
par erreur, merci d'en avertir immédiatement l'expéditeur et de
détruire le message. Toute utilisation de ce message non conforme
à sa destination, toute diffusion ou toute publication, totale ou
partielle, est interdite, sauf autorisation expresse de
l’émetteur. L'internet ne garantissant pas l'intégrité de ce
message lors de son acheminement, Atos (et ses filiales)
décline(nt) toute responsabilité au titre de son contenu. Bien que
ce message ait fait l’objet d’un traitement anti-virus lors de son
envoi, l’émetteur ne peut garantir l’absence totale de logiciels
malveillants dans son contenu et ne pourrait être tenu pour
responsable des dommages engendrés par la transmission de l’un
d’eux.
This message and any attachments (the "message") are intended
solely for the addressee(s). It contains confidential information,
that may be privileged. If you receive this message in error,
please notify the sender immediately and delete the message. Any
use of the message in violation of its purpose, any dissemination
or disclosure, either wholly or partially is strictly prohibited,
unless it has been explicitly authorized by the sender. As its
integrity cannot be secured on the internet, Atos and its
subsidiaries decline any liability for the content of this
message. Although the sender endeavors to maintain a computer
virus-free network, the sender does not warrant that this
transmission is virus-free and will not be liable for any damages
resulting from any virus transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
|