Hi Rémi,
What is your ingest setup like? If you're seeing thread exhaustion,
my first guess would be that you are not cleaning up resources
correctly (e.g. closing feature writers). Are you distributing your
ingest? Usually for data sets that large you would distribute your
ingestion across a cluster using spark or map/reduce, so I'm not
sure we've tested billions of points from a single JVM.
The metadata batch writer is configured to use 2 threads, so there
may be others generated by Accumulo but I wouldn't expect it to
swamp the JVM by itself. It will also only write mutations when you
call 'flush' or 'close' on a feature writer, so minimizing those
calls would reduce the writes.
There are other batch writers associated with each feature writer
you create; those are more likely to be causing issues than the
metadata writer as there are more of them, with more threads and
greater throughput.
You can control the underlying batch writer configuration via system
properties as described here [1], although the metadata writer
overrides max memory and max threads [2].
[1]
http://www.geomesa.org/documentation/user/accumulo/configuratiissues.on.html#batch-writer-properties,
[2]
https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/data/AccumuloBackedMetadata.scala#L30
If you code is public, I'd be glad to give it a quick check for
resource cleanup.
Thanks,
Emilio
On 08/29/2017 09:55 AM, GRISOT, REMI
wrote:
Ce message et toutes les pièces jointes (ci-après le "message")
sont établis à l’intention exclusive des destinataires désignés.
Il contient des informations confidentielles et pouvant être
protégé par le secret professionnel. Si vous recevez ce message
par erreur, merci d'en avertir immédiatement l'expéditeur et de
détruire le message. Toute utilisation de ce message non conforme
à sa destination, toute diffusion ou toute publication, totale ou
partielle, est interdite, sauf autorisation expresse de
l’émetteur. L'internet ne garantissant pas l'intégrité de ce
message lors de son acheminement, Atos (et ses filiales)
décline(nt) toute responsabilité au titre de son contenu. Bien que
ce message ait fait l’objet d’un traitement anti-virus lors de son
envoi, l’émetteur ne peut garantir l’absence totale de logiciels
malveillants dans son contenu et ne pourrait être tenu pour
responsable des dommages engendrés par la transmission de l’un
d’eux.
This message and any attachments (the "message") are intended
solely for the addressee(s). It contains confidential information,
that may be privileged. If you receive this message in error,
please notify the sender immediately and delete the message. Any
use of the message in violation of its purpose, any dissemination
or disclosure, either wholly or partially is strictly prohibited,
unless it has been explicitly authorized by the sender. As its
integrity cannot be secured on the internet, Atos and its
subsidiaries decline any liability for the content of this
message. Although the sender endeavors to maintain a computer
virus-free network, the sender does not warrant that this
transmission is virus-free and will not be liable for any damages
resulting from any virus transmitted.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
|