The number of thread which could be allocated by the JVM is
limited by severals OS settings.
I suppose you are running on a linux machine as you are talking
about "ulimit".
On my laptop I was limited around 10,800 threads by default.
After tweaking my debian configuration I was able to run ~120,000
threads.
A default Leshan client uses 10 threads :
- 1 UDP sender (from element-connector)
- 1 UDP receiver (from element-connector)
- 2 Deduplicator (from californium)
- 1 Coap Server (from californium)
- 1 DTLS sender (from scandium)
- 1 DTLS receiver (from scandium)
- 1 DTLS retransmit (from scandium)
- 1 DTLS connectionHandler (from scandium)
- 1 registration engine (from leshan)
You can reduce it by removing 3(UDP+1deduplicator) or
5(DTLS+1deduplicator) threads if your are using only UDP or only
DTLS.
To do that LeshanClientBuilder.disableUnsecuredEndpoint() or
disableSecuredEndpoint()...
But this is still 5 or 7 threads by clients for a simulator which
would run thousands of clients that's means thousands of threads
... there is no way this was a good design... We should fix this
so I open an issue about that :
https://github.com/eclipse/leshan/issues/491
I suppose we should change the code a bit to be able to use only
one thread pool executor for all the clients.
Currently this is partially done, you can set a common executor
for DTLSConnectionHandler and CoAP server so you will save 2 more
thread by clients.
With Leshan 1.0.0-M5, the tricks looks like this :
final ScheduledExecutorService executor = Executors.newScheduledThreadPool(1); // the executor to use for all clients
builder.setEndpointFactory(new EndpointFactory() {
@Override
public CoapEndpoint createUnsecuredEndpoint(InetSocketAddress address, NetworkConfig coapConfig,
ObservationStore store) {
return new CoapEndpoint(address, coapConfig);
}
@Override
public CoapEndpoint createSecuredEndpoint(DtlsConnectorConfig dtlsConfig, NetworkConfig coapConfig,
ObservationStore store) {
DTLSConnector dtlsConnector = new DTLSConnector(dtlsConfig);
dtlsConnector.setExecutor(new StripedExecutorService(executor));
return new CoapEndpoint(dtlsConnector, coapConfig, null, null);
}
});
client = builder.build();
client.getCoapServer().setExecutor(executor);
Waiting we are better you can try to play with linux limits
configuration but maybe you will face Azure limitation.
Personnaly, I was first limited by UserTasksMax=33% in
/etc/systemd/logind.conf and /proc/sys/kernel/pid_max set to
32768. (~10900)
Some resources which could help you :
https://stackoverflow.com/questions/344203/maximum-number-of-threads-per-process-in-linux/26190804#26190804
https://web.archive.org/web/20111209081734/http://research.cs.wisc.edu/condor/condorg/linux_scalability.html#Process%20Identifiers
http://geekswing.com/geek/quickie-tutorial-ulimit-soft-limits-hard-limits-soft-stack-hard-stack/
I was using this kind of code to check my modification :
public class MaxThreadCounter {
private static int count = 0;
public static void main(String[] argv) {
System.out.println("Counting ...");
while (true) {
count++;
try {
new Thread(new Runnable() {
public void run() {
while (true) {
try {
Thread.sleep(1000);
} catch (Throwable e) {
e.printStackTrace();
}
}
}
}).start();
} catch (Throwable e) {
System.err.println("Number of thread :" + count);
e.printStackTrace();
try {
Thread.sleep(5000);
} catch (InterruptedException e1) {
}
System.exit(0);
}
}
}
}
(This is a bit painful, some modification need reboot, others
does not persist after a reboot ...If I well understand this is
also not a so good practice to allow so much thread...)
If you are interested by the issue 491, do not hesitate to
contribute code, I can even guide you a little.
HTH
Simon
Le 21/03/2018 à 01:48, Bhonde, Jyotsna
a écrit :
Hi
We are scale testing. Our simulator is in
Azure and what we have noticed is that we get following
exception when we have about 3500 leshan simulators.
EXCEPTION:OutOfMemoryError:
unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:717)
org.eclipse.californium.elements.UDPConnector.start(UDPConnector.java:169)
org.eclipse.californium.core.network.CoapEndpoint.start(CoapEndpoint.java:418)
org.eclipse.californium.core.CoapServer.start(CoapServer.java:215)
org.eclipse.leshan.client.californium.LeshanClient.start(LeshanClient.java:173)
....
....
Has anybody come across this problem and
any suggestions on how to go about resolving this problem?
Available memory to the container is 25gb
ram and it has used up 10gb.
Settings at the docker engine level are are
follows:
/ # ulimit -a
-f: file size
(blocks) unlimited
-t: cpu time
(seconds) unlimited
-d: data seg size
(kb) unlimited
-s: stack size
(kb) 8192
-c: core file size
(blocks) 0
-m: resident set size
(kb) unlimited
-l: locked memory
(kb) 82000
-p:
processes unlimited
-n: file
descriptors 1048576
-v: address space
(kb) unlimited
-w:
locks unlimited
-e: scheduling priority
0
-r: real-time
priority 0
I was looking in to tweaking these
properties: NETWORK_STAGE_RECEIVER_THREAD_COUNT and
NETWORK_STAGE_SENDER_THREAD_COUNT in californium.properties.
But I don't think that will help much since it seems the
resource issue is at the system level.
Thanks
J.
NOTICE OF
CONFIDENTIALITY:
This message may contain information that is considered
confidential and which may be prohibited from disclosure under
applicable law or by contractual agreement. The information is
intended solely for the use of the individual or entity named
above. If you are not the intended recipient, you are hereby
notified that any disclosure, copying, distribution or use of
the information contained in or attached to this message is
strictly prohibited. If you have received this email
transmission in error, please notify the sender by replying to
this email and then delete it from your system.
_______________________________________________
leshan-dev mailing list
leshan-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/leshan-dev