Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Jetty server crashes
Kapil,

if your server is running out of memory, then it is most likely the application rather than jetty itself.
I would advise:
 + increasing the memory allocated to your production JVMs
 + running a test system with realistic load and profiling the applications memory usage both over time (for leaks) or just for memory hungry operations.

regards


On 12 January 2018 at 05:38, kapil gupta <kapilgupta82@xxxxxxxxx> wrote:
Thanks everyone for your replies. 

Some more points into it:
  1. @Silvio - As you mentioned, yes sometimes OOM killer invokes jetty process to stop, but many of times it doesnt.
  2. @Greg - We are using cloud, and running on CentOS 7. We have stopped updates due to some reasons.
  3. @Joakim - Yes, checked the thread dump and some times OOM killer invokes jetty crash, sometimes I see below properties in dump file:
    2. Current thread (0x00007f18e01b9800):  VMThread [stack: 0x00007f18d33f4000,0x00007f18d34f5000] [id=11584]

    4. Stack: [0x00007f18d33f4000,0x00007f18d34f5000],  sp=0x00007f18d34f3140,  free space=1020k
    5. Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    6. V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
    7. V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
    8. V  [libjvm.so+0x82191e]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
    9. V  [libjvm.so+0x821e69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
    10. V  [libjvm.so+0x81bb6a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
    11. V  [libjvm.so+0x88d623]  PSVirtualSpace::expand_by(unsigned long)+0x53
    12. V  [libjvm.so+0x88e9f8]  PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
    13. V  [libjvm.so+0x88db62]  PSYoungGen::resize(unsigned long, unsigned long)+0x22
    14. V  [libjvm.so+0x88af1d]  PSScavenge::invoke_no_policy()+0xf3d
    15. V  [libjvm.so+0x88b761]  PSScavenge::invoke()+0x41
    16. V  [libjvm.so+0x843f40]  ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
    17. V  [libjvm.so+0x9a4a97]  VM_ParallelGCFailedAllocation::doit()+0x97
    18. V  [libjvm.so+0x9abf35]  VM_Operation::evaluate()+0x55
    19. V  [libjvm.so+0x9aa2fa]  VMThread::evaluate_operation(VM_Operation*)+0xba
    20. V  [libjvm.so+0x9aa67e]  VMThread::loop()+0x1ce
    21. V  [libjvm.so+0x9aaaf0]  VMThread::run()+0x70
    22. V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108
I do not have much of knowledge in to these configuration, need you guys help.
Is this because of my server goes OutOfMemory, if yes then why it doesnt show the memory occupied more in VisualVM when we check heap size


On Thu, Jan 11, 2018 at 3:00 PM, Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx> wrote:
Kapil,

If you are running on Linux check the system log for kernel panics due to lack of memory. Linux will ruthlessly kill the process with highest memory load in such a situation. That has happened to me plenty of times and also left me wondering where my JVM process had gone.

And to add to what Greg said: the System.exit thing can also happen in a library. I used a library once that would do a System.exit in some error situations killing my server now and then. That was a real puzzler.

Cheers,

Silvio



On 10-01-18 12:50, kapil gupta wrote:
Greg,

Actually below things are observed:
  1. No process of jetty server running
  2. MySQL still runs
  3. CPU usage shows data fine
  4. RAM is also fine
Not sure how can we isolate if issue could be because of JVM, can you please guide on setting some parameters or moniotring them.

Thanks

On Wed, Jan 10, 2018 at 1:40 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:

Kapil,

I'm sorry but you've not given us enough information to say anything.  What do you mean by "crashed"?  If it is the JVM that stops, then that is not a Jetty problem.  Is there a stack trace? Is there any logging?  Is the process still running? Can any requests be served? Is the connector still listening?  Can you run with debug?  Why do you think it is jetty and not some other component?

regards



On 10 January 2018 at 05:37, kapil gupta <kapilgupta82@xxxxxxxxx> wrote:
We are using jetty server and when it is on load then the server is crashed without showing much of information.
We have also put it on JMX. The CPU usage and memory looks fine, but still Jetty server crashes. Please let me know what parameters we should look for and how can we isolate issue which is causing jetty server crash.

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users


_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

Back to the top