[orion-dev] Orion out of memory errors explained

[John Arthorne <John_Arthorne@xxxxxxxxxx>]

Last week we started getting out of memory
errors on orion.eclipse.org. I have finally figured out what was going
on, and am posting a synopsis here in case anyone else running into this
problem on their own Orion servers (and for future google searches on the
topic).  The errors we were getting looked like this:

Caused by: java.io.IOException: Map
failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:770)
        at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:705)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4400)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

The problem was persistent - on server
restart it would fail again within 30 minutes. The end user symptom was
that indexed search operations on the client side were completely broken.


This is not a JVM heap space problem.
It is trying to allocate direct non-heap memory for the purpose of doing
memory-mapped I/O. The physical machine we are running the server on has
6GB of RAM, of which 4GB was allocated as JVM heap space. The server is
a SLES 11 machine, configured to set a memory limit per process (ulimit
-v)  of roughly 5GB.  It looks like essentially we were allocating
too much memory to the heap, and there was not enough memory left available
to the process for allocating non-heap memory, such as memory mapped file
buffers, class file space, etc. I resolved the problem by actually *decreasing*
the JVM allocation to 3GB, therefore leaving more memory space available
to the process for non-heap memory. After decreasing the heap size to 3GB
we have run for 4 days with no further errors. The problem could alternatively
be solved by increasing the per-process virtual memory limit. Today I have
increased this on orion.eclipse.org to 10GB just to give some breathing
room. That will enable us to increase the heap again if needed without
running into this problem again.

John