[smila-dev] OutOfMemoryException during Crawl

[<Daniel.Stucky@xxxxxxxxxxx>]

Hi all,

during testing we encountered an OutOfMemoryException. My first thought
was that the cause is DeltaIndexingManager, as it holds its state in
memory. But it was not the case, as the problem remained after disabling
it. So we did some more tests (disabling different components) to track
down the root of the problem. Here are some interesting observations we
made:

BinStore: the BinStore allocates as much memory as possible, without
freeing any. With the BinStore active the system reaches the defined XmX
quickly. The OOM takes a lot longer to come in effect though. I guess
it's related to the caching mechanisms of Commons VFS. I don't
understand why no memory is freed if it's near reaching XmX. The
behavior is the same whether XmX is 512m or 64m. If BinStore is
disabled, the system does not reach XmX, but uses an average of 15 MB !

However, the BinStore seems not to be the real problem, but it makes the
OOM easier to occur, as it uses so much memory.
We did a test (indexing ov ofer 200.000 documents) with only
Connectivity, XMLStore and BinStore (no routing, no BPEL, no processing
was executed, the services where started though). The Memory was quickly
near reaching XmX, but all documents where processed (that means
converted to records, send and stored in XML- and BinStore) in 2 hours.
Several hours later the OOM exception occurred, without the system doing
anything.

Then I noticed that by just starting EILF.exe and waiting the following
happens:
- every minute 3 threads are created and deleted. The number of active
threads is avg 40, but the total number of created threads increases
constantly. The names of those threads were "Persistence Adaptor Task"
and "RMI TCP Connection(8)-172.24.187.35". I guess it's related to
activmq
- the memory usage also increases. It increases very slowly, but the
graph is growing linear. I think that this behavior finally causes the
OOM Exception. 

So there are 3 questions I cannot answer:
1) why does BinStore use so much memory and does not free anything ?
2) why are those threads created continually and what are they for ?
3) what causes this slow but linear increasing consumption of memory ?

Any ideas, suggestions, or solutions ?

Bye,
Daniel