Eclipse Community Forums: Remote Application Platform (RAP)

Help

Home

Home » Eclipse Projects » Remote Application Platform (RAP) » Memory-Problems

Show: Today's Messages :: Show Polls :: Message Navigator

Memory-Problems [message #119486]

Sat, 24 January 2009 15:38

Andrej Dimic

Messages: 77
Registered: July 2009

Member

Hi

i know this gone be difficult but maybe someone can point me in a
direction i didn't think of.

This is the problem:
we run a RAP-website that loses constantly heap-memory in the JVM until
all allocated Heap (750MB) is used up. This takes about a weeks time.

Environment:
Suse Linux Enterprise Server 10
JVM 6.0.11
tomcat 6.0.18
RAP 1.2 M5
Sql Anywhere 10 database

the application uses primarily the TableViewer Class (slightly adapted)
taken from the RAP Demo. Of course also Database connection. But the
database access classes have been developed for a very long time (since
JDK 1.1) and have been used in many tomcat containers without losing
memory ever.

what we have tried so far:
- eliminated all statics except final constants and the database
connection pooling class.
- profiling the java classes in JVM. Here we see only that primitive data
types and basic classes are growing (char[], Object, TreeMap).
- stress the RAP-Application with JMeter Testplans against our
developement environment (which is windows / netbeans). There we could not
reproduce any memory loss!!!

I knwo this description is rather vague

I must admit, i am at the end of my knowledge.

any helpful hints would be really appreciated.

Regards
Andrej

Report message to a moderator

Re: Memory-Problems [message #119506 is a reply to message #119486]

Sat, 24 January 2009 19:26

Benjamin Muskalla

Messages: 338
Registered: July 2009

Senior Member

Hi Andrej,

you're right - your problem description is really vague.

One thing you could try is to tell the VM to write a heapdump on a OOME
(-XX:+HeapDumpOnOutOfMemoryError). If you app crashes again, just
inspect the heap dump with memory analyzing tool.
I can really recommend MAT (www.eclipse.org/mat) to inspect the heap
dump to see what's going wrong / what is causing the lost memory.

Cheers
Benny

adiinfo wrote:
> Hi
>
> i know this gone be difficult but maybe someone can point me in a
> direction i didn't think of.
>
> This is the problem:
> we run a RAP-website that loses constantly heap-memory in the JVM until
> all allocated Heap (750MB) is used up. This takes about a weeks time.
>
> Environment:
> Suse Linux Enterprise Server 10
> JVM 6.0.11
> tomcat 6.0.18
> RAP 1.2 M5
> Sql Anywhere 10 database
>
> the application uses primarily the TableViewer Class (slightly adapted)
> taken from the RAP Demo. Of course also Database connection. But the
> database access classes have been developed for a very long time (since
> JDK 1.1) and have been used in many tomcat containers without losing
> memory ever.
>
> what we have tried so far:
> - eliminated all statics except final constants and the database
> connection pooling class.
> - profiling the java classes in JVM. Here we see only that primitive
> data types and basic classes are growing (char[], Object, TreeMap).
> - stress the RAP-Application with JMeter Testplans against our
> developement environment (which is windows / netbeans). There we could
> not reproduce any memory loss!!!
>
> I knwo this description is rather vague
>
> I must admit, i am at the end of my knowledge.
>
> any helpful hints would be really appreciated.
>
> Regards
> Andrej
>
>

Report message to a moderator

Re: Memory-Problems [message #119519 is a reply to message #119506]

Sat, 24 January 2009 22:44

Andrej Dimic

Messages: 77
Registered: July 2009

Member

Hi Benny

first of all, thanks for your reply.

I don't let a OOME occur, i do restart Tomcat before. But that is not the
point.

What you suggest i already did, not with MAT but with Netbeans (in
development Environment) and with jvmstat / jconsole in production
environment.

The classes that use up most memory are char[], TreeMap and Hashmap. Also,
memory consumption of these classes is growing.

Strange enough, we do not use any Hashmap's at all, and rarly TreeMap's or
char-Array's. And i checked several times that i release al elements in my
TreeMap's. And also, considering the load on the Homepage, it would be
impossible to use up all the Heap with the TreeMap's we used in our code.

Our application uses extensivly images (Graphics.getImage(...). Could that
be a problem?

Thanks
Andrej

Report message to a moderator

Re: Memory-Problems [message #119532 is a reply to message #119519]

Sat, 24 January 2009 22:57

Benjamin Muskalla

Messages: 338
Registered: July 2009

Senior Member

Hi Andrej,

the char[] arrays are most likely due to usage of Strings.
Why I suggested to use MAT is because it can show you a dominator tree
of your objects which in turn could reveal who is holding all these
objects. I don't say that RAP has no memory leaks so I would suggest to
check that - maybe the workbench code has a leak somewhere.

In general using images should be no problem as they are only references
by their pathname.

Cheers and have a nice weekend!
Benny

adiinfo wrote:
> Hi Benny
>
> first of all, thanks for your reply.
>
> I don't let a OOME occur, i do restart Tomcat before. But that is not
> the point.
>
> What you suggest i already did, not with MAT but with Netbeans (in
> development Environment) and with jvmstat / jconsole in production
> environment.
>
> The classes that use up most memory are char[], TreeMap and Hashmap.
> Also, memory consumption of these classes is growing.
>
> Strange enough, we do not use any Hashmap's at all, and rarly TreeMap's
> or char-Array's. And i checked several times that i release al elements
> in my TreeMap's. And also, considering the load on the Homepage, it
> would be impossible to use up all the Heap with the TreeMap's we used in
> our code.
>
> Our application uses extensivly images (Graphics.getImage(...). Could
> that be a problem?
>
> Thanks
> Andrej
>
>
>

Report message to a moderator

Re: Memory-Problems [message #119556 is a reply to message #119532]

Sun, 25 January 2009 09:09

Andrej Dimic

Messages: 77
Registered: July 2009

Member

Hi Benny

as the other tools i used to analyse the heap had no possibility to show
backreference links i will definitly follow your suggestion and give it a
try with MAT. I will report back hopefully soon.

For the images: at what point these references are given free for garbage
collection? I do ask because the images do change frequently on our
website. We do read the images from database blobs and write them to disk
so we can handle them with RAP.

Thanks a lot
Andrej

Report message to a moderator

Re: Memory-Problems [message #119583 is a reply to message #119556]

Sun, 25 January 2009 12:38

Benjamin Muskalla

Messages: 338
Registered: July 2009

Senior Member

Hi Andrej,

the references to the images (as for all resources) will be held for the
whole application lifetime. This is because we share the resources
across all sessions to reuse them.
It might be possible that if you register new resources in a very
frequent way that the Hashtable in the resource manager grows. But I
can't imagine that this would lead to such memory problems.

For defining lifetimes for such resources see:

245635: Provide an mechanism to handle different lifetimes of resources
https://bugs.eclipse.org/bugs/show_bug.cgi?id=245635

I don't like to guess such things - please take a look at the heapdump
where all these objects come from.

Greets
Benny

adiinfo wrote:
> Hi Benny
>
> as the other tools i used to analyse the heap had no possibility to show
> backreference links i will definitly follow your suggestion and give it
> a try with MAT. I will report back hopefully soon.
>
> For the images: at what point these references are given free for
> garbage collection? I do ask because the images do change frequently on
> our website. We do read the images from database blobs and write them to
> disk so we can handle them with RAP.
>
> Thanks a lot
> Andrej
>
>

Report message to a moderator

Re: Memory-Problems [message #119596 is a reply to message #119583]

Sun, 25 January 2009 19:30

Andrej Dimic

Messages: 77
Registered: July 2009

Member

Hi Benny

you were right - MAT is a great tool to analyse JVM Memory problems.
I did indeed find a problem in my code. After correcting the leak I will
now monitor the application for a day or so and see what happens.

Thanks very much - you really helped me.
Andrej

Report message to a moderator

Re: Memory-Problems [message #119698 is a reply to message #119583]

Mon, 26 January 2009 21:14

Andrej Dimic

Messages: 77
Registered: July 2009

Member

Hi Benny

after analyzing the Heapdump with MAT i can make these statements:

1. 72% of retained heap usage was due to a bug on my side
2. 11% of retained heap usage was due to java.util.Hashmap. References to
jpeg-files were the vaste majority of entries (i do not use Hashmap at
all).
3. 7% of retained heap was due to org.apache.naming.resources.CacheEntry.
Again, only references to jpeg-files (ok, i'didnt check every single
entry, but these i checked were all jpeg-files).

This snapshot was taken after one single day!

If i am interpreting the entrys in java.util.Hashmap and
org.apache.naming.resources.CacheEntry correctly, you do not only cache a
string to a file (as i have read in the newsforum and i was also told so),
but there is also for each cache and hashmap entry a byte[] with the
imagedata.
MAT shows me for each java.util.Hashmap-Entry the class
org.apache.catalina.loader.ResourceEntry with the members byte[],
java.net.URL, java.net.URL. The byte[] array is exactly the size of the
jpeg-file given in the java.net.URL.

If i am right this is a desaster for me as i do have hundreds of images
that are dynamically read from a database with over 2.5GB of image data. I
can't cache all these images in the JVM heap.

Do you rally cache the images in a byte[]? MAT makes be believe that.
If so, is there any internal method so i can clear the image cache from
time to time?

Thanks
Andrej

Report message to a moderator

Re: Memory-Problems [message #119764 is a reply to message #119698]

Tue, 27 January 2009 12:17

Benjamin Muskalla

Messages: 338
Registered: July 2009

Senior Member

Hi Andrej,

it would be very interesting who references the objects mentioned in
point 2 and 3. Without having a path to the gc root I can't tell what
the source of the retained heap usage is.

btw: how do you create your images? do you use the ImageRegistry of
JFace? Do you use ImageData in any way?

Cheers
Benny

adiinfo wrote:
> Hi Benny
>
> after analyzing the Heapdump with MAT i can make these statements:
>
> 1. 72% of retained heap usage was due to a bug on my side
> 2. 11% of retained heap usage was due to java.util.Hashmap. References
> to jpeg-files were the vaste majority of entries (i do not use Hashmap
> at all).
> 3. 7% of retained heap was due to
> org.apache.naming.resources.CacheEntry. Again, only references to
> jpeg-files (ok, i'didnt check every single entry, but these i checked
> were all jpeg-files).
>
> This snapshot was taken after one single day!
>
> If i am interpreting the entrys in java.util.Hashmap and
> org.apache.naming.resources.CacheEntry correctly, you do not only cache
> a string to a file (as i have read in the newsforum and i was also told
> so), but there is also for each cache and hashmap entry a byte[] with
> the imagedata.
> MAT shows me for each java.util.Hashmap-Entry the class
> org.apache.catalina.loader.ResourceEntry with the members byte[],
> java.net.URL, java.net.URL. The byte[] array is exactly the size of the
> jpeg-file given in the java.net.URL.
>
>
> If i am right this is a desaster for me as i do have hundreds of images
> that are dynamically read from a database with over 2.5GB of image data.
> I can't cache all these images in the JVM heap.
>
> Do you rally cache the images in a byte[]? MAT makes be believe that.
> If so, is there any internal method so i can clear the image cache from
> time to time?
>
> Thanks
> Andrej
>
>

Report message to a moderator

Antw: Re: Memory-Problems [message #119869 is a reply to message #119764]

Tue, 27 January 2009 19:15

Eclipse User

Originally posted by: admin.kabe-farben.ch

Hi Benny

i did send you the screenshots from MAT directly to your email account - i
was not able to post it to the newsgroup because of size restrictions.

You can see how images are created in my code in the snippet below:

public void saveThumbnail(File file, String imageType) {
if (thumb != null) {
BufferedImage bi = new BufferedImage(thumb.getIconWidth(), thumb
.getIconHeight(), BufferedImage.TYPE_INT_RGB);
Graphics g = bi.getGraphics();
g.drawImage(thumb.getImage(), 0, 0, null);
try {
ImageIO.write(bi, imageType, file);
} catch (IOException ioe) {
System.out.println("Error occured saving thumbnail");
}
} else {
System.out.println("Thumbnail has not yet been created");
}
}

Thanks
Andrej

>>> Benjamin Muskalla<bmuskalla@eclipsesource.com> schrieb am 27.01.2009 um
13:17 in Nachricht <glmu04$tla$1@build.eclipse.org>:
> Hi Andrej,
>
> it would be very interesting who references the objects mentioned in
> point 2 and 3. Without having a path to the gc root I can't tell what
> the source of the retained heap usage is.
>
> btw: how do you create your images? do you use the ImageRegistry of
> JFace? Do you use ImageData in any way?
>
> Cheers
> Benny
>
> adiinfo wrote:
>> Hi Benny
>>
>> after analyzing the Heapdump with MAT i can make these statements:
>>
>> 1. 72% of retained heap usage was due to a bug on my side
>> 2. 11% of retained heap usage was due to java.util.Hashmap. References
>> to jpeg-files were the vaste majority of entries (i do not use Hashmap
>> at all).
>> 3. 7% of retained heap was due to
>> org.apache.naming.resources.CacheEntry. Again, only references to
>> jpeg-files (ok, i'didnt check every single entry, but these i checked
>> were all jpeg-files).
>>
>> This snapshot was taken after one single day!
>>
>> If i am interpreting the entrys in java.util.Hashmap and
>> org.apache.naming.resources.CacheEntry correctly, you do not only cache
>> a string to a file (as i have read in the newsforum and i was also told
>> so), but there is also for each cache and hashmap entry a byte[] with
>> the imagedata.
>> MAT shows me for each java.util.Hashmap-Entry the class
>> org.apache.catalina.loader.ResourceEntry with the members byte[],
>> java.net.URL, java.net.URL. The byte[] array is exactly the size of the
>> jpeg-file given in the java.net.URL.
>>
>>
>> If i am right this is a desaster for me as i do have hundreds of images
>> that are dynamically read from a database with over 2.5GB of image data.

>
>> I can't cache all these images in the JVM heap.
>>
>> Do you rally cache the images in a byte[]? MAT makes be believe that.
>> If so, is there any internal method so i can clear the image cache from
>> time to time?
>>
>> Thanks
>> Andrej
>>
>>

Report message to a moderator

Re: Antw: Re: Memory-Problems [message #120040 is a reply to message #119869]

Thu, 29 January 2009 11:50

Benjamin Muskalla

Messages: 338
Registered: July 2009

Senior Member

Hi Andrej,

the next time please use an image service (eg. ImageShack) so everyone
can follow the discussion on the newsgroup.

To the memory problem: there are no references to anything RAP related.
As you can see the HashMap Entrys and
org.apache.naming.resources.CacheEntry all belong to the Catalina
classloader. It seems that catalina caches some of the images for
delivering them later. But there seems to be no leak in your app / in
RAP itself.

Cheers
Benny

Andrej Dimic wrote:
> Hi Benny
>
> i did send you the screenshots from MAT directly to your email account - i
> was not able to post it to the newsgroup because of size restrictions.
>
> You can see how images are created in my code in the snippet below:
>
> public void saveThumbnail(File file, String imageType) {
> if (thumb != null) {
> BufferedImage bi = new BufferedImage(thumb.getIconWidth(), thumb
> .getIconHeight(), BufferedImage.TYPE_INT_RGB);
> Graphics g = bi.getGraphics();
> g.drawImage(thumb.getImage(), 0, 0, null);
> try {
> ImageIO.write(bi, imageType, file);
> } catch (IOException ioe) {
> System.out.println("Error occured saving thumbnail");
> }
> } else {
> System.out.println("Thumbnail has not yet been created");
> }
> }
>
>
> Thanks
> Andrej
>
>
>>>> Benjamin Muskalla<bmuskalla@eclipsesource.com> schrieb am 27.01.2009 um
> 13:17 in Nachricht <glmu04$tla$1@build.eclipse.org>:
>> Hi Andrej,
>>
>> it would be very interesting who references the objects mentioned in
>> point 2 and 3. Without having a path to the gc root I can't tell what
>> the source of the retained heap usage is.
>>
>> btw: how do you create your images? do you use the ImageRegistry of
>> JFace? Do you use ImageData in any way?
>>
>> Cheers
>> Benny
>>
>> adiinfo wrote:
>>> Hi Benny
>>>
>>> after analyzing the Heapdump with MAT i can make these statements:
>>>
>>> 1. 72% of retained heap usage was due to a bug on my side
>>> 2. 11% of retained heap usage was due to java.util.Hashmap. References
>>> to jpeg-files were the vaste majority of entries (i do not use Hashmap
>>> at all).
>>> 3. 7% of retained heap was due to
>>> org.apache.naming.resources.CacheEntry. Again, only references to
>>> jpeg-files (ok, i'didnt check every single entry, but these i checked
>>> were all jpeg-files).
>>>
>>> This snapshot was taken after one single day!
>>>
>>> If i am interpreting the entrys in java.util.Hashmap and
>>> org.apache.naming.resources.CacheEntry correctly, you do not only cache
>>> a string to a file (as i have read in the newsforum and i was also told
>>> so), but there is also for each cache and hashmap entry a byte[] with
>>> the imagedata.
>>> MAT shows me for each java.util.Hashmap-Entry the class
>>> org.apache.catalina.loader.ResourceEntry with the members byte[],
>>> java.net.URL, java.net.URL. The byte[] array is exactly the size of the
>>> jpeg-file given in the java.net.URL.
>>>
>>>
>>> If i am right this is a desaster for me as i do have hundreds of images
>>> that are dynamically read from a database with over 2.5GB of image data.
>
>>> I can't cache all these images in the JVM heap.
>>>
>>> Do you rally cache the images in a byte[]? MAT makes be believe that.
>>> If so, is there any internal method so i can clear the image cache from
>>> time to time?
>>>
>>> Thanks
>>> Andrej
>>>
>>>

Report message to a moderator

Re: Antw: Re: Memory-Problems [message #120052 is a reply to message #120040]

Thu, 29 January 2009 12:56

Eclipse User

Originally posted by: admin.kabe-farben.ch

Hi Benny

> the next time please use an image service (eg. ImageShack) so everyone
> can follow the discussion on the newsgroup.
Sorry for that, i know it's bad behavior. I just didn't know how to do
otherwise, the only newsread i have is groupwise.
I don't know what ImageShack is but i will have a look at it.

> To the memory problem: there are no references to anything RAP related.
I also suspectecd that it was tomcat doing this.

A great thank you for your help
Andrej

>>> Benjamin Muskalla<bmuskalla@eclipsesource.com> schrieb am 29.01.2009 um
12:50 in Nachricht <gls561$c03$1@build.eclipse.org>:
> Hi Andrej,
>
> the next time please use an image service (eg. ImageShack) so everyone
> can follow the discussion on the newsgroup.
>
> To the memory problem: there are no references to anything RAP related.
> As you can see the HashMap Entrys and
> org.apache.naming.resources.CacheEntry all belong to the Catalina
> classloader. It seems that catalina caches some of the images for
> delivering them later. But there seems to be no leak in your app / in
> RAP itself.
>
> Cheers
> Benny
>
> Andrej Dimic wrote:
>> Hi Benny
>>
>> i did send you the screenshots from MAT directly to your email account -

> i
>> was not able to post it to the newsgroup because of size restrictions.
>>
>> You can see how images are created in my code in the snippet below:
>>
>> public void saveThumbnail(File file, String imageType) {
>> if (thumb != null) {
>> BufferedImage bi = new BufferedImage(thumb.getIconWidth(), thumb
>> .getIconHeight(), BufferedImage.TYPE_INT_RGB);
>> Graphics g = bi.getGraphics();
>> g.drawImage(thumb.getImage(), 0, 0, null);
>> try {
>> ImageIO.write(bi, imageType, file);
>> } catch (IOException ioe) {
>> System.out.println("Error occured saving thumbnail");
>> }
>> } else {
>> System.out.println("Thumbnail has not yet been created");
>> }
>> }
>>
>>
>> Thanks
>> Andrej
>>
>>
>>>>> Benjamin Muskalla<bmuskalla@eclipsesource.com> schrieb am 27.01.2009
um
>> 13:17 in Nachricht <glmu04$tla$1@build.eclipse.org>:
>>> Hi Andrej,
>>>
>>> it would be very interesting who references the objects mentioned in
>>> point 2 and 3. Without having a path to the gc root I can't tell what
>>> the source of the retained heap usage is.
>>>
>>> btw: how do you create your images? do you use the ImageRegistry of
>>> JFace? Do you use ImageData in any way?
>>>
>>> Cheers
>>> Benny
>>>
>>> adiinfo wrote:
>>>> Hi Benny
>>>>
>>>> after analyzing the Heapdump with MAT i can make these statements:
>>>>
>>>> 1. 72% of retained heap usage was due to a bug on my side
>>>> 2. 11% of retained heap usage was due to java.util.Hashmap. References

>>>> to jpeg-files were the vaste majority of entries (i do not use Hashmap

>>>> at all).
>>>> 3. 7% of retained heap was due to
>>>> org.apache.naming.resources.CacheEntry. Again, only references to
>>>> jpeg-files (ok, i'didnt check every single entry, but these i checked
>>>> were all jpeg-files).
>>>>
>>>> This snapshot was taken after one single day!
>>>>
>>>> If i am interpreting the entrys in java.util.Hashmap and
>>>> org.apache.naming.resources.CacheEntry correctly, you do not only cache

>>>> a string to a file (as i have read in the newsforum and i was also told

>>>> so), but there is also for each cache and hashmap entry a byte[] with
>>>> the imagedata.
>>>> MAT shows me for each java.util.Hashmap-Entry the class
>>>> org.apache.catalina.loader.ResourceEntry with the members byte[],
>>>> java.net.URL, java.net.URL. The byte[] array is exactly the size of the

>>>> jpeg-file given in the java.net.URL.
>>>>
>>>>
>>>> If i am right this is a desaster for me as i do have hundreds of images

>>>> that are dynamically read from a database with over 2.5GB of image
data.
>>
>>>> I can't cache all these images in the JVM heap.
>>>>
>>>> Do you rally cache the images in a byte[]? MAT makes be believe that.
>>>> If so, is there any internal method so i can clear the image cache from

>>>> time to time?
>>>>
>>>> Thanks
>>>> Andrej
>>>>
>>>>

Report message to a moderator