[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [eclipselink-users] memory usage
|
I had this problem a while back.
I was importing millions of rows using JPA, and it started out relatively fast, but after a while got very slow. Nothing I did fixed this (flushing, clearing, batch inserts etc).
The way I got around it was as follows. Split import into batches of 5000 (each with a separate transaction), and monitor the output from the console. When it got slow, I killed the process, and started it up at the next batch index.
Not perfect but it worked.
In hindsight it would probably have been better to have imported it using DBA tools instead of JPA.
Thanks,
Vackar
>>> David Parker <dap@xxxxxxxxxxxxxxxxxxxx> 10/21/09 6:04 PM >>>
I added a call to EntityManager.clear() after each transaction, but
these object arrays still seem to be piling up. I am doing another
test run with tracking allocations turned on, so hopefully I can pin
down better where these objects are being allocated.
Does EntityManager.clear() clear the whole cache? I don't see a
clear() method on EntityManagerFactory, which I understand wraps the
ServerSession - is there something I can try re-setting the whole thing?
Thanks.
- DAP
On Oct 20, 2009, at 11:57 AM, David Parker wrote:
> Thanks, everybody, for all the responses. I appreciate your help.
>
>> Are you using JTA, or RESOURCE_LOCAL?
>>
>
> Local
>
>> Are you using the same EntityManager, or do you create a new one
>> for each
>> transaction?
>
> This thing runs as a big loop sucking records out of one database
> via a simple JDBC Connection, and for each record creating the JPA
> objects for that record, and persisting them. As part of creating
> the JPA objects there is some querying against the target database
> via SQL queries.
>
> There is one EntityManager, and a single transaction per record.
>
>> What cache setting are you using?
>>
>
> SoftWeak, but I am confused about what the best caching option is.
> I've read the doc, and I ~think I understand the technical
> description, but I'm still not sure when to use what.
>
>> I would recommend you create a new EntityManager per transaction,
>> otherwise
>> the previous objects will still be part of the persistence
>> context. You
>> could also call clear().
>
> I've started another run with the clear() after each commit.
>
>
>> You can try disabling the cache (shared=false), or using a weak
>> cache.
>>
>> Do you run out of memory, or just notice a memory increase? You
>> may wish to
>> try a memory profiler such as JProfiler.
>
> Memory increases slowly but inexorably. I've had it blow the heap at
> -Xmx6G -d64 -server. I am hopeful that just being more aggressive
> about the cache clearing
> will fix the problem. The YourKit profiler doesn't flag much it
> thinks is a leal - just a few k.
>
>> Also ensure that your application is not holding onto any or the
>> objects.
>>
>>
>> DavidParker wrote:
>>>
>>> Yeah, we are using multiple transactions, i.e.
>>>
>>> get an EntityTransaction
>>> do work
>>> EntityTransation.commit()
>>> set the local tx to null
>>>
>>>
>>> On Oct 19, 2009, at 5:59 PM, Dave Brosius wrote:
>>>
>>>> Is the import all in one transaction? I'd be curious if you broke
>>>> up
>>>> the import (even for a test) into multiple transactions whether the
>>>> issue would go away.
>>>>
>>>> -----Original Message-----
>>>> From: "David Parker" <dap@xxxxxxxxxxxxxxxxxxxx>
>>>> Sent: Monday, October 19, 2009 5:51pm
>>>> To: "EclipseLink User Discussions" <eclipselink-users@xxxxxxxxxxx>
>>>> Subject: [eclipselink-users] memory usage
>>>>
>>>> I am running a long data import that uses eclipselink 1.1.2 for
>>>> persistence. I see memory increasing slowly but surely as it goes
>>>> along. The YourKit profiler does not identify anything as a link,
>>>> but
>>>> I do seem to have an increasing "retained size" for
>>>>
>>>> org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
>>>> [Stack Local]
>>>>
>>>> I see there is a clear method on TransactionWrapperImpl, but that
>>>> does
>>>> not seem to be in the JPA EntityTransaction API.
>>>>
>>>> Is there a best practice for long running data imports? Should I be
>>>> clearing something?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> =============================================
>>>> David Parker
>>>> dap@xxxxxxxxxxxxxxxxxxxx
>>>>
>>>
>>>
>>
>>
>> -----
>> http://wiki.eclipse.org/User:James.sutherland.oracle.com James
>> Sutherland
>> http://www.eclipse.org/eclipselink/
>> EclipseLink , http://www.oracle.com/technology/products/ias/toplink/
>> TopLink
>> Wiki: http://wiki.eclipse.org/EclipseLink EclipseLink ,
>> http://wiki.oracle.com/page/TopLink TopLink
>> Forums: http://forums.oracle.com/forums/forum.jspa?forumID=48
>> TopLink ,
>> http://www.nabble.com/EclipseLink-f26430.html EclipseLink
>> Book: http://en.wikibooks.org/wiki/Java_Persistence Java Persistence
>> --
>> View this message in context: http://www.nabble.com/memory-usage-tp25966341p25974597.html
>> Sent from the EclipseLink - Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> eclipselink-users mailing list
>> eclipselink-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>>
>
> =============================================
> David Parker
> dap@xxxxxxxxxxxxxxxxxxxx
>
>
>
>
> _______________________________________________
> eclipselink-users mailing list
> eclipselink-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
=============================================
David Parker
dap@xxxxxxxxxxxxxxxxxxxx
The University of Dundee is a registered Scottish charity, No: SC015096
