Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jdt-dev] parser question 
Hey Stephan,

thanks for the additional details, I went ahead and did some deeper analysis on this, with more data.

It seems like I could improve performance a bit by skipping that step, but not as much as I have hoped - attached is a performance trace.
Will file an enhancement request for that nevertheless.

It turned out that memory consumption becomes a major problem when running the ASTParser for as many files as some of our users have.
Attached is part of a head dump analysis, running out of memory while running the ASTParser for about 4000 java source files.

Do you have any advice how to do this without consuming so much memory?
I am running: ASTParser.createASTs(javaFiles, null, new String[0], requestor, null);

Should I run this in batch sizes? Or in a totally different way?

Thanks a lot for your help!
-Martin
Title: Merged callees

Merged callees

Merged Call Tree
Time (ms) 		Own Time (ms) Count
org.eclipse.jdt.internal.compiler.
Compiler.process
(CompilationUnitDeclaration, int)
Compiler.java
401,554100 %0544
org.eclipse.jdt.internal.compiler.ast.
CompilationUnitDeclaration.resolve
()
CompilationUnitDeclaration.java
179,66945 %0544
org.eclipse.jdt.internal.compiler.lookup.
CompilationUnitScope.faultInTypes
()
CompilationUnitScope.java
158,89140 %0544
org.eclipse.jdt.internal.compiler.ast.
CompilationUnitDeclaration.generateCode
()
CompilationUnitDeclaration.java
29,5387 %0544
org.eclipse.jdt.internal.compiler.parser.
Parser.getMethodBodies
(CompilationUnitDeclaration)
Parser.java
15,6014 %0544
org.eclipse.jdt.internal.compiler.ast.
CompilationUnitDeclaration.analyseCode
()
CompilationUnitDeclaration.java
11,4823 %0544
org.eclipse.jdt.internal.compiler.lookup.
CompilationUnitScope.verifyMethods
(MethodVerifier)
CompilationUnitScope.java
6,3362 %0544
org.eclipse.jdt.internal.compiler.ast.
CompilationUnitDeclaration.finalizeProblems
()
CompilationUnitDeclaration.java
250 %10544
java.lang.
System.currentTimeMillis
()
System.java (native)
50 %51,280
org.eclipse.jdt.internal.compiler.lookup.
LookupEnvironment.methodVerifier
()
LookupEnvironment.java
30 %1544
Generated by YourKit Java Profiler 2019.1-b112    March 11, 2019 06:12:13 PM
Title: Object explorer

Object explorer

Name
Retained Size 		Shallow Size
java.lang.Thread
[Thread, Stack Local]
"pool-2-thread-1" tid=20 [RUNNABLE]
1,649,940,088120
<local variable>
org.eclipse.jdt.core.dom.CompilationUnitResolver
[Stack Local]
878,737,672104
<local variable>
org.eclipse.jdt.internal.compiler.lookup.AnnotatableTypeSystem
[Stack Local]
5,241,32840
contextClassLoader
org.springframework.boot.loader.LaunchedURLClassLoader
1,456,61380
<local variable>
java.lang.String[3935]
[Stack Local]
1,139,43215,760
<local variable>
org.eclipse.jdt.internal.compiler.ast.TypeDeclaration
[Stack Local]
380,864136
<local variable>
org.springframework.ide.vscode.commons.java.JdtLsJavaProject
[Stack Local]
145,67232
<local variable>
java.util.ArrayList
[Stack Local]
size
=
39
25,36024
<local variable>
java.util.HashMap
[Stack Local]
size
=
489
19,85648
<local variable>
org.eclipse.jdt.internal.compiler.env.ICompilationUnit[3935]
[Stack Local]
15,82415,760
<local variable>
org.eclipse.jdt.internal.compiler.ast.AllocationExpression
[Stack Local]
14,976104
<local variable>
org.eclipse.jdt.core.dom.ASTParser
[Stack Local]
8,40072
<local variable>
java.util.concurrent.ThreadPoolExecutor
[Stack Local]
6,92880
<local variable>
org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration
[Stack Local]
5,328120
<local variable>
org.eclipse.jdt.internal.compiler.lookup.MethodBinding[51]
[Stack Local]
4,720224
<local variable>
org.eclipse.jdt.internal.compiler.lookup.SourceTypeBinding
[Stack Local]
2,696144
<local variable>
org.springframework.ide.vscode.boot.java.utils.SpringIndexerJava
[Stack Local]
2,22424
<local variable>
java.util.ArrayList
[Stack Local]
size
=
281
1,50424
<local variable>
org.eclipse.jdt.core.dom.NameEnvironmentWithProgress
[Stack Local]
1,34440
<local variable>
org.eclipse.jdt.internal.compiler.batch.FileSystem$Classpath[281]
[Stack Local]
1,1441,144
threadLocals
java.lang.ThreadLocal$ThreadLocalMap
57624
<local variable>
org.eclipse.jdt.internal.compiler.ast.FieldDeclaration
[Stack Local]
52880
<local variable>
org.eclipse.jdt.internal.compiler.impl.CompilerOptions
[Stack Local]
456192
<local variable>
org.eclipse.jdt.internal.compiler.lookup.BinaryTypeBinding
[Stack Local]
296128
<local variable>
org.eclipse.jdt.internal.compiler.lookup.MethodScope
[Stack Local]
256112
group
java.lang.ThreadGroup
24048
<local variable>
org.eclipse.jdt.internal.compiler.lookup.MethodBinding[51]
[Stack Local]
224224
<local variable>
org.springframework.ide.vscode.boot.java.utils.SymbolCacheKey
[Stack Local]
20024
<local variable>
org.eclipse.jdt.internal.compiler.lookup.ParameterizedTypeBinding
[Stack Local]
136112
<local variable>
org.eclipse.jdt.internal.compiler.lookup.ParameterizedTypeBinding
[Stack Local]
136112
<local variable>
org.eclipse.jdt.internal.compiler.lookup.SyntheticFactoryMethodBinding
[Stack Local]
11288
<local variable>
org.eclipse.core.runtime.SubMonitor
[Stack Local]
9648
<local variable>
java.util.concurrent.CompletableFuture
[Stack Local]
8824
inheritedAccessControlContext
java.security.AccessControlContext
8840
<local variable>
org.eclipse.jdt.internal.compiler.lookup.ParameterizedMethodBinding
[Stack Local]
8080
name
java.lang.String "pool-2-thread-1"
7224
<local variable>
java.util.concurrent.ThreadPoolExecutor$Worker
[Stack Local]
4848
target
java.util.concurrent.ThreadPoolExecutor$Worker
[Stack Local]
4848
<local variable>
org.eclipse.jdt.internal.compiler.lookup.ClassScope
[Stack Local]
4040
<local variable>
org.springframework.ide.vscode.boot.java.utils.SpringIndexerJava$1
[Stack Local]
3232
<local variable>
org.springframework.ide.vscode.boot.java.utils.SpringSymbolIndex$InitializeProject
[Stack Local]
2424
<local variable>
boolean[1]
[Stack Local]
= {false}
2424
<local variable>
org.eclipse.jdt.internal.compiler.lookup.TypeBinding[1]
[Stack Local]
2424
<local variable>
java.util.concurrent.CompletableFuture$AsyncRun
[Stack Local]
2424
<local variable>
org.eclipse.jdt.internal.core.CancelableProblemFactory
[Stack Local]
2424
<local variable>
org.eclipse.jdt.internal.compiler.lookup.TypeBinding[0]
[Stack Local]
1616
<local variable>
org.eclipse.jdt.internal.compiler.lookup.AnnotationBinding[0]
[Stack Local]
1616
<local variable>
java.lang.String[0]
[Stack Local]
1616
blockerLock
java.lang.Object
1616
daemon
=
boolean
false
1
eetop
=
long
466,176,000
0x000000001BC94800
8
nativeParkEventPointer
=
long
0
8
priority
=
int
5
0x00000005
4
single_step
=
boolean
false
1
stackSize
=
long
0
8
stillborn
=
boolean
false
1
threadLocalRandomProbe
=
int
0
4
threadLocalRandomSecondarySeed
=
int
0
4
threadLocalRandomSeed
=
long
0
8
threadStatus
=
int
5
0x00000005
4
tid
=
long
20
0x0000000000000014
8
Generated by YourKit Java Profiler 2019.1-b112    March 11, 2019 06:15:51 PM





> Am 10.03.2019 um 20:43 schrieb Stephan Herrmann <stephan.herrmann@xxxxxxxxx>:
> 
> Hi Martin,
> 
> since you need bindings, the internal Parser wouldn't suffice for your case. You'd need almost all internal classes of the compiler, and you'd have to interact with internal AST and internal bindings instead of the public DOM varieties. Not a good idea.
> 
> The gap from ASTParser may be fairly small: where it calls into the CompilationUnitResolver see that this guy is a subclass of Compiler.
> Then two steps further inside (process(..)) one simply needs to suppress unit.generateCode().
> 
> Unless I'm missing s.t., little should be standing in the way of an API enhancement. Even less with a proposed patch :p
> Obviously, measurement data also help promoting a performance feature.
> 
> Stephan
> 
> On 10.03.19 20:24, Martin Lippert wrote:
>> Hey Stephan,
>> I don’t need errors and warnings, the resolved AST is enough... :-)
>> And I would be fine with using the internal API directly, don’t really need any additional API guarantees or compatibility. The only missing piece of the puzzle for me is a simple example how to use the interval parser to generate the AST for me (after feeding the API with jars for the classpath and source files to parse, much like I do with ASTParser.parse...).
>> Any advice?
>> Cheers,
>> Martin
>> P.S.: adding an option to avoid code generation on the existing API would be fine, if course, too
>>> Am 10.03.2019 um 19:17 schrieb Stephan Herrmann <stephan.herrmann@xxxxxxxxx>:
>>> 
>>> Hi Martin,
>>> 
>>> Do you need errors and warnings to be reported?
>>> Reason for asking: some problems are detected as late as during code generation.
>>> 
>>> If you don't need those problems, it should theoretically be possible to skip code generation, but this option exists only in internal API. If you need this option exposed in public API and if you have numbers confirming that time is unnecessarily burnt during code gen, the an enhancement request in bugzilla could make sense.
>>> 
>>> Stephan
>>> 
>>> PS:
>>>> I found this:
>>>> org.eclipse.jdt.internal.compiler.parser.Parser
>>>> but I have no idea how to use that or how to configure that to generate an AST from source files.
>>> 
>>> You probably saw the word "internal" in the package name? :)
>>> That's exactly what org.eclipse.jdt.core.dom.* encapsulates.
>>> 
>>>> On 08.03.19 21:12, Martin Lippert wrote:
>>>> Hey!
>>>> I need to create ASTs (incl. type resolving) for Java source files (outside of an Eclipse project environment, just plain Java and some source files).
>>>> At the moment I am using:
>>>> ASTParser parser = ASTParser.newParser(AST.JLS11);
>>>> This works nicely, but I need to get ASTs for method bodies, so I configured the parser to NOT ignore method bodies. But this causes the parser to somehow call out to a compiler to generate bytecode - which I don’t need at all for my purpose. I am interested in the resolved AST only, no byte code necessary. Since this piece of the code is extremely sensitive to performance and memory footprint issues, I am looking into this.
>>>> Is there an alternative to this approach to avoid the bytecode generation?
>>>> I found this:
>>>> org.eclipse.jdt.internal.compiler.parser.Parser
>>>> but I have no idea how to use that or how to configure that to generate an AST from source files.
>>>> Can you shed some light on this? Any examples how to re-use this parser for generating ASTs (without bytecode generation)?
>>>> Thanks so much for your help!
>>>> -Martin
>>>> _______________________________________________
>>>> jdt-dev mailing list
>>>> jdt-dev@xxxxxxxxxxx
>>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>>> https://www.eclipse.org/mailman/listinfo/jdt-dev
>>> 
>>> _______________________________________________
>>> jdt-dev mailing list
>>> jdt-dev@xxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://www.eclipse.org/mailman/listinfo/jdt-dev
>> _______________________________________________
>> jdt-dev mailing list
>> jdt-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://www.eclipse.org/mailman/listinfo/jdt-dev
> 
> _______________________________________________
> jdt-dev mailing list
> jdt-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jdt-dev

Back to the top