Memory Analyzer

The project has been created. Please visit the project page.

Introduction

The Memory Analyzer is a proposed open source project under the Eclipse Technology Project. This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process document) and is written to declare its intent and scope. This proposal is written to solicit additional participation and input from the Eclipse community. You are invited to comment on and/or join the project. Please send all feedback to the http://www.eclipse.org/newsportal/thread.php?group=eclipse.technology.memory-analyzer newsgroup.

Description

The Memory Analyzer is a tool for the analysis of HPROF binary heap dumps, a de-facto standard for heap dumps supported by the Sun Java SE Runtime Environment (JRE) . It is useful to any Java programmer. Any application or more complex application systems like an Eclipse installation or an application server can be closely analyzed. The analysis is offline, once the heap dump is written. Such a heap dump can be written on termination of the Java Virtual Machine or at any arbitrary point in time through other tools, such as JConsole.

Further documentation, tutorials, screenshots, JavaOne 2007 slides, user feedback, performance numbers, a list of the most important features and the download as Eclipse RCP application for Linux, Linux (x86_64), Mac OS X Carbon, Windows, Windows (x86_64), Solaris or just as a bunch of plugins are all available here on the SAP SDN Wiki. We plan to make an Eclipse Wiki available some time after project initiation.

Background

The motivation for this project is to analyze failures of critical development and production systems that crash from OutOfMemory errors (OOM). These crashes usually leave no trace of the actual root cause of the problem and support personnel are usually unable to tell whether product or customer code deployed on the server caused the problem. Analyzing logs and other less-direct indicators of what could have gone wrong are usually not reliable methods for finding the root cause of the problem. Solving an OOM can easily take weeks or months.

Since 1.4.2_12, this situation improved a lot for Sun HotSpot-based Java VMs writing a so-called HPROF binary heap dump on the first OOM. The only setting necessary is the JVM parameter -XX:+HeapDumpOnOutOfMemoryError. This parameter has no overhead at runtime and is therefore accepted widely by companies running those production systems, whereas profilers were not accepted for the performance overhead they impose. Writing heap dumps became more convenient, and other tools now support writing them at runtime on button-click.

Independent software vendors like SAP are obligated to provide support for huge customers running critical production systems for years. Support teams need a tool for helping to resolve memory issues at customer sites. Since heap dumps have been the only affordable artifacts from a crash with enough data to find the root cause, the Memory Analyzer project is being proposed to build an extensible framework and exemplary tools for effective analysis of the heap dumps. SAP intends to offer an initial code contribution based on the code of Memory Analyzer, which is used to analyze critical customer situations related to OutOfMemory errors.

Before starting to develop analysis tools, we investigated and tried to use all kinds of commercial and open source tools. They all failed on huge production heap dumps. Neither their performance nor their functionality could match the demands. Performance was a big problem and a major showstopper. No tool could parse/open a medium-sized heap dump from a 32-bit box of only 1 GB on a 32-bit box (which was and often still is the default for a developer/support workplace), not to mention heap dumps many gigabytes in size. Some tools could handle them on big 64-bit boxes, but most operations took minutes or hours to complete even with multi-core CPUs. Functionality, when not looking for a single 1 GB memory leak, was not sufficient to meet the demands of complex object-retention problems either. Those tools were and are mostly bare heap walkers without much use when confronted with 100 million or more objects.

The performance problems were solved with DBMS-like data management, a highly tuned cache and algorithms from graph theory for object graph transformation to quickly detect complex object retention patterns. Later efforts went more and more into the area of functionality. Each heap dump for which the algorithm failed to show the root cause was a source of inspiration to add new functionality that would show it next time, so the tool evolved in a very practical and useful way to diagnose OutOfMemory errors in application servers and standalone Java applications alike. We started to submit papers for conferences and were accepted at JAX, Jazoon, EclipseSummit and JavaOne 2007. At the same time, the tool was offered as a free download and received very positive feedback, much of it focusing on its superior performance and the fact that users could find the root causes in a short time:

"Bravo to the Memory Analyzer team. I downloaded it and opened a 2GB heap dump on 32-bit Windows (not possible with JHat) within minutes. Also, the tool was exceedingly easy to use and quickly exposed an object retention problem we had been struggling to find. Thanks!"
From http://forums.java.net/jive/thread.jspa?messageID=221859&tstart=0
"It's really fast! Thanks for this contribution! It's working really fast, even with a huge heap dump!" and
"I worked a lot with "Vendor A" and "Vendor B." Also I tried "Vendor C" and "Vendor D." But from now on this tool is my favorite to look for memory leaks. It is really fast and handles large heap dumps easily. It's easy to use as well and it presents you the information in a way easy to understand."
From http://www.theserverside.com/news/thread.tss?thread_id=45764
"I just tried Memory Analyzer tool from the JavaOne DVD, and I have to say I am pretty impressed with it. Very nice UI, and performance is great also. It also works well to find Classloader leaks."
From http://blogs.sun.com/fkieviet/entry/javaone_memory_leaks_revisited
"On one of the JavaOne sessions this year - there was a tool presented - Memory Analyzer. A certain set of tries on heap dumps, different in size, showed significant performance advantage for SAP solution. Voted: Very Cool"
From http://www.sdnshare.com/view.jsp?id=283
"I was quite impressed what kind of information you can gain by using this tool and how easy it is to find the memory hogs in your application. Finding memory leaks is usually far easier than identifying the greedy classes which are not leaking but nevertheless take up huge amounts of memory. Any serious Java application on an enterprise scale needs this kind of analysis since memory tends to be a constrained resource."
From http://canoo.com/blog/2007/06/28/jazoon-day-3

The biggest motivator was always to provide a useful tool pointing out the root cause as fast as possible to reduce costs spent in application (server) support for customers. Beside the feedback received via e-mail, blogs, and forums, the tool was rated "Developer Tool of the Week" by the JavaPosse.

The initial code contribution will contain the basic inspection functionality and the first wave of queries that will later (at the end of this year/beginning of next year) become the basis for an expert system. This expert system is planned to incorporate the practical knowledge gained through real-life experience and to help to find issues or at least get hints of them right from the first screen.

Organization

Project mentors

  • Jochen Krause (Innoopract GmbH), Project Lead Eclipse RAP
  • Jeff McAffer (Code 9), Project Lead Equinox OSGi, RCP, Orbit

Initial committers

The proposed initial committers are:

  • Andreas Buchen (SAP)
  • Elena Nayashkova (SAP)
  • Erwin Margewitsch (SAP)
  • Krum Tsvetkov (SAP)
  • Vedran Lerenc (SAP)

Contributions

The initial committers will focus on evolving and stabilizing the basic inspection and future expert system platform. This is the core of the tool and provides extension points and an API for external usage, e.g. by the exemplary UI or in customized headless heap dump tests.

Specifically, the core allows defining so-called memory queries which operate on the heap dump. A heap dump contains a rich snapshot of the state of the system and can be probed in great detail with generic memory queries (those which work with all applications, e.g. probing the state of the FinalizerThread) and specific memory queries (those which can be applied only to a specific application, e.g. active users in the system). It is even possible to use a heap dump to check for applied patches, non-memory related problems, or just to get a good overview on the system at the time the heap dump was written.

The committers will continue to improve the UI and make new core functionality available in it. Further generic memory queries will be added when identified. The architecture cleanly separates the core from the UI and the concrete memory queries. Contributions in all areas are welcome. It would be even possible to replace the whole UI or distribute the tool with a whole different set of memory queries.

The project's agile development process will follow eclipse.org's standards for openness and transparency. The goal is to provide an even more useful tool. The tool would greatly benefit from the community. There are many different contributions that could be made to this tool:

  • Other innovative ideas to "get a clear view" on complex object retention patterns, such as how the dominators in the tool instantly show the retained size and set of an object as the Garbage Collector would see it.
  • Correlation engine that is expected to automatically discover correlations (for example, indicating that a great share of memory is held by arrays of a certain length or held mainly by instances of those classes having this value in that field).
  • Additional generic queries--basically hundreds more could be conceived. They all will help to uncover common or less common memory anti-patterns in every heap dump they are confronted with.
  • More specific queries for concrete applications or platforms. SAP will continue to develop its own queries for their own Java application server. Other companies could add queries for their application servers or their applications. Thereby individual companies could differentiate their offerings built on top of the stable core.
  • Currently only HPROF binary heap dumps from Sun HotSpot-based JVMs are supported but the team was often asked whether IBM JVMs are supported. As IBM doesn’t support the HPROF binary heap dump format, but offers an alternative with system dumps and DTFJ, a converter from system dumps using DTFJ to HPROF binary heap dumps would be a very valuable contribution.

The project actively encourages contributions to the Memory Analyzer. We also plan to help improve the Eclipse platform and other Eclipse projects by submitting patches and extension point suggestions.

Developer community

The project expects to extend the initial set of committers by actively supporting a developer community. The goal is to have the initial developer community harden the core of the basic inspection and expert system platform to the point where it is easy for any other extension to leverage on the technology, and the Memory Analyzer project will encourage all active contributors to become committers. The project will make it easy for interested parties to take an active role in the project by making the planning and process transparent and remotely accessible.

User community

The Memory Analyzer is a new approach to technology that promises to improve the quality of Java applications in general. As such, supporting and soliciting feedback from a large user community of commercial developers is critical to create an even more useful tool in the future. The project plans on doing this by using the standard eclipse.org mechanisms of supporting an open project and a community of early adopters.