One Year of Automated Error Reporting

Since June 2015, with the Eclipse Mars simultaneous release, all Eclipse packages were shipped with the Automated Error Reporting client (called AERI). Since then, two minor releases of Eclipse Mars were made (Mars.1 and Mars.2) and its successor, Eclipse Neon, was released on June 22 of this year – with AERI, of course. Now is a good time to look back at how the Automated Error Reporting Initiative has helped to fix Eclipse project bugs.

Ever since AERI became a part of the Eclipse packages, eclipse.org has seen a steady stream of incoming error reports. Eclipse users currently send around 70,000 reports per week. That is a whopping total of 3 million reports in the past 12 months!

Of course, not every error report is indicative of a new problem. Often multiple users encounter and report the same problem. For the initial Mars release the Eclipse projects received error reports for 17,253 distinct problems – and immediately began fixing them. One thousand problems were fixed in the three months prior to the Mars.1 release. With Mars.2, we saw the number of problems drop significantly, from 16,266 to 11,967.

Open Problems per Release. From Mars.0 to Mars.2, 7.800 problems have been fixed. Overall the number of known problems dropped by more than 30%.

This means that the three million error reports that Eclipse projects received for Eclipse Mars had a visible effect on the overall quality of the releases. All together, more than 7,800 problems have been fixed over the course of the three Mars releases.

Although most users don’t bother reporting problems they encounter in their day-to-day work using classic bug trackers like Bugzilla, they are apparently still eager to help out the various Eclipse projects – if only reporting problems is made easy enough. The statistics, 70,000 reports per week, show that using AERI makes reporting errors very convenient. Many users even take the time to send comments with their error report to describe the situation in which the problem occurred.

While these comments are usually short, they are helpful when fixing the bugs. As it turns out, problems with a user comment are more than twice as likely to be fixed than problems without a user comment. In other words, it pays off to spend a bit more time providing some steps to help the project developers reproduce the issue in question. The best part is, users don’t even need to log into Bugzilla for that! They can submit the entire error report directly from within their Eclipse IDE.

Leveraging Error Reporting

Along with the comments, the detailed technical information (stack traces, plug-in versions, operating system details) included in the report help Eclipse projects diagnose and fix problems faster than ever before. Some of the projects really stepped up to the plate in fixing a large percentage the problems reported by their users. The projects that stand-out are: Code Recommenders, Xtext, and most all Oomph, a project that like AERI made its big entrance with Eclipse Mars.

Closed vs. open (unreviewed) problems per project. Top 15 projects (in absolute numbers) fully embracing your error reports.

Even though other projects may have more open problems than closed problems, the Automated Error Reporting Initiative is still a success for them. As Matthias Sohn, the project co-lead of JGit and EGit, notes “the provided details and error statistics are a great help to fix these bugs and to decide which bugs to fix first. Automated error reporting helped us to quickly detect and fix dozens of bugs in JGit and EGit.”

To summarize: the Automated Error Reporting Initiative has been a clear success for Eclipse Mars. Many thanks go to the 350,000 Eclipse IDE users who helped committers identify and fix problems! Keep up reporting all errors you encounter! And remember to include a comment.

But what’s in store for Eclipse Neon?

AERI for Eclipse Neon

Based on user feedback and AERI’s initial success in Eclipse Mars, automated error reporting has been made available to any Eclipse plug-in through a simple extension point with Eclipse Neon. In other words, with the June release it’s even easier for an Eclipse plugin (whether it be commercial, private, or open-source project) to receive error reports and find out where their software breaks in the field. This allows developers to get closer to the user and make their project even better.

Get Started Today

• Watch our webinar to learn more about how to set up error reporting for your plugins.
• Login to our demo server for a tour of the tool.
• Sign up for the free Automated Error Reporting Service hosted by Codetrails and get started in less than an hour!

Many thanks again to all reporters and all bug fixers. Congratulations to all of you for creating a great Eclipse Neon release!