Eclipse Test and Performance Tools Platform Project
A Guide to Using TPTP's Reports
A Guide to Using TPTP's Reports

This document focuses on the various reports within TPTP that were created using the Business Intelligence and Reporting Tools (BIRT) project. These reports include reports for test execution histories, trace information and common base event logging. The reader should have an understanding of the test, logging, profiling and monitoring infrastructure provided by the TPTP project before reading this document.

Table of Contents:
1.0 Introduction - Description and Purpose
1.1 Installing TPTP BIRT reports
2.0 Viewing and Understanding the Test Pass Report
2.1 Viewing and Understanding the Trace Report
2.2 Viewing and Understanding the Common Base Event Logging Report
3.0 Replacing Existing TPTP Reports
3.1 Modifying the Test Pass Report
3.2 Modifying the Trace Report
3.3 Modifying the Common Base Event Logging Report

1.0 Introduction - Description and Purpose

TPTP report wizard extension point was exploited to provide a set of BIRT reports for test execution history, trace and Common Base Event logging models. A set of report templates were created using the BIRT report designer. These report template files are bound to the corresponding TPTP models to visualize the information as a report. Various charts and tables help organize the data so that end users can find trends and analyze the data.


1.1 Installing TPTP BIRT Reports

Follow the steps below to install the TPTP BIRT reports:

  1. Click on the 'Latest Downloads' link from TPTP's home page: http://www.eclipse.org/tptp
  2. Select the version and the build that you wish to download. The BIRT reports were introduced in TPTP 4.1.0.1. Select a build corresponding to TPTP 4.1.0.1+.
  3. Once a desired build is selected, find and download the following items:
    • TPTP Runtime or SDK (ALL - Which includes Platform, Monitoring Tools, Tracing and Profiling Tools, and Testing Tools)
    • Obtain all items listed under 'Requirements'
  4. Click on the 'Download' link from BIRT's home page http://www.eclipse.org/birt
  5. The report templates were created using BIRT 2.0. Select a build corresponding to BIRT 2.0+.
  6. Once you find the desired build download the following items:
    • BIRT Report Designer.
    • Obtain items listed under 'Requirements'. Note: Apache Axis, IText and prototype.js are not necessary to generate the TPTP reports. These dependencies are required to create report template files using the BIRT report designer.,

2.0 Viewing and Understanding the Test Pass Reports

Viewing the Test Pass Report

The following steps assume that the user has gone through the phases of creating and running a set of test suites. The user is now ready to generate a report base on the test execution history data.

  1. Open the Test perspective: Window > Open Perspective > Other... Select the "Test" item from the list of perspectives.
  2. Select the test elements you want to generate a report on. The report will generate information associated with all test executions associated with the specified list of test elements.
  3. Right click the selection and select the "Report..." menu item.
  4. Select the "HTML Test pass report (BIRT)" from the list. Then click "Next".
  5. On the next page specify the location where to store the html report. Then click "Next".
  6. On the next page specify a time interval. The report will consist of test executions that have run between this time interval. Then click "Finish".
  7. The report will be shown to the user.

Understanding the Test Pass Report

The following table outlines the structure of the Test Pass report.

Report Section Notes Image
Overview
  • The pie chart shows the total number of test cases that have a verdict value of failed, error, inconclusive and pass
  • The table shows the same information and uses color to highlight the total failed, total error, total inconclusive and total passed test cases.
  • The report is generated based on all the test execution information between the start and end time
  • The creation date indicates when the report was generated.
Overview Table
  • The table shows all the test executions that were used to generate the report
  • Each row links to a detailed table report
  • Each row is highlighted to indicate if the overall test execution has an error, fail, inconclusive or pass verdict
 

 - error
   - fail
   - pass
   - inconclusive
  • The failure rate column shows the percentage of test cases that have failed. A bar visually illustrates the percentage value.
Details
  • The details section shows more detailed information for each test execution.
  • The name of the test execution is shown along with the date the test was run, failure rate, number of fail, error, inconclusive and pass verdicts
  • The name column is highlighted to a specific color to indicate the overall verdict for the test case.
 

 - error
   - fail
   - pass
   - inconclusive
  • The appropriate verdict column in the table is also highlighted. For example, if the overall verdict is a fail, the fail column is highlighted. In contrast, if the overall verdict of the test execution is a pass, the pass column is highlighted.

 

Viewing the Trace Pass Report

The following steps assume that the user has collected memory and execution statistical information. The user is now ready report on the trace information.
  1. Open the Profiling and Logging perspective: Window > Open Perspective > Other... Select the "Profiling and Logging" item from the list of perspectives.
  2. In the "Profiling Monitor" view. Select a Monitor, Host, Process or Agent you want to generate a report on. The report will generate information associated with the selected element.
  3. Right click the selection and select the "Report..." menu item.


  4. Select the "HTML trace report (BIRT)" from the list. Then click "Next".
  5. On the next page specify the location where to store the html report. Then click "Finish".
  6. The report will be presented to the user.

Understanding the Trace Report

The following table outlines the structure of the Trace report.

Report Section Notes Image
Heap Overview
  • A set of pie charts are presented that shows the top 10 packages grouped by total size, total instances and collected instances.
  • A link is provided that jumps to a detailed table with more information.

Execution Overview
  • A set of pie charts are presented that shows the top 10 packages grouped by base time, cumulative time and number of calls.
  • A link is provided that jumps to a detailed table with more information.
Heap Details
  • The heap details section shows heap data for each package
  • The table is ordered by total size, then total instances and then collected instances
Execution Details
  • The execution details section shows execution data for each package.
  • The table is ordered by base time, cumulative time, and number of instances

The following steps assume that the user has imported a log file. The user is now ready to generate a report base on the Common Base Event log information.

  1. Open the Profiling and Logging perspective: Window > Open Perspective > Other... Select the "Profiling and Logging" item from the list of perspectives.
  2. In the "Log Navigator" view. Select a log file that has been imported.
  3. Right click the selection and select the "Report..." menu item.


  4. Select the "HTML Top 10 Report by Event Severity (BIRT)" from the list. Then click "Next".
  5. On the next page specify the location where to store the html report. Then click "Finish".
  6. The report will be shown to the user.

Understanding the Common Base Event Logging Report

The following table outlines the structure of the Common Base Event Logging report.

Report Section Notes Image
Overview chart
  • The stack bar shows the top ten sub components that contains the highest total number of events.
  • Each bar illustrates the number of severe, warning and informational severity events.
Pie charts
  • The pie charts shows the top 10 sub components that have the highest number of severe severity events, highest number of warning severity events and highest number of informational severity events.
Details
  • The detail section shows a table that organizes the events grouped by sub components.
  • The host name and sub component name is shown along with the creation time, severity, message name and extension name of the event.
  • Each column is highlighted to a specific color to indicate the severity of the event.
 

 - warning
   - severe
   - info

 


One of the advantages of creating reports using the BIRT infrastructure is that the presentation information (i.e. layout of report items, font color, font size, etc.) is separated from the raw model data. TPTP fully exploits this concept and creates report template files using the BIRT report designer. The report template files contain the presentation information that bind to a XML data source. This XML data source contains the raw model data whether that is trace information, Common Base Event logging information or test execution information.


The report template files are deployed in various plug-ins within TPTP. It is possible to modify these files using the BIRT report designer and redeploy the modified files in the TPTP runtime. All the template files are bound to the xml representation of the trace, test execution and Common Base Event logging data.

To modify the existing test pass report the report template file must be extracted from the TPTP runtime and sample XML test execution data must be generated. The sample XML data is not necessary but will be helpful to the user modifying the report template file. The Test Pass template file is called "TestExecution.rptdesign" which is located under the "templates" directory in the "org.eclipse.tptp.test.report.birt.ui" plug-in. A XML representation of test execution history data can be generated using the reporting wizard as follows. The steps assume that the test data is already imported in the workspace.

  1. Open the Test perspective: Window > Open Perspective > Other... Select the "Test" item from the list of perspectives.
  2. Select the test elements you want to generate a XML report on.
  3. Right click the selection and select the "Report..." menu item.
  4. Select the "XML Test Report" from the list. Then click "Next".
  5. On the next page specify the location where to store the XML file. Then click "Next".
  6. On the next page specify a time interval. The file will consist of test executions that have run between this time interval. Then click "Finish".

The next step is to get the BIRT report designer and open the report template file.

  1. Click on the Downloads' link from BIRT's home page: http://www.eclipse.org/birt
  2. Select the version and the build that you wish to download. Note that the report template files were created using BIRT 2.0.
  3. Once a desired build is selected, find and download the following items:
    • RCP Report Designer
    • Obtain all items listed under 'Requirements'
  4. Open the "TestExecution.rptdesign" file in the RPC Report Designer.
  5. The data source should be configured to point to the xml data that was generated in the previous steps. Select the "Test Execution" data source in the data explorer.
  6. Right click the selection and select the "Edit" menu item.


  7. Modify the URL for the XML source. Change the URL to the location of the XML file generated in the previous steps.


  8. Select the preview pane to make sure that the report is bound to the generated XML file.


  9. The report template file is now ready to be modified. For more information about the report designer refer to the help files.
  10. Once the modifications have been completed the new "TestExecution.rptdesign" file can be redeployed to the TPTP runtime by overriding the file stored under the "templates" directory in the "org.eclipse.tptp.test.report.birt.ui" plug-in.
  11. The report wizard description for this report can be modified by modifying a couple of entries in the plugin.properties file. This file is stored under the "org.eclipse.tptp.test.report.birt.ui" plug-in directory. The following properties contain the name and description of the report.

    BIRT_TEST_REPORT_NAME = HTML Test pass report (BIRT)
    BIRT_TEST_REPORT_DESC = Create a report with only the most recent execution results for each selected test within a specified period of time.

To modify the existing trace report the report template file must be extracted from the TPTP runtime and sample XML trace data must be generated. The sample XML data is not necessary but will be helpful to the user modifying the report template file. The Trace template file is called "TraceReport.rptdesign" which is located under the "templates" directory in the "org.eclipse.tptp.trace.report.birt.ui" plug-in. A XML representation of trace data can be generated using the reporting wizard as follows. The steps assume that the trace data is already imported in the workspace.

  1. Open the Profiling and Logging perspective: Window > Open Perspective > Other... Select the "Profiling and Logging" item from the list of perspectives.
  2. In the "Profiling Monitor" view. Select a Monitor, Host, Process or Agent you want to generate a report on. The report will generate information associated with the selected element.
  3. Right click the selection and select the "Report..." menu item.
  4. Select the "XML trace statistics report" from the list. Then click "Next".
  5. On the next page specify the location where to store the XML file. Then click "Finish".

The next step is to get the BIRT report designer and open the report template file.

  1. Click on the Downloads' link from BIRT's home page: http://www.eclipse.org/birt
  2. Select the version and the build that you wish to download. Note the report template files were created using BIRT 2.0.
  3. Once a desired build is selected, find and download the following items:
    • RCP Report Designer
    • Obtain all items listed under 'Requirements'
  4. Open the "TraceReport.rptdesign" file in the RPC Report Designer.
  5. The data source should be configured to point to the xml data that was generated in the previous steps. Select the "Trace Data Source" data source in the data explorer.
  6. Right click the selection and select the "Edit" menu item.


  7. Modify the URL for the XML source. Change the URL to the location of the XML file generated in the previous steps.


  8. Select the preview pane to make sure that the report is bound to the generated XML file.


  9. The report template file is now ready to be modified. For more information about the report designer refer to the help files.
  10. Once the modifications have been completed the new "TraceReport.rptdesign" file can be redeployed to the TPTP runtime by overriding the file stored under the "templates" directory in the "org.eclipse.tptp.trace.report.birt.ui" plug-in.
  11. The report wizard description for this report can be modified by modifying a couple of entries in the plugin.properties file. This file is stored under the "org.eclipse.tptp.trace.report.birt.ui" plug-in directory. The following properties contain the name and description of the report.

    BIRT_TRACE_REPORT_NAME = HTML trace report (BIRT)
    BIRT_TRACE_REPORT_DESC = Generate a html report that shows the top 10 trace statistics
3.3 Modifying the Common Base Event Logging Report

To modify the existing Common Base Event logging report the report template file must be extracted from the TPTP runtime. Currently sample XML data cannot be generated. The Common Base Event logging template file is called "LogReport.rptdesign" which is location under the "templates" directory in the "org.eclipse.tptp.monitoring.report.birt.ui" plug-in.

The next step is to get the BIRT report designer and open the report template file.

  1. Click on the Downloads' link from BIRT's home page: http://www.eclipse.org/birt
  2. Select the version and the build that you wish to download. Note the report template files were created using BIRT 2.0.
  3. Once a desired build is selected, find and download the following items:
    • RCP Report Designer
    • Obtain all items listed under 'Requirements'
  4. Open the "LogReport.rptdesign" file in the RPC Report Designer.
  5. The report template file is now ready to be modified. For more information about the report designer refer to the help files.
  6. Once the modifications have been completed the new "LogReport.rptdesign" file can be redeployed to the TPTP runtime by overriding the file stored under the "templates" directory in the "org.eclipse.tptp.monitoring.report.birt.ui" plug-in.
  7. The report wizard description for this report can be modified by modifying a couple of entries in the plugin.properties file. This file is stored under the "org.eclipse.tptp.monitoring.report.birt.ui" plug-in directory. The following properties contain the name and description of the report.

    BIRT_LOG_REPORT_DESCRIPTION= Generate a html report that shows the top 10 sub components by event severity
    BIRT_LOG_REPORT_NAME= HTML Top 10 Report by Event Severity (BIRT)