TPTP Platform Project

Documents

How to write a TPTP Data Collection Agent

 

Author: Antony Miguel (antony.miguel@scapatech.com)

 

Last Updated: 20th January 2004

 

Overview

This document describes the steps necessary to create a TPTP statistical data collection agent.  The user will be able to run their agent locally or remotely as a part of the TPTP Agent Controller (AC) and gather data from it using the Profiling perspective.

 

It is assumed that the reader has prior knowledge of C, Java, Eclipse and Java development using Eclipse.

 

Contents

 

How to write a TPTP Data Collection Agent 1

Overview.. 1

Contents. 1

Basics. 1

Remote Side C Implementation. 2

Remote Side Java Implementation. 3

Sample Agent pluginconfig.xml 4

Client Side Implementation. 5

User Interface. 6

Underlying Agent Client 6

Standard XML Fragments and TPTP Data Models. 7

Statistical Data Fragment 7

Common Base Event Fragment 8

 

Basics

The TPTP data collection framework basically consists of an AC client (Agent Controller client - commonly some java code running in eclipse), AC server, and an agent running on the AC server.  The task of the data collection agent implementor is to write parts of the AC client and the agent that runs on the AC server.

 

The client uses the AC APIs to connect to the AC server and connect to an existing agent or start a new agent.  When connected, the agent sends back data in a standard TPTP XML fragment format (e.g. statistical data fragment format) and a standard loader on the client side populates a TPTP data model.

 

 

 

 

 

 

 

 

 

 

 

 


A concrete example of this would be the Windows Perfmon statistical data collection agent sending back standard XML fragments which are then loaded by the standard TPTP loader into a TPTP statistical model.

 

NOTE: for more information on TPTP Data Models see http://www.eclipse.org/tptp/home/archives/hyades/data_models/index.htm

 

Remote Side C Implementation

A data collection agent consists of a native executable which uses AC dynamic libraries to connect to the AC server and register itself as an agent.  This allows the client to see it and interact with it (e.g. ask it to start monitoring and send back data).

 

The plugin “org.eclipse.hyades.datacollection” in TPTP CVS contains many of the TPTP data collection agents source code.  The agents under this plugin will be used as a reference.

 

The collection agents can be found under:

 

·                    (CVS)org.eclipse.hyades.datacollection/collection/collectors

 

The AC framework code can be found under:

 

·                    (CVS)org.eclipse.hyades.datacollection/collection/framework

 

The libraries and headers your agent will need (from the framework code) are:

 

·                    RABindings / RABindings.h

o                                           The functions in this library are used to set up and register the agent with the AC.

·                    RADataTransfer / RADataTransfer.h

o                                           The functions in this library are used to transfer data from the agent to the AC for forwarding back to the client.

·                    RASharedMemory / RAShm.h

o                                           The functions in this library are used to access the shared memory associated with a particular client

 

An example lifecycle of an agent would be:

 

·                    (agent started)

·                    Agent calls ‘ra_initializeBindings’ to register itself as an agent with the AC

·                    Agent calls ‘ra_startListener’ to notify the AC it is ready to receive commands

·                    Agent creates a new block of memory for messages using ‘ra_allocateMessageBlock’

·                    (client connects)

·                    Agent receives a START_MONITORING_AGENT_REMOTE or LOCAL command in ‘ra_commandHandler_t’ callback implementation

·                    Agent attaches to clients associated shared memory using ‘ra_attachToShm’

·                    Agent sends back XML fragments using ‘ra_writeMessageBlock’

·                    (client disconnects)

·                    Agent receives a STOP_MONITORING_AGENT command in ’ra_commandHandler_t’ callback implementation

·                    Agent detaches from client’s associated shared memory using ‘ra_stopFlushingShm’

·                    Agent frees message block memory using ‘ra_freeMessageBlock’

·                    Agent calls ‘ra_stopListener’ to notify AC it will no longer process commands

·                    Agent calls ‘ra_finalizeBindings’ to deregister itself as an agent from the AC

·                    (agent finished)

 

For more detail on the functions themselves and their arguments see the framework header files.  For a complete example of a data collection agent using the AC APIs, see:

 

·                    (CVS)org.eclipse.hyades.datacollection/collection/collectors/perfmon/rac

o                                           rac.c, rac.h

 

Once you have a working agent you may wish to add your agent to the AC config so it can be launched by the client side.  To do this you will need to create a plugin directory under your AC ‘plugins’ directory and create a pluginconfig.xml with an entry that points to your agent executable.  See the other plugins in the AC plugins directory for examples.

 

 

Remote Side Java Implementation

A data collection agent can also be written as a Java agent which uses standard AC java classes which in turn use JNI to speak to the AC.

 

The important plugins for development of a Java agent are:

 

·                    (CVS)org.eclipse.hyades.execution.remote

 

A Java AC agent uses the class ‘org.eclipse.hyades.internal.execution.remote.RemoteComponentSkeleton’ to connect to the AC and access its APIs.  This class can be found at:

 

·                    (CVS)org.eclipse.hyades.execution.remote/src/org.eclipse/hyades/execution/internal/remote/RemoteComponentSkeleton.java

 

For a complete example of a remote Java agent see:

 

·                    (CVS)org.eclipse.hyades.logging.core/src/org/eclipse/hyades/logging/core/LoggingAgent.java

o                                           LoggingAgent(…) constructor

§                                                                     The LoggingAgent sets up the remote component skeleton (AC API) and registers the necessary listeners.

o                                           write(…) method

§                                                                     The logging agent sends back messages to the client here

 

As with the C agent, you may wish to add a plugin config so that your agent can be launched by clients connecting to the AC.  See the plugins in the plugin directory of the AC for examples of how to set up these plugins.

 

The AC jars your Java agent will likely need to include are

 

·                    hexl.jar (data collection local execution framework)

·                    hexr.jar (data collection remote execution framework)

 

Sample Agent pluginconfig.xml

The following sample pluginconfig.xml file is for a Java data collection agent.

 

Sample pluginconfig.xml:

 

<?xml version="1.0" encoding="UTF-8"?>

<PluginConfiguration>

 

<!-- (set up a AC application to represent our agent) -->

<Application configuration="default"

executable="JBossAgent"

location="%SYS_TEMP_DIR%"

path="%JAVA_PATH%">

      

<!-- (add our JAR to the Java CLASSPATH variable) -->

      <Variable name="CLASSPATH"

position="append"

value="%RASERVER_HOME%/plugins/org.eclipse.hyades.perfmon.jmxagent/lib/jmxagent.jar"/>

 

<!-- (add the necessary AC JARs to the Java CLASSPATH variable) -->

      <Variable name="CLASSPATH"

position="prepend"

value="%RASERVER_HOME%/lib/hexr.jar"/>

 

      <Variable name="CLASSPATH"

position="prepend"

value="%RASERVER_HOME%/lib/hexl.jar"/>

 

<!-- (add a parameter (argument) when our agent is called) -->

      <Parameter position="prepend"

value="-DagentXML=%RASERVER_HOME%/plugins/org.eclipse.hyades.perfmon.jmxagent/agentXML/JBoss.xml org.eclipse.hyades.perfmon.jmxagent.JmxAgentExecute "/>

       </Application>

 

      <Option name="org.eclipse.hyades.perfmon.jmxagent"

type="version"

value="3.0.0"/>

 

</PluginConfiguration>

 

Client Side Implementation

The client for the data collection agent may depend slightly on the nature of the data that is being collected but all AC clients listening to agents use a common infrastructure.

 

A standard TPTP data collection client uses the TPTP data models to build a common structure of objects (which are serialised and persisted using EMF).  This structure is described in detail in the TPTP Data Models link earlier but the structure in brief is:

 

·                    TRCMonitor (e.g. monitorfile.trcmxmi)

·                    TRCNode (e.g. nodefile.trcnxmi)

·                    TRCProcess (e.g. processfile.trcpxmi)

·                    TRCAgent (e.g. agentfile.trcaxmi)

o                                           EList (Common Base Events)

§                                                                     CBECommonBaseEvent 1..N

o                                           EList (Statistical Model Descriptors)

§                                                                     SDDescriptor 1..N

 

The EList of Common Base Events under the agent is a flat list of CBECommonBaseEvent objects.

 

The EList of Statistical Model Descriptors under the agent is a list of hierarchies.  Again the statistical model is described in detail in the TPTP Data Models link above but basically each SDDescriptor may contain other SDDescriptors ad infinitum and SDDescriptors of different types (e.g. SDCounterDescriptor) may contain SDSnapshotObservations (e.g. SDContiguousObservation), which are essentially two lists representing time/value pairs.

 

A single trace will consist of all four of the above files although traces may share a TRCNode and/or a TRCMonitor object.  The TRCAgent object and file (.trcaxmi file) contains the bulk of the information such as Common Base Event messages and Statistical Model data.

 

The plugins important to the client side are:

 

·                    (CVS)org.eclipse.hyades.models.hierarchy

o                                           This plugin contains the classes representing the TPTP hierarchy model (TRCMonitor, TRCNode, TRCProcess, TRCAgent etc.)

·                    (CVS)org.eclipse.hyades.models.statistical

o                                           This plugin contains the classes representing the TPTP statistical model (SDDescriptor, SDCounterDescriptor, SDContiguousObservation etc.)

·                    (CVS)org.eclipse.debug.core

o                                           This plugin contains the Eclipse Launch Configuration classes you will need to extend to launch your agent.

·                    (CVS)org.eclipse.hyades.execution.local

o                                           This plugin contains the AC client side APIs

 

A AC statistical data collection client can be logically split up into two parts; user interface and the underlying agent client.

 

User Interface

The user interface for a standard ACclient borrows a lot from standard Eclipse and TPTP UI components.

 

Generally, the user will launch your agent using an Eclipse Launch Configuration and then monitor it using standard TPTP views.

 

The steps to writing a standard UI for your agent are therefore:

 

·                    Extend the Eclipse Launch Configuration extension point (org.eclipse.debug.core.launchConfigurationTypes)

·                    Have your launch configuration invoke your underlying agent client which should set up all the relevant TPTP data models and structures

·                    Monitor your agent using the standard TPTP views:

o                                           Profiling Perspective / Profiling Monitor

o                                           Profiling Perspective / Statistical Data View

 

Underlying Agent Client

The underlying agent client for a standard AC data collection agent also uses many standard TPTP classes.

 

 

Creating the object hierarchy:

 

Your client will first need to create the proper hierarchy of TPTP data model classes to be populated by the standard loaders.

 

For an example of how these should be created see:

 

·                    (CVS)org.eclipse.hyades.perfmon/src/org/eclipse/hyades/perfmon/utils/internal/TRCAgentUtil.java

o                                           newAgent(…) method

§                                                                     This method creates a new hierarchy based on the names for each component and any resource URIs specified for existing hierarchy components

o                                           saveAgent(…) method

§                                                                     This method saves a TRCAgent and the hierarchy above it to their associated resources (files).

 

 

Launching your agent on the AC:

 

Once you have your hierarchy and a TRCAgent object, you can connect to the AC and start up your agent.

 

For an example of how to connect to the AC and start a new agent see:

 

·                    (CVS)org.eclipse.hyades.perfmon/src/org/eclipse/hyades/perfmon/utils/internal/DCAgentGenericUtil.java

o                                           init(…) method

§                                                                     This method initiates the connection to the AC and starts up a new agent. 

§                                                                     Initially, it checks to see that some necessary bits of hierarchy model are there and creates them if necessary. 

§                                                                     It then adds a listener to these bits of hierarchy model.  This listener allows it to track changes to the statistical model.

§                                                                     Finally, it connects to a AC Node (host machine) and starts a new agent, which it then starts monitoring.

§                                                                     When the agent is connected, it creates a new XMLLoader to parse the XML fragments coming from the agent and add them as objects to your hierarchy model objects.

§                                                                     NOTE: this class listens to data coming back from the agent and passes it directly on to the XMLLoader for parsing/loading.  However, the class first generates a <TRACE> tag on the beginning of the stream it sends to the loader to make the trace seem like a parseable XML document.

 

Once you have connected to the AC and are passing your agent’s XML fragments into the XMLLoader, the loader will add to your hierarchy objects as appropriate.  A reference to your agent should appear in the Profiling Monitor view and you should be able to monitor the statistical data coming back via the Statistical Data view.

 

Standard XML Fragments and TPTP Data Models

 

Statistical Data Fragment

 

Example Trace:

 

<TRACE>

<Descriptor id="4" name="CPU" >

<description>The CPU usage of this machine</description>

</Descriptor>

<CounterDescriptor parent="4" id="4C12" name="CPU 0" >

<description>CPU 0 usage</description>

</CounterDescriptor>

<CounterDescriptor parent="4" id="4C24" name="CPU 1" >

<description>CPU 1 usage</description>

</CounterDescriptor>

<ContiguousObservation memberDescriptor="4C12" time="109431771000" value="45.2"/>

<ContiguousObservation memberDescriptor ="4C12" time="109431772000" value="30.0"/>

<ContiguousObservation memberDescriptor ="4C12" time="109431773000" value="12.1"/>

<ContiguousObservation memberDescriptor ="4C12" time="109431774000" value="4.1"/>

<ContiguousObservation memberDescriptor ="4C12" time="109431775000" value="9.1"/>

 

Schema:

 

http://www.eclipse.org/tptp/platform/documents/resources/dtd_xsd/statistical.xsd

 

 

Common Base Event Fragment

 

Example Trace:

 

<TRACE>

<CommonBaseEvent creationTime="2004-10-26T19:32:52Z" severity="10" msg="Hello World">

       <sourceComponentId component="MyMachine" subComponent="Log" />

</CommonBaseEvent>

<CommonBaseEvent creationTime="2004-10-26T19:32:54Z" severity="10" msg="Hello World 2">

       <sourceComponentId component="MyMachine" subComponent="Log" />

</CommonBaseEvent>

 

Specification:

 

http://www.eclipse.org/tptp/platform/documents/resources/cbe101spec/CommonBaseEvent_SituationData_v1.0.1.pdf

 

Schema:

 

http://www.eclipse.org/tptp/platform/documents/resources/cbe101spec/commonbaseevent1_0_1.xsd