Adapter Configuration File structure

An adapter configuration file contains a collection of contexts that are to be run by a Generic Log Adapter runtime instance. Each context describes how to process a single log file. Because the adapter configuration file can contain many of these contexts, the adapter can process many log files at the same time.

Each context associated with a particular log file is constituted of a series of components. Each component has a role to play in processing a log file.

The following diagram shows the structure of a typical Generic Log Adapter configuration.

Context

The context describes the ordered grouping of components as they are chained together for log file processing. Each log file has a separate context, and multiple contexts can be defined in a single configuration. The Generic Log Adapter provides two context implementations:

Property	Type	Behavior
Description	string	A text description for the context. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this context. The context refers to this string to identify which component instance to use.
Continuous Operation	boolean	Whether or not this log file should be monitored for changes based upon the Maximum Idle Time and Pause Interval properties.
Maximum Idle Time	integer (milliseconds)	The maximum amount of time the context will wait for a log file to change before shutting itself down.
Pause Interval	integer (milliseconds)	The amount of time the context waits after reaching the end of the log file before trying to read again.
ISO language code	string	The language code of the Locale that the log file was generated in. These codes are the lower-case, two-letter codes as defined by ISO-639. You can find a full list of these codes at a number of sites, such as: http://www.ics.uci.edu/pub/ietf/http/related/iso639.txt. If this value is not specified, the language of the current Locale where Generic Log Adapter is executing will be used. This value is used when parsing timestamps in the log file.
ISO country code	string	The country code of the Locale that the log file was generated in. These codes are the upper-case, two-letter codes as defined by ISO-3166. You can find a full list of these codes at a number of sites, such as: http://www.chemie.fu-berlin.de/diverse/doc/ISO_3166.html. If this value is not specified, the country of the current Locale where Generic Log Adapter is executing will be used. This value is used when parsing timestamps in the log file.
Encoding	string	The character encoding of the log file to be parsed. The encoding value specified must be a value listed in the IANA Charset Registry (http://www.iana.org/assignments/character-sets) and be supported by the JVM that Generic Log Adapter executes in. If this value is not specified, the log file will be parsed using the default file encoding of the JVM that Generic Log Adapter executes in, as specified in the file.encoding JVM system property.
Log file creation date - Year	integer	The year corresponding to the log file creation date. You can substitute this value in Common Base Event records using the @YEAR parser substitution keyword. This substitution is useful when log file records do not include year information. For more information, see the Parser Substitute property.
Log file creation date - Month	integer	The month corresponding to the log file creation date. You can substitute this value in Common Base Event records using the @MONTH parser substitution keyword. This substitution is useful when log file records do not include month information. For more information, see the Parser Substitute property.
Log file creation date - Day	integer	The day corresponding to the log file creation date. You can substitute this value in Common Base Event records using the @DAY parser substitution keyword. This substitution is useful when log file records do not include day information. For more information, see the Parser Substitute property
Timezone GMT offset	string	The time zone corresponding to the log file creation date. The time zone is measured as an offset from GMT (or UTC). For example, Eastern Standard Time in Canada and the United States is GMT - 5 hours. In this case, you can enter -0500 for the Context Timezone property. You can substitute the time zone value in Common Base Event records using the @TIMEZONE parser substitution keyword. This substitution is useful when log file records do not include time zone information. For more information, see the Parser Substitute property.

Sensor

The sensor provides the mechanism to read the content of the log for processing. The Generic Log Adapter provides the following sensors:

SingleFileSensor: A sensor that reads files from a local storage medium. This can be used for a single file or multiple files that have the same name, for example, in cases where the application uses log file rotation. This sensor allows a log file to be monitored, even if the log file does not exist at the time that monitoring is started.
Executable Class: org.eclipse.hyades.logging.adapter.sensors.SingleOSFileSensor
StaticParserSensor: A sensor that integrates transforms that are performed by Java based implementations. It directly invokes the Java parser and exports the Common Base Event objects that the Java parser produces.
Executable Class: org.eclipse.hyades.logging.parsers.adapter.sensors.StaticParserSensor
AdapterCBESensor: A sensor that monitors the adapter's execution. It is used internally by Generic Log Adapter to log messages from the Generic Log Adapter itself. All internal logging within the adapter is sent to this sensor.
Executable Class: org.eclipse.hyades.logging.adapter.internal.util.AdapterSensor

The following table lists the attributes that are common among the sensor types.

Sensor attributes

Attribute name	Type	Behavior
Sensor Type	String	The sensor type to be used by the adapter.
Description (required)	String	A text description for the sensor. This value is used only by the Adapter Configuration Editor.
Unique ID (required)	String	A unique identifier for this component. The sensor refers to this string to identify which component instance to use.
Maximum Blocking (required)	Integer	The maximum number of lines the sensor reads before forwarding to the extractor for processing. For the StaticParserSensor type, this field indicates the maximum number of Common Base Event objects created by the static parser before forwarding to the outputter for processing.
Confidence Buffer	Integer	The last n bytes of a file to be saved and used for determining if an append has occurred to the file since the last time it was read.
Footer size	Integer	The n bytes of a file that contains static content. Some log files append a footer to the end of the log file. This part of the log file is not considered log data. The footer portion of a file will not be used when determining if an append has occurred since the last time the file was read.

The following table lists the properties that are specific for each type of sensor. The following properties are known by the component code and must be spelled exactly as indicated in the table for them to be recognized and used by the component.

Sensor type	Properties	Application
SingleFileSensor	directory (required) The directory location of the file to be read by the SingleFileSensor. fileName (required) The name of the file to be read by the SingleFileSensor. If the application uses log file rotation, regular expressions can be used to specify the file name. Note: Case is considered when interpreting the fileName field. If you want the field to be case-insensitive, specify the file name using a regular expression. shell The UNIX shell to run the converter command in. Used only on UNIX platforms converter A command line for a preprocessing instruction to run before starting the sensor.	Used when creating an adapter for parsing a log file with regular expression rules
StaticParserSensor	directory (required) The directory location of the file to be read by the StaticParserSensor. fileName (required) The name of the file to be read by the StaticParserSensor. parserClassName (required) the name of the Java class that implements the static parser. The class must implement the org.eclipse.hyades.logging.parsers.IParser interface.	Used when creating an adapter for parsing a log file with a static parser Java class.

Extractor

The extractor takes a collection of input lines provided by the sensor and separates them into message boundaries.

The Generic Log Adapter provides the following extractors:

org.eclipse.hyades.logging.adapter.extractors.RegularExpressionExtractor
This extractor uses regular expressions to describe the delimitation of the log file into messages.
org.eclipse.hyades.logging.adapter.extractors.SimpleExtractor.
This extractor uses simple string comparisons to describe the delimitation of the log file into messages. This extractor, a high-performance alternative to the regular expression extractor, should be used with log files that contain unchanging message delimiters.

The following properties are provided as part of the configuration for the extractor:

Property	Type	Behavior
Description	string	A text description for the extractor. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this component. The extractor refers to this string to identify which component instance to use.
Contains Line Breaks	boolean	Indicates whether the messages in the log file span multiple lines.
Replace Line Breaks	boolean	Replace the line breaks in this file by another string segment. The replacement is made before the extractor searches for Start Pattern or End Pattern. Note: The side effect of using this option is that the values parsed from the log file will not contain line breaks unless explicitly added by the parser substitution rules.
Line Break Symbol	string	The string segment to replace line breaks with. This is used only if Replace Line Breaks is true.
Include Start Pattern	boolean	Whether to include the StartPattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
Start Pattern	string or regular expression	The string segment that identifies the start of a message. You can always use the RegularExpressionExtractor class. However, if your start pattern is a simple string, the SimpleExtractor class incurs less overhead.
Include End Pattern	boolean	Whether to include End Pattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
End Pattern	string or regular expression	The string segment that identifies the end of a message. If End Pattern is not specified, then the message is delimited by the next Start Pattern.

Parser

The parser takes the messages that have been delimited by the extractor and builds a set of mappings of string values to a data structure. In the case of the Generic Log Adapter the data structure is Common Base Event 1.0.1

The Generic Log Adapter provides the following parser class:

org.eclipse.hyades.logging.adapter.parsers.Parser
This parser uses regular expressions to parse values to build Common Base Event objects. The java.util.regex library is used for processing regular expressions.

The parser has two phases of execution:

The global processing phase, where a set of global regular expressions are executed against the message provided by the extractor.
The attribute processing phase, where specific sets of substitution rules are executed to determine the value to be assigned to each attribute in the data structure (Common Base Event 1.0.1).

With two phases of execution, a log record is tokenized into message attributes (in the global processing phase), which can be referenced by attribute name or index and used to form Common Base Event attributes (in the attribute processing state). The attribute values can then be referred to by attribute name or index during the attribute processing phase.

Global Processing Properties

The following global properties are specified as part of the parser:

Property	Type	Behavior
Description	string	A text description for the parser. This value is used only by the Adapter Configuration Editor.
Unique Id	string	A unique identifier for this component. The parser refers to this string to identify which component instance to use.
Separator Token	string or regular expression	A token, or a regular expression that describes a series of tokens, that breaks the message up into attribute-value pairs. For example, if each new line of your log file contains an attribute and its value, you could specify a Separator Token of `\n`. The advantage of breaking the message up is that the string segments can be referred to in the attribute processing phase. Note: If a separator token is not specified then no global processing is performed.
Designation Token	string or regular expression	A token, or a regular expression that describes a series of tokens, that breaks the string segments created by the Separator Token into attribute-value pairs. For example, if each attribute and value are separated by a `=` sign , and the value of Designation Token is `=` , the attribute and its value can be referred to in the attribute processing phase. If a separator token is specified but a designation token is not specified then the message is broken up into a list of attribute values which can be referred to in the substitution rules by index.

Attribute Processing Properties

The individual attributes of the Common Base Event have the following properties. For a complete list of Common Base Event attributes, see Common Base Event properties.

Note: The values parsed during the global processing phase will be trimmed of leading and trailing white space characters. Therefore, substitution rules that use position values can be written assuming the values have no leading or trailing white space characters.

Property	Type	Behavior
Use Previous Match Substitution As Default	boolean	If the current set of substitution rules for this attribute returns a positive match, then store the match value so that it can be used as a default to process the next record if there is no positive match.
Default value	String	If none of the substitution rules for this attribute provide a positive match or there are no substitution rules for this attribute then this attribute will be assigned the default value specified here. If the same value is desired for all records for a particular attribute, this property can be used instead of creating a default substitution rule for that attribute. Note, the Use Previous Match Substitution as Default will take precedence if there is a saved value available.
Required by parent	boolean	Allows you to specify whether or not this element must have a valid value in order for the parent element to be valid. For example, if this option is selected for the situation element, this indicates to the parser that the Common Base Event object must have a valid situation element. During processing, if the situation element returns a null value, the parser will not process any of the other child elements of the Common Base Event object since it already knows that this Common Base Event object is invalid because a required child element returned an invalid value.
Child choice	boolean	Allows you to specify whether or not processing for this element should stop once a valid child attribute or element returns a valid value. For example, consider the situation with three child elements, ReportSituation, RequestSituation and StartSituation, each with their own subtrees. If the Child Choice option is selected for the situation element, parsing will stop at its first child subtree that returns a valid value.

Substitution rules are run as part of the attribute processing phase. They are processed in order from top to bottom. Once a rule is a match, no further rules are run against this property. If no match is found, then no value is placed in the attribute unless the Use Previous Match Substitution as Default or Default Value properties of the attribute are specified.

Property	Type	Behavior
Match	regular expression	The regular expression that identifies the value to be applied to this attribute if a match occurs in the scope of the string segment being processed by this Substitution Rule. If you do not provide a regular expression, the rule is considered a match.
Positions	name-value attribute pairs	Positions let you refer to the name-value attribute pairs created during the global processing phase. If both a separator token and a designation tokens are specified, the Positions string can refer to many attributes by name using the following grammar: `$h('Attr1')@@$h('Attr2')...` `$h('Attrn')` identifies a previously created attribute name. where `@@` is the separator between the attributes. If a separator token is specified but not a designation token, then the Positions string can refer to one or more attributes by index instead of name using the following grammar: `n1@@n2..` where n1 and n2 are integers representing the indices. The index starts at 1. If the message string begins with a separator token, the attribute value at index 1 will be null. The first attribute value will begin at index 2. In the Match property, you can add the separator to indicate which regular expressions should be applied to which attribute value.
Substitute	string	The value to place in the attribute. This can be a string or it can refer to the regular expression in the Match property. The string segments that are extracted by the Match property can be referred to using the groupings in the regular expression. `$1` indicates the first grouping, `$2` the second, and so on. For creationTime attributes, you can use the substitution keywords @YEAR, @MONTH, @DAY, and @TIMEZONE to use the corresponding Context properties Year, month, Day, and Timezone, respectively. For more information about these Context properties, see Context. For example, assume that a log record's time appears in the following format, and excludes the day, month, year, and time zone: `08:40:13.380.` The following Substitute value adds the Day, Month, Year, and Timezone Context properties to the Common Base Event record: `@DAY/@MONTH/@YEAR $1 @TIMEZONE` Note, the above syntax assumes the Match property returns one grouping `$1`, which represents the time from the message.
Use built in function	check box	Use built-in function when selected for an attribute, if none of the previous substitution rules, if any exist, are matched, will cause the Generic Log Adapter runtime to generate a value for that attribute using its own internal functions. This feature is only supported for the following CBE attributes: localInstanceId globalInstanceId componentIdentification.location componentIdentification.locationType
Substitution Class	string	Specifies the name of a user created Java class that can perform custom parsing logic on the value matched by the Positions, Match and Substitute specifications of the rule. If the substitution rule matches data in the record, this class will be called and the result will be used to create the Common Base Event object. Specifications: This class must implement the interface class `org.eclipse.hyades.logging.adapter.parsers.ISubstitutionExtension` or the extension class `org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension` in order for it to be executed by the Generic Log Adapter. Use the extension class when you want to save state data between log file records. The class `org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension` causes the Generic Log Adapter to only allocate one instance of this class to parse all records of the log file. The user class must be included in the class path of the Generic Log Adapter runtime. If the Generic Log Adapter is running in an Eclipse environment, the class can be created as an eclipse plug-in extension of the Generic Log Adapter by using the substitutionExtension extension point of the org.eclipse.hyades.logging.adapter plug-in.
Time Format	string	This field only applies to substitution rules for the creationTime element and is used to simplify date time parsing. For example, some log files contain time stamps that include the names of a month instead of its numerical representation. This makes it difficult to create substitution rules to generate the string value in the form of the XML Schema dateTime data type required by the CBE creationTime element. In this case the user can specify a java simple date format string as documented in the Javadoc HTML documentation for java.text.SimpleDateFormat (java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html) in the Time Format field. The value matched by the Positions, Match and Substitute specifications of the rule will be parsed based on the Time Format string and the resulting value will be used to generate the CBE creationTime value. Note: The Locale, as represented by the ISO language and country code values specified in the Context Instance configuration, is used when parsing the timestamp with the time format string.

Formatter

The formatter takes the mapping of attributes to their values provided by the parser and builds the correct Java object instance. The adapter runtime contains a single formatter.

org.eclipse.hyades.logging.adapter.formatters.CBEFormatter: The CBEFormatter creates Common Base Event objects conforming to the Common Base Event version 1.0.1 specification.
Note: If the value for the Common Base Event msg field is longer than 1024, the CBEFormatter class sets the msg value to be the first 1024 characters of the value and then creates an ExtendedDataElement called "message" with a string array value that contains the whole message split into 1024 pieces.

The formatter contains the following properties:

Property	Type	Behavior
Unique Id	string	A unique identifier for this component. The formatter refers to this string to identify which component instance to use.
Description	string	A text description for the formatter. This value is used only by the Adapter Configuration Editor.

Outputter

The outputter externalizes the resulting Common Base Event records provided by the formatter. Outputters provide, or wrap, the mechanism for storing the final outputs of the context. The Generic Log Adapter provides the following outputters:

org.eclipse.hyades.logging.adapter.outputters.CBEFileOutputter: receives Common Base Event instances and writes the externalized records to a file.
org.eclipse.hyades.logging.adapter.outputters.CBEConvergentFileOutputter: This outputter class extends the CBEFileOutputter class and allows multiple contexts within the same adapter configuration file to write their Common Base Event instances to the same file. Each context within the adapter configuration class must use this outputter class in order to write to the same file.
org.eclipse.hyades.logging.adapter.outputters.CBEstdoutOutputter: receives Common Base Event instances and writes the externalized records to standard output.
org.eclipse.hyades.logging.adapter.outputters.CBELogOutputter: This outputter receives Common Base Event records, creates a logging agent and writes the externalized records to the logging agent. The CBELogOutputter should only be included in adapter configuration files that will be used outside of eclipse. The user can specify the logging agent name as an Outputter property in the adapter configuration file and must manually attach and start monitoring the agent within Log and Trace Analyzer in order to see the data in the Log View.
org.eclipse.hyades.logging.parsers.adapter.outputters.LogImportOutputter: This outputter receives Common Base Event instances and writes the externalized records to the Log Import loader of the Log and Trace Analyzer and is used for both static and rules-based adapters. It should only be used for parsers that are extensions of the org.eclipse.hyades.logging.parsers.logParser extension point.

Properties of the outputter:

Property	Type	Behavior
Unique Id	string	A unique identifier for this component. The outputter refers to this string to identify which component instance to use.
Description	string	A text description for the outputter. This value is used only by the Adapter Configuration Editor.
Outputter Type	enumeration	The type of outputter this is.

Outputter types and required properties:

Outputter type	Required properties
NotificationOutputter	None
StandardOutOutputter	None
SingleFileOutputter	directory, fileName
LoggingAgentOutputter	agentName - name given to the logging agent when it is created waitUntilLoggingTime (optional) - the time, specified in milliseconds, the outputter will wait for the user to attach and start monitoring the agent before it starts writing to the agent. If this property is not specified, the outputter will not wait for the agent to be monitored before writing to it. If this property has a value of 0 the outputter will wait forever. If the value is greater than zero the outputter will wait up to that amount of time for the agent to be monitored.

Copyright © 2005 IBM Corporation. All rights reserved. This document and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which is available at http://www.eclipse.org/legal/epl-v10.html