Re: [linuxtools-dev] TMF and pluggable state providers

[Alexandre Montplaisir <alexmonthy@xxxxxxxxxxxx>]

Hi Aaron,

Wow, this is really impressive! :)

So far we had a vague idea of how to move forward to support
"GUI-defined-states" and custom views, but as they say code talks, so
your implementation could really drive how it's done in TMF in the end!

One of the first steps we will have to work on is to have a really
extensible Events View. Right now number of columns is
kind-of-hard-coded, and it's really hard for specific trace types to
extend it to show their own custom columns. As was discussed in the
other thread, each trace could show type-specific information coming
either from its events, its contexts, the state system, etc. I think we
will be able to tackle this next month.

Some more comments,

On 13-03-14 11:48 AM, Aaron Spear wrote:
> [...]
>
> 1) I changed the TMF code base so that it was possible to have a plugin contribute a named factory that can be used to create state systems. The base implementation of TmfTrace.buildStateSystem checks to see if there is a property for a given trace that specifies which state system factory to use and creates it using that if so. This is nice because you can still have the state system be a function of the derived trace if you want to (like it is for LTTng currently), but you can also have a state system implementation that works with different traces regardless of the underlying file format. 

The way we've done it so far is to make the trace type object (so, the
class that extends ITmfTrace) be the central point used to specify new
trace types. It defines its event type, timestamps, statistics backend,
state system(s), etc., if applicable. That way we don't need a separate
extension points for timestamps, event types, event fields and so on.

For state systems, as you might have noticed, there is a base
implementation in TmfTrace.buildStateSystem(), that can be extended by
specific trace types if they want to add (or override) state systems to
this trace. All assigned state systems have a global ID, so they can be
accessible from anywhere (and not only those that have a reference of
the specific object type).

I haven't seen your implementation so I can't comment, but is there
anything that couldn't be done with the current method, and that
required you to add an extension point for state systems too?

> 2) Using the infrastructure above, I built a framework that does data driven state extraction. The idea is that you have some metadata that describes your hierarchy, event types that correspond to context and state changes, and rules for how to extract the new state and context from the events. It uses the TMF abstraction for events, and so it works independent of how the underlying trace is structured which is nice. 
>
> 3) I have an implementation of the data driven state extraction above that right now is populated using a simple convention whereby the factory looks for an XML file that describes the state data right next to the trace. The right way to do this is to use some nice GUI to either create rules or manually choose which state types you want to use for a given trace, but it works for now. Attached is the XML file used to create the VM trace above so you have some idea of how this works. Again, there are layers to this. In the case of the java function trace I actually have a derived trace type, so for that one the XML file can be in the plugin (or you can hard code it and not bother with the XML)

Awesome. State-definitions in XML or similar has been a hot topic
recently. In the base TMF framework we could have a special state
provider (implementing IStateChangeInput) that would read such a file
(and eventually, the custom contents that was defined in a GUI), that
could then be used to populate custom views.

>
> My plans/hopes: 
> -figure out data driven choice in columns in the display.  Right now the columns are generic and there are only two of them.  User should be able to choose somehow, or it comes with the particular event/state schema.
> -extend the view so that it can display multiple traces at the same time, whichever traces are open in editors currently and support state display should be able to be aggregated.

There is definitely some work to be done in making the views more
generic. Right now the Control Flow and Resource View have a lot of
redundant code, which could be abstracted in a base class. That base
class could then be the basic for the custom view, to which you simply
pass a map of <table entries, state attributes> to populate.

> -Figure out generic way to define rules for defining properties of contexts over time.  While strictly speaking this is still "state" of the contexts, it is not state that is displayed via colors or something, but rather it would be simply interesting to know.  In my example say that a property of a virtual machine changed during some running state (e.g. the amount of memory allocated to it from the host), it would be great to see the events that correspond to property changes as simple black vertical lines on top of the state, and if I hover over the state for that object, I see all of the name/value properties that are current at that point in time.  Perhaps using my data driven approach I could simply push all of these property changes into the state system, e.g.  /properties/MyDataCenter/MyHost/MyVM/thePropertyName = thePropertyValue ?

Yep, anything can be stored in the state system, and it doesn't have to
all go in a timegraph-like view. Some information could be shown only in
the Properties view, or the Events view, or not at all.

For a custom trace type we use internally, we use such a view with
"events" on it, which are simply states with a duration of 1 time unit.
The view then knows that this particular type of event must be shown at
one pixel (or maybe it's three? a fixed amount, in any case), no matter
the zoom level.

> comments? Should I contribute this in small chunks? e.g. the changes to add the new extension point for state system factories first (that seems relevant to other efforts as well)?

Thanks a lot for working on this and contributing it! I don't know if
you are familiar with how Eclipse handles external contributions, it's
not too complicated but let's say it's not trivial either. Right now we
have passed the deadline for accepting "big" (>250 lines) contributions
from non-committers for Kepler. (That could be worked-around if VMWare
had a committer in the Linux Tools project, but I don't think this is
the case.)

But you could always post what you have on Gerrit so people can start
trying it out and reviewing it, but we won't be able to merge it into
master until after Kepler is released, in June.


Cheers,
Alex