Re: [platform-releng-dev] improved checking for internal references

[Jim des Rivieres <Jim_des_Rivieres@xxxxxxxxxx>]

Hi Jeff, 

There's no notion of "plugin buddies" that would express that two plug-ins 
had some secret arrangement between them.  The only thing is a component, 
and a plug-in is either on the inside or the outside. If it's on the 
inside, all inter-plug-in references are considered fair game. If a 
plug-in is on the outside a component, it's expected to live by rules and 
use only official APIs.

For the Core Runtime and Core Resources, I would consider them separate 
components. I think it's fair to say that they were always viewed as 
separate concerns, and that there is no particular reason in principle why 
the Resources plug-in should require special access to Runtime. Since 
there are only a few references from Resouces to Runtime internals, I'd 
want to understand whether 
(a) are they accidental or deliberate
(b) does the coupling suggest inadequacies in Core Runtime's API (i.e., 
there's something important that Core Runtime has that it should be making 
available to all clients)
(c) does it suggest that Core Resources is over-extending itself by making 
unsupportable assumptions on Core Runtime internals
(d) is there anything that can be done to better decouple them

Cheers, jim




Jeff McAffer/Ottawa/IBM@IBMCA 
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
02/08/2005 11:20 PM
Please respond to
platform-releng-dev


To
platform-releng-dev@xxxxxxxxxxx
cc

Subject
Re: [platform-releng-dev] improved checking for internal references







Jim, this is great.  I can't wait to see the real platform-wide results. I 
have a question about how this will work 

Is there way of defining "plugin buddies"?  That is, often times there are 
sets of plugins that know a bit more about each other.  Often there exist 
an informal agreement between plugin authors regarding the use of classes 
or methods that are not formally API.  Of course, this happens all the 
time between elements of the same part of Eclipse.  For example, I'm sure 
that PDE UI uses non-API classed in PDE Core.  For this I assume PDE would 
just define one <component> and call it a day. 

But I was looking at users of non-API runtime code the other day and there 
were several from the Resources plugin.  Runtime and Resources are done by 
the same team but its not clear that you would really want them as the 
same component.  What should we do in that case? 

Jeff 



Jim des Rivieres/Ottawa/IBM@IBMCA 
Sent by: platform-releng-dev-admin@xxxxxxxxxxx 
02/08/2005 05:52 PM 

Please respond to
platform-releng-dev


To
platform-releng-dev@xxxxxxxxxxx 
cc

Subject
[platform-releng-dev] improved checking for internal references








Thanks to Jeffrey Liu of the Web Tools Project, we now have an improved 
tool that will run as part of our builds to help us detect common cases 
where our APIs are being overstepped or circumvented.

Unlike the previous tool, which explicitly checks for references for 
internal packages in another plug-in, this tool works by check to see 
whether a reference is to a bona fide API class. The resulting report of 
violations is the same in either case.

What this tool will do is let us (Eclipse Platform project) police our 
internal cross-component references. More importantly, it will also enable 

any downstream customer to detect illegal references into the Eclipse 
Platform components, as well as between their components. And customers 
can do this even if they adhere to API naming conventions different from 
those used by Eclipse.

The tool takes as input a set of component definition files 
(component.xml) that provides the following information for each 
component:

- what plug-ins are to be treated as a unit for the purposes of drawing an 

API boundary; all references between the plug-ins are considered a private 

matter (even if to internal packages)
- a list of packages that are to be considered API packages
- a list of classes and interfaces within the API package that have 
additional constraints on them (e.g., clients are not supposed to 
implement an API interface, instantiate an API class, or instantiate an 
API class)

Each component definition file is a development-time resource that makes 
sense being maintained in the root of some plug-in. This means the file is 

under version control and can be maintained by the team as their component 

API evolve. (At some point I expect we'll embed this information in the 
source code using J2SE5 annotations, and dispense with the component.xml 
file entirely. Someday.) The component.xml file will be picked up in place 

by the RelEng build script and used to check for APi violations. The 
component.xml files will not be included in the normal binary for of the 
plug-in. Rather, they will be treated like other development-time files 
(e.g., extension point schema files) and included in the special source 
features/plug-ins we build for the SDK drops.  This allows downstream 
builds to use the component.xml files from there so that they can check 
their components for references to Eclipse Platform APIs. And so on down 
the line.

As an example, here's a component.xml file I crafted for the Core 
Resources component:



I'd like to ask each of the teams to construct a component definition file 

for their component. In some cases, it may make sense to lump several 
plug-ins together as a single component; in other cases, it may make sense 

to treat otherwise intimate plug-ins as separate components. You're 
probably in the best position to decide exactly what component boundaries 
make the most sense (if not, I can help).  Once the component boundaries 
are decided, it's very easy to fill out the information on the API 
packages and classes.

The key thing for downstream customers is that the component API 
descriptions are created and maintained absolutely faithful to the API 
specifications at all time, so that they can infer from a clean check that 

their component are in good shape (within the known limitations of what 
the tool is capable of checking). Customers like WTP will be running this 
tool are part of their builds, and so will others.

Once a component.xml is in place, I suggest trading with another team to 
perform an independent cross-check.

Remember to add component.xml to the src.includes= line in the 
build.properties file, so that the file is included in the SDK drops.

Sonia is already running the new tool with each build and posting the 
report (inside the Test Results page). (The tool is currently running 
against some bogus component.xml files that I had created; these will be 
superceded by your authentic component.xmls coming from your plug-ins.) 

The complete details of the component.xml file are here:
http://dev.eclipse.org/viewcvs/index.cgi/~checkout~/working/apitools/api-usage-checking-design.html?&cvsroot=WebTools_Project


For further details on the tool: 
http://eclipse.org/webtools/development/apiscanner/apiscanner.html 

Let me know if you have any questions or concerns.

---jim




Jim des Rivieres/Ottawa/IBM 
12/10/2004 03:35 PM

To
platform-releng-dev@xxxxxxxxxxx
cc

Subject
checking for internal references





We have a tool that reports suspicious inter-plug-in references to classes 

in internal packages.

I'm proposing that we run this tool as a standard step in each build 
(takes < 1 minute to run) and post the results along with the build. Like 
the performance results, this step would be for infomation only; i.e., a 
less than clean check build would not fail the build. The information will 

allow everyone to see where we are breaking (bending ?) our own API rules, 

and motivate us to address them. 

For 3.1, we should make it our goal to come clean and use our APIs as we 
expect everyone else to. I realize that some of the cases that are 
currently being flagged incorrectly, particularly cases where a single 
component is split across multiple plug-ins (e.g, o.e.ui and 
o.e.ui.workbench). In time, we can eliminate these false positves by 
introducing a more sophisticated notion of which plug-ins make up a single 

component.

I'd like to see this new check turned on as soon as possible. Since some 
plug-ins/teams will be put in the spotlight once the checks start running, 

I thought I'd give everyone a chance to tidy up their houses before the TV 

crews arrive<g>, so I'm suggesting we turn it on after 3.1M4.

Let me know if you have concerns.

---jim

p.s. The tool is named "Internal" and can be found on the JDT Core team's 
development resources page.
Here's a sample of the kind of report it generates (the report dates back 
to the days of 2.1):
http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-core-home/tools/internal/examples/index.html

[attachment "component.xml" deleted by Jim des Rivieres/Ottawa/IBM]