Re: Re[8]: [equinox-dev] OBR2

[Olivier Gruber <ogruber@xxxxxxxxxx>]

You are right Jeff, the point is not
to boil the ocean. 
The point is to try to bring to general
availability some of the finer points about provisioning bundles and groups
that we discovered through the years.
We do not want to dismiss or forget issues or problems that 
we already know about. This is hardly
boiling the ocean.

The issue of sharing

As we both know, the question of grouping
introduces the issue of sharing... 
I agree with you that allowing sharing
between groups is more preferable. 
But it is not as simple as you put it,
unless one has simplifying assumptions. 
This is fine, but we need to be really
clear about these.

You are correct, deployment packages
in MEG owns their bundle, it is somewhat unfortunate, but it is 
quite unavailable in the MEG context.
You see, groups "configure" their bundles, which includes 
both creating configurations for services
as well as potentially having side effects on the contents 
of the bundle themselves (such as creating
and initializing files or local databases in bundles). 
Unless I am mistaken, both MEG and Eclipse
are potentially in this situation.

If bundles would not be "configured"
or "modified" in any way during installation, and there would
be no side
effects to the environment because a
bundle is installed as part of a group, then I would agree sharing is no
real issue... a simple reference counting
scheme between the groups and individual bundles would do the trick
of knowing when a bundle can be uninstalled.


Notice though that this suggests a local
shared repository for downloaded bundles (where the reference 
counting or some other garbage collection
form happens)... 

Notice also that we do not have that
in Eclipse, as far as I know---but please correct me if I am wrong.

Let's look at an example. I will use
one that I know very well, but I am sure we can create more Eclipse-centric
ones through the scripting abilities
of features...

Configuring OSGi services is one typical
case... that Eclipse is not too accustomed to for it seldom uses or promotes
services.
With services, the configuration happens
at the level of a group... because several bundles need to be configured
together... 
if any bundle is shared, then configuring
becomes a more complicated issue. It is a problem when installing a new
feature 
that needs to reconfigure... but also
when uninstalling one of the features that had done some configuration...


Beyond that point about sharing, I like
the scenario that you outline: discover - acquisition - develop.

Discovering... the first question
is about what a developer is trying to discover... is it a bundle?
Well it depends, and in most of the
case, it is not really a bundle... He or she is trying to fix 
missing dependencies. Eclipse has a
simpler environment because it promotes mostly dependencies 
between bundles. This is a very usefull
case, but it is a specific and simpler case. The same is true
about dependencies on extensions or
extension points, since an extension point name is prefixed 
by the bundle name.

But for OSGi, with services for example,
or even with most Java package dependencies (such as interfaces,
specifications, etc.), the dependencies
are not on bundles. One may need a Java package or a service,
not knowing anything about bundles.
Therefore, the dependencies need to be translated at 
some points to individual bundles or
groups of bundles. Independently from groups or bundles,
we need a discovery infrastructure going
from a dependency to a bundle or a group of bundles.

This is one point I was making about
we still have work at the level of bundles, not even considering 
groups. By the way, groups do not really
help discovery... it is a matter of scale... for a few thousands
bundles, one may hope few-enough groups
for humans to handle. But in a world-wide ecosystem, with 
an environment as successful as Eclipse
and OSGi, there will be thousands of groups, if not more.
The numbers are a loosing game, with
bundles or groups.

So groups do not help discovery... but
they help packaging. Groups do help in downloading 
coherent sets of bundles. You are correct.
It is a useful concept for packaging bundles together into one 
coherent unit. They carry a constraint
of installing or uninstalling all the parts of the group.

Groups therefore helps acquisition...
although they also increase the risk of conflicts.

Acquisition, you are correct,
this strongly suggests a resolver of some sort, working hand-in-hand
with the discovery process. Indeed,
knowing what group of bundles or individual bundles handle
a dependency is not enough. Conflicts
may arise (between versions) or between service 
configurations. Also, licensing issues
may prevent using a given bundle or bundle group.

Again, the fact that we have groups
is somewhat orthogonal to this. We have the same problems
if we consider bundles alone. While
groups help through carrying de-facto assembly constaints
and by a matter of granularity, they
also make conflicts more likely if bundle sharing happens.
I am not sure any one knows where the
right balance is.

Develop, I totally agree with
you... that acquisition shall not mean installation... in fact, I would
even go 
further... that acquisition of the different
forms of bundles should appear more seamless and more 
integrated than they are today. We have
CVS-like path for sources. We have import/export of 
binary or binary with sources. We also
have the update manager... to "install" binary plugins...
some of which carry sources for other
binary plugins... 

Having one concept of acquisition would
be great.
Having one place to put acquired bundles
would also be great, which you call "target area".
Being able to acquire different facets
of a bundle would also be great...
Like for instance first acquire a binary
version of a bundle... to compile against... 
and then easily acquire the source for
that binary bundle to be able to debug...

Hope this clarifies things a bit further.
Best regards,

Olivier Gruber, Ph.D.
Persistent & Distributed Object Platforms and Frameworks
IBM TJ Watson Research Center

Jeff McAffer <Jeff_McAffer@xxxxxxxxxx> Sent by: equinox-dev-bounces@xxxxxxxxxxx 10/02/2005 02:34 PM Please respond to Equinox development mailing list

To:        Equinox development mailing list <equinox-dev@xxxxxxxxxxx>         cc:                 Subject:        Re: Re[8]: [equinox-dev] OBR2

Fine points Olivier.  We should be careful to not "boil the ocean"
as they say.  There are some very real pragmatic problems of consumability
here.  A repo of 1000 bundles with no grouping and/or categorization
will not acheive the mission we need to acheive.  It seems the basic
problem is that developers cannot easily discover, acquire and code against
the bundles that are available.  As such, it is not apparent that
OSGi is a happening place and the barrier to entry is higher.  The
runtime deployment and updating of bundles is certainly interesting but
IMHO we are not that far yet and the solution to that problem may will
be domain specific. 

With that in mind, consider the workflow followed by a developer on a quest
for function. 
1) discover - They need to browse/search for bundles that meet their requirements
or given a bundle, find all the prerequisites.  Catalogs, queries,
categories, groupings, etc figure large here as do constraint based traversals.

2) acquisition - The developer wants to get coherent sets of bundles against
which he can compile and with which he can test.  This implies some
sort of dependency validator or resolver.  Even if grouping mechanisms
are used validation is still needed.   

3) develop - Source must be easily obtained and acquisition should not
imply installation.  Acquired bundles and source should be put into
a "target area" whatever that means for your development setup.

The problems Olivier outlined are real but (thankfully) many do not apply
in this usecase.   

As for groups, I am not proposing groups be standardized or imposed on
anyone.  Rather groups should be available as *a* means of describing
sets.  Another alternative would be JSR124 data and a resolver.  These
approaches (and others) should be able to co-exist.  The key is the
use of structures of metadata that "talk about" the bundles rather
than structures that "contain"/hide the bundles.  For example,
multiple Eclipse features can "include" the same bundle.  Features
are just metadata about bundles.  Further, the repository structure
(update sites) are open and, for example, you are free to go to an update
site and get individual bundles.  (I'm not saying features or update
sites should be used, but they are an example).  MEG DeploymentPackages
on the other hand are not just metadata as they *contain* their bundles.
 This is unfortunate. 

I hope that clears up some things. 
Jeff

Olivier Gruber <ogruber@xxxxxxxxxx>
Sent by: equinox-dev-bounces@xxxxxxxxxxx

09/29/2005 06:19 PM

Please respond to
Equinox development mailing list

To	Equinox development mailing list <equinox-dev@xxxxxxxxxxx>
cc
Subject	Re: Re[8]: [equinox-dev] OBR2

Peter, Jeff, 

Groups have value, but they have problems. But I would like to make sure
that we don't impose groups. 
We may want to promote one model for groups, through tool support in Eclipse
for e.g., but I don't feel 
we are in a position to impose a concept of groups, nor that we have a
good general solution. 

First of all, even we would all agree that groups have value, we have different
grouping models. 
At least, Eclipse features and MEG deployment packages. I also know of
Linux distribution 
using RPM deployment to bring down plugins (after all, this is just about
downloading files). 
I don't think we are in any position to impose a grouping... You are right
Peter, there is a lot 
of semantics associated with grouping... and it is mostly a concern for
packaging and 
management tools. 

As we all remember, grouping introduces many headaches, especially when
sharing 
between groups are entertained. Installing and uninstalling groups with
non empty 
intersections is tricky. As far as I know, no one has a decent and robust
solution so far. 
This is especially true because groups never stay simple about sets of
bundles... 
Sooner or later scripting appears... configuration data... etc...
It is the case for Eclipse and it is the case for MEG Deployment Packages...
Then groups get into fragments... and optional downloads based on environment
variables... 
because applications are formed out of different  configurations in
different environments... 
and this is not to consider different offerings (different prices for different
configurations). 

So it is hard to reason about groups in these contexts... to even predict
if it can be installed or not. 
Then again, you are right Peter... dependencies between groups open up
again the semantic 
question about flexibility (import-export versus provide-require). Usually
dependencies are more like 
provide-require... but we need the import-export flexibility... While for
Java packages, we can 
rely on the specification versus implementation semantics, this is no longer
true for groups... 

However, there are very good reasons for not using bundles instead of groups...
Admin granule is usually much larger than the runtime bundle granule...
You want to allow flexibility in the configuration of a product or a product
suite... 
You want fine-grain ability to update... fine-grain security control...
fine-grain lifecyle and lazy activation... 

All of which demonstrates that the runtime modularity of a software does
not match 
very well the admin modularity of it... or its provisioning modularity...
Something that we should may think about more... :-)

But here is my point, we may allow groups... but it seems that we don't
want to specify 
them in the OSGi standard. In fact, I like that OSGi is mostly about runtime
artifacts, and not 
about provisioning. I would argue that we don't want to force the adoption
of a group semantics. 
I would argue that we need to fit in existing provisioning efforts... like
JSR 124... or like OMA... 

I would say that we need the following, just for bundles, without talking
about groups. 
We need to have a provisioning infrastructure for bundles, and bundle alone.
We need to be able to find where a bundle can be downloaded from (technically
easy 
but not that easy in practice to put in place...) We need to be able to
override 
a bundle's definition of its dependencies,  so to make it fit in a
configuration 
when provisioning it (a flexibility we don't have so far).
Above this layer, it should be easy to build the notion of groups, if one
wants to. 

There is a lot to do at the level of bundles alone... 
We have to consider extensibility there for dependencies...
We already have import-export statements... and require-provide...
We also have declarative services... and extensions and extension points
(I know some people that may not want to deploy a plugin that provides
extensions 
to non-available extension points)... 

I like the JSR 124 approach... where we talk about resources, providing
and requiring capabilities. 
The approach is extensible with the ability to define new capabilities...
with new matching rules. 
It would be interesting to see if we can model 

       - RFC 79 
       - SCR 
       - extensions and extension points

What's missing I believe would be a discovery infrastructure... 
mapping from capabilities to resources... 
some of which will be OSGi bundles.... some of which may be other things.

I feel that OSGi provisioning needs to fit in a larger story that allows
to deploy not only bundles... 
I don't feel like we will be able to impose a unique grouping model...
but through an open approach 
we may allow OSGi to be a premier choice for Java component models.

Groups are surely important, and I would suggest that we promote one model
through Eclipse 
and a good tooling support... mostly as a proof that we can built it on
top of the bundle-level 
infrastructure... but I don't see why we would want the provisioning infrastructure
to be imposing 
a grouping model. Promote... but accomodate... in other words... apply
modularity and flexibility 
to our own designs and architectures... :-) 

Best regards, 

Olivier Gruber, Ph.D.
Persistent & Distributed Object Platforms and Frameworks
IBM TJ Watson Research Center

Peter Kriens <Peter.Kriens@xxxxxxxx> Sent by: equinox-dev-bounces@xxxxxxxxxxx 09/29/2005 08:58 AM Please respond to Equinox development mailing list

To:        Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>       cc:        Equinox development mailing list <equinox-dev@xxxxxxxxxxx>       Subject:        Re[8]: [equinox-dev] OBR2

I think this is the discussion require-bundle versus Import-Package
all over again ... :-)

a priori grouping will favor popular platforms until the extent that
there will be only one dominant platform because is too hard to
support all. (Law of increasing returns). E.g. Microsoft windows.
Without grouping I think it is easier for a management system to
compose a set that is needed, allowing third parties to provide the
glue components.

And I never understood why things are not just merged in a single
bundle if they really have to be together ...

The diversity of the embedded market requires imho a flatter model.
But then again, I guess I am only a theoretician :-(

Kind regards,

   Peter Kriens







JM> For the most part.  I believe that explicit groups are
JM> necessary.  It is natural to group things and then group the
JM> groups etc.  Doing this with queries is possible but ends up
being
JM> roundabout.  Your point about explosion is valid but queries
don't
JM> solve the problem. For example, who is going to manage the
JM> queries? How do I tell you about my group?  Do I send you my
JM> query?  What if someone adds something that matches the query
but
JM> I did not intend for that to be part of the group?  How do
you
JM> compose multiple query-based groups to make a composite group?

JM> Groups can be done in a way that allows you to have your
JM> groups and me to have mine.  We can share the group definitions
if
JM> we want but do not have to.  If shared they can be inthe repo
as
JM> first class enties.  They (perhaps) can be versioned and express
JM> dependencies of their own.

JM> Jeff



JM> Peter Kriens <Peter.Kriens@xxxxxxxx>
JM> Sent by: equinox-dev-bounces@xxxxxxxxxxx
JM> 09/25/2005 11:14 AM
JM> Please respond to
JM>  Equinox development mailing list

JM> To
JM> Jeff McAffer/Ottawa/IBM@IBMCAcc
JM> Equinox development mailing list <equinox-dev@xxxxxxxxxxx>Subject
JM> Re[6]: [equinox-dev] OBR2




JM> I think we agree ...

JM>  The repo metadata should have no semantic knowledge of its
contents.
JM>  The requirements/capabilities should be generic. When you
import you
JM>  translate the specific (import-package, require-bundle, scree=128x256,
JM>  etc) requirements to generics. This means that the repo does
not need
JM>  to have semantic knowledge.

JM>  However, a group should be the result of a query and not explicitly
JM>  submitted to the repo. If you do that, you always end up in
a
JM>  combinatorial explosion of groups. The diversity out there
is so high
JM>  ... which en then usually means only 1 configuration is supported.
JM>  Works maybe for Windows, but OSGi needs something more flexible
imho
JM>  because portability is at the root.

JM>  Kind regards,

JM>      Peter Kriens

JM>> Just to clarify what I after is not having to author two sets
JM>> of metadata.  From a repository USER's point of view I
have a
JM>> bundle, I add it to the repo.  I need a bundle, I query
the repo
JM>> and get the thing I need/want.  When adding a bundle to
the repo
JM>> the repo itself should interrogate the bundle and find any
JM>> dependencies.  

JM>> There is a challenge in that there may be dependencies that
JM>> cannot be detected.  For example, we have this in our Help
setup
JM>> where help needs an appserver.  Typically Tomcat is supplied
but
JM>> there may well be others.  You would not want to spec help
to
JM>> depend on Tomcat.  If we were using services then you would
spec
JM>> Help to need the webapp service.  Perhaps when we have
declarative
JM>> services that will be a possibility but there will still be
such
JM>> scenarios where the public contract implied by services is not
JM>> appropriate and alternative mechanisms (e.g., the extension
JM>> registry) are used.

JM>> Grouping bundles is an interesting notion that solves this
JM>> issue as well as allows function producers to explicitly declare
JM>> sets of bundles that go together.  A bundle group can simply
be
JM>> viewed as a bundle that aggregates the constituent parts.  However
JM>> it is actually represented it is just a unit that exports the
JM>> union of the exports in its contents and imports/required the
JM>> union of its prerequisites. etc.  

JM>> Expressing or representing these capabilties and requirements
JM>> in an an open, extensible way is goodness.

JM>> Jeff



JM>> Peter Kriens <Peter.Kriens@xxxxxxxx>
JM>> Sent by: equinox-dev-bounces@xxxxxxxxxxx
JM>> 09/23/2005 12:16 PM
JM>> Please respond to
JM>>  Equinox development mailing list

JM>> To
JM>> Thomas Watson <tjwatson@xxxxxxxxxx>cc
JM>> Equinox development mailing list
JM>> <equinox-dev@xxxxxxxxxxx>, heavy@ungoverned.orgSubject
JM>> Re[4]: [equinox-dev] OBR2




JM>> Good idea, but I assume it can be used to generate the meta
data?

JM>>  I think this is an example where the generic requirements
capability
JM>>  model shines. We will have lots of dependencies that need
to be
JM>>  modeled. Modeling them semantically in the repo format
will make the
JM>>  repo format a major spec work undertaking.

JM>>  Kind regards,

JM>>      Peter Kriens

TW>>> Can we use SCR's component XML definition for service
TW>>> dependancies?  Even if the bundle does not use SCR
directly can we
TW>>> still use this schema to define service dependancies?  I
would not
TW>>> want to define a whole new schema for this.

TW>>> Richard, are you subsribed the the equinox-dev mailing list?
TW>>>  If not, please do so we don't have to keep cc'ing
you on the
TW>>> mails :)

TW>>>  Thomas Watson
TW>>>  Pervasive Development
TW>>>  Phone: 512-838-4533 Tie: 678-4533
TW>>>  tjwatson@xxxxxxxxxx


TW>>> equinox-dev-bounces@xxxxxxxxxxx wrote on 09/23/2005 08:34:11
AM:

>>>> Agree ... if it is not in the manifest, we can't generate
it. But it
>>>> is nice to populate a repo with only the data from the
manifests and
>>>> get reasonable results.
>>>> 
>>>> Kind regards,
>>>> 
>>>>      Peter Kriens
>>>>      
>>>> RSH> Peter Kriens wrote:
>>>> 
>>>> >>It absolutely MUST be possible to generate the
metadata from the
>>>> >>manifest.
>>>> >>  
>>>> >>
>>>> 
>>>> RSH> Well, it is not possible given the current state
of bundle manifests to
>>>> RSH> generate all repository metadata. For example,
there is no way to
>>>> RSH> generate service dependencies.
>>>> 
>>>> RSH> If someone invents a new capability/requirement
pair, there will be no
>>>> RSH> way to generate it either.
>>>> 
>>>> RSH> For the standard package/module-related stuff,
we can generate them from
>>>> RSH> the manifest.
>>>> 
>>>> ->> richard
>>>> 
>>>> RSH> _______________________________________________
>>>> RSH> equinox-dev mailing list
>>>> RSH> equinox-dev@xxxxxxxxxxx
>>>> RSH> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>>>> 
>>>> 
>>>> -- 
>>>> Peter Kriens            
                 Mob +33633746480
>>>> 9C, Avenue St. Drézéry          
         Tel +33467542167
>>>> 34160 Beaulieu, France          
         Tel +15123514821
>>>>                
                     
    Tel +33870447986
>>>> AOL,Yahoo, Skype pkriens          
       ICQ 255570717
>>>> 
>>>> _______________________________________________
>>>> equinox-dev mailing list
>>>> equinox-dev@xxxxxxxxxxx
>>>> https://dev.eclipse.org/mailman/listinfo/equinox-dev
JM>>   


JM>>  -- 
JM>>  Peter Kriens            
                 Mob +33633746480
JM>>  9C, Avenue St. Drézéry          
         Tel +33467542167
JM>>  34160 Beaulieu, France          
         Tel +15123514821
JM>>                  
                     
  Tel +33870447986
JM>>  AOL,Yahoo, Skype pkriens          
       ICQ 255570717

JM>>  _______________________________________________
JM>>  equinox-dev mailing list
JM>>  equinox-dev@xxxxxxxxxxx
JM>>  https://dev.eclipse.org/mailman/listinfo/equinox-dev

JM>   


JM>  -- 
JM>  Peter Kriens              
               Mob +33633746480
JM>  9C, Avenue St. Drézéry          
         Tel +33467542167
JM>  34160 Beaulieu, France          
         Tel +15123514821
JM>                    
                     
Tel +33870447986
JM>  AOL,Yahoo, Skype pkriens          
       ICQ 255570717

JM>  _______________________________________________
JM>  equinox-dev mailing list
JM>  equinox-dev@xxxxxxxxxxx
JM>  https://dev.eclipse.org/mailman/listinfo/equinox-dev




-- 
Peter Kriens                  
           Mob +33633746480
9C, Avenue St. Drézéry              
     Tel +33467542167
34160 Beaulieu, France              
     Tel +15123514821
                    
                   Tel
+33870447986
AOL,Yahoo, Skype pkriens              
   ICQ 255570717

_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev
_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev
_______________________________________________
equinox-dev mailing list
equinox-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/equinox-dev