Re: [iot-wg] Why use concierge?

["Carrer, Marco" <marco.carrer@xxxxxxxxxxxx>]

Really interesting email thread. 

My understanding is that FrameworkAdmin is specific to equinox and Pascal was referring to the following admin support for equinox:
http://wiki.eclipse.org/index.php/Equinox_FrameworkAdmin

In Kura, we use Deployment Admin because it is a standard OSGi API to install and update bundles.
Apache Felix provides one implementation of the Deployment Admin that runs fine on Equinox.
We also successfully tested the portability of the solution with other containers and Deployment Admin implementations.

The limit of the Deployment Admin is that it requires packaging all the bundles and their dependencies in a single deployment package which is installed in a single transaction.
What Deployment Admin lacks is an external repository and dependency management framework like Felix OBR, p2 or maven.
It would certainly be interesting to explore how to combine these two technologies in a single solution.

>>>>  If you are talking about OBR as found in Felix, know that it is not more standard than p2 is.... What has been standardized a couple years ago is a repository format, a service to access repositories and an API to perform dependency resolution. This is far from solving the problem of downloading bundles and then installing them in a transactional way in your runtime.

Pascal, can you please provide few more details on the standard repository format you are referring to?

Thanks.
-Marco



On May 13, 2014, at 9:39 PM, Kai Kreuzer <kai@xxxxxxxxxxx> wrote:

> Hi Pascal,
> 
> Thanks a lot for the insight, I have to admit that I am not yet in all the technical details of p2. But as it sounds, it should not be too hard to have a lightweight version of p2 available for Concierge.
> 
>> - Provide an implementation of the frameworkadmin service to deal with concierge.
> 
> As far as I know, the framework admin is a core API that every compliant OSGi fw has to bring, right? So imho there should be nothing to do. @Jan, can you confirm?
> 
>> - Provide a touchpoint implementation specific for concierge. 
> Well, I am not familiar with these touchpoint implementations. Is this something that requires deeper access to the framework than the framework admin offers and thus has to be framework specific? Where would such code belong? To Equinox/p2 or to the Concierge project?
> 
>> An even lighter way would be to just use simpleconfigurator
> I wonder if this could be also an option for Kura? As far as I know, Kura currently uses the Felix deployment admin service, which is imho also not necessarily the most flexible solution for remote provisioning. @Wes, Marco, how do you see the current state of provisioning alternatives and are there any plans to move away from Felix in Kura?
> 
> Best regards,
> Kai
> 
> 
> Am 12 May 2014 um 20:58 schrieb Pascal Rapicault <pascal@xxxxxxxxxxxxx>:
> 
>> Hi Kai,
>> 
>> p2 does not offer any support for Concierge. That said I'm attaching a target definition called engine.target that contains the bundles required to just get the engine going on Equinox (note that you will need to wait for tonight's I build to be available because when trying to set this up I realized that there was a missing optional flag on an import package (http://git.eclipse.org/c/equinox/rt.equinox.p2.git/commit/?id=445e1dbcd4a4fa72662d0455c2b4e325b8037a67)). Trying to use this target definition against Kepler would work but it would work but would bring sat4j and the director (but they would not be used at runtime).
>> 
>> An even lighter way would be to just use simpleconfigurator (and the manipulator) to read and write the bundles.info file (see configurator.target). However in this case, it would be your code responsibility to download the jars, store them somewhere, and finally update the bundles.info. So now you may ask why bother with bundles.info? Because you need a way to do the install and update without disturbing the running system, and these 2 bundles, by letting the framework run undisturbed provide just that. And btw, iirc, these two bundles would run on concierge unmodified.
>> 
>> Now how to use the engine on concierge?
>> There are two ways:
>> - Provide an implementation of the frameworkadmin service to deal with concierge. This service is abstracting behind an interface the format on disk of the various configuration files that are specific to a fwk. For example frameworkadmin.equinox takes care of the details of reading and writing the various files dealt with by equinox (config.ini, eclipse.ini, bundles.info (through the simpleconfigurator bundles))
>> - Provide a touchpoint implementation specific for concierge. In this case you create a custom touchpoint that only cover your needs and can do everything you need :) If you wanted to be able to reuse the metadata published into the p2 repositories (to be specific, the touchpoing actions like setStartLevel, installBundle, etc.) you would need to provide actions that cater for this but this is not complicated.
>> In any even, some code would have to be written
>> 
>> HTH
>> 
>> Pascal
>> 
>> On 10/05/2014 3:23 PM, Kai Kreuzer wrote:
>>> Hi Pascal,
>>> 
>>>>   For small devices, you most likely don't want p2 dependency resolution (or any other) to run on the device because it will likely be too much computation. However if you still want a transactional installation you could go by using the p2 engine to perform the download and the installation in the runtime. Then how the engine gets the list of IU to install / uninstall / update is an implementation detail :) For example you could use the OMA DM from OSGi (not sure how it would work with the LWM2M server), or implement something custom where the device contacts a server, the server does the resolution and returns the list of things to install, or use remote service, ... I'm sure there are many other ways to do this and these are just things that I know could be made to work.
>>> As Jens, I am also looking for a very lightweight update mechanism for Eclipse SmartHome. What you describe sounds like a good option - moving the dependency resolution to the server and just sending the runtime the list of things to install.
>>> What is the minimal set of bundles required for the plain p2 engine to run on e.g. Concierge?
>>> 
>>> Best regards,
>>> Kai
>>> 
>>> Am 10 May 2014 um 19:21 schrieb Pascal Rapicault <pascal@xxxxxxxxxxxxx>:
>>> 
>>>> HI Jens,
>>>> 
>>>> Sorry for the late reply. Find below more details
>>>> On 29/04/2014 3:58 AM, Jens Reimann wrote:
>>>>> Hello Pascal,
>>>>> 
>>>>> Thanks for your answers.
>>>>> 
>>>>> We already start Eclipse without the native launcher in most cases.
>>>>> Wrapped by a Python script to find the correct jar. However you
>>>>> sometimes get some native bundles in there. I am not sure if this is
>>>>> Equinox or the Eclipse Runtime though.
>>>>    The eclipse.exe companion jar comes to mind (e.g. equinox.launcher_win32.win32) and it may be brought in by some features you depend on but equinox itself has no native dependencies.
>>>> 
>>>>> For P2 I find it rather hard to find god starting points for alternative
>>>>> solutions. I think it does what it is intended to do, and it does it
>>>>> good. But you pretty soon get lost when it comes to alternative
>>>>> launching and provisioning ways (beside the Eclipse IDE). I don't blame
>>>>> anyone, I know how hard it is to write documentation ;-)
>>>>   Some of these other starting points are not documented because they never got beyond the prototype stage a few years ago because nobody ended up need them.
>>>> 
>>>>> Also is the whole Eclipse way a bit more powerful than plain OSGi. As
>>>>> far as I know OSGi proposes OBR for provisioning. However I did find no
>>>>> support for that in Equinox or P2. I think it was Felix that allowed
>>>>> installing bundles directly from Maven repositories, which would also be
>>>>> an alternative way.
>>>>      If you are talking about OBR as found in Felix, know that it is not more standard than p2 is.... What has been standardized a couple years ago is a repository format, a service to access repositories and an API to perform dependency resolution. This is far from solving the problem of downloading bundles and then installing them in a transactional way in your runtime.
>>>> 
>>>>> I also have to admit that I am not sure what the best way actually is
>>>>> for small devices. On the one hand you want to be flexible and only
>>>>> install the bundles that you need. It may be nice to do this manually
>>>>> ("clicking" together your system with dependencies). On the other hand
>>>>> if you have 100s of devices and every device is special you don't want
>>>>> to maintain this manually.
>>>>   For small devices, you most likely don't want p2 dependency resolution (or any other) to run on the device because it will likely be too much computation. However if you still want a transactional installation you could go by using the p2 engine to perform the download and the installation in the runtime. Then how the engine gets the list of IU to install / uninstall / update is an implementation detail :) For example you could use the OMA DM from OSGi (not sure how it would work with the LWM2M server), or implement something custom where the device contacts a server, the server does the resolution and returns the list of things to install, or use remote service, ... I'm sure there are many other ways to do this and these are just things that I know could be made to work.
>>>> 
>>>> HTH
>>>> 
>>>> Pascal
>>>>> But I guess all this should be subject to a different post :)
>>>>> 
>>>>> And yes "more independence from Equinox in your application" correct.
>>>>> 
>>>>> Jens
>>>>> 
>>>>> On 04/27/2014 04:17 AM, Pascal Rapicault wrote:
>>>>>> Hi Jens
>>>>>> 
>>>>>> Answer embedded.
>>>>>> 
>>>>>> On 24/04/2014 4:52 AM, Jens Reimann wrote:
>>>>>>> Hello Pascal,
>>>>>>> 
>>>>>>> I don't want to sound negative, but from the viewpoint of Eclipse SCADA
>>>>>>> we are pretty happy with Equinox ;-)
>>>>>>> 
>>>>>>> But we also learned, the hard way, that Equinox seems to make a lot of
>>>>>>> "special things". That might be easier at first, but creates problems
>>>>>>> when other OSGi containers become a topic. At the moment we are not able
>>>>>>> to deploy to other OSGi containers since we need P2 and EMF
>>>>>>> 
>>>>>>> For EMF we really do need EMF. But for P2, we only require a some sort
>>>>>>> of build -> provision toolchain. Which is Maven Tycho (p2) -> P2
>>>>>>> Director at the moment.
>>>>>>> 
>>>>>>> Our long term goal is to also have other, more pure, OSGi containers.
>>>>>>> Also not needing any native code in order to launch the framework. And
>>>>>>> for this we would like to use technology coming from the Eclipse
>>>>>>> eco-system.
>>>>>>    The framework can be started w/o the native launcher. For example
>>>>>> you can start the Eclipse IDE by running java -jar
>>>>>> plugins/org.eclipse.equinox.launcher_....jar and you can also directly
>>>>>> start equinox itself in a similar way (java -jar
>>>>>> org.eclipse.osgi_...jar). You can also see code embedding Equinox in
>>>>>> Tycho.
>>>>>> 
>>>>>>> For this we would need Declarative Services (DS) and some sort of
>>>>>>> provision mechanism. P2 is a bit bulky, but works fine for our case. We
>>>>>>> do use the P2 director to assemble our applications (depending from the
>>>>>>> System Configuration the user created). This done on either the
>>>>>>> configuration machine or the target machine. There is no "online
>>>>>>> update". So if we have a system configuration with 10 servers and each
>>>>>>> server has 2 different applications, all applications are provisioned
>>>>>>> using the P2 director from the initial set of P2 repositories.
>>>>>>    It is true that p2 is a bit bulky and is currently known to only
>>>>>> run on Equinox. However p2 has been designed to run and provision
>>>>>> other frameworks. In fact back when we started we were able to
>>>>>> provision Felix. At this point I don't know exactly what it would take
>>>>>> to make that work again, but I think this is in the real of
>>>>>> possibilities.
>>>>>>    The other thing you may be interested in is in knowing that there
>>>>>> has been designs and prototypes done in p2 to make it work "after the
>>>>>> fact". Meaning that you would not need to have the p2 folder and all
>>>>>> that in your application. Instead p2 would be able to reason about the
>>>>>> currently installed bundle and take it from there to perform the
>>>>>> installation (still bringing transactionality and dependency resolution).
>>>>>> 
>>>>>>> However we also see a shift at the moment from this deployment scheme,
>>>>>>> to a more distributed scheme. In the the past we had two big servers,
>>>>>>> and that's it. At the moment systems go up to 20 different nodes which
>>>>>>> are (compared to then) less powerful. And we now start to have ARM based
>>>>>>> devices which will have even less performance and greater numbers.
>>>>>>> 
>>>>>>> So in the end the three most important reasons for us to move to a
>>>>>>> different OSGi container are:
>>>>>>> -) Better provisioning
>>>>>>    Maybe a topic of the p2-dev mailing list, but I would be
>>>>>> interested in knowing what are the criteria for a better provisioning.
>>>>>> 
>>>>>>> -) Less (or better no) native dependencies (launcher)
>>>>>>    See note above.
>>>>>>> -) Better compatibility with OSGi
>>>>>>    I m assume you mean more independence from Equinox in your
>>>>>> application, or do you have something else in mind
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Pascal
>>>>>> 
>>>>>>> Jens
>>>>>>> 
>>>>>>> On 04/23/2014 06:19 PM, Pascal Rapicault wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> Since Concierge has been released, I noticed a lot of interest from
>>>>>>>> the variou IoT projects to move to concierge from Equinox and I would
>>>>>>>> like to understand the motivation of such a move? I can obviously get
>>>>>>>> the size argument, but is there more to it?
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Pascal
>>>>>>>> _______________________________________________
>>>>>>>> iot-wg mailing list
>>>>>>>> iot-wg@xxxxxxxxxxx
>>>>>>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>>>>>>> _______________________________________________
>>>>>>> iot-wg mailing list
>>>>>>> iot-wg@xxxxxxxxxxx
>>>>>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>>>>>> _______________________________________________
>>>>>> iot-wg mailing list
>>>>>> iot-wg@xxxxxxxxxxx
>>>>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>>>>> _______________________________________________
>>>>> iot-wg mailing list
>>>>> iot-wg@xxxxxxxxxxx
>>>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>>>> _______________________________________________
>>>> iot-wg mailing list
>>>> iot-wg@xxxxxxxxxxx
>>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>>> _______________________________________________
>>> iot-wg mailing list
>>> iot-wg@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/iot-wg
>> 
>> <configurator.target><engine.target>_______________________________________________
>> iot-wg mailing list
>> iot-wg@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/iot-wg
> 
> _______________________________________________
> iot-wg mailing list
> iot-wg@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/iot-wg