Re: [cdi-dev] CDI future - problems and proposal

[Matej Novotny <manovotn@xxxxxxxxxx>]

Hi,

comments inline.

Matej

----- Original Message -----
> From: "Graeme Rocher" <graeme.rocher@xxxxxxxxx>
> To: "cdi developer discussions" <cdi-dev@xxxxxxxxxxx>
> Sent: Friday, November 13, 2020 1:39:19 PM
> Subject: Re: [cdi-dev] CDI future - problems and proposal
> 
> 
> 
> On Fri, Nov 13, 2020 at 8:57 AM Matej Novotny < manovotn@xxxxxxxxxx > wrote:
> 
> 
> I am finding this "discovery" concept cumbersome and probably unable to cover
> some of the use cases. But I am probably misunderstanding it, so please bear
> with me. I am assuming there would be a tool generating the content of
> META-INF/something so that user doesn't need to do that manually. So, here
> are some thoughts/questions on it:
> 
> 1) CDI allows you to register new annotations programmatically - qualifiers,
> stereotypes, scopes all of which are bean defining annotations. Then there
> is also interceptor bindings. How does your model allow for these?
> Will you also have to somehow list these? Because you cannot read them from
> annotations - there are no annotations in the code itself. And you will need
> to "discover" beans with those newly registered annotations.
> 
> I answered a similar question here
> https://github.com/Ladicek/quarkus-fork/pull/7#issuecomment-726709349 and
> provided examples of how the Small rye example can be implemented today with
> annotation processing.

Thanks for the pointer!

> 
> 
> 
> 2) CDI extensions have many uses, but one major use case is to be able to
> integrate a non-CDI library into your application and turn some of its
> classes into beans (synthetic or otherwise). That means, you need to include
> a JAR that certainly was not precomputed in any way and still work with it
> in context of CDI. And I am sure there will be lots of those - many java
> libs don't care, need or know about CDI, yet you still want to utilize them.
> How do you achieve that?
> 
> This could be implemented by looking at precompiled code in an annotation
> processor in the same way and could potentially be controlled by the
> presence of bean.xml. We already support this for certain use cases as
> annotation processors can actually visit the ast nodes of compiled byte
> code. It would just be a matter of specifying to the compiler the external
> libraries that have CDI annotations on them.

I think I understand this better now, thanks!

> 
> 
> 
> 3) A follow-up on second question. For any JAR that doesn't come with
> precomputed values I suppose you will want to (somehow) additionally process
> this to generate the metadata. However, that means additional processing at
> build time, so you have just lost all the extra speed you had by not having
> to process it in the first place. Not to mention the app you are developing
> will have to recomputed every time you build it to accomodate for changes.
> You might as well read bytecode or do anything else to create the static
> "archive knowledge" at build time and go from there. To me, this makes your
> approach look like a good implementation-specific way to further optimize
> build time if users choose to, but a poor choice on specification level.
> 
> Furthermore, I have two concerns about the pre-computation tool.
> 
> I fail to understand why you would want to recompute the DI metadata, seems a
> pretty niche requirement, however if metadata really needs to be recomputed
> it should be done as per above with some kind of compilation time scanning
> configurable on a per extension basis.

Sorry, I wasn't clear. I meant that for the app you develop, you will need to always re-generate that information because of possible additions of beans (just like you would need to, for instance, re-generate Jandex index).
For 3rd party library, you theoretically only need to do it once and keep that information in between builds.

> 
> The reason for favouring precomputing is that you don't have to rerun the
> computation again in downstream consumer code. Imagine a project that has
> 1000s of beans and potentially millions of lines of code. Whichever build
> time implementation has to think about things like compiler speed,
> incremental compilation, compiler performance etc. One aerospace user of
> Micronaut has millions of lines of code and reprocessing all the code in the
> world in each compile is just not viable. It may be ok for hello world or
> tiny microservices but not for real world usage.

I get your point.
Although I should say that lots of containerized microsevices are pretty small. Not hello-world-small but not 1000s of beans either.
So I think for majority of cases, we'll be shooting somewhere in between.

> 
> 
> 
> a) With specification requiring such thing, CDI would have to provide and
> upkeep this tool. Possibly in several forms because I can imagine it being a
> maven plugin, but then you might also need something for gradle and who
> knows what else.
> That's a lot of extra work not to mention extra explanation on user side to
> make it clearer what are they supposed to do and with what tooling in their
> given environment.
> 
> If the tool is an annotation processor (a Java standard) and the metadata
> format standardized then no additional tooling is needed since all build
> tools and IDEs understand annotation processors already. Now if the tool
> uses something else that is non-standard then additional build tool plugins
> will clearly need to be maintained by the provider (in your case Quarkus).

I am still failing to see it. Standardized or not, you need to have some tool that users can have to run their libs through so that it would contain such metadata in a format CDI specifies.
What I can imagine is for instance a plugin doing this during build that spits out a JAR already containing this information, or a standalone tool that takes JAR as input and generates this information for you additionally.
To me that (probably wrongly) implies the need to have such app/tool/plugin on CDI side?

In my opinion, CDI should keep an implementation agnostic definition of how you get to Lite. E.g. IMO it shouldn't require users to provide neither jandex index, nor annotation processor metadata.
If your implementation chooses the way you described, that's perfectly fine. What we need to align on, is whether, for instance, beans.xml works as a marker file for bean archive, what beans will be discovered or what capabitilies can extensions have so that both parties can implement it. WDYT?

> 
> 
> 
> b) What means would this tool use to build up the metadata information you
> are after? Due to the ability to register bits and pieces programmatically,
> it looks like just annotation processor won't be able to do that anyway...
> 
> Yes it can. See
> https://github.com/Ladicek/quarkus-fork/pull/7#issuecomment-726709349

I see, thanks.

> 
> 
> 
> 
> Regards
> Matej
> 
> ----- Original Message -----
> > From: "Graeme Rocher" < graeme.rocher@xxxxxxxxx >
> > To: "cdi developer discussions" < cdi-dev@xxxxxxxxxxx >
> > Sent: Thursday, November 12, 2020 6:24:48 PM
> > Subject: Re: [cdi-dev] CDI future - problems and proposal
> > 
> > SmallRye Fault Tolerance (an implementation) would provide any necessary
> > beans already precomputed in the JAR file. This would save on build times
> > as
> > downstream consumers wouldn't have to recompute anything. If an
> > implementation chose not to include the necessary bean metadata then an
> > implementation could optionally import a library (or we could keep the
> > beans.xml requirement as an indication that this library should be
> > imported).
> > 
> > I don't see the need to add annotations to the mp-ft.jar personally, you
> > could add this binding to the classes themselves of the consumer project.
> > 
> > In Micronaut we have managed to support many different annotation models
> > using this approach such as JAX-RS, Spring MVC etc. and it works fine
> > 
> > 
> > 
> > On Thu, Nov 12, 2020 at 5:47 PM Ladislav Thon < lthon@xxxxxxxxxx > wrote:
> > 
> > 
> > 
> > 
> > 
> > Hi,
> > On 21. 10. 20 17:17, Tomas Langer wrote:
> > 
> > 
> > 
> > From Matej Novotny:
> > - Bean Metadata - there is a need for build-time processing API, but the
> > runtime model I think should be built into the existing model in CDI to
> > make
> > it capable of reflection free handling, with the reflected methods being
> > deprecated. If two models are really needed, we can work on your PR to come
> > up with a model good for both annotation processing and byte code
> > processing. We could have a `jakarta.cdi.lite` model for build time
> > annotation metadata, and have the current CDI meta model extend it to add
> > the reflection methods (that may be used in EE or existing extension and
> > beans).
> > - Extensibility - agreed, the extensibility model should be designed in
> > cooperation, to make sure it works for either build time approach; I will
> > add the link to the document. Luckily there are not three parties, we have
> > written the document with Graeme, so we should be on the same page here
> > - beans.xml - see below, this one is a bit longer; can you please explain
> > how
> > you approach this for build time and runtime in your proposal? Maybe I am
> > missing some use case that requires a different approach
> > - reduction of scope - this is a complicated topic, I can see problems
> > starting with jakarta.inject (Provider should be deprecated and extend
> > Supplier, and Supplier used instead, maybe consider adding NonBinding as it
> > belongs to Qualifier, and I would consider some additional scopes as they
> > make sense in general, such as Dependent and RequestScope), and
> > jakarta.annotation-api (I have doubts about the security, sql package,
> > ManagedBean class, Resource class - seems like a mixture of classes that
> > are
> > from different areas, not all of them have meaning in SE; we should also
> > add
> > Nullable and Nonnul annotations), there are some areas in interceptor API
> > that may be improved, and of course the areas in the CDI spec itself. The
> > question here is whether these should be dropped from the specification
> > (e.g. deprecated and eventually removed), or just not part of a profile, or
> > a combination of both
> > - Extensibility #2:
> > - “Lite” would only implement build-time
> > - SE and EE would support existing portable extension
> > - I think we should aim at providing a capability to “transform” output of
> > a
> > build-time extension into a portable extension; this would not work for
> > synthetic beans created at runtime and for beans added by other portable
> > extensions (as the build time extension would not know about them). I think
> > we should go through a few use cases to see if this would be useful at all,
> > or would just bring confusion
> > - My concern - if we create “Lite” with build time extension (only), and
> > “SE”
> > and “EE” with portable extensions (only), we would end up with two sets of
> > extensions for each feature (such as a “OpenTracingLite” and
> > “OpenTracingExtension”) - this may be a good solution, I am myself not sure
> > about it
> > 
> > Beans.xml and bean discovery:
> > You (Matej) consider class path scanning is needed at build time, we were
> > thinking about class path scanning free approach. Each archive that is a
> > bean archive should have appropriate “META-INF/services” with a service
> > that
> > provides the list of beans in that archive. If done this way, there is no
> > need for beans.xml. This of course implies that each bean archive is
> > compiled with build-time processing. We could have a component, that would
> > add such metadata on existing bean archives (yet I think this can be vendor
> > specific).
> > 
> > How would this work?
> > - build time of a bean archive: the processor (annotation or byte code)
> > generates a META-INF/service/jakarta.cdi.lite.BeanDefinitionProvider with a
> > list of (generated) implementations for each bean in the archive
> > - build time of my application: we process the current application source
> > (or
> > byte code) to create the service list (either via annotation processing or
> > byte code scanning)
> > - run time: we read all the BeanDefinitionProvider instances and create our
> > bean landscape from them (the bean landscape is almost fully pre-computed,
> > may be using some conditions to limit the list of beans being loaded)
> > - in case we want to “enhance” a bean archive that is not our application
> > (such as adding an interceptor to an existing bean), we would use a concept
> > of “bean import”, with a configurable way to mark classes to be re-computed
> > and provide an alternative with higher priority than the original bean
> > 
> > 
> > I wanted to get back to this, as I find it the most important open topic.
> > It's not just about bean discovery, it's also about what extensions can do.
> > 
> > I'll use my favorite example of MicroProfile Fault Tolerance and SmallRye
> > Fault Tolerance, as I'm familiar with it, but the setting is quite general.
> > 
> > So, let's assume there's a MicroProfile Fault Tolerance API in the
> > mp-ft.jar.
> > (That's basically what you can get from Maven Central at
> > org.eclipse.microprofile.fault-tolerance:microprofile-fault-tolerance-api.)
> > This JAR has no beans.xml and no classes with bean defining annotations.
> > 
> > 
> > Further, let's assume there's a SmallRye Fault Tolerance implementation of
> > the previous API in srye-ft.jar. (In reality, it's several JARs at
> > io.smallrye:smallrye-fault-tolerance*, but let's assume it's just one.)
> > This
> > JAR has no beans.xml either, but there are some classes with bean defining
> > annotations, and there's also an extension (in current reality, that's a
> > Portable Extension, but it could also be a Build Compatible Extension).
> > 
> > The extension does a bunch of things, but for the purpose of this
> > discussion,
> > 2 are important: - it takes some classes from mp-ft.jar and "adds" an
> > annotation to them, for the purpose of interceptor binding;
> > - it also registers some classes from srye-ft.jar as beans (srye-ft.jar has
> > no beans.xml, so beans from it are not discovered automatically, but we
> > need
> > them).
> > 
> > 
> > That all works nice in current CDI. It also translates relatively
> > straightforwardly to Quarkus, or to the Build Compatible Extension proposal
> > I created -- which assumes some form of access to the whole world
> > (classpath
> > scanning, Jandex, whatever).
> > 
> > What I'd like to understand is -- how do your constraints apply here?
> > 
> > 
> > Assuming extensions are discovered via ServiceLoader, you would be able to
> > run the extension in srye-ft.jar just fine. But would the extension be able
> > to add the beans from srye-ft.jar to the application? Would it be able to
> > "modify" the classes from mp-ft.jar?
> > 
> > Do you think that mp-ft.jar and srye-ft.jar would require some other
> > build-time processing during _their_ build? (What kind of processing? Who
> > would maintain that tool?) And do you think that would be enough?
> > 
> > 
> > For example, the extension in srye-ft.jar could "install" a custom scope
> > (it
> > doesn't in reality, but it could), which extends the set of bean defining
> > annotations. Would beans with such new bean defining annotation still be
> > discovered?
> > 
> > 
> > I'm sure I'm missing a lot, but it seems to me that we would either have to
> > carefully restrict all things that might possibly require access to the
> > whole world, or... well... require all JARs to carry not just a list of
> > beans, but an index of all their classes and annotations.
> > 
> > LT
> > _______________________________________________
> > cdi-dev mailing list
> > cdi-dev@xxxxxxxxxxx
> > To unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/cdi-dev
> > 
> > 
> > --
> > Graeme Rocher
> > 
> > _______________________________________________
> > cdi-dev mailing list
> > cdi-dev@xxxxxxxxxxx
> > To unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/cdi-dev
> > 
> 
> _______________________________________________
> cdi-dev mailing list
> cdi-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/cdi-dev
> 
> 
> --
> Graeme Rocher
> 
> _______________________________________________
> cdi-dev mailing list
> cdi-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/cdi-dev
>