Taking a look at how the deployers are implemented today, I figured I should start by creating a non-flat version of WeldDiscoveryEnvironment first, so we could have a new environment input fot the new non-flat Deployment implementation.

There a few assumptions I'm making right now that I would like to validate:

- the deployment continues to exist per top level only, the only difference we are going to have is in the BeanDeploymentArchives (BDAs)

- if Deployment.loadBeanDeploymentArchive requests for a non-existing BDA, I should search for all archives in the deployment that are not currently BDAs. If one archive containing the requested class is found, I create a BDA for it and return it.

If the assumptions above are correct, the main question I have right now regards two different deployments that can see each other. So, in the scenario described by the Deployment interface javadoc example, whereas we have ejb-jar A and ear B. This is going to be translated in two different deployments, each one containg one or more BDAs (in the case of the ejb-jar, one BDA only). Since those BDAs can see each other, there must be a way for me to reach Deployment A BDAs from Deployment B BDAs. Is there some SPI in the current implementation of weld integration that supports that?

Pete and I had a face-to-face meeting where we discussed all my questions and what would be the best way to implement this.

Regarding the assumptions I wanted to validate, both are correct.

But, regarding visibility of deployments. He told me that the best approach is to mirror the ClassLoader structure. The point is that one BDA A can see another BDA B if the module it represents, say A', can see classes from the module B', that corresponds to BDA B.

So, instead of reinventing the wheel, we should use the already existent ClassLoader structure to implement this correspondence in the BDA structure. These are the steps that Pete suggested:

1) Find Modules which are BDA's

2) List <Module>, List<ClassLoader>, Map<ClassLoader, Module>

3) Iterate over classes and build graph of modules

4) Module -> BDA

a) Explore CL graph & for each CL & Module

b) Convert

A few examples of complex things that we would have to implement if we don't use the ClassLoader structure:

- we would have to able to see if a deployment is scoped or not, so we can define the visibility it has from other modules and to which modules it is visible

- we need to keep track of modules being deployed and undeployed, so to know when it becomes visible and when it becomes invisible

- we need to support Class-path attributes in manifest files

- we need to define when a previously defined archive is available to a archive being deployed now

So, these features that we need to support provide the arguments as to why we should mirror the ClassLoader structure instead of implementing this from scratch.

Pete also pointed out an important behaviour that should be fixed. In a nutshell, all archives visible to a BDA are scanned. In the given scenario:

ejb.jar A reaches lib2. In lib2 we have a ClassPath reference in the manifest to lib3, which is not in the classpath of the server... it is somewhere in the filesystem. In this case, for what I understood from Pete's explanations, lib3 is going to be scanned for CDI stuff. But this shouldn't happen, so we would also need a way of differentiating lib3 from lib2. I'll stop my explanation here, as I think Pete can fill in the blanks and explain this much better :-)

Anyway, this would be a future step, as the priority is to first mirror the CL structure effectively.

So, Ales, how do you think this should be done? I think we have plenty of code in the ClassPools that do exactly that, mirror the ClassLoader structure in a way that allows us to find the classes and xml files in META-INF dirs. Maybe there is a way of reusing part of the ClassPool code to do this?

So, these features that we need to support provide the arguments as to why we should mirror the ClassLoader structure instead of implementing this from scratch.

Yeah, re-inventing this would be a nightmare. ;-)

Pete also pointed out an important behaviour that should be fixed. In a nutshell, all archives visible to a BDA are scanned. In the given scenario:

ejb.jar A reaches lib2. In lib2 we have a ClassPath reference in the manifest to lib3, which is not in the classpath of the server... it is somewhere in the filesystem. In this case, for what I understood from Pete's explanations, lib3 is going to be scanned for CDI stuff. But this shouldn't happen, so we would also need a way of differentiating lib3 from lib2. I'll stop my explanation here, as I think Pete can fill in the blanks and explain this much better :-)

I guess this is the discussion we already had once: actual Weld deployment vs. static "library"?

So, Ales, how do you think this should be done? I think we have plenty of code in the ClassPools that do exactly that, mirror the ClassLoader structure in a way that allows us to find the classes and xml files in META-INF dirs. Maybe there is a way of reusing part of the ClassPool code to do this?

It is similar in a way, but I don't see how you would be able to re-use it.

In new CL layer it's one Module per ClassLoader and vice versa.

Each Module / ClassLoader belongs to some Domain.

So, we would have to keep track of Modules and how are they wired hierarchicaly and grouped by Domain.

A Module can see any ancestor Module and all Modules in its Domain and ancestor Domains.

I would differentiate between application's explicit Modules and what it can reach,

hence you don't scan yourself too deep; e.g. into default Domain.

This way it should be also easy to support a notin if libraries as long as they are deployed the same way / over new CL layer.

So, Ales, how do you think this should be done? I think we have plenty of code in the ClassPools that do exactly that, mirror the ClassLoader structure in a way that allows us to find the classes and xml files in META-INF dirs. Maybe there is a way of reusing part of the ClassPool code to do this?

It is similar in a way, but I don't see how you would be able to re-use it.

You are right. The point is that ClassPools contain the algorithm for finding Classes and other resources in the classpath. This means we need to take care of imports and exports and other complex stuff.

For WELDINT-1, all we need to do is to see the module hierarchy as you suggested, as that allows us to map the modules to BDAs and to yet see which BDAs are visible to a specific BDA.

The only thing that I see is duplicate here is RegisterModuleCallback/DomainRegistry. We need to keep track of modules and to map those to ClassLoaders. I'm duplicating this stuff for now, but we should definetly review this in the future.

You are right. The point is that ClassPools contain the algorithm for finding Classes and other resources in the classpath. This means we need to take care of imports and exports and other complex stuff.

Thinking about it a bit, I'm not sure we actually need that complex strucutre in Classpools.

I don't see why simple domain mapping wouldn't work -- which would super speed up things.

I need to think about it a bit more, and have a more detailed look.

Or perhaps Kabir can do this, if we cannot get current Javassist based Reflect to behave faster as Introspection. :-)

The only thing that I see is duplicate here is RegisterModuleCallback/DomainRegistry. We need to keep track of modules and to map those to ClassLoaders. I'm duplicating this stuff for now, but we should definetly review this in the future.

No, you don't need that registry module callback. It should all be possible to do in deployers.

And at the same time you are able to get access to other things via DeploymentUnit.

Pete also pointed out an important behaviour that should be fixed. In a nutshell, all archives visible to a BDA are scanned. In the given scenario:

ejb.jar A reaches lib2. In lib2 we have a ClassPath reference in the manifest to lib3, which is not in the classpath of the server... it is somewhere in the filesystem. In this case, for what I understood from Pete's explanations, lib3 is going to be scanned for CDI stuff. But this shouldn't happen, so we would also need a way of differentiating lib3 from lib2. I'll stop my explanation here, as I think Pete can fill in the blanks and explain this much better :-)

Something has gone wrong here, in this case lib3 should be scanned for CDI stuff.

The other thing that Flavia and I discussed, probably as a later step, is that we should be able to cache the scanned BDA for a scanned archive - this makes a lot of sense for things like installed libraries. Though, with the caching we get from MC of the real scanning work, perhaps it won't have any impact...

Pete also pointed out an important behaviour that should be fixed. In a nutshell, all archives visible to a BDA are scanned. In the given scenario:

ejb.jar A reaches lib2. In lib2 we have a ClassPath reference in the manifest to lib3, which is not in the classpath of the server... it is somewhere in the filesystem. In this case, for what I understood from Pete's explanations, lib3 is going to be scanned for CDI stuff. But this shouldn't happen, so we would also need a way of differentiating lib3 from lib2. I'll stop my explanation here, as I think Pete can fill in the blanks and explain this much better :-)

Something has gone wrong here, in this case lib3 should be scanned for CDI stuff.

Now I see. So, this is someting that is not currently implemented and that should be implemented, right?

I have a few questions:

- regarding jars in the lib of the server, they should never be treated as BDAs, right? (I know we discussed about this, but I'm so deep in so many details that now I got confused :-) )

- the Deployment.loadBeanDeploymentArchive(Class<?> beanClass) method should search only for the archives included in the deployment, or for all archives visible from the deployment? A small example to ilustrate my question:

if there are two separate ejb-jars deployed, A and B, and I invoke loadBeanDeploymentArchive at the deployment that represents ejb-jar A, using a class that is in ejb-jar B as parameter, should the deployment be able to find it?

- what do I do if loadBeanDeploymentArchive fails to find any archive containing the class, should it return null? I'm assuming yes, but I would like to have an ok here, as the javadoc is not explicit about this

- this one is still about loadBeanDeploymentArchive method, and I think Ales knows the answer. Since the parameter of this method is a class, I don't need to look for an archive that contains a class with the same name. I could use instead the ClassLoader to determine the module and, hence, to find the corresponding archive. Should I do it this way?

- regarding jars in the lib of the server, they should never be treated as BDAs, right? (I know we discussed about this, but I'm so deep in so many details that now I got confused :-) )

No, I think they should -- but, only if they have beans.xml in their metadata location.

- the Deployment.loadBeanDeploymentArchive(Class<?> beanClass) method should search only for the archives included in the deployment, or for all archives visible from the deployment? A small example to ilustrate my question:

if there are two separate ejb-jars deployed, A and B, and I invoke loadBeanDeploymentArchive at the deployment that represents ejb-jar A, using a class that is in ejb-jar B as parameter, should the deployment be able to find it?

I think yes, you should be able to "jump" from deployement to deployment if they can "see" each other.

It would be your (hidden) impl that would easily allow this; e.g. keeping proper overall structure with easy-to-access mapping.

- this one is still about loadBeanDeploymentArchive method, and I think Ales knows the answer. Since the parameter of this method is a class, I don't need to look for an archive that contains a class with the same name. I could use instead the ClassLoader to determine the module and, hence, to find the corresponding archive. Should I do it this way?

Yes, more or less.

It depends on the impl details how you're gonna map the overall structure.

Pete also pointed out an important behaviour that should be fixed. In a nutshell, all archives visible to a BDA are scanned. In the given scenario:

ejb.jar A reaches lib2. In lib2 we have a ClassPath reference in the manifest to lib3, which is not in the classpath of the server... it is somewhere in the filesystem. In this case, for what I understood from Pete's explanations, lib3 is going to be scanned for CDI stuff. But this shouldn't happen, so we would also need a way of differentiating lib3 from lib2. I'll stop my explanation here, as I think Pete can fill in the blanks and explain this much better :-)

Something has gone wrong here, in this case lib3 should be scanned for CDI stuff.

Now I see. So, this is someting that is not currently implemented and that should be implemented, right?

I have a few questions:

- regarding jars in the lib of the server, they should never be treated as BDAs, right? (I know we discussed about this, but I'm so deep in so many details that now I got confused :-) )

They are BDAs if they have a META-INF/beans.xml

- the Deployment.loadBeanDeploymentArchive(Class<?> beanClass) method should search only for the archives included in the deployment, or for all archives visible from the deployment? A small example to ilustrate my question:

if there are two separate ejb-jars deployed, A and B, and I invoke loadBeanDeploymentArchive at the deployment that represents ejb-jar A, using a class that is in ejb-jar B as parameter, should the deployment be able to find it?

All archives visible. In your example it should be able to do it.

- what do I do if loadBeanDeploymentArchive fails to find any archive containing the class, should it return null? I'm assuming yes, but I would like to have an ok here, as the javadoc is not explicit about this

Throw an exception, as this shouldn't happen. (IAE I think). Can you file a WELDINT issue so we update tha javadoc.

- what do I do if loadBeanDeploymentArchive fails to find any archive containing the class, should it return null? I'm assuming yes, but I would like to have an ok here, as the javadoc is not explicit about this

Throw an exception, as this shouldn't happen. (IAE I think). Can you file a WELDINT issue so we update tha javadoc.

Now we finally have something concrete to use in our discussion. So, my draft version of WELDINT-1 is fully documented in order to make it easier for others to understand and to estimulate discussion :-)

Basically, I created an Archive class that maps to a Module/ClassLoader and contains all the info necessary for creating a corresponding BeanDeploymentArchive. An Archive can reach other archives through Classpath, which maps one-to-one to ClassLoader domains and contains a list of Archives/BDAs. A Classpath can internaly point to another classpath, which corresponds to the parent domain relationship. These classes set up the graph structure Pete suggested, which is actually a tree.

At first, when a deployment is created, BDAs are created only for those Archives that contain one or more CDI xml file. After that, as a result of Deployment.loadBeanDeploymentArchive, a BDA may be created by an Archive. All lthe BDA collections are updated accordingly, mainly by the use of a BDALyfecycleListener.

Take a good look at the ClasspathImpl class, as it contains the glue to put together the archives in the tree structure This is a very naïve but simple to understand implementation of the Classpath interface. It returns always a flat collection of all visible BDAs or archives, including the elements of the whole Classpath hierarchy it belongs.

A more clever implementation is NoDuplicateClasspath, which is currently not being used. I just created this class because it would be better to write it than explaining my idea of how to improve ClasspathImpl in the forum. To replace ClasspathImpl by NoDuplicateClasspath, just replace the two ClasspathImpl constructor calls in ClasspathFactory by the corresponding constructors of NoDuplicateClasspath. I think that the intention here is to get rid of ClasspathImpl... still I thought it would be better to keep it there now as NoDuplicateClasspath is just a suggestion for improvement.

Finally, the tests are very poor. I just ported the already existing ones to test the new deployers. So writing real tests is what I'll be doing next, and I'm sure I have quite a few bugs to catch ;-)

At this stage, I decided to not delete the previous deployers, as I'll be waiting for approval and improvement suggestions on this implementation before doing so (unless I should not delete those deployers?).