NuPattern.Extensibility ?

I cant beleive I dont know this by now, or maybe I have lost sight of it, but I think its worth challenging even at this late stage.

What is the purpose of the 'NuPattern.Extensibility' assembly?

It seems that 'Runtime.Interfaces' is the lowest common denominator assembly containing types that are used by all other runtime, library and authoring assemblies, including extensibility.

I know at some point in teh near future we will have an assembly called 'NuPattern.VisualStudio' that will contain all the VS extensibility stuff, that will then be the lowest common denomiator assembly (i.e. referenced by all others).

I just wonder if we need to refactor NuPattern.Extensibility and move some of the types out of there into NuPattern.VisualStudio and NuPattern.Runtime.Interfaces ??

OK, I finally got around to addressing this directly.
As a pre-step to integration of the FERT stuff, I decided to take some initial pre-work steps and refactor and re-establish some order back into the runtime code, and fix some of the technical debt we have evidently carried for a long time. I know, no one ever
wants to do this kind of work, but it needed to get doen at some point, and now we have some time to address it in preparation for future contributors.

At this point in the history of the project we have learned some valuable things about how the runtime has been developed to date, and more importantly how it is consumed by toolkits and toolkit authors:

We need to start separating and abstracting all the VS specific stuff to a separate assembly, and take that initial step in the direction to providing an abstraction layer around VS. (a) primarily because we need this to have any hope of deploying a single
VSIX that works with any version of VS in the future, and (b) if potentially we are looking at supporting other IDE's in the distant future (i.e. MonoDevelop, Eclipse etc?).

We have many extension methods to foundational .NET classes that are scattered all over the code base (as they were needed), and now need factoring out into a foundational assembly for consistent reuse. Prior to this we were file sharing these classes,
which felt like avoiding the inevitable.

We know that toolkit authors get confused about which assemblies from the platform (read the 'Runtime' solution as being the platform here) to reference in their toolkits to reuse the various services, utilities, extension methods and declarations. Today
they reference Runtime, Runtime.Interfaces, Extensibility, Library (and probably others illegally. All these assemblies use different namespaces, and interfaces and extension methods are scattered all over the place. Hence the confusion, with lack of guidance
and boundaries in what they should be using and where. In short, we need to make this more discoverable and clearer for toolkit authors. AND we need to make the codebase more understandable (and maintainable) for new contributors.

We know that over the course of the last 4-5 years as the codebase has evolved and priorities have changed, many shortcuts may have been taken and tech debt accrued. (i.e. the overuse of InternalsVisibleTo is evidence of this). Understandably in any mid
size project. A milestone review and analysis of the codebase is overdue to ensure that we re-align the intention of the codebase, and establish some guidelines for future contributors to follow to maintain the necessary boundaries, layers and abstractions,
as they face understanding of this software. Basically, intentions and boundaries should be much clearer in the codebase.

And that's just the stuff I can think of on the top of my head for now, there is more of course. I am just brain storming at this point.

As I actually got into this exercise, and tried to start defining some clear rules and guidelines about access, and layers etc, alot of side-issues jumped out at me, Like:

I really need to establish some core guidelines about where new types land in code base (for future contributors), based upon a set of objective rules and decisisons, rather than just landing it where it is convienient. I started this document as a flow
chart of key decisions that end up telling you what assebmly and what access your code should have when it lands, but this is taking time to refine, as I pour into the information I am learning through this exercise.

We needed to move all implementation out of Runtime.Interfaces to Extensibility or Runtime depending on consumer, that obvious, but there were many exceptions made historically, why? Fortunately, resolving these has been reletively painless, but understanding
the good reasons for these exceptions should have been clearer from the code.

Where to define extension methods to runtime types that are expected to be shared with toolkit builders? We should have one assembly that toolkit builders reference for all runtime implementation, and it aint Runtime.dll. Currently, I have always understood
(perhaps poorly) that Extensibility.dll was the intended place to share this stuff, but then it had internal stuff and interfaces scattered within it also? That was easy to resolve, but still left me puzzelled about how other contributors have viewed this
beast, and I get the sense that I am not the only core contributor who is confused about the original intention of this assembly. Hence the thread at the top.

If Extensibility.dll is the assembly with all the shared runtime implementation stuff in it, and Runtime.dll is the assembly with all the internal runtime stuff in it, shouldn't we rename Runtime.dll to Runtime.Core.dll (or somethign like that)? Afterall
it should only contain the core engine stuff that should all be internal to it. No other assembly (except Runtime.Shell which necessarily packages and initializes it) should access this internal stuff at all - ever?

Shouldn't Exensibility be renamed to Runtime.Extensibility.dll, and shouldn't we get rid of the NuPattern.Extensibility namespace altogether? Why does this namesapce exist at all? All the types in here are implemntations of runtime types or runtime utility
types and extension methods anyway. Why the extra namespace confusion for consumes like toolkit authors? Why not just put everyting in this assembly in structures namespaces under NuPattern.Runtime? That si show they will get discovered easier by toolkit authors.

IIRC, the Extensibility assembly is one that we use internally in the runtime as well as the design-time, but we wanted to keep public as it would be useful to toolkit authors looking to extend the runtime (or design-time, via new types of bindings) in
a deep way. I don't think by default toolkit authors should be referencing that assembly directly. It's just like the "SDK" for the runtime. Just like users install just xunit.dll but they can also use xunit.sdk.dll if they want to extend the runner
or do some more deep extensions to the unit testing framework (like theories and the like).

Runtime.dll should never be referenced by anyone, and Runtime.Interfaces should probably not be called "Interfaces" and instead just Runtime.dll. And the "engine" should be Runtime.Engine? Placing extension method implementations on the
"interfaces" assembly makes me uncomfortable, but it's kinda unavoidable. Ideally, you'd like to be able to ship updates to those implementations via runtime updates, without having to worry about toolkits having references to the actual implementations,
but rather just the interfaces...

I agree on the namespace unification, it doesn't help to have an extra "SDK/Extensibility" namespace.

In the refactor so far, I have discovered and sort of enforced the following guidelines:

Public, runtime types (i.e. interfaces, enums, constants etc) go into Runtime.Interfaces.dll. This assembly contains no implementation (i.e. no concrete classes, no extension methods, no exported services etc.). It also has no dependencoes on any other
assembly. This assembly is intended to be shared by everyone: runtime, library, authoring and toolkits.

Public runtime types (i.e. extension methods, concrete classes, exported services etc.) go into Extensibility. This assembly has all public implementation and extensibility 'nature' to runtime, and little 'internal' implementation (except the exported types
that it implements and exports to MEF). Almost all public extension methods used outside the runtime assemblies are contained in this assembly. This assembly is intended to be shared by everyone: runtime, library, authoring and toolkits.

Runtime.dll contains only internal implementation. With a rare exception case of types that have to be public to be imported by MEF (not a constraint of MEF, but an apparent constraint of the importing type). This is not intended to be referenced by anything
except Runtime.Shell, that MEF/VSX registers, initializes and configures it at runtime.

Runtime.Store.dll and Runtime.Schema.dll are entirely internal, and in some expected cases permits access of its internals by Runtime.dll. These assemblies are not intended to be referenced by anything except Runtime.Shell, that MEF/VSX registers, initializes
and configures them at runtime.

All VS related stuff (i.e. ISolution, all service abstractions, helpers, extension methods, etc) is contained in VisualStudio.dll. It is intended to be shared by everyone: runtime, library, authoring and toolkits.

All extensions and helpers to basic .NET types are contained in Common.dll. This assembly is intended to be shared by everyone: runtime, library, authoring and toolkits.

We also have a couple of other ancillary assemblies (i.e. Common.Presentation.dll and Extensibility.Serialization.dll). These assemblies are intended to be shared by everyone: runtime, library, authoring and toolkits.

Given your affirmation above, I propose the following further changes:

I think the decision on whether to have Runtime.Interfaces.dll or Runtime.dll depends on one remaining aspect now. Extension methods in the interfaces.dll or not?
Do you see extension methods as part of the declaration or definition of a type (i.e. interface, base class etc)?

To me, with my toolkit author hat on: extension methods are helpers. Handy bits fo functionality; utilities. When I 'see' the interfaces, I get these helpers also?

The extensions should not really add new features to a type per-se, they just make using the type easier and cut down on code maintenance. But that is not really an enforceable distinction to make or control. They are also very much like (duck-type like) methods,
and can have implementation details in them. This is their duality in how we 'see' their intention that I am addressing now. (not whether or not they actualy are methods - of course they are).

If they are only helpers, then why not have them in the interfaces.dll?
As a consumer of these assemblies, if I reference the Runtime.Interfaces.dll shouldn't be able to use the extensions methods that make it easier to use the types defined in the interfaces? Should I be required to add another assembly reference to get the extension
methods?

I can't quite resolve what is the best (most usable, but still functional) thing to do here. Clearly, I want to improve usability of the API, but not at risk of the Interfaces.dll being poluted with implementation again.

Fair enough, so the distinction between Runtime.dll (Runtime.Interfaces.dll) and Runtime.Extensibility.dll would need to be the intention of the implementation within it.
Which I am finding hard to distinguish myself.

What guidelines would we have, that a new contributor can use, to decide which of these assemblies to put new stuff into?
How do we distinguish stuff for the runtime (as a whole) versus stuff for extensibility?

Are we saying that all interface, enums, constant definitions, and only some minimal implementation (read: extensions) goes in Runtime.dll (Runtime.Interfaces.dll), and only stuff to be shared with Library, Authoring and all Toolkits goes in Runtime.Extensibility?

Here is what I have come up with from the ongoing experience of actually going through the refactoring exercise as we have been discussing this.

Lets define the 'Runtime.Internal' components as being the Runtime.Core, Runtime.Schema, Runtime.Store assemblies. These assemblies need to collaborate together, but they expose no public signatures. They are not referenced directly by any other assemblies
except Runtime.Shell. Runtime.Shell is simply the packaging container that registers and initializes them with VS and MEF.

Lets consolidate Runtime.Interfaces and Extensibility assemblies into one assembly called 'Runtime.Extensibility'. This assembly exposes all public runtime types and implementations that are consumed by Runtime.Internals, Library, Authoring and all Toolkits.
Namespaces are rooted at NuPattern.Runtime.* and break out into the logical areas of the services that runtime provides.

NuPattern.VisualStudio contains all types, interfaces, extensions and helpers for anything related to VS. Namespaces are rooted at NuPattern.VisualStudio.* and break out into the areas of abstraction of an (ideal) IDE. (a little forward thinking to other
IDE's in the distant future)

NuPattern.Modeling.dll contains extensions for anything related to the DSL Tools and related types. Namespaces are rooted at NuPattern.VisualStudio.Modeling.*. As a note it became evident to move this stuff into a separate assembly to avoid Authoring and
all Toolkits having to depend on the DSL Tools, which is highly undesirable, and unecessary.

In terms of the distinction between runtime declarations and types that should go into Runtime.Core.dll versus types that should go into Runtime.Extensibility.dll, I think the guidance should include these determinations:

Types should initially be declared as 'Internal' in Runtime.Core, and therefore become accessible to all assemblies in the Runtime.Internals group. This is the lowest level in the architecture.

If a type is required to be consumed by either Library, Authoring or other Toolkits, then it can be promoted into Runtime.Extensibility, and be exposed as 'Public'.

If the type is an interface that requires a concrete implementation that is solely used as a feature or function of the Runtime.Core assembly, then it is declared as 'Internal' in the Runtime.Core assembly, Otherwise delcared as 'Internal' in the Runtime.Extensibility.dll
assembly, or 'Public' in the Runtime.Extensibility.dll assembly if consumers are required to take a direct reference to the type (i.e. base class).

Most of this work is either complete or underway now, and any exceptions to this set of rules, I will post here. Of course, I will create a wiki page with all of these finding and guidance to future contribs once the refactor is complete.

Thanks for you involvement in the discusssion. This has been a tricky one to figure out on the outset, but nonetheless very necessary to ensure that we have some solid guidance for new contributors in the future.