Equinox Summit 2007 Results

The Equinox Summit 2007 was arranged as a set of lightning talks and a series of breakout sessions. Each session had a number of parallel discussions with self-assigned and self-organizing participants. The groups then reported to a plenary session, summarizing their results and possibly adding more topics to be discussed in further breakouts.

Mandatory: your bundle isn't started until all the mandatory services are available

This only guarantees availability at a moment in time: at the moment of startup all mandatory services are available, but they could disappear after that moment.

If a mandatory service goes away, the services exported by the bundle needing the service is unregistered.

For stateful services, there is a listener model - bind an unbind events occur as service implementations arrive and depart

It would be interesting to be able to declare a default service implementation that would be used whenever no real service is available

Spring and DS require eager activation, but only for service producers. Many bundles are just service consumers, and so they can be lazily started

Requirements for Spring service support: Spring core, Spring AOP, a few other essential Spring core pieces

Service Application Toolkit: IBM toolkit for making it easier to work with OSGi services

How do we reconcile and combine the various component toolkits available: Spring, DS, SAT, extension registry

Spring initializes the app context asynchronously. Typical Eclipse programming model assumes that all initialization has completed by the time any classes are loaded in that bundle.

To be scalable, we need a model where service wiring is persisted across sessions, so we don't have the massive event storm of services being started and bound as the system comes up.

Most bundles are POJO - how do they specify that they need Spring or some other component system to be available at runtime? Ideally 95% of bundles don't know or care about the active component system - but they need some component system to be around at runtime to inject the services they need.

Can we come up with a common syntax where a bundle can specify its service and extension registry imports/exports, and be agnostic about the runtime that will parse it and provide the runtime capabilities.

Scoping: extension registry / services are global - how can I publish extensions/services that are only available in a particular scope* only particular set of bundles have access to it. In J2EE world, sometimes only want to register/bind services within a particular web app. Scoped registries - local versus global extension registry.

p2 Design Deep Dive

Interesting topics:

Profiles/ hierarchy of profiles

Headless agent; managed installs

Everything is possible

Version range lineups,

Security? JAAS

Resolver - where does it go? when?

Plugable download manager

Plugable communications - within ECF. replacing ECF?

Tooling (builds)

Enhancements to current export

pack/unpack/sign support

export to multiple targets (exporting platform dependent fragments)

Current Export

leave behind generated files for debugging purposes

Project Builder (can we get this into the pde.incubator)

a new builder project (like existing update site project)

Provides a mechanism to bootstrap from export to headless build

generates artifacts (features/build.properties/customTargets.xml/allElements.xml) that are input into pde.build

P2 Security

Day Two Results

Third breakout meeting

ECF/Transports/Repositories

We started with a discussion of requirements for the p2 download manager and transport layer:

Robustness of download manager

Mirroring

Different repositories have different characteristics - some repositories can support many connections

Adaptive re-prioritization - if you are downloading 1GB of data, you may need to switch to a different mirror dynamically

Weighting of repositories based on latency, availability, etc.

For downloading many small pieces, may be more efficient to combine them into a single large artifact for download purposes (minimizing round-trips)

BitTorrent:

Not good for many small files

Great for Eclipse release periods where there are large download peaks.

Could work well in a corporate environment where a large group of people are provisioning the same software around the same time.

Where do we embed all this intelligence - in a single uber-smart download manager, or should some of that smarts be hidden away in repository implementations? Could have a single smart repositories that acts as a facade to a group of backing repos.

ECF:

Scott: ECF is just intended to be a slim asynchronous transfer API - while some of these details could be handled at the transport level, some of this complexity should be built in a layer above ECF

Requirement for ECF: be able to introspect download stats in order to make adaptive decisions, alter repo weightings, etc. Want to be able to ask a transport up front for its anticipated throughput/latency, and get progress details that allow to update with actual throughput/latency as transfer progresses

Also need new API from ECF to start artifact downloads at a particular offset in an artifact. This is needed both for resume of downloads, and for multi-threaded download of a single file (each thread downloads a chunk, potentially from different mirrors, and then chunks are reassembled upon completion of download). See bug #205011.

Getting more concrete on the design and how the pieces fit together. First cut: Intelligence and design making largely in download manager. However, repository implementations need to be able to answer various questions in order for the download manager to be able to make its decisions:

Memory and disk space requirements

CPU time requirements

Latency/throughput of transport

Download manager also needs to account for local machine characteristics - how many concurrent downloads can it run, etc

Aiming for a relatively simple download manager with pluggable download strategies. Can start with a simple d/l strategy much like our current implementation. Then be able to plug in a much fancier d/l strategy that does mult-threading, adaptive behaviour, etc.

Work items:

Modify current d/l manager to have pluggable strategies

API for asynchronous interaction with repositories

API for querying progress of repos

Repo API for starting artifact downloads at some offset in the artifact

p2 Tooling

"Everything is possible with enough contributions"

self-hosting has been difficult historically

provision from workspace

imagine target platform + your workspace are one big repository

push from workspace

"Export -> metadata repository"

build tooling

ability to batch IU generation

are we generating a repository as an archive?

all plug-ins are pushed and published in a repository

archiving of a repository is a build operation

installing agent and installing and downloading a runnable piece should give the same thing

deployment tooling

when do we integrate the generator during build/exporting?

authoring tooling

long-term we want people to author IUs (editors)

probably not for 3.4

for 3.4, we will probably have some sort of feature.xml -> IU translation

It should not become too fine grained like Java permission models (finer grain -> administration headache)

Need an "administration":

"policy" on what user can install (especially new extension fragments)

associate names <-> roles <-> permissions

Login timing: different users have different needs

Server side: "late" login (app is well and running)

Shared install: login before workspace selection (so that it can drive workspace and preferences based on the user)

Server-side: JAAS approach itself is not sufficient for authorization. Role-based model of implemented by framework and UI might help

Cool new idea: login determining user preferences

Real keystore and secure data storage would be nice to have

Question: what keystore format should we use

OS-based single login can work with headless runs

Framework Deep Dive

Why is ContextClassLoader hard?

in an enterprise situation you always know the context (this war, this ear).

all lower level libraries can get the context

OSGi has no clear entry point, you don't know who your client is

the specification is mute on the subject of CCL. It defaults to the system class loader

Spring OSGi does some manipulation of the CCL. Ensures that CCL can see bundle classes during activation. Can also manage CCL on service invocation (configurable to service client bundle, or service provider bundle)

ContextFinder

Equinox has the ContextFinder that tries to find the context of who is calling. CCL is set to the ContextFinder. When forName is called, the *caller's* bundle is asked to load the class.

Note: you can use the context finder on other OSGi implementations.

BuddyPolicy

The Equinox buddy policy allows you to attach as a buddy of another bundle. e.g. to say that you are a buddy of Hibernate. Buddy policy does not required classes to be exported in order for them to be visible.

The "dependent" policy says anyone who depends on me directly or indirectly is a buddy.
The "registered" policy forces bundles to say they want to be a buddy of X.

There is no way for a bundle A to declare that it wants access to private classes in bundle B.

What if bundle A doesn't know that bundle B will need to access it? For example, a framework creates this requirement without A's knowledge. Peter Kriens has a blog entry describing some possible solutions to this.

One proposal is to trap "ClassNotFoundException" and then go hunting for matching classes. Has a weakness that would force classes to be loaded this way to be exported. Can be problems if two bundles both have a private type with the same qualified name.

Some problems with the Buddy mechanism:

If two buddies both have an internal class C, when the buddy bundle asks for C, which should it get?

Allows static references, not just dynamic ones. We noted that the Eclipse tooling will prevent static references if you develop using eclipse.

Equinox sets the CCL to the ContextFinder. Inherited by spawned threads. If any bundle ever sets the CCL to something different and forgets to set it back, this thread will never recover. In some app servers it might recover on next request (but not all, e.g. OC4J).

System Bundle Fragments

There are 2 types:

1. To supply extra classes to the framework implementation

Added for conditional permission admin, types that are sensitive from a security perspective and should not be loaded from a bundle.

2. Boot class path extensions. (Adding things to VM boot classpath). For example, if you need to add types in the "java" namespace. This is not implemented in equinox. Requires vm restart.

An "Extension fragment" is an extension to the fragment classloader.

Other:

The EclipseStarter code is not pretty.

Fourth breakout meeting

p2 Engine

Install IUs:

collect (download)

install

configure

IUs are installed by a particular touchpoint. Have touchpoint data

Engine talks to touchpoint

How to contribute "actions"

actions are things like unzip, copy, change file permissions.

Each touch point registers actions that other touchpoints can use

do actions have preconditions?

are they context free

how to contribute them

dependencies between actions -> are they transactional

Touch point data contains javaScript (Ant?, perl?, contributable?)

Actions are implemented in Java

each language used in touch point data "connector" that can expose these java actions

Reaction to cancel? (failure) -> cancel itself is long running

rollback/undo (hard -> backup old artifacts that are overwritten etc...)

How long to keep undo data, undo a whole install?

p2 User Interface

Currently update sites are global, but should that information be profile-specific?

Could just have a browser-based UI. Use a Browser widget within Eclipse, and hook link handling to initiate installs/updates from UI

Update: no UI needed. Just a dialog asking if you want to restart

Better support for browsing/filtering the list of available things to install. Organize by topic, filter like preference dialog

Would be nice to have an aggregator for update sites, so there is some central registry to browse for available software (such as EPIC)

However, we can't really resolve such a large set of data because the resolution algorithm likely won't scale up that much

Springing into Equinox

Two topics were discussed: Springs and Service Activator Toolkit (SAT)

Springs

An overview of Springs was provided:

Service provides, service consumers, proxies, dependency injection, all good things :-)

Springs has "smoothing" effect hiding services coming and going in a highly dynamic system (this works for proxies themselves, but not necessary for objects obtained from those proxies)

Supports "scopes", in particular "bundle" scope added for OSGi

Distribution:

Distribution includes Core and Additional modules. Current version comes in a form of bundles that can be immediately used in Eclipse / OSGi.

The OSGi-specific pieces need to be downloaded separately <please insert a link here>

if service goes away bundle that requires it gets deactivated, no re-wiring (?)

describes why bundles don't start

includes tools to avoid circular dependencies

it is an Eclipse project, small size (170K), used of Open health Care Framework (OHF)

Overall:

It feels like there are some common concepts in different component models. Extension registry, OSGi services, DS, Springs all work by wiring together "producers" and "consumers". All have they interesting points.

Is there a "kernel" in the framework that could be used by OSGi services, DS, Springs, extension registry?

On another point, could we combine "core" of Springs and SAT to get a small "OSGi service helper"?

Bug 178927: launcher: way to pass arguments from launcher to a running application instance

Bug 4922: EditorMgmt Need ability to open a file in eclipse from the command line

Dynamic programming planning

Need a concrete programming model for service registration and use. Adoption of OSGi services in the Eclipse world is seriously hampered by lack of a simple idiom for service lookup and use

Want to end up in a world where we're back to POJOs, without needing to know what service delivery mechanism is used

Currently in p2 we are using a simplistic approach that isn't reliable in a dynamic world, but has a simple use model

We should investigate using SAT (Service Activator Toolkit) for wiring/initialization of services. It is easy to use, relatively lightweight (about 150KB of code). It cuts down a lot of duplicated code in bundle activators that is traditionally needed for service wiring

What can we do to help make use of the extension registry dynamic enabled? We currently have ExtensionTracker as a way for clients to add hooks for handling extensions being added/removed

Dynamic markup: allow bundles to declare their level of dynamic enablement/awareness. That way when a bundle is added/removed/updated, we can determine if a restart is required.

This is a good thing, but there isn't agreement on whether this should be a manifest header, or a separate piece of metadata (for example in the corresponding IU)

What is the default assumption for a bundle that is missing dynamic markup? Should we assume the bundle is dynamic or not?

Action plan:

Investigate use of SAT

Look at Spring core when Spring M3 comes out (mid-October)

Settle on dynamic markup syntax and start rolling it out

Review extension registry usage

p2 planning

Backwards compatibility:

Compatibility with existing update.core APIs

Data compatibility with feature.xml, platform.xml, site.xml, etc

Generally data files such as platform.xml are considered implementation details of the platform and are subject to change

Compatibility support for people implementing install handlers

Co-existence model: Update.core APIs are still there, but if you invoke them, they will forward to corresponding function in p2 API. Main body of update.core is essentially thrown out in favour of p2 implementation

Compatibility model: update.core does its own (old) thing, and then run some generator or other tool on the resulting platform.xml files, etc, to reconcile with p2 world

Community will continue to craft feature.xml files in 3.4 timeframe, and we will use Generator to build p2 metadata

Source bundles: Want a better story for packaging source. 1-1 correspondence between source bundles and binary bundles is most flexible for packaging/install purposes

A rough outline of the current plan:

M3:

Rename packages and bundles

Be able to install to any platform/os from a single update site

Shared install

Refine end-user UI

Establish concrete set of functionality to be available in 3.4 final

Pluggable download manager strategy

M4:

Graduate p2 bundles from incubator into Equinox proper

Firm up APIs - javadoc, etc

M5:

p2 goes live in SDK in first M5 build (first I-build after M4)

M5a:

Fix critical p2 bugs found the day after M5 release that caused community panic

Summary

Day one summary: Everything is possible

Day two summary: Everything is possible with sufficient contribution. Contribution is not just code.