Last year, at about this time, I wrote a lengthy, omnibus post called Eclipse has a future. It’s purpose was to capture some of the challenges that are facing the Eclipse community, and describe how the work that we were doing in the e4 Incubator Project could help to address them. Since then, our investment in that work has continued to grow, and we are now ready to ship a new version of the Eclipse SDK, built with technology that has graduated from the incubator, called the Eclipse SDK 4.0 Early Adopter Release.

I’m going to talk about what it is about Eclipse 4 that makes it interesting and important, but first I wanted to say a bit about why we grew the tagline “Early Adopter Release”.

An Early Adopter Release

As I noted last year, the Eclipse Project developers have known for quite a while that we were going to have to innovate in some significant ways in order for the base platform to continue to be compelling in the face of changing pressures in the industry — notably, a range of new technologies for building desktop apps (e.g. AIR, Silverlight, etc.) and the push towards using the web to deliver “desktop grade” applications. At the same time, however, many of our consumers were looking for the strongest possible backwards compatibility / stability so that they could continue to move forward with the release train without needing to invest in responding to changes in our layer (or any other one they depended on, for that matter).

In addition, we had begun to feel that the complexity of the platform code itself was preventing us from growing our community. More than ten years of development had left the code in a brittle state where almost any significant change would break more places than it fixed, making it extremely difficult for new developers to get involved.

The result of this was a conscious splitting of our focus into an ongoing development stream, Eclipse 3.x, where we could continue to offer the strongest possible backwards compatibility and stability, and a brand new project, the e4 Incubator, where we could work on the kinds of major innovations and code transformations that we believed were necessary. As the work in this incubator matured, it would move outward to the most appropriate place: some to other projects at Eclipse.org, some to our Eclipse 3.x SDK (as the flexible resources work did in 3.6), and some to a new “forward looking” Eclipse 4.x SDK, which would “organically” become the focus of Eclipse based development as it proved its worth.

So here we are in 2010. We’ve gone from a prototype — last years “0.9” release that used significant chunks of old code as “shims” to get things running — to a real Eclipse SDK built on some powerful new underlying technology. The thing is though, this really is new technology. It’s not like we went in and tweaked the implementation a bit; we actually gutted the workbench implementation and started again.

The good news is that it seems like our bet has paid off. If you look at the pace at which the new version of the workbench has come together, it’s obvious that this code is much easier to work with. Personally, I’d say we’ve seen about a milestone’s worth of improvement each week for the last month or so. Over all, the Eclipse SDK 4.0 is completely usable for development. The team has been running on it exclusively for at least a month, and the drive to eat-your-own-dog-food has gone from a painful experience to one that feels just like working with Eclipse 3.x (except that it looks nicer ;-) ).

The thing about dog-fooding though, is that it works through the scenarios that are most important for the people working on the code, and basically nothing else. It’s great at getting something we can use, but not so good at covering all of the myriad ways that the community has built on top of us, which gets me back to why we’re calling this an “Early Adopter Release”.

We need your help.

To get from the current state to something that is solid, bug free, and complete enough to be included in a release train (or used to ship product) is going to take your input. We need to know where the changes we’ve made impact you and your uses of Eclipse. We need to know whether there are things that are missing that you must have. We need to know if there have been performance impacts… Basically, we’re asking you to invest enough time to take the Eclipse 4.0 release and load your plug-ins and projects on top of it, and then tell us what is working and what isn’t. And, in case that wasn’t enough, we’re asking you to continue to work with us over the course of this year, re-trying milestone builds and helping us verify that we’re converging to where we need to be.

Now, I wish I could tell you that it would be immediately obvious how this was going to benefit your work. I think there are some very compelling things about the new workbench, but I honestly can’t tell how important they’ll be to you. Really, I’m asking for your help here because I really do believe that this is the future of Eclipse, and I know that it will be great with your help, or it will die on the vine without it.

The one thing I can offer in return is an absolute commitment to be responsive to your input. Our goal is to have every milestone on the road to 4.1 be a “mini-release”; one that is visibly more complete, more capable and more solid than the ones before it. To do this, we will be strongly focused on the issues that you report to us — to the early adopter goes the spoils (er… bug fixes).

What’s there now

If you’re looking for the full list of what’s new in Eclipse 4 you should check out the New and Noteworthy. I’m not going to reiterate all that, but I would like to talk a bit about some of the technological pieces that have graduated from the e4 Incubator, which are the basis for this “newness”.

For example, the most immediately visible change you see when you start up Eclipse 4 is the new look and feel:

This is a great, fresh new look with a modern style and excellent use of whitespace to reduce visual clutter. But, in addition to being some best-of-breed design work [no, I didn’t have anything to do with it] this also represents the first time where the appearance of the workbench — everything from the background colour of the main toolbar to the radius of the curve on the folder tabs — is controlled by CSS.

Why we believe this is important, is that it simultaneously gives people building RCP apps and products based on the SDK much more control over the look (i.e. the “branding”) while making the “language” of that control be one that makes sense to designers (i.e. instead of requiring them to know the internals of our Presentation API. Ugh!). It also opens up opportunities for people building apps that contain embedded HTML content to share more of their design resources.

Just as an example, here is an ugly, but definitely um… different look we rolled using only CSS changes:

The underlying SWT changes to support this work were done by the SWT team as part of 3.6 development (with a couple of tweaks since then). The CSS Engine itself was contributed by Angelo Zerr, with ongoing development from several others. To see an example of how you can use this new support in an RCP app, to excellent effect, take a look at Kai Tödter’s Contacts Demo.

Also, if you are interested in pushing the limits of what you can do with the CSS support, please come talk to us. There are definitely places where we would like to provide even more flexibility, and we know the workflows around editing the CSS could use work. Knowing what would be most valuable to the community here would be excellent.

Despite being less immediately visible, the most critical change in Eclipse 4 is the switch to a modelled user interface.

Essentially, the entire structure of the application’s user-interface is represented as an EMF model. This model has been carefully tuned so that it has strong separation between unrelated concerns, and is layered so that Eclipse 4 based applications can incrementally take only those pieces they actually need. This is a significant advance over the “take everything and prune out the bits you don’t need” approach used in Eclipse 3.x (and earlier) RCP applications. What this actually means is that people building new Eclipse 4 based applications do not have to include any of the scaffolding that is specific to the workbench’s use of the model, nor do they have to include any of the code that supports backwards compatibility for 3.x apps, if they don’t need it. We believe this new way to work is sufficiently important that we have christened it the “Eclipse 4 Application Platform”.

Note that this does not imply that you must give up 3.x compatibility in order to build RCP applications in 4.x. We continue to be committed to supporting full API compatibility between 3.x and 4.x, so building Eclipse 4 based, traditional RCP applications is still supported. It’s just that once developers start using the Eclipse 4 Application Platform, the benefits will be obvious.

At the workbench level, the flexibility we get from the use of the modelled user interface, together with another piece of technology newly graduated from the incubator, hierarchical contexts, also opens up brand new capabilities for 3.x compatible applications. In this case, by separating the application code from the context in which it is running, we can remove many of the artificial constraints imposed by the 3.x workbench. Dependency injection is used to provide the application code with the values it needs without requiring it to know where they came from, and in turn, the 3.x compatibility support makes use of this to ensure that all traditional change notification, data binding, etc. behaves as expected.

You know, reading that last paragraph, it’s clear that this is a case where a picture would definitely be worth the proverbial thousand words. Here is an example of a common workflow for us on Eclipse 4.0 that would be simply impossible to do in 3.x:

What you are seeing here is the completely unmodified 3.x Console view being displayed in the same stack as the editors. We like this because it’s a great way to give views like the Console more screen real-estate, but every 3.x user will tell you “You can’t put views in the editor area(!)”.

Really, this is about making the decisions relating to where views and editors show up (for example) a matter of policy rather than being hardwired into the implementation. Because, in 4.0, these policy “add ons” can be done by simply controlling what transforms are possible on the model, the code to implement them is very easy to write, is typically all held in a single class, and can be changed to support the needs of your particular application. For anyone who understands what the equivalent code in 3.x looks like, this is one of the clearest pieces of evidence we have that the new modelled UI is much easier to work with.

Where we go next

Before I go on, I have to say I’m absolutely humbled by how much effort everyone has put in to get us to where we are today. From the “old school” platform committers, to the new ones we gained because their work graduated from the incubator (Brian de Alwis, Kai Tödter, Yves Yang), to all those who continue to work on cool new technology in e4, to the many who have helped us with the design, and the many more who tested (with special thanks to Stefan Mücke), commented, documented… All of you have my greatest respect. I hope you all know we have built something great here, and we ought to be proud of it.

The thing is though, we also know one other important thing: We’re not done.

And we’re not done in an at least three significant ways:

1) plugging the holes

You don’t have to look far when playing with Eclipse 4.0 SDK Early Adopter Release to find places where there are bugs and things that are just outright missing. I’d apologize about this, but the truth is we got as far as we needed to so that our “early adopters” can help to guide us the rest of the way.

We have bugzilla for the bugs and a wiki page that captures the most obvious missing features. We know that there’s lots more work to do, and we’re going to do it.

2) graduating more

Even though we did see some important new capabilities graduate from the incubator, there is lots more work being done there that hasn’t graduated yet. We know that’s the natural order of an incubator — things will graduate when they’re ready — but what’s important is that we continue to push to make the level of investment it takes for that work to reach maturity.

As an example, for those who’ve been following e4 since the beginning, one of the most obvious missing pieces in Eclipse 4.0 is the Eclipse Application Services (a.k.a. “the 20 things”). Fundamentally, we still believe the EAS are going to be important for the future of Eclipse; particularly if we want to start looking at building plug-ins in languages other than Java. We did make some progress on this, this year, but we didn’t get even close to far enough for it to graduate as part of the 4.0 release. I’m disappointed, but I know we won’t let this one go. Stay tuned for more in the weeks ahead.

3) continuing to innovate

So listen, let’s say you’ve got this great idea about the next big innovation that we need, and that you are looking for just the perfect place to make it happen. Well, have I got an incubator for you!

Seriously, the e4 Incubator is intended to be a place to experiment, push the boundaries, whatever-it-takes on an ongoing basis. Just because Eclipse 4.x has started doesn’t mean its one burst of innovation is over. If you have an idea for something new, please come talk to us.

I was very happy to see that Olivier suggested bringing a JDT related project into e4 on improving support for Java-like languages. I think that’s a great idea, and I hope he can get enough critical mass to get it going. If you’re interested in that possibility check out the discussion on jdt-dev.

Anyway, even for me, this has been a marathon post. I hope that it helps to give you an idea of what we’ve been up to. Please give Eclipse SDK 4.0 Early Adopter Release a try, and tell us about your experiences.

We’re getting close to releasing 0.9 of e4, so I wanted to take some time to talk a bit more about the context for why we started on this path, and how far we’ve gotten in the last year. To give this some structure, let’s look at a few of the big questions…

1) Does Eclipse have a future on the desktop?
2) What would an ideal Eclipse platform look like?
3) What is e4?

… and maybe a couple of more pragmatic ones too…

4) When is e4 done?
5) What can I do with e4 0.9?

Does Eclipse have a future on the desktop?

Easy answer: “Yes, but not if we sit still”.

I’m sure it’s not news to anyone in our community that, unlike in the early days of Eclipse, there are many credible threats out there when it comes to technologies for building desktop applications. Adobe AIR, Silverlight, even pure HTML5+JS, for example, can all be used to build rich, cross-platform desktop apps. For Eclipse to be a viable platform in this environment we need to innovate on many dimensions, including packaging/distribution/size issues, flexibility and richness of presentation, integration with other technologies and, most especially, ease of development.

We’ve got to get to the point where developers are choosing Eclipse because it’s both easier to use than the other technologies out there and because the results work better and integrate better than anything built any other way.

What would an ideal Eclipse platform look like?

Well, obviously it needs to address the issues above, but there’s more to it than that. To me, the ideal Eclipse platform is one that has the ability to move when there are new pressures on it. Technologies change, uses change, the community itself changes over time, and whenever that happens, either the platform has to be able to move with it, or we die.

The ideal platform should have a strong separation between presentation and implementation, and it should be dynamically configurable to run in the widest possible range of contexts. It should have the smallest possible number of “pre-canned” constraints, and it has to be easy for new people to start using it so that it can be discovered and consumed by whole new communities.

<intermission>

“But wait!”, some of you are saying, “for my uses of Eclipse, I don’t need any of those things. I just use the IDE to develop; I don’t care what it looks like. And besides, we’ve already built all our plug-ins. The most important thing is don’t break us.”

I get it. You’ve invested in Eclipse long enough to get something that works for you, and you just want the world to stop so you can get as much value from that investment as possible. That’s ok. You’re a part of this community too, and we need to support you. That’s why the focus for the R3.x stream of the SDK is the strongest possible backwards compatibility: If you ran on R3.4 or R3.5, we want you to move to R3.6 because it’s effortless.

But the thing is…

This post isn’t for you.

What we are talking about here is what it’s going to take for Eclipse to be relevant in another ten years. This is for those of you who care about making that happen in addition to getting the great day-to-day value you currently get. If you believe, like me, that Eclipse is simply too important to allow it to fade away, then read on.

</intermission>

What is e4?

A few years ago, the Eclipse Project developers were facing what seemed like an insoluble problem: We could see that our industry was heading toward another period of big change and the platform was going to have to react to it, while at the same time we continued (and still do!) to get more and more real-world, important products built on us — products that could not afford to deal with the potential for instability caused by major work on platform internals.

At the same time, we were (yes, finally) beginning to realize that we needed new blood; that the monolithic nature of the team, which had helped us make progress early on, was now hurting our ability to grow our community and potentially risking our future.

This was one of those times when things “just worked”. From those starting points, we have become a new community, based around the e4 Incubator project, that is made up of a wide range of people who are working together to build that “ideal Eclipse platform” for the future.

It’s been a wild ride (and we’re not close to done) but I have to say that it blows my mind how far we’ve come in the last year. I’m not going to go into the individual technology details, but I will point you at John Arthorne’s excellent e4 Technical Overview white paper for the full description.

When is e4 done?

Hopefully, e4 the incubator is never done; it will be there as long as there is a need for the platform to innovate. Today though, you’re probably more interested in:

· when things that are currently being worked on become part of the platform· when Eclipse 4.0 appears· when Eclipse 4.0 gets on the release train

We have always said that code that is developed in e4 will mature at different rates. Some of the things that are being worked on could simply be moved to the R3.x stream, once they meet the backwards compatibility and stability requirements. Personally, I’d like to see CSS skinning of the widget layer, improved web browser integration, and the flexible resources work all move to R3.6, but that’s still to be determined.

Some of the changes that are being made, however, are simply too fundamental to ever make it into the R3.x stream. Those will only appear in a new, separate development stream of the Eclipse SDK, with the first version, called “4.0”, shipping a year from now.

The thing to keep in mind though, is that a year from now the odds that Eclipse 4.0 will have been subjected to the kind of testing and tuning that it absolutely must have for it to be used as the basis for product delivery are pretty low. It will be, in the truest sense, an entirely new platform that is capable of hosting your plug-ins. Will it be backwards compatible? Yes. Will it behave in exactly the same way as R3.x, so that every possible subtle interaction with your existing plug-ins is supported? No. Wait… what?

This is probably the most critical part of understanding the Eclipse 4.x stream: Even though we plan to do everything we possibly can to make Eclipse 4.0 backward API compatible, either because of use of internals, or dependencies on unspecified timing/ordering behavior, or any number of other subtleties, it’s possible that your plug-ins may not run. Since the only way we will know about that is if you tell us, our success will depend on you making an investment in trying to run on Eclipse 4.0 builds:

We will work with everyone who does this and finds problems to either improve our own code, or help them fix the incompatibilities in theirs.

Believe me, I understand that we’re imposing on you; if you don’t see value in moving to the next Eclipse, that’s ok, we really do plan to develop the R3.x stream for as long as anyone is using it. But at some point, one of two things will have happened: either the Eclipse platform itself will no longer be relevant, or so much of the community will have moved to the R4.x stream that your consumers will be asking you to move too. My crystal ball doesn’t say when that happens, but I know it will.

In case that sounded too harsh, I must also say that this is going to be an organic process that is controlled by the consuming projects, not by us. The best example of this the answer to the question about when Eclipse 4.x gets on the release train:

We will not ask to put Eclipse 4.x on the release train until every other release train project with a platform dependency runs on it.

So, no it won’t be in Helios. However, it will be real enough at that point that you should take it, work with it, and help us understand what needs to be done. If you’re an RCP developer, you will be very pleasantly surprised by how much easier it is to get the look and feel you want. For the SDK, we plan to have every single view and editor that is in R3.6 also working on R4.0. That should be a good start on backwards compatibility.

What can I do with e4 0.9?

All that’s great, but we’re shipping something this week too. :-)

The “e4 Compatibility Platform” is what we’re calling the main download for the 0.9 release. It’s not even close to being complete enough to be an “Eclipse SDK” yet, but it is enough for self-hosted development (using views and editors from R3.5(!)) and demonstrates well all of the things we’ve been talking about. Kevin McGuire wrote a great post that describes a perfect example of why we we’re so psyched:

Despite some immediately obvious missing pieces (like no save/restore of workbench layout, no main toolbar, no min/max behavior, and numerous small bugs), it’s definitely usable. For the last week or so, I’ve been spending my evenings using the e4 Compatibility Platform (with (SchemeWay) and Ahti Kitsik’s Virtual Word Wrap[edit: updated link] plug-ins installed) to write some wiki software in Gambit Scheme. For my use, this worked great.

I also know some people on the team are using the e4 compatibility platform for all development. Currently, that takes more perseverance than any of us would like, but the good news is that we know what the issues are, we just ran out of time to fix them.

In any case, just so there are no illusions about exactly how “young” this code is, here are some hints that I learned while using it:

If you can’t get a menu or keybinding to work, try the popup menu instead.

This is a symptom of all the myriad ways we have for hooking up menus in R3.x, some of which just haven’t been wired off yet. That’s a perfect example of the kinds of things that make it hard for new people to understand Eclipse R3.x development.

Don’t check the “always exit without asking” check box when you close the shell.

Until save/restore starts working, it’s painful to have to re-open/re-layout all of your views. You don’t want to exit by accident.

If you close a perspective, don’t immediately open that one again.

If you try, it silently fails. Instead, open any other perspective first, then you can open the original. I’m not sure why this is happening, but it’s just a bug.

Well, that’s where we are. Once R0.9 is out, we’ll be reviewing all of the work areas, identifying the missing pieces, and building the plan to get from here to Eclipse SDK 4.0.

In Closing…

Please, give the e4 Compatibility Platform a try. If you see the value in what we’re doing, please consider participating. We are entirely open to new people who want work with us on the existing development areas or on anything else that you think is interesting. Even if you don’t think we’re on track, come talk to us: if we’re missing something important, we need to know.

For me, I think we have hit the “sweet spot”: we’ve found a way to continue to work with interesting people, on interesting problems, innovating a future for Eclipse on the desktop, while still providing the backwards compatible, it-just-has-to-work, world that we must have on the release train.

Thanks to Boris, we actually have a place to record whether or not you are coming to the e4 Summit. I know many people, who said that they wanted to come, whose name is not on that list yet, so if you plan to be there, please add your name.

Note that there are two distinct sections to that page:

If you are planning to come to the summit, add yourself to the Attendees section

If you are interested in working on a particular area, even if you can’t come to the summit, please add yourself to the Work areas section. Our expectation is that those lists really will be the starting points for the major work areas, so if you want to work on something that isn’t covered by any of those lists please add a new row.

For example, one area that I believe is not covered yet, which I am sure I heard a lot of people say they were interested in at EclipseCON, is UI macro recording/playback. Although the work on Make it easy to script Eclipse (e.g., expose DOMs) will help people who want to implement UI macro recorders, it isn’t the same — the existing item is talking about making it easier to drive eclipse via scripting languages. So if you want to work on macro recording, you will have to drive it.

Over the last couple of days, there has been a lot of discussion about the announcement of a new component, called “e4″ (“efour”), as part of the Eclipse Project Incubator. I believe that most of this stems from the fact that a rather poor job was done of explaining the intent of the component. Several of the committers have made admirable efforts at answering peoples questions, and I took a stab at it in the eclipse.org-architecture-council mailing list. It was suggested on the list that this message should get out to a wider audience, so I’m going to paraphrase it here (with some edits via sober second thought):

Eclipse 4.0 and the need for community participation

I know that everyone involved in the Eclipse community is trying to do the right thing. I also know that we all believe that the level of communication and community participation that will be required to make Eclipse 4.0 succeed is orders of magnitude more than what you have seen from us in the past.

So, like everyone who is learning a new way to work, what we absolutely need from the community is positive re-inforcement when we get things right, and gentle:-) chastisement when we mess up. It’s clear that, the way that the e4 component has been described so far has confused things more than it has helped. I’ll make another attempt at clearing that up below, but I want to make one other point first.

I care deeply about Eclipse. I have been fighting for its success since day zero. When I look at Eclipse.org today, I am truly humbled by the sheer magnitude of the successes we (all) have built, and I really do GET IT that it is the community that has generated that success. So I’m going to be completely open here: I am royally pissed that something so critical to all our successes (i.e. the Eclipse SDK) is being built by a very small team, most of whom come from one company. There are exactly two things that I believe are threatening the continued success of Eclipse right now:

lack of a diverse, growing committer community on the SDK

a codebase that is so constrained by history that it can’t respond to a rapidly changing computing environment

Addressing those two things is exactly why the e4 effort was started. No hidden agendas. No extra direction from IBM to solve some new business problem. Nothing else. So what it all comes down to is this: If someone sees us doing something that doesn’t look like it matches one of those goals, absolutely speak up! But, for God’s sake, give us the benefit of a doubt. We will make mistakes, but they will be honest ones. For my part, I personally commit to making e4 be the kind of community driven project we all want.

Let me talk a bit more about the e4 component…

The only code that currently exists that is related to e4, is a couple of cool(ish) demos (neither of which is called “e4″) that we built to help us figure out whether some ideas we had about new directions were valuable or not — As Steve Northover says, “We think with our fingers”. I’d been planning to show that code at EclipseCON, because I also believe that demos speak louder than powerpoint, but we really only had them in publishable shape in the last couple of weeks. In retrospect, what we should have done was just dump them in some existing place in CVS and be done with it, but we thought that making them more visible was important. The idea was that by building a separate component for them, they would both be easy to find and, when we did get the e4 effort rolling, they would be easy to move to another home (assuming that made sense).

Obviously, we misjudged the implications of the proposal that went out. If I had know that it was going to have the impact that it did I would have spent more time wordsmithing it when Steve showed it to me. I apologize to the community for the confusion that it caused. [Heck, when I went back and re-read it, I thought it was bogus.]

I do believe that there will need to be a new shared area to work, once e4 actually starts to happen. My current theory is that the best place for that would be as a new incubator project unto itself, under the Eclipse Project, with the initial committer list being *everyone* who wants to get involved. That’s just my opinion, and I’m looking for feedback from the community. Let’s talk.

"Gaff, Doug" wrote on 03/07/2008 13:00:14:
> ...
> (warning: gentle chastisement coming) the project lead should have a
> blog on planet eclipse.
>

You’re darn right he should, Doug.

For those who don’t know me, I am the project lead for the Eclipse Platform project and co-lead of the Eclipse Incubator project. I’m also on the Eclipse Project PMC. I’m interested in all aspects of UI design, programming languages, large-scale systems, massively-multiplayer online games, and playing the cello.

I’m not sure how often I’ll post here, but I’ll try and keep the (rest of the) posts relevant.

I decided to call this blog “I am a committer” mostly to remind me that I really am a committer (on the SWT team). Man, I’d love to have enough time to go work on the Cocoa port…