For reasons I will better explain later I have been investigating alternatives to GWT. In particular, I spent the last week working with Java2Script. Here are my initial impressions of this project:

First, what it is: Java2Script fulfills a greatly similar role to GWT, in that it acts as a Java-to-JavaScript language compiler. It provides a similar emulation layer for the core JRE class library. Additionally, where GWT provides its own custom widget toolkit API, Java2Script provides an emulation layer for SWT. Java2Script is, in fact, older than GWT. It seems to be less focused on optimizing for performance, and as such allows for such things as dynamic class-loading, a by-design limitation of GWT.

Java2Script only has a few developers, and I think only one main one (Zhou), but so far, I have been very impressed at how supportive and responsive they have been. My first task was to attempt to compile the draw2d API, such that it was able to load without errors. In attempting to accomplish this task, I discovered several bugs in the j2s compiler. I reported them, and within a day or two, Zhou had published fixes. Zhou and the other developers have also been very supportive in terms of offering me guidance with respect to my particular project. And so, by Saturday evening, I had accomplished the first task I set out: the draw2d Hello World example successfully compiled and loaded in the browser. It didn’t do anything, but that is to be expected as GC is currently only implemented as stub code.

The next step is to get simple examples of GC working. I don’t have a clear idea of how I will accomplish this, yet, as most patterns of using GC rely on setting a Paint event listener on a SWT widget, and using the GC returned on the event. The browser, unfortunately, doesn’t expose a native Paint event (only Firefox 3 nightlies, at the moment, as far as I know) so I’ll have to find another way to accomplish this. In any case, I have a thread going on this, some sample code to work from, and a good dialogue in progress with one of the developers, so I’m optimistic that I will get a reduced example of GC working by the end of this coming week.

I don’t know that I’m such a big fan of IDE’s. They seem to run counter to the Unix Philosophy, which is to have a large number of small tools that do one specific thing, and do it very well. These small tools are all bound together through dead simple, absolutely standard and rock-solid message-passing interfaces (stdin, stdout, stderr; all parameters are Strings). A big IDE that attempts to do everything and the kitchen sink often just feels like the wrong level at which to be working. Certainly, for my current project, being strongly bound to Eclipse for development has so far been counterproductive. This is because the GWT-Eclipse tooling is not yet matured… which is to say that the GWT module concept is really very different from the basic Java package concept, and this is a gap that the current tooling does not yet bridge. The result, for me at least, has been a great deal of confusion. Then again, I’m attempting to use GWT for things for which it was never intended to be used

In any case, it has felt very good to ditch the IDE for a while and return to my humble yet powerful development tools: GNU Screen, Vim, Bash and Ant. I won’t describe here what these tools are or how to use them; this is covered very well in other places. But I will post a nice screenshot of my current desktop, which I feel is very close to complete User Interface zen:

One component currently missing from my toolbox is a good, preferably console-based file managers. I don’t really have very strong preferences with regard to file managers, but I really should, because there are many times when I could really use some powerful features. I just haven’t quite figured out what those are yet. Vi keybindings are a major plus, though. That rules out the famous midnight commander, unfortunately, as its keybindings are not configurable. Really, it only leaves two options: vifm, and a Vim plugin called VimExplorer. vifm was quite nice, a nice solid little file manager with vi-like keybindings. Unfortunately, it had some trouble installing on Ubuntu Hardy, and it was just different enough to what I was used to with a proper Vim instance to be annoying. That left me VimExplorer. I was a bit wary of this, because it hadn’t been updated since 2007, and the last update was to fix a critical bug that could cause data loss. But I installed it, tried it out, and found it to be quite good. It gives you a small file manager instance that lives right inside of vim. All of your vim skills then can be applied to managing files, including: regular expression search, macros, manipulating vim windows and buffers, and using “!” to trigger shell scripts. There’s something really great about being about to just say “yy” and then “p” to copy and paste a file. Same for move and delete.

Unfortunately, I still feel like this tool is lacking. 90% of what I do when developing is in done in just a few files and directories. What I want is to be able to just type the name of a file on my filesystem (e.g. “Accessibility.gwt.xml”), and immediately have an editor up for that file. The system should behave intelligently when there are naming collisions. I believe that GNOME-Do might actually be able to do this, and I should look into that. For the most part though, I think most of this could be accomplished by just tagging certain files or directories as favorites, and maintaining a simple hash to map short variable names to long paths. In any case, I’m continuing to iterate on this component of my set of simple tools.

I worked really hard today, and I have lots to report. I started at 6:30am and worked straight through on GSoC until about 7:30pm. I also spent some time investigating alternate tooling, and I’ll talk a bit about this in another post.

So, to recap, we have this stack: Demos/GEF/Draw2d/SWT/JRE. The goal of this week is to get Draw2d to compile under GWT. After that, I’ll have to try to get it to run, but for now, I’m just concerned with getting it to compile. This, of course, means figuring out what Draw2d’s dependencies are in SWT and the JRE, and where they are not satisfied, writing out stub code and small TODO comments. Also, I will atempt, where possible, to reuse the existing, non-functional code from the previous project that used GWT to compile SWT to js.

To summarize my experiences of today: JRE == SUCCESS, SWT == FAIL.

First some background: GWT already provides some JRE emulation, but it is not sufficient to compile the SWT codebase. To get SWT to compile, one must at least implement certain stub interfaces that GWT does not provide. There are additional issues, such as reflection, that may be more serious, and I’ll talk about those in a later post. The extension to the GWT emulation classes live in the project org.eclipse.swt.e4.jcl. Left over from the previous project were three GWT modules which I was never able to compile. Additionally, it was unclear to me how these modules were supposed to compile, as they used the GWT module “source” tag, rather than the “super-source” tag, which, according to all of the documentation I have read, is what one is supposed to use when implementing JRE classes.

Today, I successfully compiled the previous project’s JCL classes. In order to accomplish this, I applied Technique 1 from my previous post, which is to say, I added a prefix (“e4″) to the JCL packages, where before there was no prefix, and I used <super-source path=”e4″/> in my GWT modules. I had to futz a bit to discover in which order I should import the modules, but after that, they compiled without complaint, w00t

//TODO: insert complete project structure here

Note that, since yesterday, I have been developing this prototype outside of the Eclipse environment. My toolset has consisted of GNU Screen, Vim, a Vim plugin called VimExplorer, and Ant. This has allowed me to ignore all of the IDE-specific confusion that I have so far encountered. My next post will be about my experiences using some of these tools.

I was feeling very optimistic after succeded in getting JCL to compile, so I set it as my goal to get the necessary SWT libraries to compile as well. Unfortunately, I failed in this task, because, I believe, I took the wrong approach. The approach I took was to attempt to reuse directly the SWT emulation classes left over from the previous project, just as I reused the previous project’s JCL classes. Unfortunately, these SWT emulation classes had many missing dependencies. My strategy was to just keep hacking on it (adding in missing classes, stub methods, etc.), until the thing worked. I spent most of the afternoon working on this, probably about 4 hours. Ultimately, I ended up pulling in most of the code in “org.eclipse.swt/Eclipse SWT/common”. I didn’t get to the end of the dependency chain, but I think I got far enough along to recognize a trend that I wasn’t happy with, namely, pulling in dependencies that in turn rely on other dependencies, etc. It should be a major priority to limit dependencies, because the more deps I cut out, the less complex my project will be. Given that this project is still at the prototyping stage, reducing complexity is of critical importance.

Note that I also tried compiling “org.eclipse.swt/Eclipse SWT/common”, but this pulled in some native code which GWT was unhappy with. So, I really do need to be selective.

The alternative to today’s approach is to start at the top of the stack (work top-down) and discover what the dependencies are for the Draw2d demos; then, find out what the dependencies are for Draw2d; then, very carefully, attempt to pull in those deps from org.eclipse.swt, or stub them out. I intend to use Doxygen to figure out the complete dependency chain. This will be tomorrow’s work.

In short, there is a kind of impedance mismatch between the functionality that GWT provides the functionality that GWT’s Eclipse tooling provides, at least with respect to super-source. Here’s my understanding of how all of this breaks down:

1. You can use a package structure in which the native packages you would like to emulate (e.g. java.*) are prefixed by some other package (e.g. hack.java.io). So you might have a project structure that looks like this:

The Pros: This works great in both GWT hosted mode and compiled mode.The Cons: The Eclipse IDE will think that something is wrong with your package declaration (it thinks it should be hack.java.io), and will give you errors.

2. You can use a package structure which does not use prefixed packages. So, you have something like:

testsuper/

super/

test/

TestSuper.gwt.xml

test.java.io/

OutputStream.java

Your GWT module super-source tag would then use path=””.The package declaration in your emulated Java classes would then say “java.io”, as per normal.

The Pros: GWT Hosted mode works (although you must set the runtime classpath to point to whatever dependencies TestSuper has; you set this as an Eclipse run configuration), and the Eclipse IDE doesn’t throw errors.The Cons: GWT Compiled mode fails for the Eclipse GWT tooling, if you have projects that import this project, because the super/ folder is not on the Eclipse build path. If you were using Ant, this would not be a problem, because you can simply set the build-time classpath, but it does not seem to be possible to do this yet with the Eclipse tooling. Also, because super/ is not on the build path, you don’t get all of the advantages of the Java tooling that you would normally get from Eclipse.Workaround: Use ant within Eclipse to Compile, and use regular GWT for Hosted mode.

It’s important to point out that all of these issues are related to the GWT Eclipse tooling, not GWT itself. Nevertheless, for my project, I am strongly dependent on Eclipse, so it is necessary for me to find the best way to resolve these issues. I’m not 100% certain about this, but at the moment, I feel like my best option is to go with option 2.b., so to stay in Eclipse, but move all of the build logic to Ant.

Moving forward. In the post before my last post, I mentioned the swtlibs.gwt.xml file which was confusing me. Here it is again:

<module>

<source path=""/>

<source path="events"/>

<source path="graphics"/>

<source path="layout"/>

<source path="widgets"/>

<source path="dnd"/>

<source path="printing"/>

<source path="animation"/>

<source path="accessibility"/>

<source path="custom"/>

<source path="graphics2"/>

<source path="internal"/>

<source path="net"/>

<script src="swt/loading.js"/>

</module>

The thing that was confusing me was the path entries. One would assume that they are relative paths… and I’ve now decided that, in fact, they are. First of all, the GWT documentation makes it pretty clear that this is the case:

<source path="path" /> : Each occurrence of the tag adds a package to the source path by combining the package in which the module XML is found with the specified path to a subpackage. Any Java source file appearing in this subpackage or any of its subpackages is assumed to be translatable. The <source> element supports pattern-based filtering to allow fine-grained control over which resources get copied into the output directory during a GWT compile.

But this was nonobvious in this case, because all of the paths referenced except for graphics and widget, are not accessible as relative paths. One could interpret them as being poorly-documented indicators of “future work”, but only if the it was certain that the GWT compiler wasn’t doing something clever like pulling in other, actual SWT folders on the classpath, that have the same name; for this to be true, the GWT compiler needed to be silently ignoring the missing referenced paths. To test this, I set up a sample GWT project with a source package that does exist. I also added a reference to a source package that did not exist:

End result: the Java classes in subpackage exists were found usable in both hosted mode and compiled mode. The compiler just ignored the reference to a package that did not exist.

So what does all of this finally mean for my project? It shows me the scope of the SWT emulation that they had at one point presumably been working. At the very least, it sets an upper bound on the work that was done in this regard: emulation of graphics and widgets package. That’s all. But they were aiming to emulate the entire SWT library.

With that in mind, I dug a bit deeper into the widgets package. As expected, there were a bunch of methods using JSNI. Not the nicest JavaScript code… But I won’t criticize it to much, as I understand that it never got past the prototype phase.

I’m ignoring the work that has already been done on this, for the moment, and imagining how I would do it if I were starting from scratch.

There are two components that would need to be implemented: the jre emulation layer, and the SWT emulation layer. GWT already provides a JRE emulation layer, but it will likely be necessary to enhance it.

SWT is big. All we really care about is GraphicsContext and event handling.

The graphics part should slot nicely into the way SWT graphics is already organized, in that it has different drawing backends. We just need to use the API exposed by GWTCanvas. It should be possible to look to swt graphics for guidance on this, in that it already supports many different target implementations.

Draw2d may have other dependencies on SWT beyond GC and events. Here’s a bit of script I used to found out what dependencies org.eclipse.draw2d has on SWT:

So, my strategy for my GSoC project is going to simultaneously encompass two approaches: top-down and bottom-up. The top-down approach would have me re-use the pre-existing code as a starting point as much as possible. The bottom-up approach would have me start from scratch. I think the success of my project depends on me embracing both: I should try to understand the code as much as possible in order to maximally reuse it. At the same time, said code appears to be in a rather broken state, so I need to forge ahead and dolve problems in my own way. It would be really nice if the code was not in a broken state, though, as I can see where my contribution would slot right in. Oh well…

Here’s what I’ve determined so far. The goal of this phase of the project is to use GWT to implement the SWT API, so that GWT may then be used to compile any Java code that targets SWT down to JavaScript. It appears that the work that has been done so far has been to implement this API for most of the SWT controls. A notable exception to this is the GraphicsContext (GC) API. This is where my contribution would slot in, if any of the previous work was functioning correctly.

The best place to zoom in on this is org.eclipse.swt.e4.examples/dojo/controlexample/controlexample.gwt.xml. This top-level GWT module illustrates what other projects I would likely need to depend on, and where the GWT modules have been inserted into that project structure.

<module>

<!-- //////// FIXED PATHS //////// -->

<!-- Inherit the SWT-Dojo libs and the SWT common libs -->

<inherits name="org.eclipse.swt.swtlibs"/>

<!-- Inherit the base Dojo libs -->

<inherits name="dojoLib"/>

<!-- Inherit the SWT javascript libs, css styles, etc. -->

<inherits name="resources"/>

<!-- Inherit java.io + java.lang emulated libs. -->

<inherits name="JCL"/>

<inherits name="extended_JCL"/>

<inherits name="extended_js_JCL"/>

<!-- Inherit the core Web Toolkit stuff. -->

<inherits name='com.google.gwt.core.Core'/>

<!-- //////// FIXED PATHS //////// -->

<!-- Specify the app entry point class. -->

<entry-point class="org.eclipse.swt.examples.controlexample.Main"/>

</module>

So, basically, we’re pulling in the SWT lib, dojo, some resources, classes for custom JRE emulation, and GWT core. Pretty straightforward so far. There’s an ant build.xml file that accompanies this which sets up the appropriate classpath and runs the GWT compiler against the module. If I was using the GWT plugin for Eclipse, I imagine that I could do away with the ant script and simply press the GWT compile button to begin compilation.

So, that part is fine.

It gets more complicated, though, when you drill down into the above modules. I spent the first week-and-ahalf of this project trying to make sense of the *JCL modules, before finally deciding that they were, in actual fact, in a disfunctional state. See, for example, the swtlibs.gwt.xml module:

<module>

<source path=""/>

<source path="events"/>

<source path="graphics"/>

<source path="layout"/>

<source path="widgets"/>

<source path="dnd"/>

<source path="printing"/>

<source path="animation"/>

<source path="accessibility"/>

<source path="custom"/>

<source path="graphics2"/>

<source path="internal"/>

<source path="net"/>

</module>

These might appear to be relative paths, but they are not, because the directory structure in which they find themselves is as follows:

org.eclipse.swt/EclipseSWT/

common/

org/eclipse/swt/

events/

graphics/

internal/

layout/

widgets/

package.html

SWT.java

SWTError.java

SWTException.java

dojo/

org/eclipse/swt/

graphics/

widgets/

swtlibs.gwt.xml

By process of elimination, it appears that they are referencing libs in the org.eclipse.swt project.

jacob@jacob-laptop:~/workspace/eclipse$ find -name layout

./org.eclipse.swt/Eclipse SWT/common/org/eclipse/swt/layout

That’s the only folder named “layout” in the entire workspace.

There’s a bunch more strangeness here involving importing the project, which I’ll cover in a later blog post.

Right now, I’m running Kubuntu 8.04, which uses the very excellent KDE 3.5.10 desktop environment. Unfortunately, Kubuntu 8.04, unlike its close cousin Ubuntu 8.04, is not an LTS (Long-Term Support) release. This means that, instead of support being discontinued in April 2011, it will be discontinued in October of this year. I suppose this makes sense, as KDE technology has moved on to the 4.* branch. The KDE developers apparently no longer even fix bugs in KDE3 apps. This makes me sad, because I’m totally happy with my current setup. I really don’t want to change my system when everything is working so nicely, but I’m afraid I’m just going to have to bite the bullet and do it. The question is, what should I change to? It seems like my main options are to either upgrade to Kubuntu 9.10 (which will be out in October), which should ship with the KDE 4.3 desktop; or, I stay in 8.04, but step over to the Ubuntu desktop.

One reason I’ve been avoiding using GNOME/Ubuntu is that by using Kubuntu 8.04, I’ve been able to avoid PulseAudio. I have had terrible experiences with PulseAudio so far. It is really not ready for normal users. My roommate Chris has discreetly switched back to using Windows because we couldn’t adequately resolve his audio problems. I don’t blame him; PulseAudio is not ready to go. Neither is Phonon (the KDE4 media backend) for that matter.

BUT, I have used PulseAudio enough to apprehend the vision behind it, and I like it. I like the idea of having audio sources, sinks and streams; having the ability to redirect those streams arbitrarily, even across a network. It’s a lovely, simple vision. I would love to write a beautiful, graph-like UI on top of it. I want to believe in PulseAudio. I want it to work. So, if I can get the minimum apps that I need to work, including the proprietary ones (Flash and Skype, although I’m hoping to finally begin to migrate rapidly away from Skype very soon), given my very minimal hardware requirements, and have it work reliably, then I will be a happy camper. Anyways, I’ve sudo apt-get installed ubuntu-desktop, and I’m going to try it and see what the state of things are.

The other reason why I’ve been hesitant to switch to GNOME/Ubuntu is because of GNOME’s use of Mono. I’m in the camp that says that relying on Mono is bad. Microsoft is litigious and Mono is patent-encumbered. It should therefore be avoided; it’s as simple as that. What that ultimately means in terms of how I use my desktop, though, is not clear. The only applications that come bundled with Ubuntu that use Mono appear to be FSpot and Tomboy. That’s not so bad; these are two apps I can live without. But I was hoping to try out Gnome-Do, which is also based on Mono. It’s tempting to just ignore the issues surrounding it, and just install the app, but it feels dirty and wrong, so I just won’t do it.

There are certain KDE apps that I would miss if I used GNOME: Amarok is a big one. Konqueror is another. KDESVN rocks my socks. Of course, I can use all of these apps in GNOME as well, but then it pulls in all of the KDE libraries.

In any case, these are all KDE3 apps running on top of KDE3. From what I’ve seen, the KDE4 apps are often radically different. One reason I’m even engaging in this discussion is because my experience of KDE4 has so far been quite bad. On every platform I’ve used it, KDE4 has felt slow and buggy. I got a chance to really sink my teeth into it in Kubuntu 9.04 (the last *buntu I installed on my roommate Chris’s computer before he finally gave up on it), and everything regarding usability was just awful. For just an example, look at what they did to the available apt front-ends. Kubuntu 8.04 shipped with aptitude_manager. It was fast, had a fast incremental search, a simple direct UI. I never once had a problem with it. Not a single instance of it not being able to install packages correctly or crashing. Even slightly weird packages like the java-sun* packages that require you to agree a special license installed easily and reliably. In KDE4, on the other hand, you have KPackageKit. The app feels slow, the UI is terrible, and it choked multiple times when trying to install software. In particular, it choked on java-sun, and actually asked me to try using a better apt front-end. Ridiculous. There is also a version of Aptitude for KDE4, although though it doesn’t ship by default, but the UI is also very different from the KDE3 version, and, in my opinion, is not very usable.

My experience of KDE4 was like this constantly while I was setting Chris up with Kubuntu 9.04. So I feel I may not be very happy in KDE4. Too much has changed, most of it for the worse.

So where does that leave me as a full-time GNU/Linux user, looking toward the future of my OS of choice?

…

Probably GNOME, sans Mono. Although I keep waffling about this. Hopefully, in six months Kubuntu will have turned things around, and I can reassess my decision. But who knows, by that time I might be a happy GNOME user.

I met a bunch of smart people at the Montreal Ubuntu 9.04 release party, and I talked to one person a lot about the distributed SCM Git. I had heard of distributed SCM’s but had never had any reason to try anything other than SVN. Still, I was intrigued, and so decided to look into it. Here are some useful links:

My goal was to use Git to solve a particular problem, which was as follows:

There is an SVN repository A, and a CVS repository B. The code in B was probably forked at some point from the code in A. B was iterated on in a private sandbox before being put into CVS, so B has no history, just HEAD. A has also changed since the time it was forked. I need to take the code in B and compare it to each of the major release tags in A, so as to determine which tag is the most similar to B, and therefore which tag B was most likely to have been forked from. This basically means diffing two different filesystem trees. Because Git is supposed to be very good at merging different branches based on content, I assumed it would be able to give me some very intelligent diff output.

In this contex, A is the GWT SVN repo, and B is a project in Eclipse’s CVS repo. The Eclipse project org.eclipse.swt.e4.jcl appears to be forked from GWT, so I needed to determine when it was forked, what changes were made after, and (hopefully) why.

The person I met at the Ubuntu party, Derek, wrote me very excellent instructions on how to do this, which I have gratuitously copied and pasted here.

Here are the general steps:

- Use git-svn-clone to clone the Subversion repository A.

- Checkout the CVS HEAD of repository B into a separate workspace.

- Create a new branch in the git-svn repository. You can branch

starting from any tag or commit, but the closer that the branch origin

is to the original fork tag, the fewer differences you will likely

encounter. So, try to correctly guess the fork tag and branch from

that guess.

- Overwrite the workspace associated with the new branch with all the

files in the CVS HEAD workspace. Stage and then commit the changes

into the Git repository.

- For every Subversion "tag" branch in Git (Git stores every

Subversion tag as a Git branch, not as a Git tag), display the

differences between the branch that contains CVS HEAD and the Git

Subversion "tag" branch. Note the one that gives you the fewest

differences.

Specific commands (untested):

# Clone the Subversion repository A. Option "--stdlayout" tells Git

that the repository has the standard branches, tags, trunk layout

git svn clone --stdlayout http://domain.com/repositoryA

# Create a branch from a tag

git branch subversiontag cvsbranch

# Switch to branch "cvsbranch"

git checkout cvsbranch

# Overwrite files in Git "cvsbranch" workspace with those from CVS

HEAD workspace. Make sure that you don't copy the CVS metadeta

directory in each workspace directory. Consider using rsync which can

ignore CVS and Subversion metadata directories.

# Stage all CVS changes

git add --all

# Commit CVS changes

git commit --message "Added changes from CVS repository B"

# List all Git Subversion tag branches

git branch | grep svn

# Compare each Git Subversion tag branch with tag "cvsbranch". Option

"--name-only" displays only the names of the files that changed:

for tag in `git branch -r | grep tags`; do git diff --name-only $tag

cvsbranch; done

# Note the tag that produced the fewest differing files.

While this mostly worked great, I ran into one gotch using Git from the Ubuntu 8.04 repo (git version 1.5.4.3), which was that git-svn did not capture all of the tags from svn. When I tried it with git from the 8.10 repo (git version 1.5.6.something), it worked fine. Really, I should be git-cloning the latest git, but I haven’t quite figured out how to do that successfully. Nevertheless, beware.

I also tried using svn2git, which is supposed to basically be svn-git with some manipulations to make the imported svn repo more git-like. I spent an hour trying to get this to work, and met with marginal success. It turned out not to be neccessary.

In the end, I ended up using the following code to guage similarity.

The first one just counts the number of files that changed between each tag and the cvs branch.

So, it looks like 1.5.2 has significantly fewer changes than any other tag. The diff to 1.5.2 is attached to this email. It basically looked like 48 new files were added, 0 were removed and many were modified.

Once I had established which GWT SVN tag was most similar, it was trivial to diff the branches and get meaningful output:

I won’t post the resulting diff here, but suffice it to say that it was exactly what I was looking for, and I am now moving onto the task of analyzing the diff to figure out exactly what was changed, and why.

My final impressions of git: I’m very happy with it. The ability to import an entire SVN repo so that it can be used offline; the ease with which one may branch and merge; and the ability to push changes back up into the svn repo, are all things that will be very useful to me. I think I’m actually going to get to use Git a lot on my GSoC project.