by James Coglan

Menu

Last week, my former employer theOTHERmedia open-sourced the last project I
worked on there: Helium. It’s a web application that lets you deploy
JavaScript packages from Git and load them on-demand into any website by
including a single script tag. There’s been a lot of innovation in JavaScript
deployment recently, and Helium fits a particular set of needs that I think most
web agencies will be all too used to struggling with.

First, some background about the problems we were trying to solve. theOTHERmedia
is a design and development agency with dozens of clients and a lot of code to
keep under control. Their JavaScript stack is based around YUI and
Ojay, as well as other common tools like Google Maps, Analytics etc. As
part of client work, we inevitably ended up producing a lot of reusable
components that didn’t really fit into the Ojay core but nevertheless needed to
be shared between client projects.

Because they had no obvious place to live, these little libraries ended up being
copy-pasted from project to project, producing a maintenance headache. If I
fixed a bug in one copy, I’d have to track down all the other copies to patch
them. Also, some clients required some small customisations: too small to merit
generalising the code to accommodate them, but large enough that managed
branching, merging and cherry-picking between branches would really help with
maintenance.

The main day-to-day problems I’d run into were that these projects became hard
to maintain, hard to track down, and that it wasn’t clear how to set them up.
Developers would copy a component into their project and come to me when it
didn’t work: inevitably some dependency or config setting was missing. I wanted
to make it as transparent as possible to load any JavaScript object you liked
into a site without running into these sorts of problems, and I wanted each
library to be maintained in one repository and farmed out to client sites as
required. I should be able to apply a bug fix and see it show up in all our
clients’ sites immediately.

The solution to the maintenance problem was obvious: these libraries should have
been under their own version control, and the need to fork libraries for certain
clients and keep the forks up-to-date with bug fixes made Git the perfect choice
for this. Git makes branching and merging easy, and lets you cherry-pick changes
between branches if you don’t want to do a full merge. Problem one solved.

The second problem is deployment: we need to get the libraries out of Git, build
them and distribute them to client sites. Clients need to specify which version
(i.e. which branch or tag) of each library they want to use. And, dependencies
should be handled automatically: I’m a big fan of package managers like aptitude
on Ubuntu or RubyGems in the Ruby community as they make it easy to get software
running without worrying about its internal dependencies.

This is where an number of other tools come in. Helium is really a wrapper
around a set of smaller components that were all designed to be flexible enough
that they can be trivially integrated. The main “front end” of Helium is powered
by JS.Packages, a pure-JavaScript dependency manager that ships with
JS.Class. It works at a very high level, letting you load JavaScript objects by
name rather than by URL, figuring out dependencies for you and downloading files
in parallel where possible. For example, it lets me write this:

This is very powerful, but it requires some configuration to tell JS.Packages
where to find objects, what other objects they depend on and so on. To make the
above work, you need the following configuration:

Aside from the fact that writing this is fairly tedious, I didn’t want these
config files to live in the client projects where they’d need just as much
maintenance as the original libraries, especially since different branches of a
library often have different dependencies.

Now most of this code tends to be of the file/provides/requires variety
rather than the custom loader function variety, and this is code that’s very
easy to generate. The final piece of the puzzle, Jake, is what ties this
all together.

Jake is a build tool for JavaScript projects that mostly handles code
concatenation and minification for larger JavaScript projects. The neat thing
about it is that it lets you embed metadata (such as dependency information) and
has an event system to let you know when build files are generated and where
they live on disk. This means you can use it to build a manifest of all the
files a project build generates, which JavaScript objects each file contains and
which objects it depends on, just by placing some metadata in each project’s
build configuration. We’ve now moved this data out of the client projects and
into a central location: each library manages its own dependency data, and
Helium uses Jake to extract this and build a manifest of all the libraries on
your server and what their dependencies are.

Putting it all together, the whole process pulls projects out of Git, uses Jake
to build every branch of each project and extract its dependency data, then uses
this to generate a JS.Packages manifest: it’s this file that client sites
include in their head section:

You then need to tell Helium which branch/tag of each library to use, and
thereafter you can use require() to load any object you like. On the client
side, Helium essentially acts as a versioning layer on top of JS.Packages.

The intent is for Helium users to host their own installations of it, since
providing a single package manifest for all the JavaScript libraries on the web
would produce a huge file and obviate the benefit of on-demand loading. If you
want to check it out, the best place to start is the documentation on
GitHub, which covers a ton of stuff I’ve not even mentioned here. In
particular it explains how to make your own JavaScript projects
Helium-deployable, a process I’ve tried hard to make as easy as possible. I’d
love to see this adopted as a package distribution system for JavaScript, so if
you have any feedback on how it can be improved, get over to GitHub and let
us know!

—

If you’ve enjoyed this article, you might enjoy my recently published book
JavaScript Testing Recipes. It’s
full of simple techniques for writing modular, maintainable JavaScript apps
in the browser and on the server.