Thursday, May 31, 2012

I have been working on volo, which is a real, working attempt at providing a solution for front end Javascript package management.

It seems like more people are starting to look at this, and here are my suggestions on what the solution should look like. I do not expect everyone to use volo, but if we all agreed to some basics, then it will allow some easier interop (see the end of this post).

Resist the temptation to make your own registry

One of the harder parts is to agree on a registry for code. With volo I have gone with using GitHub because:

It already exists and works/scales.

It has social/feedback tools and search.

Used by many JS libraries already, it is already part of developer workflow.

I suggest starting with GitHub as the registry. They have a nice API that volo uses to get tags and files out of a repo.

It would be easy to replicate a stand-alone server that has the same API later if it seemed like GitHub as the registry did not make sense, but by using GitHub and its API, some of the mundane bikeshedding over API and registry construction goes away.

The "owner/repo" IDs with github, vs just the "repo" single namespace that is in something like npm, is a distinct benefit to have for a registry. Forks should be possible, with the default search on just "repo" giving the most popular version, which is usually the original repo. Given GitHub's social tools, they have a way to measure popularity, and it seems to work out pretty well.

Easy convention

Try to find a convention so that configuration is not required. The basic convention that volo uses: it checks for any explicit configuration (see below) but if none is found, it pulls down the zipball of a version tag, and if there is one JS file at the top level of the zipball, that is the script that is installed.

There is a bit more to the convention, but this basic convention works for many JS libraries. It helps encourage providing small libraries that can be composed together well with other libraries in other repos.

Easy configuration

The above convention does not work for every project. Some folks do not want to host their code on github, and some projects need to do a "build" step to deliver the final code, and like hosting that built code outside the git repository. So there needs to be a way to configure what to download for a dependency.

It supports {version} substitution with a version value from a version tag.

More is documented in the package.json page. volo also understands a volo.archive, but I want to reduce that config to just volo.url, and have it do content type detection to know if it is a single JS file or an archive zip/tarball.

Do not require server devs to change

volo has a "shims" repo it checks if a library does not have the package.json "volo" property it is looking for, so it makes it easy to bootstrap new libraries into the system without requiring the library author to do anything.

Of course it is best and more distributed if the library author supports the package.json property/properties directly, but for now it has been easy to consume scripts without getting complete buy-in from library authors.

Let's coordinate

I would love it if there were other tools besides volo for this functionality, as long as we agreed to the above GitHub bootstrapping and some package.json properties we can all read and understand.

In particular, I do not want these properties under a "volo" name in the package.json. I am doing that for now just so I do not claim a more generic name without any agreement with others. What is a name we can use instead? "frontend"? "browser"? I'm up for a more generic name so that we can open up the client tool building space.

I can be reached on gmail at jrburke. If you want a public space to talk, there is the volo list, but I am happy to talk on another list if that is preferable.

While the 2.0 code has gotten some use, it is still new, and it is naturally more subject to issues than the 1.0.x releases. If you do see an issue, please report it.

I am really grateful for the quality and quantity of community feedback that prompted the 2.0 release, with community members paving the way for core changes by doing implementations via the loader plugin API.

Plugin APIs are awesome, and the loader plugin API is one of the great strengths of the AMD module ecosystem. In addition to reducing the "pyramid of doom" that can occur in async resource fetching, it helps try out ideas for the loader, and supports transpiled languages like CoffeeScript.

I sometimes see some confusion about AMD modules and RequireJS, and I want to take a moment to reiterate the problem they are trying to solve.

The AMD API is about getting workable module syntax in JavaScript that meets all the async, networked nature of browsers without having the hidden costs down the road (no eval, CORS/cross-origin concerns, needing a transform to ship code).

Node's module API and using something like browserify is not enough for browser modules. It works for a certain case of problems, but to be a complete solution, a standardized callback-based require for on-demand async loading after initial module load and a wrapped module format that supports named modules for bundling are needed at a minimum. Loader plugins are also incredibly useful for reducing the async "pyramid of doom" for resources that are needed as part of module initialization. If node was able to integrate something like amdefine into their core it would really make it a complete JS module solution.

ECMAScript harmony modules account for the async network IO in browsers, and it has the ability to change the internals of browser script loading so that it will not hit the eval and CORS issues that node/commonjs modules have. But it is not done, and still has quite a few kinks to work out. It will still have the same "how do I use browser globals scripts that do not declare their dependencies" issues that people see when starting to use AMD modules with older scripts. The shim config in RequireJS 2.0 is an attempt to ease that problem, and I hope that AMD and RequireJS can continue to help inform the ECMAScript effort.

So until node closes the gap on some things needed for browser loading, or ECMAScript harmony modules work out the kinks and ship, there is AMD, and RequireJS aims to be the reference implementation for the AMD APIs.

If you want to support AMD loaders in your library code, there are some code templates that can help you do that in a way that still allows your code to work in traditional "browser globals" situations.

AMD: the worse JS module API, except for all the others. Because it works better in more cases.

The big change since that last post: Dan Mosedale had some good feedback about treating GitHub Pages as a deploy target, so I changed around the template to do do that.

As part of these changes, volo has a new release 0.1.2 that has support for doing GitHub authorization to do OAuth-based API calls with GitHub. The app template uses this to create a GitHub repo on the fly for GitHub pages deployment.

The nice thing about this setup is that the app creation is simpler (no more questions), and you can decide later to deploy to GitHub pages without needing to make the decision up front:

See the video for a more complete walkthrough, but basically to get started now it just looks like this:

volo create myproject volojs/create-responsive-template

cd myproject

volo appcache (generates the appcache-enabled build)volo ghdeploy (copies the built contents and pushes them to GitHub Pages)