This is the first post from my company’s new engineering blog. I’m hoping that in the near future, we’ll be able to share more of the cool stuff we’re doing. Please let me know if you have any feedback!

I really hate this land of langauge specific build tools that we’ve created. We started off on the right foot with make and then, when we ran into limitations, everyone just decided to go their own way rather than make things better. Now every language has its own dependency manager, packager, build system, etc. Ugh, it really sucks. Can’t we reverse this?

I’ve been getting into OCaml and Haskell again after a few years away and I’ve been navigating the maze world that is OPAM and Cabal. For example, getting Agda running means getting the Haskell Platform working, which, for those of us who want to understand what we’re installing on our systems, is an awful lot of work (and navigating a lot of options). Add to this the fact that I’ve had to work with Grunt, npm, setup.py, among a whole host of others I can’t recall right now (package management and build systems tend to form together) it starts to wear real thin in a real hurry.

Analyzing the faults of all these is too much to get into here, but suffice it to say that all of them are reinventing the wheels of make and Maven. Poorly. The desire to have a homogenous build system and code base is as fallacious as thinking that we’ll one day have “the ultimate programming language”.

I understand that it can be frustrating having to learn a lot of different tools, but in general why would we want to reverse this?

The current approach allows people to write specialized build tools and package managers targeted at their particular language’s features, and use the language they know best in order to customize things to their needs. That seems much more useful than trying to create a single monolithic build system/package manager to rule them all.

The problem is not as much having to learn (at best) one tool per language. The problem is that those tools reinvent the wheel continuously. And very rarely they are a good wheel. For example, how is dependency management different in jode, ruby, java, c, erlang, ocaml, haskell, python, emacs packages, … different? Why do we need a different tool for each of them? Most of the tools I use don’t solve the problem correctly, whereas a community trying to solve just dependency management has much better chances to get it right (e.g. rpm, deb, nix). How is building a tree of building steps different in any of those languages? A properly build framework could be pluggable so that adaptors can be build for different technologies, but instead each language has its on implementation of make (usually with worse characteristics), and so on.

I believe that by spreading the efforts in so many different silos we (the industry) are not solving the problem but just making it bigger.

A properly build framework could be pluggable so that adaptors can be build for different technologies

That single line made your perspective a lot more understandable to me. I’ve frequently encountered people who say, in effect, “You’re an idiot for wanting to use the language you know best to manage your build tasks. Instead learn #{ obscure_dsl_language } and all your problems will be solved.”

I don’t care whether make, rake, or cake is supplying the underlying technology as long as I can define my JavaScript build steps in JavaScript, and the technology actually works cross-platform as well as Node build tools do.

I don’t care whether make, rake, or cake is supplying the underlying technology as long as I can define my JavaScript build steps in JavaScript

I apologize for being crass here but the truth is that you’re wrong to want this. JavaScript, and pretty much any other turing complete language, is terrible at defining builds. As an industry we learned long ago that builds are declarative.

Looking at the blog post, I don’t see how learning something like make is worse than using gulp. Gulp is JavaScript but it’s incredibly verbose, it’s a whole API that someone needs to learn. On top of that, since the inderlying language is JavaScript, I’m not protected at all from some other component doing crazy things, nullifying any invariants I want. Is that better than spending a few minutes to learn make? And I mean literally a few minutes. The builds described in this blog post are a half dozen lines in a Makefile.

I apologize for being crass here but the truth is that you’re wrong to want this

No need to apologize. Forthright disagreement isn’t crass.

However, I would like to know what standard you are using to judge an approach right or wrong. Otherwise, it’s hard to evaluate whether I am missing some objective universal CS truth, or whether we just value different things.

As an industry we learned long ago that builds are declarative.

I’m a little confused by this statement. Wasn’t the whole point of your original complaint that “we as an industry” haven’t learned that?

Wasn’t the whole point of your original complaint that “we as an industry” haven’t learned that?

No, my point in the original complaint that every language has its own build system and package manager. Some of them get declarative builds correct, some of them don’t.

However, I would like to know what standard you are using to judge an approach right or wrong.

I’m using almost half a century of build systems as standard. Even the painful ones, like Maven, value declarative builds. Pretty much every build system that has gotten serious use adheres to the same set of base principles.

I don’t believe a build system is as subjective as people would like to think. There is a task, building software, there are clear principles which have been discovered decades ago for how to solve this problem correct and consistently. Everyone that solves the problem of building software ends up in the same place. I’d prefer a world were we just figure out how to solve it once and for all for everyone and move on with life rather than people spending a weekend building something that hasn’t benefited from history at all.

I would love to hear which ones you think get things right and which don’t, just for my own education. I understand if you don’t have time, though.

There are tons of build systems, but ones that get it mostly right includes the various flavors make, bazel, maven/graddle/ant. Ones that get it wrong are things like the JavaScript ones, and I’m sure there are plenty of others. Note that I’m not saying these tools are good in total, I really really dislike Maven, just that they get this particular aspect of builds correct.

Well, sure, but I was hoping for something a little more concrete that can be referred to by someone who doesn’t have the half century of build systems in their head. :-)

The internet is full of information on these things, just look up make and go from there. Bazel, from Google, will probably become rather popular.

Another common mistake, IMO, is many of these build tools conflate building with dependency management. I have yet to see this end well, building and dependency management are differen tthings that people just generally need to be solved around the same time. maven is an offender here and, probably the worst is, rebar in the Erlang world. Rebar is bad at dependency management, it’s bad at building, it’s okay at creating Erlang things called ‘releases’, and that is it. Rebar should probably just be a CLI tool to A) help produce a Makefile B) turn a built project into a release.

The current approach allows people to write specialized build tools and package managers targeted at their particular language’s features

Like what? Almost every language-specific build system I’ve seen poorly re-implements the actual act of building and the language specific things should just have been stand alone CLI tools that are useful outside of the build system. If you look at the gulp + npm examples you have, all of those are clearer and shorter if expressed as a Makefile.

and use the language they know best in order to customize things to their needs

No, it’s not the language everyone knows best, some people are forced to maintain tools in a wide assortment of langauges and this everyone-gets-its-own-build-tool world is miserable.

That seems much more useful than trying to create a single monolithic build system/package manager to rule them all.

What’s the value of having node trigger grunt/gulp? Users still have to install grunt/gulp and understand the grunt/gulp config if they’re going to modify the build? I’d rather have a single canonical way to invoke the project build (probably gulp), and have the readme explain how to call it.

So, to use Grunt/Gulp, you have to have a way of invoking the build system. The “standard” way of doing that is to install a global package whose sole job is to invoke the build system.

Grunt: npm install -g grunt-cli

Gulp: npm install -g gulp

From that point on, you have a grunt/gulp cli that will run your build. But that CLI tool is managed separately from the local node_modules directory and package.json, so you have to instruct everyone on the team how to set it up.

Using npm run, though, looks into your local node_modules/.bin/ directory for executables before the rest of the system. That means you don’t have to manage an extra global CLI dependency. You just tell people to use npm run build, and everything just works.

An additional benefit is that if you want to change build tools for some reason, you don’t have to go around telling people to switch their CLI. Just edit the npm script to point to a different tool. The only person that needs to deal with it is the person changing their build.

I’d rather have a single canonical way to invoke the project build (probably gulp), and have the readme explain how to call it.