meme: an idea, behavior, or usage that spreads from person to person within a culture agora: a gathering place

Welcome to Neal's gathering place for ideas.

Thursday, November 15, 2007

Ruby Matters: Frameworks, DSLs, and Dietzler's Rule

As an industry, we've been engaged in an experiment for the last decade or so. This experiment started back in the mid to late 90's, largely driven by the fact that the demand for software vastly outstripped the supply of those who could write it (this wasn't a new problem then -- we've had this problem almost since the idea of business software started). The goal: create tools and environments that would allow average and/or mediocre developers to be productive, regardless of the messy facts already known by people like Fred Brooks (see Mythical Man Month). The reasoning goes that if we create languages that keep people out trouble by restricting what damage they can do, we can produce software without having to pay those annoying software craftsman ridiculous amounts of money (and you'd probably never be able to find enough of them even then). This thinking gave us tools like dBASE, PowerBuilder, Clipper, and Access: the rise of the 4GL's.

But the problem was that you couldn't get enough done in those environments. They created what my colleague Terry Dietzler at the time called the "80-10-10 Rule" for Access: you can get 80% of what the customer wants in a remarkably short time. The next 10% of what they want is possible, but takes a lot of effort. The last 10% is flat out impossible because you can't get "underneath" all the tooling and frameworks. And users want 100% of what they want, so 4GLs gave way to general purpose languages (Visual BASIC, Java, Delphi, and eventually C#). Java and C# in particular were designed to make C++ easier and less error prone, so they built in some fairly serious restrictions, in the interest of keeping average developers out of trouble. The problem is that they created their own version of the "80-10-10 Rule", only this time the stuff you couldn't do was much more subtle. Because they are general purpose languages, you can get pretty much anything done...with enough effort. Java kept bumping into stuff that would be nice to do but was way to much work, so frameworks were built. And built. And built. Aspects were added. More frameworks were built. It is so bad that meta-frameworks were built: the Avalon framework was a framework for...building other frameworks!

We can see what this trend has done to productivity when building complex software. What we really want is the productivity of 4GLs with the generality and flexibility of powerful general purpose languages. Enter frameworks built with Domain Specific Languages, the current exemplar being Ruby on Rails. When writing a Rails application, you don't write that much "pure" Ruby code (and most of that is in models, for business rules). Mostly, you are writing code in the DSL part of Rails. That means that you get major bang for the buck:

You get a huge bunch of functionality with this little bit of code. 4GL levels of productivity, but with a critical difference. In a 4GL (and the current mainstream statically typed languages), it is cumbersome or impossible to do really power stuff (like meta-programming). In a DSL written on top of a super powerful language, you can drop one level of abstraction to the underlying language to get done whatever you need to get done.

This is the best approach currently available. The productivity comes from working close to the problem domain in the DSL; the power comes from the abstraction layer simmering just below the surface. Expressive DSLs on top of powerful languages will become the new standard. Frameworks will be written using DSLs, not on top of statically typed languages with restrictive syntax. Note that this isn't necessarily a dynamic language or even Ruby tirade: a strong potential exists for statically typed type-inference languages that have a suitable syntax to also take advantage of this style of programming. For an example of this, check out Jaskell and in particular the build DSL written on top of it called Neptune.

8 comments:

Can you explain more about your idea of frameworks written in DSLs?I'm thinking that maybe once we have a DSL, we build DSLs over it instead of a framework. For a sufficient complex domain, we can have layers and layers of DSLs one over the other... not sure if I want to go the "framework" way once I have a DSL ;-)

I think you had the direction wrong. You don't build frameworks on top of DSLs, but rather use the dsl to configure the framework. This happens all the time now--what is the use of XML in Java but a DSL used to configure a framework? This is of course not the only way people use DSLs, but it is very common.

Far be it for me to disagree but aren't we really just trading one 80-10-10 for another? For example, taking the given example of Rails if you go off the...well, rails from what RoR was designed to do (i.e. babysitting a database with a web front end) you hit a "10" in either category. I'm wondering how extending RoR's DSL to cover the 10-10 is that much different than extending a framework?

How dare you disagree with me? Don't you know who I am? Oh, wait, you do...nevermind!

This is a good point, but the thing that keeps you out of the 80-10-10 space with Rails is exactly what's underneath. There are very few things you can't do with Ruby, and having that powerful language just one level of abstraction down is nice. You cannot do this with Java Frameworks without going to something complex like aspects, and that just solves one of the types of meta-programming you can do.

Mark my words: simple DSLs on top of super powerful languages is going to be the next paradigm shift (and it's already partially here with Rails).

Having spent the last few months all warm and cuddly with the Groovy source code I don't believe that meta-programming in Java is that far off from what you can do in Ruby. Now training Java developers to think in a world where you can monkey patch new methods into String is another matter. :)

I guess what I was trying to get at is that when the framework/DSL breaksdown you end up taking the same action (extension) to fix both. For example, as far as I know in (maybe there is a plug-in now) core RoR trying to schedule an event to fire off at a given time is a job that you have to hit cron for. Just not sure how you'd meta-program your way out of that one. I just seems to me the problem is trying to account for what people don't think of as design-time is a problem in a framework or a DSL.

In the example you just gave, you are correct: having RoR vs. Java talk to cron is going to be the same thing. However, here is a counter-example. Look at the Dust gem in Ruby. It allows you to automatically fail unit tests if you touch the database, and functional tests automatically fail if you mock anything out. Not impossible in Java, but so much work you'd never do it.