what I learned over the years

Menu

programming

There’s a joke in computer science that everything can be solved with another layer of abstraction or indirection. It’s true. In our line of work, tools are essentially good abstractions for the problem we’re solving. Our craft and discipline is built with abstractions, and yet, I find it’s something hard for programmers to grasp.

Why use abstractions in the first place? We build abstractions to make it easier to work on the problem we’re trying to solve, without getting distracted by details from the big picture. We want to be able to drive a car using a an abstraction called a gas pedal, rather than timing the firings of the spark plugs in the engine, which is what actually moves the car forward. This abstraction lets us worry about that “what”–getting from point A to point B–without worrying about the “how”–the specifics of how that occurs.

I think we all know the above logically, but emotionally and in practice, we often disregard the ideas above. Ideally, a programmer not only gets the program to do what it wants, but also produces usable abstractions for other coworkers along the way. Often times we laud the quickest programmers, but I think the best programmers are ones that build usable abstractions for other programmers. The ones that build tools for others to use and increases the productivity of those around them. And yet, I’ve met programmers that didn’t like to build abstractions.

There are some programmers that like to have everything on a single page. . I suspect these are programmers that can hold an inordinate number of things in their head. They can look at all the details of which pistons are firing at which time all at once, and still understand the whole thing. But even this is unsustainable at some point, as there’s no physics capping the upper limit on the size of code bases. In addition, coding, at its core, is a social activity. If you write code that others find hard to follow because you’re mixing the “what” with the “how”, then your code is less useful to others, both to use and to maintain.

There are yet other programmers that fear it makes the code slow. I’ve seen Ruby code written as if it was C code. This is counter productive. Often times you don’t know which part of your code is the slowest until you measure and profile. Often times, your intuition is wrong about which paths get executed the most. There are diminishing returns to make a piece of code faster. If it takes 10 seconds to run, then a 50% reduction in time is great! But if it already takes 10 milliseconds to run, then the extra 5 milliseconds may not be worth it, as there are often many other pieces of code that would take 5 milliseconds or more to run. In addition, a user will not notice 5 ms as much as 5 seconds.

So while not every programmer should build tools for others, I believe it’s something every good programmer should aspire to. However, this is often a difficult skill to learn and get right in practice. It’s not easy to do under the time crunch of deadlines and fire drills. Even if you didn’t have those time pressures, it’s still hard to get right.

Why is it so hard to build useful abstractions? For one, you can often hide the wrong details. And when you hide the wrong details, the abstraction is useless because you need to manipulate the details. On the other hand you can veer too far the other way, by building abstractions that are too flexible, requiring too many new concepts for users to grasp before they’re productive.

The culprit to both problems is that we often don’t understand the problem well enough when we sit down to write. When you don’t yet understand all the different circumstances that you’d use an abstraction, you expose either too much or too little.

There are two ways to combat this. First, you should set out to write the application, and extract the framework or platform for it, rather than setting out to write the platform. I’ve seen some useless web frameworks because they set out to write the framework, rather than be informed by experience writing applications. Second, you should do something akin to sketching. You write the core of what is still unknown about the problem domain then test it. If you’re building a blog engine, you start writing the posting and commenting, not the login. Then refine it as you understand more about the domain.

Make tools for yourself that’s informed by the problems you face when you’re trying to get something done. It’s not a waste of time. It’ll make you sharper as a programmer, and you may get better technical or business ideas as a result. It’ll make you more productive, and you’ll be able to delight others once you learn this skill that requires experience and taste.

I knew in college that some dudes were faster than I was in terms of programming. Since peer programming wasn’t exactly encouraged in college, and at work I did mostly prototyping work, I never really knew how fast other programmers worked.

So when I read Paul Graham (and Joel’s) claim that great hackers are at least ten times as productive as average programmers (too lazy to cite right now), I was kinda shocked. Surely, ten times is an order of magnitude! Something that takes an average programmer a 40 hour week to do the great hacker can do in a 4 hour afternoon?

I wondered about that, since there are times when I get stuck on something, then I just start stabbing around randomly out of frustration. I had assumed that great hackers were faster only because they had either the experience or insight to side-step whatever I was doing wrong.

But lately, I’ve been re-reading everyone’s essays that write about programming productivity. And one thing that caught my eye the second time around was when Paul Graham was talking about bottom up programming and how he didn’t really believe in objects, but rather, he bent the language to his will. He was building new blocks for himself so he could think about the problem at a higher level of abstraction.

This is basic stuff! I mean, that’s the whole point of higher-level programming. When you refactor methods out, you’re creating a vernacular so that you can express the problem in terms of the problem domain, not in terms of general computing. This is much like if you want to drive a car, you’d want to be able to step on the gas, rather than time the firings of the pistons. And if you want to control traffic in a city, you’d rather tell all cars to go to a destination, rather than stepping on the gas and turning the steering wheel for each one.

But taken into the light of bending a language to your will, it makes it more clear for me as to how great hackers are ten times as productive. Great hackers are productive not only because they know what problems to sidestep and can problem solve systematically and quickly, but they also build a set of tools for the problem domain as they go along. They are very good pattern recognizers and will be able to generalize a particular pattern of code, so that they can use it again. But not only that, great hackers will create an implicit understanding attached to the abstraction, ie. what we might call common sense.

A case in point. Before Ruby, I’d used for loops over and over again, never really thinking that I could abstract a for loop. It wasn’t until they were taken away in Ruby did I realize that map, inject, and their cousins are all abstractions of the for loop. When I see “map” I know that it performs a transformation on every element. But I also know that the array I get back will be the same size, that each element operation doesn’t affect other elements, among other things. These are implicitly stated, and they allow for shorter code.

When that happens, you can simply read “map”, and get all the connotations it comes with, and hence it comes with meaning. It also becomes easier to remember, since it’s a generalized concept that you can apply in different places in the code. The more times you use it, the easier it is to remember, instead of having specialized cases of the same kind of code where the behavior is different in different parts of the code.

A great hacker will take the initial time upfront to create this generalized code, and will save in the long run being able to use it. Done over and over again, it all adds up. So it’s not that for any given problem, a great hacker will be done in 4 hours what it takes an average programmer 40 hours, but that over time, a great hacker will invest the time to create a tools and vocabulary that lets him express things easier. That leads to substantial savings in time over the long haul.

I hesitated writing about it, as it’s nothing I (nor you) haven’t heard before. But I noticed that until recently, I almost never lifted my level abstraction beyond what the library gave me. I would always be programming at the level of the framework, not at the level of the domain. It wasn’t until I started writing plugins for rails extracted from my own work and reading the Paul Graham article that a light went off for me. It was easier to plug things like act_as_votable together, rather than to still mess around with associations (at the level of the framework). I still believe you should know how things work underneath the hood, but afterwards, but all means, go up to the level of abstraction appropriate for the problem domain.

DSLs (Domain specific languages) are really just tool-making and vernacular creation taken to the level of bending the language itself. It’s especially powerful if you can add implicit meaning to the vernacular in your DSL. It’s not only a way of giving your client power in their expression, but it’s also a refactoring tool so that you can better express your problem in the language of the problem domain. Instead of only adding methods to your vernacular, you can change how the language works. It was with this in mind that I did a talk on DSLs this past weekend at the local Ruby meetup. First part is on Dwemthy’s Array, and the second is using pattern matching to parse logo. Both seemed pretty neat when I first read about it. Enjoy!

I’ve been using Ruby most everyday for about two years now. While I’m no expert, I know enough to be fairly productive in it. And beyond liking the succinctness and power that you often hear other people talk about, it’s made me a better programmer. But there’s an aspect of Ruby that worries me somewhat.

To start, programming is recognized rightfully as a means to build something from pure thought. But it’s also a form of communication, to other programmers that will touch your code later, and to yourself when you look at it months from now. We’re at a point that other than embedded and spacecraft programming, we have the luxury of using programming languages that focus ease for the programmer, rather than for the ease of the machine. Fundamentally, that’s the philosophy that Ruby takes.

And while Ruby’s nice in a lot of ways, I’m not sure about how it communicates an object’s interface. When you’re allowed to modify objects and classes on the fly, how do you communicate interfaces between modules you mixin and methods/modules you add? By interface, I mean, how do you use this class so that it does what it’s suppose to? Normally, it’s pretty obvious–you look at the names of the methods declared in the code. A well-written class has public methods exposed, or you look at its ancestor’s public methods. You might need some documentation to figure out how to call them in the right order, but generally, you have some idea just by looking at the method signatures.

However, when you throw mixins and metaprogramming in the mix, it becomes less easy to tell just from looking at the method signatures in the code–the structure of the code. You have to specifically read the code, or you have to rely on someone who knew intent to document it in detail.

An example communicating interfaces for mixins: the module Enumerable contains a lot of the Collections related methods. The cool thing is that if you wanted these functions in your own class, all you have to do is define each() in your class, mixin the Enumerable module, and you get all of these “for free”. However, outside of documentation explicitly stating it, it’s not as immediately obvious in method signatures that this is what you have to do in order to use it. It’s only after scanning through the entire code that you notice each() being used for all the methods.

Of course, Ruby contains enough metaprogramming power to protect yourself against this. one can do something like this:

This only works if you put the include after you define each(). That’s just asking for trouble when the order of your definitions in your class matter.

A fair number of people are writing mini-DSLs in ruby using metaprogramming tricks. One of the common ones is the use method_missing to define or execute methods on the fly. ActiveRecord’s dynamic finds are implemented this way. The advantage of communication of interface here in the structure of the code is obvious. Unless it was documented well, you can’t tell just by looking at the method signatures.

Why do I harp on interface signatures? I mean, in the instance of requiring each(), it works by just letting it fail in the enumerated methods, since it’ll complain about each itself. In the instance of method_missing, just read the regex in the body. While these are true, none of these allow for rdoc to generate proper documentation. The whole point of documentation is to show you the interface–how to use that piece of code. I’m just afraid that given Ruby’s philosophy of being able to write clear, powerful, and succinct code, it might fall short when people start using these metaprogramming tricks like alias_method_chain and method_missing more and more. Maybe rdoc needs to be more powerful and read code bodies for regex in method_missing?. It already documents yields in code bodies, but that seems awfully specific.

I’m not a exactly a fan of dictating interfaces like in Java. When you’re first coding something up, you’re sketching, so things are bound to change. Having plumbing like interface declaration gets in the way, imo. However, when something’s a bit more nailed down, it’d be nice to be able to communicate to other programmers your intent without them having to read code bodies all the time.

In the end, I side on flexibility. However, I kinda wish Ruby had some type of pattern matching for methods so I didn’t have to read method_missing all the time. But then again, that would be messy in all but the simplest schemes. Can you imagine a class that responded to email addresses as method calls? I guess I’d have to file this one under “bad ideas”

The power of Ruby lies partially in how one can reopen classes to redefine them. Besides namespace clashes, this is usually a good way to extend and refine classes to your own uses. However, last night, I got bitten in the ass trying to refactor a couple classes. In Rails, you’re allowed to extend associations by adding a class the association call.

So when refactoring and cleaning up, I renamed StoryAssociationExtensions to AssociationExtensions and reopened up Story class and put it in there. I just wanted to clean up the namespace, and put the association extensions somewhere that made semantic sense. Naturally, I thought putting association extensions for a class belongs in a class. Well, it doesn’t work. And don’t do it. Hopefully, I’m saving you some pain.

Well, this works if you’ve reopened the class within the same model file, story.rb in this case. However, if you reopen the class in another file elsewhere, your model definition won’t get loaded properly, which leads to associations and methods you defined not to exist.

So imagine my bewilderment when associations didn’t work on only certain ActiveRecord Models. In addition, they worked on the unit tests and script/console, but didn’t work when the server was running. All that at 3am in the morning. 😦

Good thing for source control, so I could revert (but I have to say, svn isn’t as easy to use as it could be).

I ended up creating a directory in model called collection_associations and putting the associations in there under a module CollectionAssociations namespace. Not exactly the best arrangement but it’ll do for now.

I’m still not sure why ActiveRecord::Base instances don’t like being reopened, but I’m guessing it has something to do with only getting loaded once. If anyone has an explanation, I’d like to read about it.

When I started writing some code recently, I noticed that my controllers were getting fat. There was much to do, but there was a bunch of stuff in there that didn’t have anything to do with actually carrying out the action–things like sending notifications. ActiveRecord already has observers to take action on certain callbacks. However, what I needed was to take actions on certain state transitions. Not seeing any immediate solutions in the Rails API, I decided to test myself and try writing one. I was bored too. So while I’m not sure if it was worth the time writing it, it certainly was kinda interesting. Here’s what I came up with:

Just as a contrived example, let’s say we are modeling the transmission of a car. It has three modes: “park”, “reverse”, “drive”. We want to send a notification when a user tries to change it from “reverse” to “drive”, but not when he tries to change it from “park” to “drive”. If it didn’t matter, and we just wanted to send notifications when the state changed to drive, we’d just use the observers that came with ActiveRecord. But since we do care where the state transition came from, here’s what I came up with:

So where’s the magic? It took a bit of digging around. There were two major things I had to do. I had to insert observers during initialization and I had to override setting of attributes to include an update to notify observers.

ActiveRecord doesn’t exactly allow you to override the constructor. I don’t think I tried too hard to mess around with it. Looking on the web, I happened upon has_many :through again, where he has some good tips that helped me through Rail’s rough edges. While I didn’t exactly follow his advice, I did find out about the call back, :after_initialize. It must be something new, because I don’t see it in the 2nd edition of the Rails book, and the current official API doesn’t list it. Other Rails API manuals seem to be more comprehensive, like RailsBrain and Rails Manual.

Then overridding attributes has always been a bit of a mystery. I found a listing of the attribute update semantics, which was helpful to figure out what I was looking for, but it was false, in that you can’t use the first one (article.attributes[:attr_name]=value) to set an attribute. Looking in the Rails code for 1.2.3, it shows that attributes is a read_only hash. But it’s right that you should override the second one (article.attr_name=value), since update_attribute() and update_attributes() depends on it.

I had a difficult time figuring out how to define methods for an instance of a class. The only thing I came up with was to use define_method, or to include a module with instance methods in them. instance_eval() didn’t work. The meta programming for ruby gets rather confusing when you’re doing it inside a method–it seems hard to keep track of which context you’re in.

So if you can make a use of this, great. If you think it’s worth moving it into a plugin, let me know that too. If you know of a better way, by all means, let me know. tip!

What are game rules? When browsing the rules of popular games like Monopoly and Scrabble, they seem to follow a similar format:

The initial conditions of the game (the setup)

Then given a condition,

the set of allowable actions for the player to do

the effects of the condition

I embarked on somewhat of the same problem, though on a much smaller scale. I wanted to implement a point system that gave points to those that participated in a Rails app. My first reaction was to make a Rule abstract class, and have different rule classes that subclass it. Something like this (variable names have been changed to protect the innocent):

Then I’d be able to call it from the controllers. However, on second thought, it’s rather ugly, since I’d be updating the karma everywhere in the controllers. If I understand cross-cutting concerns correctly, scoring karma would be a good example of one. I suppose it’s a good candidate for aspect orientated programming, so I scrapped the code above.

Logging is often cited as the poster-child problem to solve with AOP. Logging needs to be done everywhere in the code, but it really has nothing to do with the responsibilities of the class that it’s performed in. So you have the same code doing the same thing, duplicated everywhere because there’s no one place to put it to make things easy to change.

By the same token, game rules and scoring are of the same nature. And because game rules involve lots of different objects at once, and scoring is interspersed throughout, I think that makes it a good candidate for AOP. However, Ruby has no such direct support for AOP. Instead, the closest thing we have are observers, before/after/around filters (in Rails), and some meta-programming.

I wanted something that allowed me to list out rules like games like Monopoly and Scrabble. I’d have a setup, and some conditions and their effects. Scoring is simplified here because the only time you can score is when one of the models is created or changes state. This is a good fit to the observers and the filters available in Rails.

Here, the Rules module is what encapsulates the setup, rule, and rule_dispatch calls. I needed setup so that I can access different “game elements” (the board and the players) to update the scoring. It basically stores the setup as a list of lambdas that it can execute at a later time when the rule needs to be executed. Now, when a model is created, we ask the rule dispatcher to figure out which rules execute based on the rules we’ve named, and then execute the attached block. The block is passed a hash of different game pieces that it needs to update the score and the game conditions. That’s it.

I thought it was an interesting way about it and probably warranted some criticism. Is there any particular disadvantage of doing it this way? And if you can think of a way to not have to explicitly state the model relationships in the setup, that’d be nice. half-tip!

In any web application, we’re often just reading a collection of rows from the database and displaying it in the browser. Often times, we’ll have code that looks like this:

No friends yet

I don’t know why, but this kinda gets to me, and doesn’t look all that neat. I probably attribute it to having to upgrade and maintain a piece of C server code that was nested 8 or 9 layers deep all in one huge main(). It might be counter-productive, but I tried to see if I could do better.

No friends yet

Well, this is kinda nice in a way that it’s only one hierarchy deep. When I look at it, one section is for what to display when there are elements in the list, and one is for when there isn’t. I suppose your mileage may vary. However, I didn’t like the “if” in front. It obscures the intent of displaying the list. So, in the pursuit of more counter-productivity and perhaps in the spirit of pseudo-altering the language, I tried this out:

No friends yet

Well, that worked. I kinda like it. Since the message was so simple, I had wanted empty() to take a message, and just display it, but because it’s a “%” and not a “%=”, the message won’t get displayed, so I had to do it in a block. In a way, it’s almost like being able to write my own “else” statement. If I had used curly braces instead of “do/end”, it might look pretty close. Here’s the code for empty:

Optimization isn’t something you should do too early on, but I think a little house cleaning every so often to make sure your pages aren’t ridiculously slow is healthy. With any optimization task, you’d want to benchmark the results and see if there’s an actual gain. The very basic tool for benchmarking is the ordinary script/performance/benchmark. The easiest to find analysis tools is the rails_analyzer gem. The last time I used rails analyzer, it wasn’t that easy to use. The command line arguments seemed arcane. But its bench tool, which can benchmark controllers as opposed to just object models, is fairly easy to use.

In the listing of books, one would display whether it’s actually bookmarked by a user or not. Normally, without the :include, the listing would make repeated queries to the DB every time it displayed a book list element, since it will use bookmarked_by?(user_id) to determine if a user bookmarked the book. So instead of just 1 query, it would make n + 1 queries.

Preloading child tables isn’t necessarily wise all the time. It really depends on what you intend to do with the data after you fetch it. As the Agile rails book warns, preloading all that data will take time. If you look at your log files, you’ll see that it’s a significant amount.

If you’re only going to load a limited number of these book list elements on a single page at a time, it actually might make sense to forgo preloading of child tables, and just use a find() instead of a select.

And if you’re going to display counts of arrays, but all means, use counter caching. It’s easy to do (as long as you follow instructions!), for most situations.

Intuitively, if you want to display over a certain n number of book list elements, it makes more sense to use :include and select it. However, I wanted to point out that when you make decisions like this, you’ll always want to measure the load times, because you earn what you measure.

Also, use the right number of runs. Too short number of a number of times you run a function, the more variation you’ll have in your benchmarks. Let’s say that you get two numbers for two different methods.

So it’s obvious that method2 is better right? Well, not necessarily. While benchmarks only show averages, you’ll need to pay attention to standard deviations. The bigger the standard deviation, the more runs you’ll need to figure out the average load time, and the number of decimal points you can trust. That way, you can figure out whether the difference in load times is statistically significant or not.

That way, you can ascertain whether the optimization you made were worth the trouble or not. tip!

Javascript has been a surprising language, mostly because the difference between my experience with it now, and when I touched it in high school. Going along with the theme of seeing everything as the proverbial functional nail, there’s been a bunch of things to get use to, such as how scoping work works in javascript.

Javascript is a lexically scoped language (scope determined when function is defined, not when it’s executed). This much, I think I get.

However, unlike Java and C++, “instance methods” of Objects doesn’t include attributes in its scope. One has to explicitly refer to it. So let’s say I have something like the code below. Just something simple, where one wants to process each element in a list. Of course, one can do this with a for loop and it wouldn’t be an issue, but since playing with Ruby, I’ve found that iterators were more clear when writing code. It’s probably bad design, but for the sake of argument, let’s say that tile’s draw() method takes the Renderer object as an argument.

This will not work, because apparently, “this” inside the anonymous function of each() doesn’t refer to renderer, but (I think) to the anonymous function. So draw() will usually complain something about how the renderer doesn’t have the right properties.

This is a bit odd, since this would work in Ruby. The only way I’ve found around this is just to throw “this” into a variable, then refer to that variable.

I was reading an article on Javascript and its use of first class functions. I’d quote it if I could find it again, but basically, it extolled the virtues of being able to create anonymous functions, blocks, and closures easily in Javascript. One of the things that it claimed was that creating local functions only used within a method makes the code much more readable. I was skeptical, but as I was refactoring this function, I thought I’d give it a shot even though I was coding in Ruby. Ruby also allows for the creation of first class functions.

It’s just a helper function that generates a bookmark button–however, what gets generated depends on whether a user is logged in and whether a user had already bookmarked the page.

Well, I thought it was pretty straightforward to read, though half of it was duplicated code. It wasn’t very DRY. So instead of fudging around with another way to structure the control flow, I tried my hand at using blocks. I ended up with this:

As you can see, I put the link_to_remote() code into a block, since the only difference between the two is the label. In addition, it is possible for the block to use variables that are in this method, but outside the block, even if the execution is outside the scope of this function, if the block is ever passed outside of bookmark_button. I know in javascript, this causes memory leaks in IE. I haven’t heard that it’s a problem in Ruby yet.

But in this case, I’m just creating a function inside a method to make things more readable and cleaner. I think for the most part, it worked pretty well. As long as it’s not overdone (notice I didn’t create a proc for the labels), it should make for more readable code, and you don’t pollute the class namespace with methods that are only used in this one method. I should get use to this. tip!