Wednesday, June 14, 2006

You can't design your application protocol in a vacuum. You need to understand how the layers below your application work before you deploy it.

An excellent rule. Actually, I've often heard (and used) a more general form:

You can't be competent doing computer work at level N unless you have a good grasp of level N-1.

Programming is all about abstractions, and we as programmers are fond of thinking that our abstractions mean you "don't need to know" what's under the covers. But abstractions aren't perfect, and if you don't know what's under your current level of abstraction, then you're simply not competent.

For example, if you want to be a good MFC programmer, you need to have a decent grasp of Win32 API fundamentals. If you want to work in Python, you don't need to be a Python core hacker, but you'd better know enough about the implementation to know why, for example, repeated string concatenation is slow. And if you're using a object-relational mapper over top of a relational database, you still need to know your way around SQL.

I first heard this from Dr. Ralph Droms, one of my professors at Bucknell (who also invented DHCP). If I recall correctly, he was quoting one of the "elder statesmen" of computer science (Dijkstra, maybe?), but I can't recall just who it was.

Interestingly, in Cpython 2.4 they've optimized string concatenation when you use +=. Of course, that still points to the fact that you need to know how python is run internally (the += operations, in general, tend to be different).

I still usually resort to ''.join semantics unless there are only a few concatenations (in which case ''.join is actually slower).

I once looked into the string concatentation optimization in some detail, and it turns out that you need a fair amount of things to work out just right before any optimization kicks in. (One starting requirement is that the target string needs to already be longer than about 240 characters.)