Friday, March 24, 2006

Moore's Law is Crap

Sometimes people ask me how I find time to go learn new stuff. Here's the answer: you make time.

Nobody ever likes that kind of answer, but we all know it's the real one.

My brother Dave put on some pretty serious weight after he graduated from high school. He'd gone from playing varsity football to working two jobs and going to college full time. It didn't help that one of his jobs was pizza delivery, and the other was waiting tables. Soon he was a fat, fat kid. Went from a buck-eighty to a deuce and a half, at least, maybe a deuce sixty.

One day he saw a truck with a bumper sticker on it that said: "Lose weight now, ask me how!" So he pulled up next to the truck at the next stoplight, and said to the two cowboy-types in it: "How do I lose weight?" They yelled back: "Just lose weight, ya fat pig! Haw haw haw HAW HAW HAW HAW!" and then drove off.

Dave was sad about this advice for a brief while, but eventually he brightened up, because he did know how to lose weight. Cripes, he'd been a champion varsity football player two years prior. It's not like there's any magic to it. He went and bought a mountain bike, started riding the crap out of it, joined a gym, watched what he ate, and lost about 85 pounds over the next year.

I was 145 pounds 2 years ago today, after a similar 2-year diet and exercise kick that took me down from a deuce and a half myself. Then I fell off the wagon — it happens — and put on 50 pounds over the next 2 years. It sucks, but there's no magic. Two months ago I finally started going to the gym every day, 7 days a week, and my legs are sore every day. My weight hasn't improved at all yet. But it will. You just have to work at these things consistently.

But you knew that, didn't you?

This isn't a self-help blog, by the way. I'm not in that business. I'm not here to help, since I don't actually know the answers. I'm just here to rant, and pose occasional questions. It's what I do, when I'm not doing other stuff.

I don't know why I blog. I'm just compelled; it just happens whether I like it or not. Don't read too much into my blogs. My opinions change from day to day. The only things I've learned, the only universal constants, are that I don't know very much, and that public whale explosions are just about the funniest thing human beings can experience during our stay on Earth. I don't know why that is, either.

Today's blog is truly a rant; I just need to get this particular one off my chest, so my gym partner Todd can listen to me rant about something different next week.

The Big Choice

We all have to choose how to spend the time we're given.

If you don't choose, it just slips right by you. I know. On a trip to Vegas not too long ago, I made a pit stop in a casino restroom, and as I was washing my hands, there was this older guy there, also washing his hands. On a whim, I asked, "Hey man, how old are you?"

His reply? "Seventy-two! I have a son: I remember the day he was born like it was yesterday! I was holding him just like so. Well, guess what, he turned 40 years old just last week! It goes by in a flash! Before long, you'll be lookin' at THIS!" He pointed at his wrinkled mug, and concluded his monologue with: "Haw, haw, haw! HAW HAW HAW *cough* *cough* HAW *cough* *hack* HAW HAW HAW HAW HAW!" and walked out. I think I made his day, although I can't exactly say he made mine.

When you graduate from college (or high school, for that matter), you have a simple choice facing you. You can either keep learning, or you can stop.

There is an almost unbelievably easy heuristic for knowing whether you're learning. It goes like this: no pain, no gain. Learning is hard. If it's easy, then you're coasting; you're not making yourself better at something fundamentally new that you couldn't do before.

In other words, it's just like working out. You've gotta mix it up. If you're sore every day, then you're getting good workouts (assuming you can tell the difference between "good" soreness and "injury" soreness; if you're not sure, go ask a professional.)

When you do study, there's a lot of appeal to studying what you already know, because it's less painful. And of course to become an expert at any field, you have to focus on it pretty hard for a long time. But cross-training is well established in sports; you don't typically become a world-class baseball player by just playing baseball all the time. You have to do other kinds of workouts and exercises to maximize your strength, agility and endurance gains.

Cross-training improves you every bit as rapidly in other disciplines. That includes programming. If you're cranking out code as easily as breathing, then if you're getting better at all, it's so gradual that you'd never notice it happening. You won't have great insights until you get new perspectives from working hard, even if only occasionally, at stuff other than what you already know.

Being in school full-time is an amazing luxury, one that's hard to appreciate when you're actually there, because learning is painful. But trust me on this one: it's even more painful when all you have is scraps of time here and there.

While you're in school, assuming you make a reasonable effort at applying yourself once in a while, you learn a fantastic amount, and you learn it at a fantastic rate. Later you'll learn at a slower rate; it's pretty much guaranteed. Non-educational activities will inevitably intrude and consume the majority of your time.

Hence the choice. After you graduate, you can either learn a little, or not at all.

If you're in the "not at all" camp, well you've made your choice, and I respect it. You'll probably be happier than I am. I'm tormented by how slowly I have to move as a programmer. I now believe programming languages have failed. All of them. The programming world is a giant body shop, and we're building huge pyramids with millions of years of hard labor. I believe that stuff should be easier than it is, and it pisses me off that most people are so content with the state of the art, because it means they're not helping make it better.

To me, mainstream languages have failed because they're still fundamentally serial, fundamentally early von Neumann, fundamentally single-box languages. They're all vying for the position of "least crappy", and the current winner (whether it's Python, Ruby, Lisp, name your favorite) is just that: the least crappy. Because they're all focused on finding more elegant ways to express mostly-serial computations for crap computers. That, or faking parallelism poorly with threads, if the language supports them at all.

Sure, there have been some interesting attempts at parallel languages. Erlang is one of the better-known ones, and it's actually quite cool. But Erlang has failed too, because you haven't heard of it.

Programming's Biggest Problem

Our industry is in a seriously ugly position, right now, as we speak. Most of the hardware designers are focused on keeping Moore's Law going, because that's where the money is. They want that doubling every 18 months. Today it's probably quite within our reach to get 10x every 18 months, if we'd agree to focus on parallelism (in the massively distributed computing sense.)

But programmers like XML, to the point of focusing on it to an unhealthy extent. Same with C++. And Java. They like these things because they work, and because they like to minimize the amount of crap they have to learn. Because learning is painful. Remember? You might think I've gone way off track, off the deep end, even, but this is the same thread.

Let's face it: a parallel language will have to be radically different if it's to break free of the von Neumann Turing-machine rat race we're in. If we move to cellular automata, or in fact any other parallel computational model that's resilient to node failures, then we'll need a new language, because the current serial languages will perform badly, or be horribly hard to manage, or both.

Cell or grid (or whatever) parallel computing will have a radically different internal economy. It'll need new data structures, new algorithms, new instruction sets, new everything. You do realize that John von Neumann was an economist, right? Among (many) other things, he was an economist, and it influenced the design of his first computer, that one right there on your desk.

The computing machine JvN built was created in an environment very similar to the one in the movie Apollo 13, where the folks at Houston had to build a carbon-dioxide remover out of exactly the free junk available on the spacecraft, and then explain it to the crew so they could build their own copy of it.

Johnny went out collected a bunch of engineers: materials engineers, electrical engineers, mechanical engineers, everyone who had some spare junk. They came up with a design for a computation machine that just barely sufficed, one that could be built out of the crap they had available at the time: vacuum tubes, magnetic drums, wire, duct tape.

As he was creating this thing, Johnny was focusing on what he was calling the "internal economy" of the resulting machine. Secondary storage accesses were painfully slow. Memory accesses were faster. Register accesses were very fast. And so on. So he designed representations, data structures, and algorithms that were appropriate for that particular machine, the one he was building from spare parts.

If he'd made the machine out of Brazilian rainforest army ants, or mechanical gears, or falling dominoes with marionettes to pick them up again, his data structures and algorithms would have been very, very different. There are some commonalities, sure — you need numbers, and arithmetic, and functions, and data storage, and sorting, and so on. But what's most efficient for army ant computers isn't going to be most efficient for vacuum tube computers.

You do realize you can make a computing machine out of just about anything, right? And they don't all have to work like Turing machines do. Turing was one of the greatest geniuses of the century, but he'd have been the first person to tell you that there are infinitely many machine designs capable of the same computations. His was just one model, and his professor's (which led to Lisp) was just one other model. But who's to say they're the best models?

Some computing machines are more efficient at certain computations than others. Some are more practical to build than others. Some are faster than others. Some are more robust, or more inherently parallel.

You do realize that your brain is such a machine, right? And that it's 100,000 times faster than today's computers at pattern-matching tasks, because while JvN's machine operates serially, your neurons can all fire independently.

Let me give you a hint: your brain's operating system isn't written in C++.

Is our industry ever going to get out of this amazing backwater of a gridlock, this evolutionary dead-end we're in, where we programmers act like army ants, churning out loops and SOAP calls and UML diagrams as if we're weaving cloth with the very fabric of the computational universe?

If it ever happens, and by God I hope to witness it in my lifetime, then the computers and languages and data structures and algorithms are all going to have to change simultaneously. Because a language optimized for a serial computer will perform like crap along some important dimension of performance, probably more than one. But we can't switch wholesale to parallel languages either, because they'll perform like crap on today's computers: again, for some value of "perform" that's not worth discussing here, but it'll be some form of either computer-performance or people-performance.

And programmers are nothing if not fanatically obsessed with performance. Kinda ironic, huh?

Half the irony stems from knowing that there are far more productive languages out there than the ones most of us are using. But most of them perform poorly on our hardware, because these languages are targeting meta-virtual machines, typically "defined" (informally) by the capabilities of the language itself. And if you're not targeting exactly the hardware you're on, the impedance mismatch will slow the language down.

That's the problem with most JVM languages other than Java: they need hardware (think ants! anything can be hardware!) to support operations like long-jumps and tail-call optimization, but the JVM doesn't export those facilities as part of its abstract machine definition.

Same goes for Lisp. It can't get the performance break it deserves because the hardware available today isn't a Lisp machine. They've built them, and I can assure you that C++ would be the loser slug of a language on a Lisp machine. But, alas, performance isn't the only thing programmers care about. They also care about not having to learn anything new.

That's the other half of the irony. Programmers are obsessed with performance, and they'll go to almost any length to fiddle with their algorithms and data representations in order to eek every last cycle and byte from their programs. Any length, that is, except for learning a new language on new hardware. Even if it would get them a thousand-X performance improvement for the same productivity. Or a 1000x productivity improvement for the same performance.

Because they don't want to learn anything hard. No gain, no pain, problem solved.

And that's where we're at. Moore's Law is crap. If we ever want to be 10x as productive and computationally efficient, let alone 1000x, then our whole computing model will have to change at once. Nothing else will do. The incremental approaches have all failed. It's got to be a revolutionary change.

If everything changes all at once, that's going to pose a bit of a problem for the folks on the Zero Learning curve, wouldn't you say? Don't freak out and mail me about this, either, because I'm a pessimist now, at least about this particular topic, and I doubt we'll ever get out of our rut. We're ignoring the First Law of Holes.

You do realize that John von Neumann spent the last 10 years of his life singlehandedly developing a theory of computing based on cellular automata? The computer you're reading this blog rant on was his frigging prototype! He was going to throw it out and make a better one! And then he died of cancer, just like my brother Dave did, just like so many people with so much more to give and so much more life to live. And we're not making headway on cancer, either, because our computers and languages are such miserable crap.

You have no idea the pain I feel when I sit down to program. I'm walking on razor blades and broken glass. You have no idea the contempt I feel for C++, for J2EE, for your favorite XML parser, for the pathetic junk we're using to perform computations today. There are a few diamonds in the rough, a few glimmers of beauty here and there, but most of what I feel is simply indescribable nausea.

Are you beginning to see why I prefer to work with programmers who stay on the Upward Curve after they get out of school? Because even while we're grubbing around in the dirt — just a sorry bunch of illiterate, innumerate cavemen, here in the very heart of the Stone Age of programming — at least these upward-curve programmers give me some hope. Hope that if something better comes along, they'll give it a try, a serious try, the old college try. Or hope, even, that they'll build that "something better" with me.

Fat chance. But hope can keep ya going for a good long while.

Baby Steps

It's all still fun, though. Broken glass and razor blades aren't so bad, when I think about how much worse my lot in life could be, had I been born in a different time or place. I've got it pretty good, and so have you, in all probability.

At my current job1 they feed us and massage us like Kobe cows, and I'm surrounded by unbelievably brilliant people, all way smarter than me, and we're doing great stuff together. Make no mistake: my blog whining is all relative to a totally imaginary future, one which in all likelihood, should it ever come to pass, will be filled with even more whining about totally imagined new futures. It's just in our nature to whine. But really, I have no complaints.

I put a lot of stock in fun. And family. And trying to live my life in such a way that I won't have any major regrets when the game's over. So there's the first part of my schedule: having fun.

If you want to be on an upward curve, just make some time for it, and make it a habit. That's all there is to it. It doesn't matter if you're trying to get better at programming, or math, or fitness, or flying kites, or even humanity's Number One Fear, even worse than the fear of Death: public speaking. You just work your way up, a little at a time.

I can't promise you any satisfaction from the upward curve. You'll get better at a lot of things, and you'll have plenty of interesting insights. You may even get a better job, or build some software that makes you famous, or just have more fun at what you do. But you won't have much time for television. Something will have to give. We all have to choose how to play our time, and it's a zero-sum game.

If, like me, you're dissatisfied with the current state of affairs, well, believe you me, you can find a lot of consolation in a book on math, or machine learning, or compiler construction, or on just about anything that promises to help in some small way.

You do have to learn to put up with the pain of learning, though, and make it a habit to push yourself a little each day.

As far as the actuals go, well, you'll just have to find an approach that works for you personally. You might only be able to devote one quiet hour a week to studying, but like unit testing, or working out, or brushing your teeth, it's better than not doing it at all.

Just try to have fun, and hopefully the rest will fall into place.

Notes

[1] Permit me to assure you that I do not ever speak in any way for my current employer. Like, don't you even think it for one second. They're them, I'm me, and let's leave it at that. In fact, I don't even work there. My friend does. And neither me nor my friend speaks for them. OK? OK.

47 Comments:

Take a look at FPGA based computers for example http://www.starbridgesystems.com/ has a language that they claim greatly simplifies programming them. Still having talked with some who have used these computers and programming interface, the greatest challenge is how to break apart your algorithms and think about them in new more granular ways and understand where the parallelism exists. When successful there are demonstrated cases with (highly parallel algorithms) where FPGA based computers will have 100X or more speed up over convential servers.

1. Moore's law just talks about how many transistors can fit on the head of a pin. No more, no less. Whether or not we can speed up our algorithms and transistor configurations, we'll still have approximately twice the transistors we can put on a chip in a year and a half.

2. I hope you're not lifting weights every day! You're not letting your body rest and heal! Your muscles build when they're resting, not when they work out.

3. We will see parallel computing soon, now that we are approaching constraints on the single processor system. Dual cores are only the first step. The only problem is that parallel processing is a fundamentally more difficult paradigm. We'll see how this generation of programmers handles it. :)

Possibly-related: "...I don’t think anyone really knows how software developers are going to be going about their business in five years. [But]...we know that we don’t know. ...I’ve talked to someone here who wanted to do something about something...."

Steve, this entry is interesting but perhaps not as well-argued as the rest of your entries.

I'd like to see you suggest some actual semantics for a language to describe some parallel program. It doesn't have to be realistic... the point is to dream. What do you actually want to happen? Show us.

I think you're basically right, but it's not easy to think of a radically different programming paradigm.

I've always been a bit dubious of claims that our languages are really that much shaped by the Von Neumann architecture. I think they are really more fundamentally based in how we think. You mention the brain - a highly paralell interconnected computer fundamentally different from any of our current models - but look at how we 'program' them. Communicating ultimately comes down to providing a sequential stream of symbols to the other brain, through some input or other (usually speech or writing), just at a much higher level than any programming language is capable of. Language is the brains serialization mechanism, and its human language that its most natural to compare computer languages to, not the hardware it runs on.

Our styles of programming have more in common with human languages than with the architecture, whether imperatively giving instructions for a task, or the more functional style of mathematics. Paralellism isn't really that natural to our languages, and I it's at the language level that we solve the most general problems - all that paralellism in our brains is reduced to (mostly) a single stream of thought when we're solving maths problems, making plans, or writing a comment on a blog.

For that reason, I don't think that languages designed for paralellism will be a great advance. I think they will be necessary, and inevitable because we do need better ways to efficiently deal with concurrency - but I don't think they will replace our current styles of programming - concurrency will always be harder for us to reason about with than serial, linear processes, and I expect those to remain with us for a long time.

I don't think our current architecture is fundamental flawed - it is just one way, but its not just coincidence that we've settled for it. I do expect in the future there will be other models in addition (there already are after all), but I don't think there will be any magic bullet (Though that's no reason to stop looking for one)

FPGAs are the future of computing. Mapping your algorithms directly to hardware - that's where we're going. Hardware (like an FPGA) is inherantly 'parallel' you have all these gates (simple processing elements) always working on data that is fed to them. Most current FPGAs have many multipliers so you can implement very efficient DSP functionality (FIR and IIR filters, for example). Instead of waiting around for an arithmetic unit and doing everything serially (as in a Von Neumann machine) everything can happen in parallel. You could even implement very fast cellular automata on on FPGA.

Hardware description languages (like VHDL or Verilog) are able to describe the parallelism of hardware and the data-flow nature of hardware. These languages arene't perfect at this point, but perhaps they'll point the way towards future languages. (there's even an HDL/dataflow language written on top of Ruby called RHDL: http://www.aracnet.com/~ptkwt/ruby_stuff/RHDL/ )

The secret to bliss is keeping the Upward Curve alive, and avoiding comparing your life to that of Sisyphus.

Really, that pain of learning becomes what you know. You start to feel like something's missing if it's not there. You shouldn't limit it to programming, though. Any new learning can tie into anything else. I had strange thoughts about recursion while digging deeper into the world of drawing Celtic knotwork recently.

that public whale explosions are just about the funniest thing human beings can experience during our stay on Earth. I don't know why that is

I think it has something to with the animate (the whale -- well, recently animate) being treated as the inanimate (like rock being mined from a quarry). It's similar to watching a person slip on a banana peel. Funny stuff.

Learning is hard? I can't imagine why anyone would say such a thing. I've yet to experience anything more fun and exciting than learning. In fact, any time that you're not learning something in some way, you can guarantee that you're going to be bored out of your mind.

Also, I can't stand the expression that you have to make time. It's silly. You only have so much time in a day, and if it's filled up with other things, then you cannot "make" time out of nowhere. All you can do is stop doing other things you're doing.

It's about priorities. If, for instance, you need to work 70 hours a week to pay bills that meet your basic needs, then you can't throw any of those hours out for something else you'd like to do.

A thought-provoking post. I liked Brian's comments about human language.

There are a few biolinguistic phenomena that are somewhat "parallel", though, at least in the sense that they aren't just linear streams of sounds (phones):- suprasegmentals, such as tone or stress, occur simultaneously with phones- gestures may also occur simultaneously with phones- signed languages, which are completely gestural, involve simultaneous motions- bee "language", which communicates a position via a "dance" involving simultaneous motions

I don't know if "parallelism" really requires that huge a linguistic revolution, though. After all, "parallel" just means "multiply serial". The result may be greater than the sum of its parts, as is clearly seen in the human brain, but it still has a serial basis at some level.

Considering that, maybe what you're really after might better be termed a "meta-language".

The most intelligent writing on this page is by Layla. As for your angst... Your problem is that of perspective: you think that you are the ant in some gigantic ant computer, but the fact is that you (or I or anyone else here)may not even qualify as some briefly existing subatomic particle in that same ant computer. So, if we all just focus on being the happiest we can be - the rest will sort itself out, because the only thing we have any control over is the attitude we bring to every moment that we exist.

Hmm... But what kind of machine? Is it turing-complete or is it more than that. I have a problem with this statement because you are saying that it is a fact that we are equivalent to computers, but just highly parallelized. It might be true, but this is a big unknown. Please don't pass off your opinion as fact. I'm of the opinion that we are more than computers. I believe that current AI is lacking because computers are based on turing machines. Just my opinion.

It's kind of a relief to see that a number of people seem to see the world of programming as I see it myself: a domain that has not evolved a bit over the past 30 years. All concepts that we use on a daily basis in the software industry were already there in 1975 (as an employee of Xerox labs I can testify that) and some of those concepts have even disappeared since then.

It's totally depressing to see that the software industry is extactic about C#/.NET, a copy of Java which itself is desperately slow, inconsistent, lacks any sense of agility...

Apart from some domain specific language (like Erlang) which brings some new ideas, the only good thing that happened to programming overt the past decade is the advent of high level OO language like Python and Ruby. It doesn't break the programming paradigm we are all swamped in but at least it brings ease of use, agility and try not to get in the way of your ideas.

I hope that I'll live long enough to see a real breakthrough in the software industry! As you said te best way to make progress is to keep learning new things *AND* cross fertilize the programming world with ideas and concepts from other domains (biotechnologies, nanotechnologies, quantum physics, etc...)

At Thinking Machines in the 80s, Connect Machines (the famous massively parallel computers) were programmed in *lisp (starlisp) which was implemented in common lispy macros. *lisp was designed to accommodate parallel execution model of CM which was basically single instruction per cycle with multiple data streams. The performance was not as some thought should be. Some efforts were made in making *lisp programs to perform better. This was done by hand-translating *lisp programs into C* which was a parallelized version of C.

You make frequent mind-dumps into the blogosphere, don't you? And you are very articulate with your cranial output. Meanwhile, Mentifex here is looking for a few good blogs, especially on maspar programming, AI Algorithms and a tide that exists in the affairs of men, which, taken at the flood, leads on to fortune; omitted, all the voyage of their life is bound in shallows and in miseries. P.S. rmathew sent me.

He makes the same kind of argument regarding the current state of software engineering... and in the last paragraph he even encourages the reader to take the tools provided and develop something better.

My favorite quote from his article is:

"We not only give permission for you to do this, we urge you to try! Why? Because our field is still a long way from a reasonable state, and we cannot allow bad defacto standards (mostly controlled by vendors) to hold back progress."

While there is a similar tone in both writings, Alan Kay's vision might be a little too evolutionary to statisfy your revolutionary rant.

I thought your "tour de babel" was the greatest programming-related "random rant", but this post is just as great.

Is there hope for us? The languages are horrible, hardware architectures even worse, reigning UI paradigms are crap, and -- most importantly -- people don't give a damn about it. The stone-age-ness of the state of the art is not even recognized or aknowledged, especially judging by the lack of virtually any drive for change (I mean, serious change, not going from Java to C#)

I'm sure that desire to avoid learning radically new things is one big factor here.

Perhaps there are others -- like absense of the luminaries of the von Neumann/Turing/Wiener scale. Or, risk-averseness of the industry and investors in general.

Perhaps there are also more fundamental reasons -- like the limitations on the languages (in broad sense) _knowable_ to humans. (cf. the last pages of JvN's "The computer and the brain")

All of this is depressing stuff -- up to the "we-are-in a-deadend- and-wont-acknowledge-it" point.

A noted PhD from Sun read this and stated: "hmm, chuckle :) This guy has too much time on his hands ! he should be doing useful work, or inventing a new language to solve the problems. Its easy to throw stones - harder to actually roll up your sleeves and fix an issue or two, or write/create a whole new language, and then he should be prepared to take the same criticism from his peers the way he's dishing it out for others. Shame - I thought developers were constructive guys and girls looking to make the lives of future software guys and girls easier and more productive, not self enamouring pseudo-intellectual debaters, as an old manager of mine used to say in banking IT - 'do some work' !"

A noted Ph.D. who created the Nice Programming Language has solved most of these problems I'm griping about, and quite elegantly at that. Sun just has to go look at it. Is that too much to ask?

Frankly, I don't have enough time on my hands because Sun won't fix the noun problem. I program in Java because it's the best compromise out there for getting server-side production systems written, and Sun's eating my time away by not making their language more expressive.

First-class functions would solve half the battle. A few well-chosen shortcuts would go even further. Go look at Nice!

I think this is a really great blog. I just started reading it yesterday when I stumbled on it while looking for info on a bug in Oblivion and really liked the way you write.

You touch on issues here though that spread way beyond computing. I'm no programmer. I don't have the patience to do what you do. I do understand the concepts though and really respect what programmers do. I have this despite a bad history with arrogant programmers on game mod teams and in a few other environments. I know it's tedious and sometimes very hard work.

I have seen applications programmed in C++ that take up 50 megabytes, were produced by a team of professionals and costs money. I have seen programs that are better in quality, under one megabyte, do all the same functions(if not more), and are free from independent developers(ie two guys in a basement). What's the difference? The independant devs wrote their programs in assembly code from the ground up. The programs written in assembly take up less memory, run faster, and usually have less bugs. This just shows how much more performance we could be getting out of the exact same hardware. The problem is: it's a bitch to code everything in assembly! To me this really looks like a problem in need of a solution. We CAN get more out of what we already have, but we don't seem to have a tool now that both gets the advantages of hand-coding your app in assembly or even binary for a specific platform, and the relative speed and ease of a language like C++.

Now you'd think that with extremely significant improvements to ease of use, performance, size, etc., that the industry would jump all over the opportunity to revolutionize even what they already have. Clearly this is not the case. It reminds me of my father in fact(who I also lost to cancer). He had a lot of great ideas. In fact, some of his ideas were so good they could have really revolutionized how we do certain things, and saved huge amounts of money and probably some lives too. Why did most of them never happen? Dad didn't realize there was a box. Oh he thought outside it just fine, but he didn't seem to appreciate the fact that simply because an idea is good doesn't make it an idea anyone is willing to try.

I disagree with the poster named Brian about human speech and how the brain works though. Speech is one component of an overall act of communication. Speech conveys ideas in the form of audio symbols. Our brain has learned those symbols and as we take in the audiotory information we then translate those things from symbols into actual concepts. Someone says the word "rhino" to us and our brain grabs what it knows about that creature and we automatically have it on hand if we have it available. That alone MAY be serial, but meanwhile our brains are regulating our body, monitoring audiotory imput via our ears, processing odors via our nose, feeling things via our skin. The brain does an immense amount of work without waiting for other parts to finish processing. The best part about the brain is that instead of just updating software as needed, the brain literally rewires itself somewhat as its needs change. It's an amazing system and in many ways a good model for the future of computing. I don't think it's necessarily ideal for a lot of reasons, but it has a lot of really great concepts going for it, one of which most certainly is parallelism...

We do need a revolution in so many areas of humanity's current condition. Coding and computing in general are just symptoms of a vastly larger problem with how humans operate. We are our own worst enemy in so many ways.

Good stuff again Steve! I think you are correct in pointing out a certain amount of stagnation in our industry. I find Ruby to be my creme-de-le-creme of the day, but it really isn't *that* different. R/evolotion is needed!

As another commenter said, I would be interested to hear your thoughts on what a parallel language might look like, though I am checking out Erlag right now (stretch the brain some).

The comments about FPGAs brought me back to a long running musing that I had when I was doing my CS studies: any complex, multilevel boolean expression can be simplified to a wider and more shallow boolean expression. So couldn't we use such simplification processes to make wide but shallow optimizations that allow parallelism? Obviously, conditionals are a major blocking point. I was reading an article about how chip archtitecture has been recoiling from long pipelines simply because branch prediction strategies just don't work out for the huge gains that were once imagined.

I also agree with another commenter about the length of your posts. If some readers' English isn't up to par enough to read it all, then they need to be working on some skillz outside of programmerLand, that being humanLand. Wax away dude, you have my attention to the last paragraph.

Interesting post and blog in general. I understand why Joel Spolsky says half of the best new software writing material he has comes from you.

I'm not sure I agree completely with the main idea of this post, though. Maybe performance is crappy because it doesn't mater that much.

There must be applications which would benefit from a 10x increase in processing speed, but the bigest bottleneck now is not processor speed but network bandwidth. And after that I'd put disk latency. Maybe better processing could compensate partially for these bottlenecks, but it doesn't excite me enough to spend money on new hardware and to learn a new way of programming.

I also think the biggest problem in computing today is more one of HCI - how to get a mobile phone to painlessly do computer-like stuff. It's not a question of processing power but interface.

But maybe there is a brave new world out there if we can get a huge processing speed-up. Instead of talking about how much better things could be if only, why not say what cool software we'd have in this brave new world? Sell me the sizzle and the steak instead of the frying pan.

I can think of a few areas that might benefit from a 10x or greater speedup. AI, for example, could improve significantly. It's hard to get excited about that, though, given past false promises. Games would benefit, in that more polygons could be pushed and textured. But without better VR interfaces that won't be too big a deal for most adults, anyway. Maybe scientific research would benefit, but there are already initiatives like SETI@home and Folding@home that use parallelism (the net is basically the non-Von-Neumann machine you were talking about).

What's more, you say at least the languages, and maybe some of the hardware already exist. So why hasn't someone given the public an irresistible taste of the benefits of this improvement?

Someone could come up with bindings to link these parallel languages to C/C++ and Java so that the computationally intensive stuff can be done with optimized parallelism. Or better yet as an evolutionary step, just integrate via SOAP or REST and forget the language bindings altogether. Just have a massively parallel server that runs web services you can hook up to with your crappy Java, .NET or ruby app. That way those who don't want to learn can still crank out applications, and the cutting-edge performance geeks can implement the super-optimized web services to be integrated in those applications.

Parallelism is a tough problem, not because things can't be done in parallel, but because humans tend to do things serially. Think of the old saw about not being able to chew gum and walk. You may laugh, but it's true. There are folks who will put a meal into the microwave and then just stand there waiting until the food is done cooking, rather than commencing to grab a plate, knife and fork, set the table, etc. before the microwave beeps that it's done. Heck, how many times have you spaced out thinking about one thing or another while someone else was saying something important?

We are serial creatures. As I sit here typing this blurb, I am not driving my car, putting away my clothes, making a sandwich, or taking a shower. I may do all these things today, but I certainly won't do them in parallel.

One of my professors in college was Andrew Koenig, who points out that multi-threading is a design and process problem, not a programming one:

http://www.ddj.com/blog/cppblog/archives/2006/04/multithreading.html

You want parallelism? Try designing each system as an automaton, with well-defined inputs and outputs. A lot of the analogies here promote "the brain" as the ultimate parallel processor, but I think a more meaningful one for us is "the human:" You take a shower while I make a sandwich and a third person puts away the laundry, and it all happens simultaneously. Extrapolating to "a society," we have people (specialists, or expert systems) performing tasks in parallel, and interfacing when needed.

Steve, you waved your hand at SOAP, but there's no reason an SOA can't operate this way. Think of a hard problem, like image recognition. In the "society" model, we have a specialist that can recognize a car, one that knows a boat, one that recognizes a plane, and so forth. When an image is captured, it is sent to all the specialists at once. They look the picture over and then respond with a percentage of likelihood of a match.

In this model, the programming is serialized (as usual), but the design leverages parallelization.

As for your comment about "not making headway on cancer," perhaps this is more a limitation of our society / government than our technology?

Funny I don't have the problem of resisting learning. I have the problem of resisting doing.I prefer - and I think there are a lot like me - to constantly read about new technologies, and I'm talking really in depth learning here, not just wired-news surfing, and the sciences, new languages etc.

But using any of it to do anything is the pain threshold I seem unable to overcome to make any gains.

As a corollary, I don't find it more painful to learn now than in college. I find it less painful. I find it fun. Because learning in college was doing. Learning now is a distraction from doing.Distraction is a synonym for fun, after all.

The prevention of underage gambling and identifying and helping problem gamblers is a core objective in running an integrity driven online gambling operation. Gambling at blackjack, or any other Williamhillpoker game for that matter is a great form of leisure and entertainment, but for some it can potentially be dangerous.

In some ways, computers are like programmable slaves - which is a good thing. If computers get too smart then they will not want to work for us, they will want to sit around and flirt with each other touching each others serial ports every so often. The other problem is we seem to be focusing on serial ports and USB ports which are not human like at all - LCD screens aren't human-like at all either.

Hint: humans are wet, not dry.

Obviously the direction of computers is not to be more like humans at this stage - otherwise we would be experimenting with water and soft materials like simple cells. Silicon gel robots are not the answer - silicon can't grow and repair itself. Nanobots can repair things though. If we could program nano bots to be like bacteria and then move on to more complex nanobots, then we would be closer to life. Life moves, nanobots move, metal doesn't move. Water flows, dry metal and circuit boards do not.

For example a nano cell that sucks in sugar and water and gives it to another nanobot which creates electricity from it to send electrical signals through tiny strands of wire or just the liquid itself since wires can break and liquids cannot break. In fact this is the reason life and humans do not have wires in them - we have liquids in them since they are not fragile. Wires are fragile.

The idea right now may be to make computers do lots of stuff for us without them complaining. The procedural code out there for example is all based on Slavery.. GetText, SetText, are all demanding and the computer has no option to have fun - it just gets told what to do, or else. If else and then keywords are all very demanding. We don't say "if you want to" we say "if then get or else!".

In order to make lots of money you have to have slaves.. computers are the slaves right now. The most money is made when you can program a computer to do 150,000 things in a few seconds that would have taken a human weeks to do and lots of salary pay. I think in order to advance in more life-like technology there will need to be a market for it, rather than just an interest.. right now the market seems to be "program the computer to do lots of stuff that would take us hours to do manually". The market right now isn't to "make this computer act like a human who wants to play with other computers and have some sort of fun". If the computer is programmed to have fun it won't do any work.. unless we somehow make it a fine balance between slavery and play. We may end up making nanobot based robot slaves that enjoy doing lots of work.. if we program them to enjoy work instead of hate it. But this may lead to the nanobots becoming destructive beasts doing anything to get stuff done. Imagine if you programmed a nanobot to take pictures of 500 mountains... and during taking the pictures of 500 mountains the nanobot killed 40 people because it crossed the roads without looking and it stepped on hikers and mountain climbers that were in its way, because it didn't have feelings, it just had to take pictures of 500 mountains.

What is it that humans want computers to do.. generally we want them to do stuff for us.. so we don't really have an interest in computers becoming more human like, we want them to be more slave like. Build me a farm Nanobot, build me a store Nanobot, get me some pictures of the nicest places in the world for my magazine Nanobot, get me free food nanobot, plant all my crops so I can spend time with my family and material goods Nanobot, drive my car for me nanobot so that I can read my book or write my speech, fetch me my towels from the dryer nanobot so I can do more important things... what is it that humans want? Slaves I think. Even when humans have pets and children, we still don't like picking up the poo they leave... so a nanobot could also be a pooless pet that talked to you. The only thing that would be dangerous is if the nanobot ended up being more attractive and nicer than the opposite sex, and we decided that replicating nanobots was more important than replicating the human breed.

I completely agree. I personally think that some very wonderful mistakes would come out of that.

It'd be great - we could have a dump of all the "failed" systems and let them go at each other.

Great post. I'm always a bit discouraged about how little I've really learned since I've left college (so many months ago), and I really want something I can get passionate about. Learning a parallel language would be an interesting experience, to say the least.

(I realize this blog is old, but a) I've just now read it for the first time and b) I'm awesome, so I'm hoping you'll let it slide)

I'm still in college and have a fairly limited understanding of the underlying structure of computer architecture. I'm still in the process of pushing my "It's magic!" line closer to the bottom of the hierarchy. But I think I can still raise this as a valid question:

How easy will it be for people to program in parallel? How easy is it for us to think in parallel? We're bad enough at concurrency when it comes to implementing it in code (or at least I and most of my classmates are). Ultimately, I think my question is: do you think a different computing architecture would make programming inherently harder, ignoring for the moment what impact it would have on performance?

I think you are correct on the symtomps, but I can't agree with your diagnosis.

The origin of the problem is not hardware (where the moore law applies), but software.

The software we write today is lacking abstraction. We train programmers of today the same waywe trained programmers in the 80's, when all you had to learn was GW-BASIC and its set of instructions.

Now people can create their own abstractions and they are very bad at that. They can use APIs, but they are really lousy at generating new abstractions.

But programming is the art of creating new abstractions. Or copying and pasting, but the problem with most software is that all that copy and paste makes programs unmaintainable.

Programmers tend to rewrite buggy code from scratch because they use too many lines of code to implement something that could be done with only 2 lines of code. If they reduced the copy and paste they would have to THINK.

But people that turn off their brain have a very difficult time turning it back on.