meme: an idea, behavior, or usage that spreads from person to person within a culture agora: a gathering place

Welcome to Neal's gathering place for ideas.

Friday, April 25, 2008

The Productive Programmer - History & Future

No, I haven't fallen off the face of the earth. I haven't been blogging lately because I've been heads-down, finishing (finally!) The Productive Programmer. Understanding why it took so long requires some recent history.

Back in 2005, after having fully recovered from writing the Art of Java Web Development in 2003, I decided to look for a fresh book project. I started wondering if the world could use another book on Regular Expressions, other than the seminal work on the subject by O'Reilly. I asked a few people and the universal opinion was a resounding "No!": the current book is so good that it's pointless to try to write another. I was talking about it to David Bock, friend and No Fluff, Just Stuff colleague when I visited NovaJUG, the Java User's Group he facilitates in the DC area. He agreed with everyone else, and we started chatting about what we thought would make a good book. We had both recently observed developers struggling to make an IDE do something that was trivial on the command line, but the developers had spent their entire careers in graphical operating systems and IDE's, and couldn't bash their way out of a paper bag. We decided that it would be cool to write a book about command line tricks and techniques for developers, and that we would write said book. The original idea was to write a gigantic recipe book for programmer productivity. We contacted a publisher (who suggested the title The Productive Programmer), and we started gathering recipes and writing about them.

Then, 2 things happened. David left his job of many years and founded a consultancy, Code Sherpas, and instantly became 300% allocated. At about the same time, his wife Lorna became pregnant with triplets! Clearly, David now has his hands more than full. At about the same time, I realized that a book consisting solely of recipes would be the most boring book every written. But, by that time, I had also noticed some patterns in the productivity stuff in which I was now immersed. In fact, I remember the exact moment I realized that this book should show patterns (or principles) of productivity. I was in India, working on a ThoughtWorks project by day, and writing the book by night. I had some long conversations with some of the folks with which I was working (Hi, Mujir!) about this, and finally settled on the five principles of productivity (which later became 4). Instead of being a book of fish, I now had a fishing manual.

Of course, I've been busy as well, so the work on the book was intermittent at best. Plus the fact that the publisher was really expecting a recipe book, and it was no longer that. So, the publisher and I started going back and forth, wrestling over the format and content to try to mold it into something upon which we could agree. By this time, I had kind of taken over the book because David was so slammed. Eventually, I realized that no amount of massaging was going to create the book the publisher thought they wanted, so the publisher and I decided to part friends. I now had distilled the 5 principles down to 4 (I realized at some point that indirection is really just an aspect of canonicality) and written about them. While I was writing, I was also doing Productive Programmer talks at No Fluff, Just Stuff, honing the material that was perpetually going to be out RSN (Real Soon Now). I now had about 100 pages of refined material. But I also realized that the principles really only covered one aspect of developer productivity, the mechanical side. I also realized that I had been talking about the practical side of productivity in my older talk Clean Up Your Code and its modern sequel 10 Ways to Improve Your Code. That was the last piece. I contacted my now publisher O'Reilly with 1/2 of a productivity book and a clean vision for what the second half should be. So, from December until March this year, I wrote the 2nd half. And, fortunately, David agreed to write the forward of the book, which gave a pleasing resolution to the original vision we both had.

For all those long suffering attendees of my Productive Programmer talks at No Fluff, Just Stuff for the last 2 years: the book is ready. It's in copy editing now, and should be out soon. RSN. For real this time.

This is a thanks to all the people who've attended the talk over the last couple of years. Your questions and responses made the book better, and now you can hold one in your hands. And, now that I'm back from intensive writing, ready to reengage the blogsphere with a vengeance.

This is exciting news. Certain friends of mine have been talking about doing a command-line help and have started an open-source/creative-commons project to compare and describe (in something akin to english) Linux and Solaris CLI stuff. The wiki is started at..

Good book , Neal... I just got finished with it. As one of the hackers of yore, introduced to C and Unix in '76, the "small is beautiful" perspective rings true in your writing. It's all solid advice.

Just as a historical aside, the derivation of grep (as global regular expression print) is right on except for the editor - it came from ed and not ex. The former was written by Ken Thompson at BTL, while ex came from Berkeley and became vi. I remember Jim Seagraves first showing me vi at the labs in the early 80's - it wasn't that steep a learning curve if you knew ed. It inspired me to learn curses and write other screen-oriented tools from form entry to status displays. Ah, the good old days...