Wednesday, September 27, 2006

Good Agile, Bad Agile

Scrums are the most dangerous phase in rugby, since a collapse or improper engage can lead to a front row player damaging or even breaking his neck. — Wikipedia

When I was growing up, cholesterol used to be bad for you. It was easy to remember. Fat, bad. Cholesterol bad. Salt, bad. Everything, bad. Nowadays, though, they differentiate between "good" cholesterol and "bad" cholesterol, as if we're supposed to be able to distinguish them somehow. And it was weird when they switched it up on us, because it was as if the FDA had suddenly issued a press release announcing that there are, in fact, two kinds of rat poison: Good Rat Poison and Bad Rat Poison, and you should eat a lot of the Good kind, and none of the Bad kind, and definitely not mix them up or anything.

Up until maybe a year ago, I had a pretty one-dimensional view of so-called "Agile" programming, namely that it's an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who've never read "No Silver Bullet", the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don't know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.

You know. Chumps. That's the word I'm looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

But I've had a lot of opportunity to observe various flavors of Agile-ism in action lately, and I now think I was only about 90% right. It turns out there's a good kind of Agile, although it's taken me a long time to be able to see it clearly amidst all the hype and kowtowing and moaning feverishly about scrums and whatnot. I have a pretty clear picture of it now.

And you can attend my seminar on it for the low, low price of $499.95! Hahaha, chump!

No, just kidding. You'll only find seminars about the Bad kind of Agile. And if in the future you ever find me touring around as an Agile Consultant, charging audiences to hear my deep wisdom and insight about Agile Development, you have my permission to cut my balls off. If I say I was just kidding, say I told you I'd say that. If I then say I'm Tyler Durden and I order you not to cut my balls off, say I definitely said I was going to say that, and then you cut 'em right off.

I'll just go right ahead and tell you about the Good Kind, free of charge.

It's kinda hard to talk about Good Agile and Bad Agile in isolation, so I might talk about them together. But I'll be sure to label the Good kind with a happy rat, and the Bad kind with a sad dead rat, so you'll always know the difference.

The Bad Kind

Back in Ye Olden Dayes, most companies approached software development as follows:

- hire a bunch of engineers, then hire more.- dream up a project.- set a date for when they want it launched.- put some engineers on it.- whip them until they're either dead or it's launched. or both.- throw a cheap-ass pathetic little party, maybe. This step is optional.- then start over.

Thank goodness that doesn't happen at your company, eh now? Whew!

Interestingly, this is also exactly how non-technical companies (like, say, Chrysler) handled software development. Except they didn't hire the engineers. Instead, they contracted with software consultants, and they'd hand the consultants 2-year project specs, and demanded the consultants finish everything on time plus all the crap the customer threw in and/or changed after signing the contract. And then it'd all fall apart and the contractors wouldn't get paid, and everyone was really miffed.

So some of the consultants began to think: "Hey, if these companies insist on acting like infants, then we should treat them like infants!" And so they did. When a company said "we want features A through Z", the consultants would get these big index cards and write "A" on the first one, "B" on the second one, etc., along with time estimates, and then post them on their wall. Then when the customer wanted to add something, the consultant could point at the wall and say: "OK, boy. Which one of these cards do you want to replace, BOY?"

So the consultants, now having lost their primary customer, were at a bar one day, and one of them (named L. Ron Hubbard) said: "This nickel-a-line-of-code gig is lame. You know where the real money is at? You start your own religion." And that's how both Extreme Programming and Scientology were born.

Well, people pretty quickly demonstrated that XP was a load of crap. Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it. The rationale was something like: "well if ONE programmer sitting at a terminal is good, then TEN must be better, because MORE is ALWAYS better! But most terminals can only comfortably fit TWO programmers, so we'll call it PAIR programming!"

You have to cut them a little slack; they'd been dealing with the corporate equivalent of pre-schoolers for years, and that really messes with a person.

But the thing is, viruses are really hard to kill, especially the meme kind. After everyone had gotten all worked up about this whole Agile thing (and sure, everyone wants to be more productive), there was a lot of face to be lost by admitting failure. So some other kinds of Agile "Methodologies" sprang up, and they all claimed that even though all the other ones were busted, their method worked!

I mean, go look at some of their sites. Tell me that's not an infomercial. C'mon, just try. It's embarrassing even to look at the thing.

Yeah. Well, they make money hand over fist, because of P.T. Barnum's Law, just like Scientology does. Can't really fault 'em. Some people are just dying to be parted with their cash. And their dignity.

The rest of us have all known that Agile Methodologies are stupid, by application of any of the following well-known laws of marketing:

- anything that calls itself a "Methodology" is stupid, on general principle. - anything that requires "evangelists" and offers seminars, exists soley for the purpose of making money. - anything that never mentions any competition or alternatives is dubiously self-serving. - anything that does diagrams with hand-wavy math is stupid, on general principle.

And by "stupid", I mean it's "incredibly brilliant marketing targeted at stupid people."

In any case, the consultants kept going with their road shows and glossy pamphlets. Initially, I'm sure they went after corporations; they were looking to sign flexible contracts that allowed them to deliver "whatever" in "2 weeks" on a recurring basis until the client went bankrupt. But I'm equally sure they couldn't find many clients dumb enough to sign such a contract.

That's when the consultants decided to take their road show to YOU. Why not take it inside the companies and sell it there, to the developers? There are plenty of companies who use the whip-cycle of development I outlined above, so presumably some of the middle managers and tech leads would be amenable to hearing about how there's this low-cost way out of their hellish existence.

And that, friends, was exactly, precisely the point at which they went from "harmless buffoons" to "potentially dangerous", because before they were just bilking fat companies too stupid to develop their own software, but now the manager down the hall from me might get infected. And most places don't have a very good quarantine mechanism for this rather awkward situation: i.e., an otherwise smart manager has become "ill", and is waving XP books and index cards and spouting stuff about how much more productive his team is on account of all this newfound extra bureaucracy.

How do we know it's not more productive? Well, it's a slippery problem. Observe that it must be a slippery problem, or it all would have been debunked fair and square by now. But it's exceptionally difficult to measure software developer productivity, for all sorts of famous reasons. And it's even harder to perform anything resembling a valid scientific experiment in software development. You can't have the same team do the same project twice; a bunch of stuff changes the second time around. You can't have 2 teams do the same project; it's too hard to control all the variables, and it's prohibitively expensive to try it in any case. The same team doing 2 different projects in a row isn't an experiment either.

About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations. But where does the data come from? Companies aren't going to give you their internal data, if they even keep that kind of thing around. Most don't; they cover up their schedule failures and they move on, ever optimistic.

Well if you can't do experiments and you can't do proofs, there isn't much science going on. That's why it's a slippery problem. It's why fad diets are still enormously popular. People want fad diets to work, oh boy you bet they do, even I want them to work. And you can point to all these statistically meaningless anecdotes about how Joe lost 35 pounds on this one diet, and all those people who desperately want to be thinner will think "hey, it can't hurt. I'll give it a try."

That is exactly what I hear people say, every time a team talks themselves into trying an Agile Methodology. It's not a coincidence.

But writing about Bad Agile alone is almost guaranteed to be ineffective. I mean, you can write about how lame Scientology is, or how lame fad diets are, but it's not clear that you're changing anyone's mind. Quitting a viral meme is harder than quitting smoking. I've done both. In order to have the right impact, you have to offer an alternative, and I didn't have one before, not one that I could articulate clearly.

One of the (many) problems with Bad Agile is that they condescendingly lump all non-Agile development practices together into two buckets: Waterfall and Cowboy. Waterfall is known to be bad; I hope we can just take that as an axiom today. But what about so-called Cowboy programming, which the Agileers define as "each member of the team does what he or she thinks is best"?

Is it true that this is the only other development process? And is Cowboy Programming actually bad? They say it as if it's obviously bad, but they're not super clear on how or why, other than to assert that it's, you know, "chaos".

Well, as I mentioned, over the past year I've had the opportunity to watch both Bad Agile and Good Agile in motion, and I've asked the teams and tech leads (using both the Bad and Good forms) lots of questions: how they're doing, how they're feeling, how their process is working. I was really curious, in part because I'd consented to try Agile last Christmas ("hey, it can't hurt"), and wound up arguing with a teammate over exactly what metadata is allowed on index cards before giving up in disgust. Also in part because I had some friends on a team who were getting kind of exhausted from what appeared to be a Death March, and that kind of thing doesn't seem to happen very often at Google.

So I dug in, and for a year, I watched and learned.

The Good Kind

(cue happy rat)

I'm going to talk a little about Google's software development process. It's not the whole picture, of course, but it should suffice for today. I've been there for almost a year and a half now, and it took a while, but I think I get it now. Mostly. I'm still learning. But I'll share what I've got so far.

From a high level, Google's process probably does look like chaos to someone from a more traditional software development company. As a newcomer, some of the things that leap out at you include:

- there are managers, sort of, but most of them code at least half-time, making them more like tech leads.

- developers can switch teams and/or projects any time they want, no questions asked; just say the word and the movers will show up the next day to put you in your new office with your new team.

- Google has a philosophy of not ever telling developers what to work on, and they take it pretty seriously.

- developers are strongly encouraged to spend 20% of their time (and I mean their M-F, 8-5 time, not weekends or personal time) working on whatever they want, as long as it's not their main project.

- there aren't very many meetings. I'd say an average developer attends perhaps 3 meetings a week, including their 1:1 with their lead.

- it's quiet. Engineers are quietly focused on their work, as individuals or sometimes in little groups or 2 to 5.

- there aren't Gantt charts or date-task-owner spreadsheets or any other visible project-management artifacts in evidence, not that I've ever seen.

- even during the relatively rare crunch periods, people still go get lunch and dinner, which are (famously) always free and tasty, and they don't work insane hours unless they want to.

These are generalizations, sure. Old-timers will no doubt have a slightly different view, just as my view of Amazon is slightly biased by having been there in 1998 when it was a pretty crazy place. But I think most Googlers would agree that my generalizations here are pretty accurate.

How could this ever work?

I get that question a lot. Heck, I asked it myself. What's to stop engineers from leaving all the trouble projects, leaving behind bug-ridden operational nightmares? What keeps engineers working towards the corporate goals if they can work on whatever they want? How do the most important projects get staffed appropriately? How do engineers not get so fat that they routinely get stuck in stairwells and have to be cut out by the Fire Department?

I'll answer the latter question briefly, then get to the others. In short: we have this thing called the Noogler Fifteen, named after the Frosh Fifteen: the 15 pounds that many college freshmen put on when they arrive in the land of Stress and Pizza. Google has solved the problem by lubricating the stairwells.

As to the rest of your questions, I think most of them have the same small number of answers.

First, and arguably most importantly, Google drives behavior through incentives. Engineers working on important projects are, on average, rewarded more than those on less-important projects. You can choose to work on a far-fetched research-y kind of project that may never be practical to anyone, but the work will have to be a reward unto itself. If it turns out you were right and everyone else was wrong (the startup's dream), and your little project turns out to be tremendously impactful, then you'll be rewarded for it. Guaranteed.

The rewards and incentives are too numerous to talk about here, but the financial incentives range from gift certificates and massage coupons up through giant bonuses and stock grants, where I won't define "giant" precisely, but think of Google's scale and let your imagination run a bit wild, and you probably won't miss the mark by much.

There are other incentives. One is that Google a peer-review oriented culture, and earning the respect of your peers means a lot there. More than it does at other places, I think. This is in part because it's just the way the culture works; it's something that was put in place early on and has managed to become habitual. It's also true because your peers are so damn smart that earning their respect is a huge deal. And it's true because your actual performance review is almost entirely based on your peer reviews, so it has an indirect financial impact on you.

Another incentive is that every quarter, without fail, they have a long all-hands in which they show every single project that launched to everyone, and put up the names and faces of the teams (always small) who launched each one, and everyone applauds. Gives me a tingle just to think about it. Google takes launching very seriously, and I think that being recognized for launching something cool might be the strongest incentive across the company. At least it feels that way to me.

And there are still other incentives; the list goes on and ON and ON; the perks are over the top, and the rewards are over the top, and everything there is so comically over the top that you have no choice, as an outsider, but to assume that everything the recruiter is telling you is a baldfaced lie, because there's no possible way a company could be that generous to all of its employees, all of them, I mean even the contractors who clean the micro-kitchens, they get these totally awesome "Google Micro-Kitchen Staff" shirts and fleeces.

There is nothing like it on the face of this earth. I could talk for hours, days about how amazing it is to work at Google, and I wouldn't be done. And they're not done either. Every week it seems like there's a new perk, a new benefit, a new improvement, a new survey asking us all if there's any possible way in which life at Google could be better.

I might have been mistaken, actually. Having your name and picture up on that big screen at End of Quarter may not be the biggest incentive. The thing that drives the right behavior at Google, more than anything else, more than all the other things combined, is gratitude. You can't help but want to do your absolute best for Google; you feel like you owe it to them for taking such incredibly good care of you.

OK, incentives. You've got the idea. Sort of. I mean, you have a sketch of it. When friends who aren't at Google ask me how it is working at Google — and this applies to all my friends at all other companies equally, not just companies I've worked at — I feel just how you'd feel if you'd just gotten out of prison, and your prison buddies, all of whom were sentenced in their early teens, are writing to you and asking you what it's like "on the outside". I mean, what would you tell them?

I tell 'em it's not too bad at all. Can't complain. Pretty decent, all in all.

Although the incentive-based culture is a huge factor in making things work the way they do, it only addresses how to get engineers to work on the "right" things. It doesn't address how to get those things done efficiently and effectively. So I'll tell you a little about how they approach projects.

Emergent Properties versus The Whip

The basic idea behind project management is that you drive a project to completion. It's an overt process, a shepherding: by dint of leadership, and organization, and sheer force of will, you cause something to happen that wouldn't otherwise have happened on its own.

Project management comes in many flavors, from lightweight to heavyweight, but all flavors share the property that they are external forces acting on an organization.

At Google, projects launch because it's the least-energy state for the system.

Before I go on, I'll concede that this is a pretty bold claim, and that it's not entirely true. We do have project managers and product managers and people managers and tech leads and so on. But the amount of energy they need to add to the system is far less than what's typically needed in our industry. It's more of an occasional nudge than a full-fledged continuous push. Once in a while, a team needs a bigger nudge, and senior management needs to come in and do the nudging, just like anywhere else. But there's no pushing.

Incidentally, Google is a polite company, so there's no yelling, nor wailing and gnashing of teeth, nor escalation and finger-pointing, nor any of the artifacts produced at companies where senior management yells a lot. Hobbes tells us that organizations reflect their leaders; we all know that. The folks up top at Google are polite, hence so is everyone else.

Anyway, I claimed that launching projects is the natural state that Google's internal ecosystem tends towards, and it's because they pump so much energy into pointing people in that direction. All your needs are taken care of so that you can focus, and as I've described, there are lots of incentives for focusing on things that Google likes.

So launches become an emergent property of the system.

This eliminates the need for a bunch of standard project management ideas and methods: all the ones concerned with dealing with slackers, calling bluffs on estimates, forcing people to come to consensus on shared design issues, and so on. You don't need "war team meetings," and you don't need status reports. You don't need them because people are already incented to do the right things and to work together well.

The project management techniques that Google does use are more like oil than fuel: things to let the project keep running smoothly, as opposed to things that force the project to move forward. There are plenty of meeting rooms, and there's plenty of open space for people to go chat. Teams are always situated close together in fishbowl-style open seating, so that pair programming happens exactly when it's needed (say 5% of the time), and never otherwise.

Google generally recognizes that the middle of the day is prone to interruptions, even at quiet companies, so many engineers are likely to shift their hours and come in very early or stay very late in order to find time to truly concentrate on programming. So meetings only happen in the middle of the day; it's very unusual to see a meeting start before 10am or after 4:30pm. Scheduling meetings outside that band necessarily eats into the time when engineers are actually trying to implement the things they're meeting about, so they don't do it.

Google isn't the only place where projects are run this way. Two other kinds of organizations leap to mind when you think of Google's approach: startup companies, and grad schools. Google can be considered a fusion of the startup and grad-school mentalities: on the one hand, it's a hurry-up, let's get something out now, do the simplest thing that could work and we'll grow it later startup-style approach. On the other, it's relatively relaxed and low-key; we have hard problems to solve that nobody else has ever solved, but it's a marathon not a sprint, and focusing requires deep concentration, not frenzied meetings. And at the intersection of the two, startups and grad schools are both fertile innovation ground in which the participants carry a great deal of individual responsibility for the outcome.

It's all been done before; the only thing that's really surprising is that Google has managed to make it scale.

The scaling is not an accident. Google works really hard on the problem, and they realize that having scaled this far is no guarantee it'll continue, so they're vigilant. That's a good word for it. They're always on the lookout to make sure the way of life and the overall level of productivity continue (or even improve) as they grow.

Google is an exceptionally disciplined company, from a software-engineering perspective. They take things like unit testing, design documents and code reviews more seriously than any other company I've even heard about. They work hard to keep their house in order at all times, and there are strict rules and guidelines in place that prevent engineers and teams from doing things their own way. The result: the whole code base looks the same, so switching teams and sharing code are both far easier than they are at other places.

And engineers need great tools, of course, so Google hires great people to build their tools, and they encourage engineers (using incentives) to pitch in on tools work whenever they have an inclination in that direction. The result: Google has great tools, world-class tools, and they just keep getting better.

The list goes on. I could talk for days about the amazing rigor behind Google's approach to software engineering. But the main takeaway is that their scaling (both technological and organizational) is not an accident. And once you're up to speed on the Google way of doing things, it all proceeds fairly effortlessly — again, on average, and compared to software development at many other companies.

The Tyranny of the Calendar

We're almost done. The last thing I want to talk about here is dates. Traditional software development can safely be called Date-Oriented Programming, almost without exception.

Startup companies have a clock set by their investors and their budget. Big clients set target dates for their consultants. Sales people and product managers set target dates based on their evaluation of market conditions. Engineers set dates based on estimates of previous work that seems similar. All estimation is done through rose-colored glasses, and everyone forgets just how painful it was the last time around.

Everyone picks dates out of the air. "This feels like it should take about 3 weeks." "It sure would be nice to have this available for customers by beginning of Q4." "Let's try to have that done by tomorrow."

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.2) Grad school projects.3) Google.

Most people take it for granted that you want to pick a date. Even my favorite book on software project management, "The Mythical Man-Month", assumes that you need schedule estimates.

If you're in the habit of pre-announcing your software, then the general public usually wants a timeframe, which implies a date. This is, I think, one of the reasons Google tends not to pre-announce. They really do understand that you can't rush good cooking, you can't rush babies out, and you can't rush software development.

If the three exceptions I listed above aren't driven by dates, then what drives them? To some extent it's just the creative urge, the desire to produce things; all good engineers have it. (There are many people in our industry who do this gig "for a living", and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are better than that.)

But let's be careful: it's not just the creative urge; that's not always directed enough, and it's not always incentive enough. Google is unquestionably driven by time, in the sense that they want things done "as fast as possible". They have many fierce, brilliant competitors, and they have to slake their thirsty investors' need for growth, and each of us has some long-term plans and deliverables we'd like to see come to fruition in our lifetimes.

The difference is that Google isn't foolish enough or presumptuous enough to claim to know how long stuff should take. So the only company-wide dates I'm ever aware of are the ends of each quarter, because everyone's scrambling to get on that big launch screen and get the applause and gifts and bonuses and team trips and all the other good that comes of launching things with big impact at Google.

Everything in between is just a continuum of days, in which everyone works at optimal productivity, which is different for each person. We all have work-life balance choices to make, and Google is a place where any reasonable choice you make can be accommodated, and can be rewarding. Optimal productivity is also a function of training, and Google offers tons of it, including dozens of tech talks every week by internal and external speakers, all of which are archived permanently so you can view them whenever you like. Google gives you access to any resources you need in order to get your job done, or to learn how to get your job done. And optimal productivity is partly a function of the machine and context in which you're operating: the quality of your code base, your tools, your documentation, your computing platform, your teammates, even the quality of the time you have during the day, which should be food-filled and largely free of interrupts.

Then all you need is a work queue. That's it. You want hand-wavy math? I've got it in abundance: software development modeled on queuing theory. Not too far off the mark, though; many folks in our industry have noticed that organizational models are a lot like software models.

With nothing more than a work queue (a priority queue, of course), you immediately attain most of the supposedly magical benefits of Agile Methodologies. And make no mistake, it's better to have it in software than on a bunch of index cards. If you're not convinced, then I will steal your index cards.

With a priority queue, you have a dumping-ground for any and all ideas (and bugs) that people suggest as the project unfolds. No engineer is ever idle, unless the queue is empty, which by definition means the project has launched. Tasks can be suspended and resumed simply by putting them back in the queue with appropriate notes or documentation. You always know how much work is left, and if you like, you can make time estimates based on the remaining tasks. You can examine closed work items to infer anything from bug regression rates to (if you like) individual productivity. You can see which tasks are often passed over, which can help you discover root causes of pain in the organization. A work queue is completely transparent, so there is minimal risk of accidental duplication of work.

And so on. The list goes on, and on, and on.

Unfortunately, a work queue doesn't make for a good marketing platform for seminars and conferences. It's not glamorous. It sounds a lot like a pile of work, because that's exactly what it is.

Bad Agile in More Detail

I've outlined, at a very high level, one company's approach to software development that is neither an Agile Methodology, nor a Waterfall cycle, nor yet Cowboy Programming. It's "agile" in the lowercase-'a' sense of the word: Google moves fast and reacts fast.

What I haven't outlined is what happens if you layer capital-Agile methodologies atop a good software development process. You might be tempted to think: "well, it can't hurt!" I even had a brief fling with it myself last year.

The short answer is: it hurts. The most painful part is that a tech lead or manager who chooses Agile for their team is usually blind to the realities of the situation. Bad Agile hurts teams in several ways.

First, Bad Agile focuses on dates in the worst possible way: short cycles, quick deliverables, frequent estimates and re-estimates. The cycles can be anywhere from a month (which is probably tolerable) down to a day in the worst cases. It's a nicely idealistic view of the world.

In the real world, every single participant on a project is, as it turns out, a human being. We have up days and down days. Some days you have so much energy you feel you could code for 18 hours straight. Some days you have a ton of energy, but you just don't feel like focusing on coding. Some days you're just exhausted. Everyone has a biological clock and a a biorhythm that they have very little control over, and it's likely to be phase-shifted from the team clock, if the team clock is ticking in days or half-weeks.

Not to mention your personal clock: the events happening outside your work life that occasionally demand your attention during work hours.

None of that matters in Bad Agile. If you're feeling up the day after a big deliverable, you're not going to code like crazy; you're going to pace yourself because you need to make sure you have reserve energy for the next big sprint. This impedance mismatch drives great engineers to mediocrity.

There's also your extracurricular clock: the set of things you want to accomplish in addition to your main project: often important cleanups or other things that will ultimately improve your whole team's productivity. Bad Agile is exceptionally bad at handling this, and usually winds up reserving large blocks of time after big milestones for everyone to catch up on their side-project time, whether they're feeling creative or not. Bad Agile folks keep their eye on the goal, which hurts innovation. Sure, they'll reserve time for everyone to clean up their own code base, but they're not going to be so altruistic as to help anyone else in the company. How can you, when you're effectively operating in a permanent day-for-day slip?

Bad Agile seems for some reason to be embraced by early risers. I think there's some mystical relationship between the personality traits of "wakes up before dawn", "likes static typing but not type inference", "is organized to the point of being anal", "likes team meetings", and "likes Bad Agile". I'm not quite sure what it is, but I see it a lot.

Most engineers are not early risers. I know a team that has to come in for an 8:00am meeting at least once (maybe several times) a week. Then they sit like zombies in front of their email until lunch. Then they go home and take a nap. Then they come in at night and work, but they're bleary-eyed and look perpetually exhausted. When I talk to them, they're usually cheery enough, but they usually don't finish their sentences.

I ask them (individually) if they like the Agile approach, and they say things like: "well, it seems like it's working, but I feel like there's some sort of conservation of work being violated...", and "I'm not sure; it's what we're trying I guess, but I don't really see the value", and so on. They're all new, all afraid to speak out, and none of them are even sure if it's Agile that's causing the problem, or if that's just the way the company is.

That, my friends, is not "agile"; it's a just load of hooey. And it's what you get whenever any manager anywhere decides to be a chump.

Good Agile Should Drop the Name

I would caution you to be skeptical of two kinds of claims:

- "all the good stuff he described is really Agile" - "all the bad stuff he described is the fault of the team's execution of the process"

You'll hear them time and again. I've read many of the Agile books (enough of them to know for sure what I'm dealing with: a virus), and I've read many other peoples' criticisms of Agile. Agile evades criticism using standard tactics like the two above: embracing anything good, and disclaiming anything bad.

If a process is potentially good, but 90+% of the time smart and well-intentioned people screw it up, then it's a bad process. So they can only say it's the team's fault so many times before it's not really the team's fault.

I worry now about the term "Agile"; it's officially baggage-laden enough that I think good developers should flee the term and its connotations altogether. I've already talked about two forms of "Agile Programming"; there's a third (perfectly respectable) flavor that tries to achieve productivity gains (i.e. "Agility") through technology. Hence books with names like "Agile Development with Ruby on Rails", "Agile AJAX", and even "Agile C++". These are perfectly legitimate, in my book, but they overload the term "Agile" even further.

And frankly, most Agile out there is plain old Bad Agile.

So if I were you, I'd take Agile off your resume. I'd quietly close the SCRUM and XP books and lock them away. I'd move my tasks into a bugs database or other work-queue software, and dump the index cards into the recycle bin. I'd work as fast as I can to eliminate Agile from my organization.

And then I'd focus on being agile.

But that's just my take on it, and it's 4:00am. Feel free to draw your own conclusions. Either way, I don't think I'm going to be an Early Riser tomorrow.

Oh, I almost forgot the obvious disclaimer: I do not speak for Google. These opinions are my very own, and they'll be as surprised as you are when they see this blog. Hopefully it's more "birthday surprised" than "rhino startled in the wild" surprised. We'll see!

very interesting post. I work at a govt agency (not software development) who's general approach to projects is not so far removed from what you describe. the biggest missing ingredient is financial incentives, but we seem to get buy pretty well on praise and an attentive external audience.

Academia in general is also a bit like what you describe - there are no research deadlines (even the conferences and grant application dates come around every year) but the carrots (mostly tenure, but also the implicit ranking system engendered by the publish or perish approach) keep everyone working furiously.

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.2) Grad school projects.3) Google.

Yep, and why is that? It's because most of us in our industry are writing sotware for paying customers (who might happen to share an employer with us) who have a "soft real time" idea of the value of the features we build: ie, the value of the features is at a maximum at some time t and declines, perhaps rapidly, after that. These three kinds of development are all quite unlike this.

Google's products and services seem mostly to exist for the purpose of making online advertising through Google more attractive, and that's a continuous process.

My client's projects generally exist to make some business process 1) possible, 2) faster, 3) higher quality before the competition do that. The business sponsors of these projects want to know when the benefit is going to start acruing. So they need schedules. Rightly or wrongly, that's what they need. And that's why (in contrast to the named methods that camed before) the Agile methods tend to put an emphasis on early delivery of business value, and hence the focus on schedule. In XP, though, note that we would rather reduce scope that slip a delivery.

In a different context, different approaches are requried: how well do you think the Google approach would fit (to pull an example from the air) adapting a trading system to handle new kind of derivative? Or making a new kind of revenue earning service avilable on a mobile network? Beleive me, "it'll be done when it's done" is not an acceptable answer when the sponsor can see many millions of dollars (a chunk of which is headed for their bonus...or not) being lost if the opposition gets into the market far ahead of them.

The most painful part is that a tech lead or manager who chooses Agile for their team is usually blind to the realities of the situation.

Ah. See, if a manager chooses anything an foists it on their team, it's going to be a struggle to get it to go well. But when teams choose themselves to work in a certain way, it can really sing.

I don't doubt that the Google approach is very enjoyable for the developers, and is a good fit for Google's business model--but there are a lot of other businesses wokring in other business models. For many (not all) of them, agile with any size of "A" can be a big step up from what they've got.

Well, people pretty quickly demonstrated that XP was a load of crap. Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it.

Really? Since adopting Agile methods myself in the late 1990's I've worked with five teams on three continents, all of which pair and continue to pair because they find it beneficial to do so. YMMV, of course, but where's your contrary data?

you've got a point though. appropriate organizational/management style seems very business model dependent, with google's model being among the outliers. but maybe there's an argument for finding new ways to contract with customers (i.e. changing the business model).

But how well do you think that organisation would work at an organisation where the hiring policy is a little less stringent than your own?

Sadly for a majority of us working further down the intellectual bell curve, I just don't think you will see such a proactive approach to a "communal" task list even if developers were left to self organize. We've all worked in places with their own wally, and for this type of person, peer respect just doesn't motivate them. In my experience the smarter you are the greater your motivation from intellectual pursuits and peer respect, but as you drop further back down that bell curve you see that it just becomes a job, a 9-5 to pay the bills.

So lucky you, but I'm pretty sure that if you all did waterfall over there you'd make it work too.

You must know about this already, but you got mentioned on RedHanded! I'm proud to say I discovered your awesome blog weeks before _why declared to a stunned readership that you had rendered a brilliant wordsmith speechless.

keithb said..."And that's why (in contrast to the named methods that camed before) the Agile methods tend to put an emphasis on early delivery of business value..."

Lest we forget methods that came before: "... we use the concept of selecting the potential steps with the highest user-value to development-cost ration for the earliest implementation. This is really like skimming the cream off the top of the milk. ...I would like to stress that selection of high value/cost steps is a fundamental distinguishing feature in my own formal conception of evolutionary planning, in contrast to that which I have found elsewhere."1988 Tom Gilb Principles of Software Engineering Management

Jeebus. I'm really sorry you have such a bad attitude about agile. I'm also really sorry that I started to nod off halfway through your rant. What you are describing at Google is not agile - it's a corporate philosophy. Getting free meals has jack squat to do with your development process.

Of course, I'm sure that arguing with your position will entitle me to some jab about being an early riser - as if that were some kind of fatal character flaw.

Your queue sounds exactly like the product and sprint backlogs in Scrum, which works well.

As far as pair programming, I love it and have had the chance to exercise it at more than one company. The rationale behind it had more to do with getting a realtime code review and sharing of knowledge than some ideal of "if 1 is good, 2 is better".

Oh, and you can take my index cards when you pry them from my cold, dead hands.

Hey Steve, nice post... I totally agree with you on one point: people deliver their best when they get presented with the right level of incentives.

You probably forgot to mention that the Business people (i.e. the people who dream the projects and initiate the work queue) also are great achievers. They probably beat a few other companies in this respect as well don't they? Do they like getting huge bonuses and a dinner at Google CEO's palace (not mentioning the nice check that must hidden in the fortune cookie)? I bet they do.

Your post shouts "there should be more companies like Google where people are rewarded for delivering stuff that serves the Business Model and Business Stragegy, and hey guys, it works even better when the company they work for actually HAS a working Business Model", and I can't argue with this! But I strongly disagree when you make it a Celebrity Fight between Good Agile (ie Google) and Bad Agile (People who are trying to work in ways that put people at the centre of the delivery, but who unfortunately have not achieved the level of incentives that Google can support). There are a lot of companies out there that just cannot afford to behave like Google. There are a lot of companies out there that have a "cultural debt" as far as Project Management and software delivery are concerned. Their evolution towards a Google style change/evolution mechanism is slow, and chances are, they will never achieve it. However, they do recognise that they can do things better, and for this, they need guidance: Agile is indeed on offer when that constatation is made.

And Agile disciplines are NOT dangerous as you state: I can actually understand your comment regarding the fact that Agile discipline may not fit the Google culture: this is not due to the fact that Agile is BAD, it stems from the fact that Agile focusses on Delivering Business Value, whereas the model you live in is based on people recognition and personal incentives.

By the way, does Google ever bin a project? even some effort have been put on it, and possibly a vast amount of investment? Damn, I bet that happens. Does that mean that your Good Agile at Google is Bad then... hummm, am confused now.

Once again, great post, good ranting and great recruiting pitch, unfortunately, I think you just don't get it...

Kudos to Google, looks like the Company Culture lets you have a lot of stuff right. You sound like the guys at Microsoft and Apple did back in the 80's and 90's.

Sadly I suspect most of us poor sods work in organisations where senior management don't let you up and wander over to another team when you feel like it and expect something to ship about when the development team said it would.

Given that, and given that development team's clients sometimes change their minds about priorities, how do you manage the work queue? Personally I find a kind of bubble sort: "Is this more important than that" (with or without cards) works rather well.

(igouy, Tom Gilb certainly had many of the ideas of Agile before "Agile", Sadly, most of the software development community never heard of him)

Interesting. Recently I worked on a project that went into production right on time, despite starting months late, that had no bugs, without pulling extra hours, and without shouting at each other (too often). In a conventional organisation that does not have a virtuoso tradition of software development. We had index cards, paired on most everything, and estimated and reestimated as our understanding grew.

I've worked on waterfall projects, agile projects, DoD projects (which are their own adventure), and now Google. Steve's rant is spot on, but there's more than one way to look at things. Overall, Google generally fits the spirit of agile methods, and in fact some groups use XP, or Scrum, or whatever, but it's up to each team, and the methodology a team choses to use is definitely secondary to the results it produces, so nobody cares about buzzword compliance. The corporate philosophy seems to be "we'd rather underconstrain people than overconstrain them". Some projects are time critical--and they use schedules, milestones, etc. (though not, that I have seen, MS Project) as a way to keep themselves on track. Some people do pair programming as a way to be more productive, but that's up to them. That's what's most different from other large development organizations: process is intentionally not centralized except where absolutely necessary. It's the polar opposite of, say, CMMI.

And yes, setting the hiring bar very, very high does make a real difference.

As I understand it, every single product that Google has launched other than search is a financial failure. Just like grad school - you have a relaxed, low pressure environment and, just like grad school essentially zero impact to your life if your project isn't a success.

This is not a "knock" on Google - I am a great admirer of Google in general. But your fanboi fawning is blinding you to Google's weaknesses.

You said one thing that struck me: if 90% of the teams screw up the methodology, it's a crappy methodology.

Which is true, and a fair criticism.

Of course, this is also true of virtually every other software process. And the ones that people can follow? They don't deliver.

So how's this for a quote: Agile Software Development is the worst software development methodology ever tried - except for all the others

Because there's more to life than low key deliverables and free lunches - most companies need their projects to financially succeed in order for them to stay in business.

Thanks for sharing. It's always helpful to hear how successful software companies manage their projects since it seems to be a hard problem to solve/predict.

My company recently decided to implement SCRUM for developing the next version of our product. The more reading I do, the more I see common characteristics, like collaboration, task tracking and prioritizing, and self-organization. You can even see it in applications like BaseCamp, Joyent, and other collaborative apps springing up on the web. It sounds like Google focusses on those important characteristics in project management.

For what it's worth, I think that's an excellent way to build software. I'd love to start my own software company so I could just do the same thing. It sounds fun.

It's all about the people.At Google, I think it is pretty rare to find someone who is not an A-lister and who treats software development as only his job.That way, there is trust amongst the team, and between management, business, and developers. I agree that the initial culture is important, but it can only be sustained if your people are "right".Also, developers have the power at Google.

Compare this to the normal, everyday IT world of software, and where the managers and team members don't really have that trust, because there is wide variation of skill within a project, and sometimes the manager does not really understand technology either.

And what about if Google doesn't hit another jackpot, say, before it has some bad news and the very generous valuation and one-way ride straight up from IPO, etc. etc. starts to turn a bit? Google is a company that has never known hard times, at all. Thus, it is very young, *totally* unproven at this point. Let's see if it can maintain the same level of pampering when the finances aren't quite so rosy.

Relying heavily on advertising is a model that is very likely to fluctuate at some point. And as previous posters have pointed out, Google hasn't successfully found other revenue models. And, as previous posters have pointed out, much of the software that Stevey is so proud of is, yes, used by millions, but no, not used by millions of paying customers. It's a loss leader to promote more eyeballs and brand awareness -- to support and increase ad revenue.

So, given all of the above, might it just be possible that what works for Google isn't a universal fit for everyone? And, might it also be possible that it might not actually be how Google does things (can afford to do things) forever either?

Google is, obviously, a group of very smart people. But, as this post argues/posits/asserts/protests, if you can take just one tiny step back/out of Google-world, I think you will see these smart people are not living in the "real" world, by which I mean, *the world that their customers live in.* It is a world of "grad students" making cool products that need only be loss leaders. It is a world of very smart people who are all together feeling how smart they are, constantly reinforcing to each other how smart they are -- and thus is precisely blind to whatever such a narrow, particular, specialized culture might not consider important/relevant, might not be interested in, etc. Meanwhile, it's customers are just regular people. And if the Google culture is indeed as impressed with itself as you are Stevey, if it indeed fosters the idea that there is one way to do things that is the right way for all situations for all times, and it's the way Google does it and it's obviously better, which is basically what you are claiming, then this is a company ripe for a fall.

Humbleness wins over arrogance, every time. When you think you're the best, you have stopped learning anything about all the things you already think you're the best at.

As other people have pointed out, the priority queue you describe is basically the same idea as the scrum backlog, which I've found works pretty well. You add in everything you can possibly think of, prioritize it, and fill in more details as things bubble up to the top of the queue.

I also want to echo a bunch of other comments around how Google's delivery model makes the style of development that you describe much more feasible. Companies that have to ship software to customers (either internal or external, but especially external contracted ones) usually just don't have the option to say "you'll get it when it's done." The best you can do is pad the guess to relieve date pressure internally, but you have to give them some date, and it can't be ten years out.

The agile practices definitely work best under certain circumstances (smaller teams, usually in-house or contract development with a single customer), but that doesn't mean that the practices themselves are necessarily crazy. You always have to experiment to find what works best for your organization and your business, but it's certainly worth knowing about all the different "agile" practices so that you can make informed decisions about what to try or perhaps how to take existing ideas and change them to suit your needs. Overall, I think the agile proponents have certainly done more harm than good in pushing people towards a more sane, iterative development model.

Very interesting article, for one thing you make Google sound like a fantastic place to work.

The process you describe that Google have worked so hard to make scale sounds like massive fun to work under, and perfect for regularly generating interesting new ideas and technologies.

There are some things that Google is bad at, and that seem plausibly to be related to to process you describe. Google does not seem to care about the users of its services. They care about making fantastic services, but not about supporting people who use those services or transparency. They also don't seem to mind having software in permanent beta. Googles beta is just an excuse for not providing a full service. It's good for the developers, but it's tremendously irritating for the users who wait 4 years to get a completed service. Google seem to be very good at providing services that the kinds of people who work at Google would want to use, but not so much at stuff that the rest of the world want (luckily theres a big overlap, but that is a way the process has tied you into a specific part of the market). Also, Google are in a very specific position in the market that enables them to offer fantastic rewards and only hire highly motivated employees. They have a tonne of money, and make money simply from attracting people to their pages. Few other companies have as much resource, and most other companies have to charge for their services to make their money. If you charge for a service, it can't be in beta for ever.

The idea that (some?) programmers have up days and down days, days when code seems to just fly out of your hands, and days when the code comes out like having teeth pulled (if at all!) ... well, there isn't a lot of literature on this.

Either most programmers aren't like that, or have conquered it, or only some people experience this.

Sheffield University did a study on agile vs non-agile development, with a number of teams each working in parallel on the same requirements. I offer the link without comment.

Ah, what the hell. Without going into specifics on the paper, I'm attracted to the Agile way of doing things, largely through my own collected experience over the last (gulp) 28 years. I don't currently pair, being the only developer in the group, but there are index cards on my desk, unit test frameworks on my C: drive and numerous methodology books on my shelf.

Well I came here with the best of intentions to learn about what you have to say. I must be honest though once you started making fun of my religion I lost all respect for you. Isn't it funny? By publishing your opinions you obviously want to be respected yet you don't show respect for others.It's sad, short sighted, pathetic and offensive. I'm sure you'll be happy when I say: "I won't be back."Martin.

Great article. But you leave out a without-which-nothing that lets Google do the "good agile" you talk about: unlimited funds. The reason that most sw development is date driven is because most people need to know how long something is going to take and how much it is going to cost. It's not unreasonable, yet that causes poor sw development.

For the benefit of Damien and Lexi: as it happens my alarm clock is set for 7am and I often oversleep that.

And this I can do without any problems currently since the client site I'm most often at is only 30 minutes away from my home by train and the team I'm coaching there have (for their own convenience, using their powers of self-determination) decided to start their working day with a standup meeting at 9:30 What a bunch of chumps they must be, eh?.

Sorry to contradict your predjudices--although I did enjoy the haiku :)

Google’s culture is adapted for internally sourced innovation. What makes software hard for the rest of the world is the division between customer and developer. Customer is an insurance agent, developer is a geek. The geek doesn’t know what technology the insurance agent needs; the insurance agent doesn’t know what technology he needs either. Project management, traditionally, has existed in part to deal with this. Requirements and other forms of “agreement” are one way to solve the problem. Intense iteration of working software is another way. Yet, this remains one of the most difficult problems to solve. My advice, don’t fool yourself into thinking Google is special because of its development culture; that’s hogwash. If Google is special, it’s because it doesn’t have the customer/developer separation. Developers get to source their own requirements, customers simply use the product they are given without having to manipulate its content. If everyone had that relationship with our customers, we’d all be setup like Google. However, commissioned software (like commissioned art) must be influenced by the acquirer; and that’s where things get tuff. That’s where some of what you call “bad agile” is actually necessary. Great post though, loved it.

How dows Google know when it needs to hire more people? One of the ways I get more folks is to show that we're working appropriately long/hard/smart on problem set A but problem set B, a slightly lower priority, needs to be done too so we should hire people to do B.

This looks like an interesting post. But *please* can you provide a media="print" stylesheet in your blog layout so that I can print it without the right column and wasting twice as many pages as necessary?I just don't ready stuff as long as this on the screen.

"Back in Ye Olden Dayes, most companies approached software development as follows:

- hire a bunch of engineers, then hire more.- dream up a project.- set a date for when they want it launched.- put some engineers on it.- whip them until they're either dead or it's launched. or both.- throw a cheap-ass pathetic little party, maybe. This step is optional.- then start over."

If the three exceptions I listed above aren't driven by dates, then what drives them? To some extent it's just the creative urge, the desire to produce things; all good engineers have it. (There are many people in our industry who do this gig "for a living", and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are better than that.)

And then there are those of us that do this gig and go home and don't think about it until the next day because we're busy raising families and enjoying life with them. I've been doing software development for nearly 20 years and I don't know where this attitude comes from that says unless you code constantly or work on an open source project in your (probably sparse) spare time that you are in some way a "lesser" developer than others. Don't get me wrong, I would love to be able to contribute to the OSS movement sometime in the future when my children are older and I have more time. Until then I have to innovate and be creative in my "normal" work.

My team adopted Agile/Scrum methods 6 to 8 months ago. At first it was tough, but now our "unexpected" issues are down to less than 10 hours of development time for the entire team where previously it was above 50. We are making better quality code, faster, and our user's are a lot happier with their Software Development Unit.I'll drop agile when i'm dead thank you very much.

Do you worry about being sued by the Scientologists for simply associating them with a bad idea? Now, if the agile folks come after you with the same legal team, you may have a lot of truth to your rant.

Must be nice to work in a fairy-tale land where there are no real deadlines, and where there's seemingly infinite money to hire the best and brightest, and coddle them as well.

I'm one of those overpriced agile consultants, or at least I used to be. I now (once again) work in the trenches, something I have to go back to every couple years so that the consultant work isn't all outdated or inapplicable BS. And when consulting, my view of "agile" was always pragmatic: what is going to work best for this team?

Unfortunately, as you point out, there are many people that (a) are making a pile out of the big Scrum MLM scam and (b) overhype agile. Yet that doesn't discount the fact that "agile" was intended to eliminate much of the bullsh*t that we had to contend with before (RUP, waterfall). That also includes bullsh*t like letting hotshot developers go off into their own cubes to produce wonderful, unmaintainable crap. Or bullsh*t like we can do a "perfect" design once up front and keep it clean over time without good coverage in unit tests. I had to put up with detestable work conditions throughout the 90s, where beliefs in these poor practices were at an all-time peak.

I don't believe in Methodologies, but I do believe that agile (not Agile) thinking and use of certain "agile" techniques, more than any other way of doing things, is a great start for a path to success, as well as a good opportunity to create teams and projects that are enjoyable to work in.

I really like your writing style but it's not the most constructive possible. Any process will have good stuff and bad stuff and as embarrassing as all evangelists are it doesn't mean there isn't anything useful in Agile. On the other hand we could suggest that the over the top details of the utopian development at google coupled with a hot topic is more about maintaining an image recruitment than commenting constructively on development process !

In answer to Wesley's question: do (most) programmers really have up days and down days? Well, my reaction was "self-evidently yes", and the colleagues I asked agreed with me.

I reckon 80% of my last 2 months' worth of programming was probably achieved in 4 days of highly productive "on" time (and we're not talking about cramming work in the last few days before the deadline).

If there's really no literature on this, then someone needs to do a research project on it. I think it was one of the most perceptive things in the post.

Doesn't all creative work come in bursts? I've seen plenty of interviews with novelists who spend months writing little or nothing; they don't consider this time unproductive, they are organising their minds, planning, so when the writing starts, it will work.

First, let me say that I work for a company that has been practicing the "agile" with a lower a process for over 20 years. We love it, it works well for us. But let me echo some of the other comments, the reality is that "goggle agile" works in environments that are like "google" and may not work well in other climates. Google doesn't have customers in the traditional sense; clients don't call Google and say "I want a web-based calendar that does this-and-that but never the other". Google is a take-it-or-leave-it, you're either with-us-or-against-us, organization. If you want a very "heavy", feature-rich applications (ala Outlook) then you're out of luck when it comes to Google; you can use G-Mail or not use G-Mail, those are your only options. You also need an internal culture that supports and fosters "agile" development. Let me say, Google's approach would not work well (IMO) if you were contracted to build nuclear missiles for the DOD. It will not work if your company has a hierarchical, top-down-philosophy. My opinion is that the goal is to find the "right" approach for your particular "environment"; silver bullets don't exist outside a particular set of circumstances.

The article misses one very important point about the history of Agile Development: before all the hype, it started out as a grass roots movement.

The grunts could see that most of the time and effort went into doing crazy shit, so they came up with the ingenious strategy: "let's stop doing all this crazy shit". But in any sufficiently large organization, it's heresy to give up the crazy shit. Crazy shit is not just random crazy shit, but crazy shit with rules, crazy shit with a history, crazy shit that give people that warm fuzzy feeling of something familiar. You know where you stand with crazy shit, even if everyone knows that it is crazy.

What you can do though, is to say: "let's stop doing all this crazy shit and replace it with ...". That gives people a sense that you have a vision, a sense that you have your eye on the big picture, that you'll be taking something away but you won't leave them with a void.

Thus Agile Development crept in. And because people had stopped doing all the old crazy shit and introduced only a little new crazy shit, the total level of crazy shit drops dramatically. Hence the word spread about the Agile Development silver bullet, and the seminars started.

The rest is history.

If you want to learn more, I'm working on a series of seminars entitle "Applying the Lets Stop Doing Crazy Shit Methodology". There are still seats available.

Great post, Steve. I'll be posting my further comments on pliantalliance.org when I get a chance. I created pliantalliance.org in an effort to get rid of the word Agile. Not surprisingly, the progress is slow. It is fun however to watch people get upset when you question their methodology.

To challenege a little bit of the criticism -- I don't think Stevey is suggesting that every company can/should adopt Google's business model. He's simply saying that if you want the best software, this is the way to do it - motivate your people, don't force things out the door, and (as many, many people have mentioned) hire the best. And I don't know if I buy 'we're not sitting on a mountain of cash, we can't afford to do this kind of thing', and I DEFINITELY don't buy 'this will only work as long as Google is on top.' Remember, this is HOW Google got on top, and this is what they're doing to STAY on top. This methodology isn't a result of their success -- it is the cause of it.

Your critique of agile seems to be based entirely on the more jumped-up and complex approaches.

Strip it down and you're left with two basic principles - short development cycles and talking to your customers more. Combined nicely in showing them more prototypes to make sure what you're doing is what they actually want.

I really can't see why you find that such a bad idea. Maybe you just don't like having to talk to customers?

Long ass post, wonder if half the people read the whole thing, wonder if getting employees for google is your 20% project, this post seems more like evangelizing naive smart developers to go work for the NSA, errr, google.

Wonder how you get anything done if you have so much on your mind.

Wonder how many projects fail in google.

Wonder how many people actually do any work in google, how many actually deserve to get paid.

In any case, I don't care, I'm taking over something bigger than google one day, gotta code.

Oh, and Pair programming rocks, you do things right, make less mistakes, learn tricks, teach tricks, you should try it, maybe you're not as good as you think you are.

I learned a lot about Agile, XP in particular, from a project manager at JP Morgan Chase. He had introduced XP to his team, and they did all of it--pairing, the planning game, test-first, on-site customer. And all their metrics improved. Fewer bugs, faster progress, fewer late nights.

This PM wasn't a fanatic. He was a silver-haired lifelong programmer at a company where inefficient people are laid off at the next merger.

So I wonder where the comment "people pretty quickly demonstrated that XP was a load of crap" comes from. Sure, there's no magic bullet, but XP and Agile methods seem to work for a lot of people. And any methodology, applied inflexibly without regard for the particular realities of a company or team, will only go so far.

I admit I'm a fan of XP. I find it to be the best way I've worked, because you learn faster and you can make suggestions easier.

One other thing. Pair programming was used on NASA's Mercury project. They really didn't want any bugs.

I'll admit that I grew up with the deadline mentality and it still lingers.. but since I left the corporate world to work on my own, I find much more flexibility in the scheduling to account for those "down days" that everyone has, but apparently some won't admit to.

My previous company had processes galore. Meetings upon meetings all throughout the day. I was lucky to get a couple of hours to actually do development. Each year, new processes were added to "improve" this. Yeah, doesn't that just make you want to wake up early in the morning and head right to work.

I do agree that outside the Google-sphere, deadlines are a necessity, but deadlines are most often just an arbitrary date. The world does not end when a deadline is missed (and if it does, your skills likely won't be needed during the rapture). Most corporations do not recognize this and do believe that the world will end if the arbitrary date set by someone is not met.

Finally, one of the reasons I think Google is successful with its approach is because it isn't afraid to fail. I believe, fear of failure insures that you eventually are parallelized to the point of guaranteed failure. Look at Microsoft and Vista as a classic example.

I work for a game company, developing tools for designers and artists. I'm not a designer, I'm not an artist. I don't even care about the game industry. I'm a programmer. Here's how our projects go.

I'm going to use the example of a cinematics tool.

Take the current cinematics tool. Sit it down in front of cinematic artists, art directors, animators, designers, and see how they use it. What do they like about it? What are the important things they use? What do they want that isn't there?

Then we (the programmers) meet with them (the users). There's not that many of them, and we know them personally, so it's informal and focused on getting stuff done. We talk about what they want and in what priority. We give them an estimate about what we can do in the next two weeks. The whole process takes about a half hour.

Then we work. We get as much done as possible. We don't pair, per se, but if someone else has some insight into a problem you're working on, they help you out. We work in an open environment, so it facilitates asking questions and learning.

After two weeks, we release everything that works. We meet again and ask them what they like, and for a new prioritized feature/bug fix list.

The users couldn't be happier, and we get a lot done. It's fun, there's very little pressure, and the urge to get things done on time is high, but not out of control.

We don't have crunch.

Is it agile? It certainly borrows things from agile, but it's not Big A Agile.

To all those who are jealous here is some good news. You can make your own google. I have done it. All you need to do is stop caring.

I am fortunate enough to work for the software organisation of a large company. Through no fault of my own I was assigned to what is basically a maintenance project. It's not all bad. For example I got to travel to the nerve centre and spiritual home of the company (where honestly 50% of the building occupants are managers) and witness how people actually worship & slave to a bug tracking database. Anyway where was I? Oh yes google. Another benefit of working for a large company is that it's very easy to float around aimlessly in the vast sea of mediocrity and have a great time. We can't bring pets to work and there is no chef but for me the 20% do-your-thing rule is in full effect (company policy 0%). I'm actually developing some better tools for doing what we do, since we are so tools improverished (among other things). And it's fun! At lunch time I like to go out for a 7km run then eat a nice meat pie and drink an iced coffee. Better than a massage! The rest of the time I basically cruise spending it fixing whatever silly mistake some overstressed engineer made 4 years ago.

My ambition is to exit the workforce and live off my accumulated savings while developing free software. I intend to achieve this using patience and a positive attitude. Eventually, due to the reality of working as a software engineer for a multinational company in a relatively expensive country, I and my colleagues will be released from the tethers of employment. This is the greatest incentive and I am really looking forward to it. But in the mean time it's leisure as usual.

It's funny, since one the the Scrum guys and I got in a big argument a year ago from the opposite direction. His point was basically, if the customers demand "crazy shit" that they think they want or that's just to make them feel comfortable, tell them to take a hike. Steve's idea seems to be the same but his target is the methodology instead of the customer.

But that only works because Google doesn't have customers! Google doesn't build applications for clients- there are no deliverables and few products (if any). If folks stop eating Google's "light-and-easy" fare and advertising revenue drops I wonder how long their work environment will stay the same?

Don't get me wrong- I think Google found a formula that works for now, but its real easy to be organizationally innovative when your stock price is sitting over $300 and you're flush with venture capital.

Um. Google 'produces' advertising management software of the highest order. That's their big business. This was not a business at all worth being in five years ago.

All you're describing is a long-term gamble on the part of upper management that if enough bright people are left free to roam and given incentive to work on whatever they want, they will be the ones to produce the next big hit that makes a large pile of money.

That's it. Nothing special beyond a gamble and an attempt to lure the brightest people possible into working at Google instead of starting their own company.

The people at Google have to be driven slightly differently, or they don't make it in. Really, the approach is smart because it creates a lot of buzz among the people who are going to create the great software of the future.

To make it work there are a lot of factors that work ONLY at Google, and have to do more with the goal and people involved than the methodology.

If you were trying to make a directed product, and did not have all the ancillary need for advertisement / slashvertisement / web buzz, this would be an incredibly inefficient way to do things.

Wesley said a bit back:The idea that (some?) programmers have up days and down days, days when code seems to just fly out of your hands, and days when the code comes out like having teeth pulled (if at all!) ... well, there isn't a lot of literature on this.

Either most programmers aren't like that, or have conquered it, or only some people experience this.

How normal is this periodic motivation for y'all?

I was really glad to see Steve's mention of this because this is something that I experience all the time.

What I've found is that either: 1) this is not universal/widespread, or 2) everyone who doesn't consciously realize they experience this has already subconsciously beat this part of their psyche into submission.

At my previous job, this phenomenon baffled my superiors. Days, sometimes weeks would go by when I would plod along producing relatively little, and then for periods averaging about 3 days, I'd be absolutely on fire, which I'll subjectively quantify as "ten to fifteen times as much output per unit time." In fact, at my old job, if I felt that I was going to be on fire, I'd actually call in sick sometimes and work at home so I didn't waste the raised productivity sitting in meetings or getting interrupted every five minutes.

The problem was that this bred an impression that all the times when I wasn't "on fire," I was being "lazy," whereas other employees, who produced far less in terms of long-term average output per unit time, were perceived as "hard working."

What ended up happening was that the "on fire" periods just got beaten into submission. I'd either lie, and slowly spoon out the fruits of these "on fire" periods over time, or I'd simply direct the motivation into something not related to work. It was easier for my boss/coworkers to accept consistent mediocre output than occasional extraordinary output.

I wish there were more research/literature on this phenomenon so that I could understand it. I feel fortunate that I've been able to address it consciously and not completely lose access to those hyper-productivity periods.

One interesting note however: I once met a programmer who described much the same phenomenon w/r/t bursty productivity, but with the added data point that he was diagnosed Bipolar. His "on fire" periods in terms of productivity and output coincided with his manic swings. I have no reason to believe that this applies to me, however it raises the question of whether some people are "wired" for bursty productivity and others for more consistency.

Stevey,What this post is really about is what motivates people to do work. That is a key thing every business wants to know. The answer is different for every niche.It is nice that some companies, like Google, have figured out the whip is not the best motivator. The ideas are not new.Max DePree wrote "Leadership is an Art" years ago. He is CEO of Herman Miller. They make the cool Aeron chair you are probably sitting in now. He said hire creative people and let them create.The ideas from Herman Miller are based on a "Scanlon Plan" from http://www.scanlonassociates.org/. Google for gainsharing and open book management. I am thankful I work in a company based on the same philosophy.This stuff originated in the 60's. I just want to give some context. Google is a great place to work, but they are not inventing this stuff out of the ether.Google has discovered that food and interesting work motivate programmers. You just felt the earth move with that revelation.It is sad that other companies don't look for the right combination of incentives to motivate their employees. It works so much better than using the whip. Nice post, keep up the good work. Do you think we could start a micro ISV from this work queue idea of yours?

As a longterm resident of academe, I had to laugh about some of this because you hit the mark so well. Interestingly, after grad school, where they pay you squat and don't care what you do, what passes for management at universities veers pretty solidly into the "Bad Agile" track. Or possibly just plain bad.

Just one example: all the self-evaluations, assessments, program plans, five-year benchmarks, and so on. One year I actually kept track of the time wasted on administrative crap, like a lawyer might for billable hours, except in my case it was a labor of hate. It took almost three months out of my (paid) nine months of academic life. I could have taught two courses and published a research paper in the time it took me to fill out forms.

The reason I still got my job done, was that I spent every waking moment, including evenings, weekends and summers working. (I once figured out what my hourly pay would be if all the hours I worked were counted, and it came to about $6.25.)

Frustrating work saps energy. Working when you're bone-tired saps energy. Not being able to do your work right, when the reason you got into it was for love saps the most energy of all. Result? All that dead tenured wood you hear so much about.

Google is on to something, if half of what you say is true. It'd be great if they don't lose their way, and if anybody else learns something. You'd think that there's nothing very weird about the concept that treating people with respect works better than crapping on them.

I love how quick many are to attack this idea. It reminds me of the battered wife syndrom.

Mark, what idea are you referring to (Agile? Good/bad Agile?) and how does it relate to battered wife syndrome?

Stevey, despite your wildly distorted Agile strawman of the first two pages, you bring up some good ideas and the comments in response have been very clarifying. Like a lot of other commentors, though, I wonder how Google's culture could be applicable at all to my environment.

Also, one could easily envision "Google-like developer culture" becoming the next software 'methodology' fad that is misapplied and brings frustration to many developers and managers.

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.2) Grad school projects.3) Google.

There's another exception: Blizzard. Unlike almost anyone else in the game development business, they don't ship on a schedule- they ship when the app is done. Like google, they are on top of their field (EA's software is hit or miss, and their burnout rate is legendary).

5% of the people can't think, so they don't. 5% of the people can think and they do. The rest of 90% can think, but they don't.

Open-source software projects, Grad school projects & Google employ people of the 5% that can and do think. IMO this is what really makes a difference.

A process is a recipe. That has as its stated purpose prevention of creative thinking. That may satistically work in some emergency situations (for example paramedics). But software development is a 100% creative thinking process. So in software development processes are by definition bad.

But they are also necessary. In non-google like companies, 90% of the population thinks as much as a door knob. Then you need a process which has enough premeditated intelligence just to keep the death march going.

First, let me say that I really appreciate your blog entry. I have heard some things, like the 20% thing -- but it is nice to hear that not all companies focus more on dates instead of quality.

I read through quite a few of the commentaries left on it, and am actually quite surprised by the overall viewpoint of many of the commenters. I, for one, have worked at a few companies as they were beginning to implement [insert fad-based acronymn here] and it was always a huge time waster. Sure, we usually saw how there could be some benefit to one piece or another - but in most cases the developers rebelled against it because management wanted us to spend more time doing their "new" methodologies than actually meeting the deadlines, which were never realistic to begin with.

What you describe sounds very much like what most of us think of as "The Ideal Pure R&D Job". I, for one, wish more companies were focused on real creativity instead of just reimplementing a 10-year old wheel.

What you describe about the tools is something I have experienced in others companies - though not nearly as often as I should have. That trend does seem to be changing though.

The real problem comes down to two things, I think. First, people assume that what is tried and tested (even if it never really worked) is better than taking a risk on a novel approach. Second, FADs are very popular with people who pay the bills. Anyone else remember when J2EE was simply a buzz word describing a single download of everything Sun had on a single download page?

And people made comments about how their clients are all date-oriented, and thus they have to be, etc. I think that gets back to reason #1 in the previous paragraph. But I think they should also realize that the reason Google is able to output new services so fast is very much based on little incentives like the 20% rule.

I personally would love to work for Google. If their Project 02 ever starts hiring people to do hardcore Java (AI, TRAM, Encryption, whatever), I will probably apply.

The priority queue approach reminded me strongly of the "Getting Things Done"/43 Folders personal productivity system. That has some very good explanation for why it makes you feel less stressed, in addition to actually getting more done.

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.2) Grad school projects.3) Google.

Well, I would add Drug companies to this list. Inventing or discovering a new drug isn't something you can attach a timeline to - it takes research and experiment.

This comment really drove that home to me:

a) Come up with a ton of capital.b) Fund a bunch of smart people with good ideas.c) Hope that you make 1500% on 10% of the projects in order to offset the 90% that fail completely.

This sounds exactly like a drug company.

Google is really an engineer-driven company. They invent things for a living. HP used to be an engineer-driven company in that their engineers were free to experiment and invent, and that culture may very well still remain. 3M is somewhat like that, and it's how they have grown their product portfolio.

So, while Google's organization and process may be most appropriate for inventing things, it's perhaps not best for companies whose goals are not quite as open-ended.

It's always great to read about people so enthused about where and how they are working. What struck me most about your response was just how similar it sounded to the people who were talking about agile development in the beginning. "We're not doing Waterfall and we're not doing Cowboy Development," was a common ralying cry, followed by, "We're not doing Extreme Programming!" as other Agile methods appeared. It is interesting that Extreme Programming itself has moved away from the rigidity and workshop-based mentality that you critique as it has matured. Perhaps the rest of the Agile movement will follow suite.

Google seems to have combined many agile processes with IBM's marketplace of ideas approach to management. Of the eight main bullet points you presented for example four are core practices of Extreme Programming. Three have to do with not serving customers directly. As soon as customers are introduced priorities, communication and deliverables have to change. Personally I see frequent delivery as a way to trick customers into not caring about dates because they don't have to be afraid that they aren't going to get what they need.

Google can be seen a beacon that attracts the best developers.... I believe you when you say that peer respect goes a long way, in an environment were everyone is extremely talented it would have to.

But transferring this process to an arbitrary company X with an arbitrary talent pool doesn't seem obvious. You have your collection of talents, of work-a-days, of work-a-holics, of slackers, and these traits are not even mutually exclusive.... You can mabye create a new Google from scrath, but to transform a current soft eng shop into one....

Wow, as a fresh graduate in CS, i imagine to to have the pleasure of working in a structure fostering these idealogies.

I do not consider myself overworked or pressed by suits with unrealistic deadlines, but getting exposure to this kind of development would be awesome. Most people who haven't tried the Google apprach will probably feel this way, but then again, i havent tried XP or pairing, so i'm really being objective on this.

After reading the 'Google Story' by David Vise, i can see the importance on carying on the structure that created the company into a large scale arena. As mentioned above, it might not be the best approach for some, but it shouldn't be, so whats the issue with that? I pity the scenarios where non-programmer, business suits rule over all aspects of a project. I temp'd at couple of firms that dealt this way, and i hated it.

Question to the Agile programmers out there. I'm attempting to understand the process--I feel it should be more flexible than anti-Agile people ever seem to let on.

For instance, the original post mentioned pairing 10% of the time (or was it 5?) Anyway, that doesn't seem to be outside agile anyway. The stuff I've read suggests you wouldn't pair more than half the time at most and all prototyping should probably be done alone.

So I'd like to ask, how often do "Agile" groups actually pair and at what point is it not considered Agile any more? I think that Agile and non-Agile projects should pair as often as the programmers need it, and for some programmers or some situations, maybe 100% of the time even fits...

The concept of Bad Agile hit home with me but a different aspect. Pairing isn't an issue for me, but there are practices that Agile cannot exist without.

Some of the Agile practices eliminate necessary concepts like design and some documentation. You cannot eilminate these wihout using the practices that they replace.

If you aren't constantly refactoring, you NEED an upfront design. (By constantly refactoring I'd say that 60% of your time should be fixing your old code). This is doubly true if you go for The Simplest Thing That Works.

If you don't have the ability to pair throughout the entire team (if the team is distributed, for instance) you CANNOT eliminate documentation, in fact since Agile is so much about communication, I think a distributed Agile team must produce at least as much developer documentation as a "Normal" project.

So my main quesiton, do I just not understand Agile development yet? Is it really a rigid set of practices that must be followed exactly, or do inexperienced managers and programmers just implement it that way?

I also think a lot of people miss the fact(?) that Agile is much more values than practices. It's not really what you do, but why you do it. Practices need to adapt to fit the situation, but the values are absolute--Or am I not understanding again?

Vigorous writing is concise. A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts.

Interesting article, shows the attitude of people working inside of Google.

The comment re Google is similar to academe is apt. I've worked in IBM Research as well as Apple and they really have the same setup. They all came to an end when cash flow finially caught up with the company.

Something else not mentioned: Googles pays substandard wages with comparable job, betting on smart, motivated, and YOUNG people to go for it because of the "working environment". I happen to have a mortgage, (and at this point of > $400 share price, the upside for the stock options is also questionable) so will have turn to high BASE pay outfit...

The following quotes are from Malcolm Gladwell (http://www.gladwell.com/2002/2002_07_22_a_talent.htm):"They recruited ceaselessly, finding and hiring as many top performers as possible""We hire very smart people and we pay them more than they think they are worth." "It was a place where stars did whatever they wanted.""The reasons for its collapse are complex, needless to say. But what if Enron failed not in spite of its talent mind-set but because of it? What if smart people are overrated?"

Thanks for the excellent article. I think you're being unfairly harsh on Agile though. It's one of a plethora of toolkits to dip in to. The more tools you have your disposal, the more effective you're likely to be. It takes experience to get software 'right', experience gained by remaining open minded, trying stuff out and probably being burned along the away. To state that *anything* is simply 'Good' or 'Bad', software methods included, is an oversimplification, the world is more complicated than that.

I enjoyed your rant. It had me chuckling the whole way through. Wonderfully written.

You are presenting a very closed-minded and extreme view of agile. You're exagerations are the very "marketing" techniques that you criticize that agile consultants from using. Unfortunately, you're not alone in your experience with agile. Having been involved with the XP and Scrum communities for many years, it is disappointing that this is the case. I appreciate that you did the observing and discussing that you did in forming your opinion.

Agile is not nearly as simple as people want it to be. Just as Google has to work very hard and be disciplined to maintain its culture and practices, so to does any team that wants to be emergent and adaptive. This does not make agile bad, just hard. Hard in the sense that it requires culture change. I have met very few people in software development that understand what it means to change an organization's culture. This is not the first time that something positive has crashed on those rocks.

Although you're uncomfortable with the term agile, what you describe is in fact consistent with what agile manifesto its creators are all about. Google does not need agile as something that is defined because it is in the DNA. Google doesn't have to change in this regard. It has always been this way. The Google IPO Prospectus is a masterpiece that protects the company from letting the shareholders being too shortsighted, among other great things.

Few organizations have the market position, cash, and billiant leadership that Google has. With these legacy cultures and organizational infrastructures, it is a much more difficult situation. Given the difficulty, there will be failures. But the goal of most is to move in the direction of what you are so fortunate to live everyday.

The software development community needs to be rid of wanna be change agents (consultants and employees) that don't know what their doing. In that regard, I agree with your rant.

"If a process is potentially good, but 90+% of the time smart and well-intentioned people screw it up, then it's a bad process. So they can only say it's the team's fault so many times before it's not really the team's fault."

I work for a major dot com and can't agree more with your post. Even beyond direct "product" development, the such approaches are destroying our company. Perhaps this approach doesn't work for consultants, but then again, most of our outsourced projects turn into horribly managed disasters. Look at BOSE and their 30 year project to develop vibration dampening systems for vehicles. Companies that are only concerned with the bottom dollar, shareholder revenue, project time management, people you say "Getting free meals has jack squat to do with your development process," well they really just don't get it. Sure, they may survive. Their companies can be pumped up with millions upon millions of dollars from investors. They can provide mediocre services and keep up with the flow. They ride the tide. But what to they innovate? Why is it that the most successful car companies are in Japan? Why are so many of our programming jobs heading to India? How come manufacturing has disappeared to China?

There are valid criticisms of Agile, and there are valid praises for Google, but contrasting them doesn't actually make any sense - they aren't trying to solve the same problem. Maybe I'll write a post about how SUVs get bad gas mileage, but macaroni and cheese is delicious, so clearly mac'n'cheese is better than SUVs. That would be about as valid and meaniful as comparing Agile methods with Google's software development process.

Seriously, you are an entertaining writer, which makes it almost possible to overlook the randomness of this comparision. Since Agile (and Waterfall, for that matter) is trying to balance scope, quality, and delivery date, and Google doesn't seem to care about delivery date, the comparison isn't all that useful. On the projects I work on we have to deliver by a certain date - if we can't, another vendor will. I am not hardcore about any methodology - I think that Agile has its good points, and so does CMMI. It all depends on the business context. If I tried to use the Google method (as you describe it) I'd be looking for new work - the business I'm in has deadlines. This isn't an attack on the Google process - it just means that the Google process assumes a different business model than Agile.

It was shocking to read such a big amount of bad things about agile development.

Anyway, it makes me doubt, think, and that's always a good thing.

But, I have to laugh when he mentions "religion". Hey, he really talks like being on a sect: "they are so good to us. Everything is so nice. Nobody manages developers". Wow! So big bosses there DID have manage to make people think they can do whatever they want, and actually make them do what they ask them to do... Uhm... Looks like brain-washing to me...

What I like to hear from agile, or google's development, or whatever any other cool guy on the block that shows up, is that people goes first. I do really hate "people interchangeability-based management methods". That's it. And I guess most of the people out there, doing or pretending to do agile would agree on that.

Anyway, I liked the silver-bullet-based evangelism he introduced: put clever people doing whatever they like and you will get things done. And lovely managers upstairs won't even try to "make them converge".

(There are many people in our industry who do this gig "for a living", and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are better than that.)

This is getting to be a pervasive and I think counter productive attitude in our industry. After spending hundreds of words talking about how motivating the perks and financial incentives are at Google, how important it is to maintain rational working hours and the extent to which Google actively subsidizes personal work (including open source work, I'm guessing), Steve makes this comment denigrating the majority of developers that have priorities outside of development.

It sounds like along with everything else Google does its best to cultivate the feeling of superiority into its employees.

When I leave the studio tonight and focus my time, energy and passion on my family, I won't feel sorry that my "betters" are off making it easier to sell advertising.

Actually, it sounds like Google is an attempt to create a paid mini-OSS development environment. About the only difference is in having tangible financial incentives to reward people for contributing to "important" projects. The rest of the incentivization (peer recognition, visibility) sounds an awful lot like what drives open source development.

My question is this, what happens when a big, complex problem comes up? The kind where you need, say, a network guru, a statistician, maybe some behavioral people, etc. just to understand the problem?

Steve Yegge wrote: But it's exceptionally difficult to measure software developer productivity, for all sorts of famous reasons.

I thought the main reason was that no one tries to do it.

Question: how would you evaluate two methodologies if you really wanted to? What would you need?

Answer: conduct a social science experiment using teams of volunteers. Undergraduate CS majors might do in a pinch.

Question: who is going to do this experiment? Computer Science professors? Ha! CS profs want to scribble equations, Design Languages, maybe, once in a great while, they'll be willing towrite a program or two... but conduct a social science experiment? Feh, that's some other department, isn't?

Steve Yegge wrote:About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations. But where does the data come from? Companies aren't going to give you their internal data, if they even keep that kind of thing around. Most don't; they cover up their schedule failures and they move on, ever optimistic.

Well Duh. So don't try to get the data from corporate sources.

The comparison to "fad diets" is completly unfair: there are indeed academic sources of data on the performance of diets, it just takes awhile for the data to accumulate, but accumulate it does... consequently diet fads are much shorter lived than programming fads.

Can you tell me how even the good agile can be used in the context of a fixed price contract? I am a consultant - yes one of those. Clients hire us to do the impossible. We make it happen but struggle to reach fixed dates. How am I supposed to convince a client to work with a priorities queue instead of fixed dates? Seems to me you were right the first time.

To the people who complained that because they have other priorities besides programming (families, hobbies, etc) they've been lumped in a "lesser programmers" category I can only say this: if you have other priorities besides programming, then you are, by definition, a lesser programmer.

Not that you aren't skilled, brilliant, whatever, it just means that your footprint on the world of programming will be shallow. You won't be of a magazine, you won't be giving keynotes at OSCON. To be truly outstanding in any field requires that you be obsessed. People who influence their fields don't go home on time. The always need to stay an extra hour or eight. Not because they need money or because they have a deadline, but because they need to work out an idea.

They know going home would be pointless anyway. They might say hello to their wives and children, but their mind would be elsewhere.

Don't take it as an insult, it's just reality. The hour-a-day jogger isn't going to make the Olympics. The eight-hour-a-day programmer isn't going to write Linux. If that isn't obvious to you then no amount of hours would be likely to make you exceptional so don't worry about it.

Since there are now 119 comments I doubt anyone will see this. Nevertheless, here goes:

Your comparison to "good cholesterol, bad cholesterol" belittles the post. Just as with cholesterol, the situation is more complex than simply "good" or "bad."

I've used Agile [yes, with a capital "A" and index cards] at Google and for the most part it's been good. What I like about it is the transparency. You know how much work there is to do, and you know how fast you can do it. And you know these things from historical data. Sure, for the first few iterations your estimates might be bad. You learn from that and the estimates get better. Hence, you get a better idea of how much work there really is left. You also know historically how much work the team can do each iteration, and that comes from real, hard data, so it helps you plan.

My experience is that Agile is pretty good for these things; estimating and scheduling. Of course it won't solve all your problems, and there are lots of other factors at Google that mean that most projects turn out well, but the estimation and scheduling aspects, AFAICT, really do work.

Also, it would be much easier to take you seriously if you didn't go off insulting people randomly. I myself am a late riser and don't really understand how early risers do it. That doesn't mean I think there's something wrong with people who can get up early; indeed I respect it. It's something I've never been good at.

Since there are now 119 comments I doubt anyone will see this. Nevertheless, here goes:

Your comparison to "good cholesterol, bad cholesterol" belittles the post. Just as with cholesterol, the situation is more complex than simply "good" or "bad."

I've used Agile [yes, with a capital "A" and index cards] at Google and for the most part it's been good. What I like about it is the transparency. You know how much work there is to do, and you know how fast you can do it. And you know these things from historical data. Sure, for the first few iterations your estimates might be bad. You learn from that and the estimates get better. Hence, you get a better idea of how much work there really is left. You also know historically how much work the team can do each iteration, and that comes from real, hard data, so it helps you plan.

My experience is that Agile is pretty good for these things; estimating and scheduling. Of course it won't solve all your problems, and there are lots of other factors at Google that mean that most projects turn out well, but the estimation and scheduling aspects, AFAICT, really do work.

Also, it would be much easier to take you seriously if you didn't go off insulting people randomly. I myself am a late riser and don't really understand how early risers do it. That doesn't mean I think there's something wrong with people who can get up early; indeed I respect it. It's something I've never been good at.

Maybe Stevey was drunk when he wrote this. Definitely false advertising -- I thought he'd have something interesting to say about Agile, not a rehash of everything that's already been said about Agile methods or working at Google... boring.

He's clearly never spent 5 minutes speaking with Kent, Ward, Bob, Martin, Mike, etc etc etc. So what if there are lots of poser Agile consultants running around? There are lots of poser consultants in general, in any field. Big deal!

Well, maybe it's good that he wrote this anywho because it got mentioned on Slashdot. Never heard of him before and I now have a bunch of his old Lisp and Emacs posts to put me to sleep tonight. Hopefully some of those postings actually make a point... he must have some smarts to have kept a job at Google for a year+, and he does use Emacs after all (although he just got around to 22 ;)

Steve is letting us in on something bigger here, he has a unique perspective on the inner workings of Americas and perhaps the worlds greatest tech environment. Listen, learn, take what you can - The value of this is really between the lines.

My experience has been that agile and every other organized/engineered process is great for people who are experienced and have discipline. Organization fails for people who have yet to master their own work habits and style. Get together a bunch of experienced talented people and they'll kick ass because they are disciplined and know how to flock. Put a bunch of inexperienced people together who have not mastered their own abilities and expect them to coordinate their own clunky abilities with the clunky abilities of others and you just have a pile of clunky people.

I tried reading between the lines. Maybe I've just read too many other Google employee blogs and InfoWeek "Google Management Secrets" type articles to find anything new in this article.

My top complaint remains -- false advertising! He should have titled the post "Why I love working at Google, and don't plan to attend Agile seminars or buy more books on XP" (I'm sure Kent Beck will become despondent and cancel plans for a third edition)

For a guy who recently whined about Software needing Philosophers, he comes off as a stellar hypocrite here for bashing a grassroots effort (Agile) that fundamentally challenged the establishment of Traditional Software Development. Dissing the C3 project is such an old XP hater tactic. Couldn't think of anything more original. Oh yeah, he was drunk.

I think Google is fast turning out to be a load of arrogant and self-sufficing nerds. The google story is bound to implode like Microsoft in the next decade. Apart from the search engine and gmail, ALL their products suck real bad.

But then again I forgot to add the disclaimer. This opinion is entirely my own and does not in any way reflect that of my grandmother, my fellow villagers or my pet dog.

Word of Wisdom - "Too much tongue in cheek never made a very good geek"

You mentioned that XP is a load of crap. Furthermore you mentioned the 'good agile' @ Google. You've described developer's are free to join a project when they can. Develop's work 8-5 hours and no overtime. These are specifics of XP too. Or do you think Pair Programming == XP?

Have to mention i haven't work on XP or other Agile methods, but what are your experiences with XP?

I was actually more captivated by how development life is inside Google than your rant on Agile. Your rant lacks perspective as it seems you've forgotten life in a development environment where delivery date is included in contracts. Sounds like you've been drinking Google Koolaid...then again, I would be too.

Everything you describe about Google's working conditions sound very much Agile...in fact, it sounds like you work at the holy grail of Agile. What Google is trying to emulate is the Open-Source model, which is really the grandfather of Agile development methodologies. They have figured out that software development at the highest levels is as much art as skill, and you really can't force a creative process.

Imagine if Google would announce new software releases 3 quarters prior to release. Do you still think your development environment and conditions would be as cushy and flexibile as they are now? If not, wouldn't you agree that Agile and its various flavours offer numerous practices that would be beneficial where delivery date is important?

One thing I do agree with you on however is anyone who is dogmatic about any methodology or process, is indeed "stupid" as you put it. There is no process and methodology that works for everyone and every project. Anyone who doesn't inspect and adapt really isn't agile.

Funny. The only project I've ever worked on that was consistently (a) on time; (b) utterly rock-solid; and (c) really *fun* to work on, was the one that was developed in a mostly-XP style. You seem to be rather selective in your data.

Moreover, I'm not sure who you're talking to, but the kind of religious Agile approaches you're describing are exactly the ones that serious Agilists don't have much respect for. Everyone who's serious about Agile knows that you have to adapt the methods and mechanisms to your situation, not take it as some sort of religious dogma to be faithfully followed to the word...

cwells, you shouldn't confuse time spent with innovation and contribution to a field. The two are not the same. There are plenty of people in programming that can't produce as well in multiple obsessed days what a good, talented person can in one day.

One of the best implementations of Scrum I have seen is at Google on Adwords. You need some structure to Google style when you have multiple distributed development groups working on software that every other Goggle app has to integrate with on regular upgrades. See:

AbstractGoogle is very successful in maintaining its startup culture which is very open and engineering-centric. Project teams don’t have a project manager, but organize themselves and communicate directly with all stakeholders. Most feature decisions are made by the engineering teams themselves. As well as this works for products like search, gmail … it creates issues for the AdWords frontend (AWFE) application. AWFE is much more product management and release date driven then other Google applications. This presentation discusses how we carefully introduced agile practices to coordinate the AWFE development teams and made the process more efficient and predictable.

It seems like alot of people are mentioning googles failed project ratio. Which is broken for 2 reasons.

First they are a multi-billion dollar company. As long as that is true then by definition they are successful. Please keep in mind this was not always the case, and they had to get to that point... At any rate, these projects can go on in perpetuity until they get another home run (and actually they have had several).

The Second reinforces why the first is true. What seems the important piece of the above article... is that the code is solid.

If everyone is writing the same type of code, then it doesn't matter if the project fails, the code is very likely reusable in other areas. At least in a company that sounds like they could use it.

The process described for the google system sounds terrific. It seems like they are geared towards producing as much top quality code as can be produced times n(coders). Then its just a matter of increasing n.

The other important point that seemed to get a little muddled was the meta info disagreement. The importance of this seems to be that index cards tend to get in the way of "doing the right thing." They create an easily trackable, accountable path, without focusing on what the desired result is... which tends to change. The importance of avoiding feature creep is one thing, but if its the right thing to do...

Sounds like index cards have a place, but its more in a tuning and support capacity than core development.

In my experience, managers love Agile. Just as they loved XP, and before that RAD, and before that time cards.

Traditional managers are hand wringing expectant fathers. They pace and worry, wondering if they're going to have a bouncing baby boy, or a sled, or a big wheel of cheese.

That's because in many companies, managers are managers because they have been managers before. It's still a rarity that an actual contirbutor rises and is allowed to lead. As such they don't always trust the engineers, they don't always see things teh way an engineer would, they have little grasp of abstract concepts. And because they rise at 5am, they have little empathy for a guy who time shifts his sleep schedule to match his own biorhythms, etc.

But they do understand bullet points, and clocks, and checklists. So they cling to them. Agile is a way to try and trick clock punchers into thinking they are swift little mammals, when they are actually the same old dinosaurs keeping one short sprint out in front of the glacier.

Agile works, of course, well enough to produce *something*. Any process will work if everyone uses it. But you'll never know what the code could have been, because the reins are always on the "cowboys". You'll just produce competently mediocre code, that never rises to greatness.

My advice to anyone looking to become agile, as well as leave the door open for greatness is to study open source methodologies, that is not to say everyone has to open their preciously proprietery code, look at how they do what they do, and do likewise within your organization.

And a huge part of what Stevey said, that is getting poo pood is that you need to value the health and well being of your employees. They will produce more, better, and longer if you do.

Agile does not value them, it underestimates them. It smacks of a Special Olympics "everyone's a winner, let's all get there together" rush toward homogeny that is the difference between multi billion dollar successes and the sub million dollar assembly line coder body shops where this seems to be "working for us."

I won't comment on the good/bad agile stuff, but I would like to say something about "Date-Oriented Programming". My company produces enterprise software for Fortune 1000 companies. These customers must be provided with product roadmaps that describe what we intend to deliver, and when, several quarters in advance. You can't just drop stuff in their laps out of the blue and say "Time to upgrade, boys!". They have to plan how and when to pull together the resources required for what can be very complex migrations (of customized database schemas, for example). So yes, Stevey, "Date-Oriented Programming" is a bad thing, *unless* you actually have to have a plan, and more or less stick to it.

With my short 10 year experience in software, and reading all those comments, i can see a common thread that people seem to only partially acknowledge: it's about the people stupid!

Maybe i am one of the lucky fews, but i have had the chance to participate in three very successful projects in the last 7 years that used very different methodologies. I don't believe in methodologies but in the people. A great team with great people can adopt any methodologies and be successful.

The real question is which methodology works best, most consistently, in the real world, when the team is not all composed of A+ individuals, which seems to not be the kind of place that Google has fostered. If people are top, methodology doesn't matter one bit and freedom is what generates creativity and value. Discussing the value of methodologies in your world Steve doesn't seem to make sense.

Now of course, my big question is why would a company compromise on hiring anything but the best. That's the worst business decision, not which methodology to adopt.

Your criticism of date-driven projects, while insipiring to those of us who live in that world, fails to mention one aspect of Google's business that differentiates it from a lot of software-driven businesses -- Ad revenue sustains income while people work on new projects.

I don't know about others, but if I put a banner at the top of my company's proprietary software for the purposes of selling ad space, I'd be canned in a heartbeat.

This model works for Google because of the nature of the web. The point being that many companies are date-obsessed because they don't get paid until they ship. Many have a rough-equivalent of ad revenue called "maintenance", but much of that cost goes to fund the service and support functions.

So you say Google projects are one of the three types that are note date driven. But name me another software company (outside of Yahoo) that has the luxury of placing ads AND the user base to make it a profitable proposition?

Actually, you've got it backwards: talent without obsession is next to worthless. Talent is the raw material, obsession is what refines that raw material into real innovation. How many people have talent but never use it? Michael Jordan may have had talent, but what got him to where he was was hard work inspired by obsession. Many others may have been just as talented as him, but he's the one who obsessively shot 1000 free throws a day, who practiced in the dark, outside, just to get a few more shots in.

You might also think that obsession without exceptional talent is worthless, but it isn't. While talent is clearly a huge boon, there have been many people of mediocre talents who's obsession and commitment to their fields changed the world. The reverse simply isn't true.

I urge anyone to read these when deciding what 'agile' might be for them. Practices should flow from these values and principles and must take in your situation: your specific organisation, project, team, individual. Your "agility" is not defined by the practices you employ; but how you go about applying the values.

Like life, diversity and adaptation is crucial; there is NO silver bullet... and that includes Google's.

A nice rant of the "let them eat cake" variety. Entertaining for the royalty but not to be taken seriously by the peasants in the real world.

Personally, I'm on the fence on the value of pair programming. It doesn't seem like it is somthing that should be done 100% of the time, but the nice thing about Agile is that nothing has to be done 100% of the time. Agility is the key, and even the methodology can and should be adopted to the task at hand.

The one critical part of XP/Scrum that you conveniently left out of your diatribe is that XP/Scrum puts the management of the project squarely where it belongs: on the backs of the people doing the work. This is the most important part of XP/Scrum. When the people actually doing the work have the final say in what gets done and when, then projects actually get done on time.

Having worked for a company that put the Scrum methodology in place was so refreshing that we released 3 commercial software products in 9 months. Pair programming wasn't used except when needed and then only because the team decided it was necessary (not because some MBA manager type decided to ram it down our throats) We didn't shun it; we (the team) just didn't see at as a necessary part of our process unless a particular part of our project called for it or one of the team members asked for it.

We also eschewed index cards for whiteboards. One contained the currently active project list and the other contained our backlog. Every morning during our scrum, we worked both lists, moving things back and forth, adding to the backlog, etc. All you had to do to check the status of a project was to look up at the whiteboard. If the status of a project you were working on changed during the day, you got up, walked to the whiteboard and marked its new status. Everyone could see the change immediately.

During the scrum, No one but team members were allowed in the Scrum Room and on the rare occasion we allowed guests, no one was allowed to speak except for team members. The president of the company would on occasion request the opportunity to attend. We would allow it only if he agreed to the ground rules, ie. he wasn't so speak unless spoken to and then it was only to answer direct questions. If after the meeting he wanted to submit questions to be answered in a subsequent scrum, then an email would suffice or he could catch a team member outside of the Scrum Room at another time.

The one benefit that everyone liked is that it was forbidden to work more than 40 hours a week. We never worked more than 40 hours a week. Never.

XP/Scrum works when implemented properly. I have the practical experience to prove it works.

The only times I have seen it fail is when there is no management buy in, someone tries to graft it onto traditional software management practices, or the software developers don't or aren't allowed to run the process.

Thanks for the interesting read. I would like to comment on your comments as relates to opensource.

Opensource projects may not often have specific date deadlines, but they generally try and set asside reasonable time frames to have a release within.

Generally opensource projects without a significant update every half a year are very stagnant or dying.

Releases for F/OSS are important for a few reasons -

1) It encourages developers to finish up projects that they had planned to do.

2) It helps shake out the bugs - both in greater end user testing and in developers desire to have a low bug count for releases. Also major functionality changes can have wierd interactions, so only a few major changes per release cycle will make bugs easier to find. Whereas a long development cycle can mean multiple large scale interacting changes.

3) Attracts more endusers and developers - open source projects tend to only get publicity with releases (a few exceptions being Firefox that has exceptionally clever marketing efforts, Ubuntu - which gets additional coverage due to Mark Shuttleworths celebrity status and other factors, and Blender that gets publicity from major art projects that it is associated with). Publicity results in faster growth of both the userbase and the developer base. (Blender for instance gains both a surge of new users within a few days of release, and a long term permanent boost in the user growth rate with each major release).

For Blender we try to have between 2 and 4 significant releases per year.

just curious how the writer relates to this article about how google manages its meetings...to me it seems highly bureaucratic, controlling and condescending. Not nearly the utopia one would expect....

(1) Hire really smart people(2) Set some basic direction/goals(3) Get the hell out of the way

In addition to the steps about, there's another key: RETENTION. This is basically the "secret sauce" over at Google and a number of places. They've figured out that the above works, and everything else about how Google works revolves around RETENTION and ATTRACTION of talent. All the food, money, etc, is to make it attractive to the top performers to stick around.

How do you make developers want to work for you? Make them believe the things they hate most don't exist in your environment. I haven't worked at Google, and don't have any inside knowledge, but I do know its a publically traded company, and at the end of they day they want to make a profit and make the shareholders happy. Do I believe they truly don't have deadlines? No, I don't. They don't have deadlines they tell developers about, but I bet they have an idea of how long they'll let a project go along before they kill it off. I also bet while it may be true that you can swap teams every other day if you want that it won't serve you well when it comes to bonus time.

Bottom line, they've figured out the 3 steps about, and then built an internal economy that ensures:

I'd love for someone at Google to weigh in on this with internal insights. I can say the two things I've seen absolutely kill performance on teams is:

(1) Bad projects that just don't get killed. A company that keeps beating its head against the wall just because they don't "quit". Stupid, insane, and bad business. Worst of all the good developers smell out these failures waiting to happen and get frustrated beyond all hell when the project keeps running like the Energizer Bunny.

(2) Mediocre or bad developers lingering along because a company is unwilling, unable, or just completely clueless about performance management. Nothing will drive a project straight to hell faster than some below average performers. Not only do they drag down the project but they drive the top folks out of the organization faster than anything. Sure, there are some people who "like" being the biggest fish in a very small pond, but most top performers like to be around other top performers.

Combine the above two and you have a recipe that ensures you'll be looking for a silver bullet in any methodology anyone can come up with. Problem is the only thing that really works is to solve the real problems. This leads me to a one-word silver bullet software methodology:

One of the most interesting things I've read in a long time. I'm going to have to show this to all my friends studying Management. I've always thought that subject was pure BS, but this sheds some light on how Google works as well as it does. Thanks!

First, Steve is very fortunate to have the opportunity to work at Google. It sounds like an amazing place.

Second: Google is very much an exception. Try finding VC firms willing to find this level of employee perks: some market leading VCs were for an exceptional opportunity, but it is not as simple as just saying "well, why don't the rest of you morons do this".

Google has a very unique position and one that its backers were happy to fund. However, only a few can be the world's leading search engine.

For the rest of us on commercial deadlines, we have to come up with processes that help get things done by a set date. That is the way most of the world works. As J.P.Morgan said "I don't want it perfect, I want it Thursday". Perfection is not the goal. The goal is a solution that pretty well works ON TIME.

Your clients want to see regular progress reports and will not settle for "some of the finest people in the world are working on this".

So for the rest of us (and that would have been you Steve, were you not the beneficiary of the genius of Sergei and Larry), we do have to continually search for processes that work in the real world.

For me, big M methodologies are a bit of a have. They do have some grains of usefulness in them as they draw on accepted principles of RAD. I agree with the analysis of how they drove Agile into the corporate environment. When the consultants and courses arrive, you are being fleeced.

Software teams learn how to work together and then decide what mix of RAD principles work. They don't need to buy into a whole methodology.

Agile was a worthy thought - how projects can deal with change as the project continues. Another great disapppointment though.

Hi Stevey,I just want to say huge Thank You for this post, as it provides an outstanding presentation of thoughts that have been bothering me for far too long now. Someone had to step out and say these things loud and clear. I am forwarding this link to my friends and my management.Hail the incentive-based development!

Nice rant on Agile ... I disagree with most of your conclusions but I'll give you points for outlining a well written blog post.

Here is where I think you miss the mark:

First, you make VERY BROAD generalizations about agile practices. Does what you describe happen? I'm sure it does. But that doesn't mean every development shop trying scrum or XP is layering a ton of crap in with it (in effect making it non-agile). I don't think you have anywhere near enough experience with non-google Agile to make the conclusions that you do. Itsounds liek you were in an agile shop that either didn't have management buy-in or where the tam wasn't allowed to be self-organizing.

Second, I'm very happy for you that you are happy working at Google. Google's products, for the most part, exist for the sole purpose of driving advertising revenue. There are exceptions such as the Google Search Appliance. I am guessing that the ongoing development of that product, with it's corporate customer base might be going a little differently then the development for gmail. When someone is paying $$$ for something it is a different game. I'm sure senior management at google gets that. Eric Schmidt isn't stupid.

Third, it is easy to preach from the mountaintop with Google's $$$ war chest sitting next to you. I'm going to guess that Google's culture was not the same when it was in it's infancy. The culture you speak so fondly of evolved becuase Google's initial product (search engine) generated a ton of cash. The cash came first, then the culture evolved. That fact makes it difficult for others to attempt to recreate Google's corporate operating philosophy (aka lets integrate our advertising into every consumer facing product we create). btw, that is a brilliant philosophy because it means that as new products get released, they add volume to the advertising engine. Well, except for stuff like the Google Search Appliance. But since that technology is an off-shoot of the main search engine, any revenue from it is like a cherry on top of a sunday.

Fourth (and I'll stop at four), for all the negatives you say about more 'traditional project management', you mention a whole number of things at Google that strike me as counter to the free-wheeling environment you speak so lovingly of.

Your comments about the difficulty of scientific experiements are spot on. But I'd like to point out that one guy did try it. It took him 10 years, but eventually he did find meaningful corelations between the successful teams.

The researcher was Alistair Cockburn. Now before you dismiss his work because you know him as an Agile "methodologist", let me point out that his agile views stem from is research (not the other way round), that his work earned him a Ph.D, and that the process he documented is not as dogmatic and "cult like" as other agile processes are (or appear to be).

Drifting around the web I've heard a few comments about agile methods being imposed on a development team by upper management. Imposing a process on a team is completely opposed to the principles of agile software, and has been since its inception.

Its easy to knock Agile, or anything else, when you clearly don't get it. I guess some people will write Martin off, given he was involved in drafting what you and others deride as the kooky and cultish Agile Manifesto. Give me a break!

Great article on an exceptional corporate culture and on the short-comings of Agile! As many of the commenters have mentioned, would that everyone worked in such a culture - and most don't. So what should someone do who works in a different type of culture?

Perhaps rather than simply (or, some might say, simple-mindedly) embrace the "latest greatest" silver bullet, consider an approach (no I didn't say methodology) that matches method to situation. I'll call that adaptive, and it happens at the level between project and business - often referred to as the project management level but it doesn't look much like project management - perhaps project guidance strategies would be more appropriate.

Many of our friends, consultants in the XP and Agile community, have suggested that we promote our adaptive approach under the agile umbrella - and, no doubt, we could lure many more clients into trying us out. But we shy away from that because it seems fundamentally dishonest. Plus, one of the natural outgrowths of our approach is to disqualify the use of Agile methods where they aren't appropriate either to the culture or to the nature of the project.

I also don't think they really want us in their midst if they understood the real implications. It appears to me that Agile consultants love the long term coaching engagements that keep them working. I hear many many stories of a consultant spending months with a client - in-house, making sure they adopt the method 'appropriately' and that the organizational anti-bodies don't kill it in its infancy. I consider that creating a bubble-baby and not terribly agile. How about using a consultant for a few high value days (like 4) with a group, equipping them so they don't need the consultant anymore. Now that's what I consider real agility.

Oh yes, and this adaptive approach was distilled from the best practices of a slew of the predecessors to Google in the valley. Before the MBAs moved in.

To summarize the sentiment in most comments above: "Wow, I'd really love to work in IR&D at Google! Unfortunately, I'm on the assembly line at [not Google]. How does what you have to say impact me in any way?"

The group I'm with at my company deals with internationalization. In that pursuit we are developing guidelines and rules about proper ways to code. So we have tools we are developing that we plan to load into developer workstations that will error/warn/inform when they have violated a rule.

I personally think it's pretty pathetic and patronizing and apt to be ignored and discarded. So I asked our architect how they plan on ‘enforcing’ their usage.

His response was that they where thinking up incentives. For example a coupon for a free ice cream when the install this new ‘process improvement’ tool.

Interesting rant. The main problem I've seen with A-gile methods has been documented by Khaled El Emam. Agile teams usually fail to do a good job on the architecture -- and, when they do run into an architecture problem they do not really do the refactoring that Beck says they should.

The net result is that products of agile development teams have not endured for very long.

Of course, there are situations where the architecture is 90+% predetermined. Most web apps fall into this space. In those situations there are some successful projects and products.

If you are working on something that requires a truly new technology or super tight privacy or super tight security or super performance, etc., then the off-the-shelf architecture is not sufficient. There are some wicked problems.

The challenge of wicked problems is that they are usually solved by very, very small teams or a single person. It takes them a while to do it. That phenomenon often breaks the rapid cycle.

Google's culture sounds interesting. I always had fun developing a few tools when I was young (I'm an old fart). I used to have to hide the time I spent on tools. However, several of the tools proved to be useful to colleagues and eventually I became a development manager.

Then the challenge was to find ways to inspire others to develop tools and hide their work from my bosses.

Have you actually worked on an agile project? Observing is not doing. Watching sausage being made is not making sausage. It looks disgusting from the outside, but until you have DONE it yourself, you have no standing. Critics are a dime a dozen.

You also attempt to paint agile as bad since there are seminars that make grandiose claims. I'll bet you can find someone out there who will grandly proclaim that the best way to create software is the Google way. Oh wait, that's you.

Google is an anomaly. You have one data point, which a trend does not make. Who else has such a development environment? Not open source or grad school--they have zero compensation. Google does not have a "product" to sell. It is not in the software business, it is in advertising. Software is just a means to an end, which is to stuff more ad crap down people's throats (through their eyeballs I guess).

Google has no concept of delivering a product to a market. You do not SELL any software, so who cares if it's late?

Google's added value is cool, on-line applications that drive advertising. What a boon for mankind. More ads in more places. Don't get me started.

Another flaw in your diatribe is your assertion that all agile development is the same. One of the key tenants is that you get to pick and choose whether your team wants to follow ANY of the tenants. Too much resistance to pair-programming? Fuhgetaboutit!

I suppose you don't see the irony in that Google practices many of the agile methods. Short, focused meetings, prioritized tasks, and so on.

I just don't get the big rant thing about agile. Filtering out the unsubstantiated verbiage I come up with:

* Never tried it* Don't like some of the concepts and yet use some of the concepts* Not interested in something I have never tried

Your whole blog entry reminds me of trying to get my daughter to eat broccoli. She had never tried it, but from the look and smell she knew she would not like it. Never mind it has fiber, and vitamins, and so on. She had convinced herself that she would not like it and no amount of my logic was going to change her opinion.

So agile proponents sound like snake oil salesmen? Hah! Try selling the Google way and stand back and prepare to be bombarded.

Steve, if Google's model has worked so well how come it is not in wide use? I am not trying to be sarcastic. Could you please write a book on the inner workings of Google and what makes your company work :) I would like to read it! Then force my bosses to read it.

Google's model is in use to some degree in all healthy and innovative companies. In most companies, after the initial entrepreneurial wave, the corporate folks start to join and exert influence over the engineering dominated company. They create too many layers of management to try and control things, and before you know it, there are not many critical and contrarian thinkers left to innovate.

The google model sounds great for a company that really has one product (SEARCH!) and a bunch of other independent products that can come to maturity at their own pace. Like another commenter mentioned, the software "methodology" at Google works only because of Google's particular business model, and that Business model is a very unique one.

What you are describing is definitely related to Lean Thinking and Lean Software Development, the 20% slack time for instance has a presedence in 3M R&D. I suggest you read up on the lean movement, e.g. Lean Thinking. Very intersting indeed.

I saw a lot of critics saying "what you describe works because you have no strict deadline, what do you do when you DO have a deadline"?Apparently, a lot of people did not read the rant carefully - the "google process" is used for doing things "as fast as possible". So how do you do things "faster than possible"?Using such a software process is not necessarely impractical for all but google - of course, it needs to be adapted to each particular company. For instance - he said that at google they are rewarded for launching before end-of-quarter; so similarly you can have rewards (monetary or not) for "launching on time". If launch date is strictly correlated to the company income, make it correlated to the develpment team's income, so that they know that if they fail and loose the contract, they will actually loose money (i.e. they will miss the chance of earning money).

It's really an entrepreneurial model, not a Google model. It’s not really a process, it’s a mindset. A company has to buy into the mindset and also have individual leaders who understand development at an intimate level to appreciate the advantages of agile working. The mindset basically managing the unknown, not thinking you know it all. Most companies can’t deal with this ambiguity, adaptation, and uncertainty. They’re under an illusion that one can define up front what the customer wants, right down to the smallest User Interface elements. They spend enormous amounts of time thrashing over these elements. They also build in functionality that when released is never used by the customer. They ride the wave of the entrepreneurs who created the product before they arrived. A company can do this for awhile but eventually it will come home to roost and they will get disrupted by more agile competitors.

We worked as a small team the way you describe it happens at google back in '92 and produced a great piece of software which these days is called Microsoft Dynamics NAV. Rather than 10 people it now takes hundreds of people to make a new version. Yet they still haven't really changed what we made back then, I guess it still does the job.

Hey Stevey, your original post says "Agile is bad, google is good". What is Agile is never answered...Also google does not have a client that has a business deadline, so it can afford to amble around and hit-an-try, but this is not possible for most of the non-product companies. Do you know that there are companies who excel in fixed-bid fixed-time projects?

Neeraj, I think when it comes down to it, generalisations (including this article, excellent as it may be) are of limited value until you glean from them what works and what doesn't work in the context of whatever environment you operate in.

There are some things that the article doesn't address, such as what DO you do when your clients keep adding unreasonable requirements to projects that are already behind deadline? As the article suggests, this is one of the main motivating factors that led to agile developments in the first place. Whereas the 'methodology' itself may be flawed, the intention behind it is understandable and more importantly the PROBLEMS it sought to address are still very real, immutable facts of daily life in any organisation that has to live with the reality of 'deadlines' and thus can't be ignored.

What is the alternative to SLDC/iterative development? Unfortunately this article doesn't cover this thoroughly. Yes, the whole incentive for completion as a motivator is there, but that works for Google, it may not work for say a bank that really, really needs to fix a financial module thats leaking thousands of dollars by the day. So really, these articles serve as a useful start for beginning a thought process, the answers should come from you and your own research, don't seek silver bullets but work to tailor ones of your own. That's one of the points of this article, I think.

I was watching you blog for several months now, since I stumbled on great articles about Lisp (from your Amazon times). But ever since, it was getting worse and worse. This one is over the top.I hope the comments will open your eyes. You just cannot afford such scheme outside of Google's financial wealth.One thing that will get you more readers is attacking everything - from software methodologies to religions. Controversy will get you popular, for a time. Unfortunately, the audience quality dwindles rapidly in such cases.Expect to get more and more people commenting on your blog and linking to you, together with less and less insight.

Your biggest point seems to be that Google's business model ignores one of the three variables of most systems engineering: schedule. You retain cost and performance, and get exceptional performance at an excruciating cost because your customer doesn't even know that a product is coming.

The downside of this is that you risk Segway-style failure: having no customer ahead of time, you risk coming up with a product for which no validated need exists.

With enough brilliant people left free to challenge each other's assumptions, you can have an Edisonian inventors' paradise. But at some point, you have to realize that you're playing a different game than everyone else.

The challenge of any systems engineering project is balancing cost, schedule, and performance. Agile, XP, and many fads pin down a schedule and then suggest approaches to managing performance within cost. As long as Google can afford to ignore one of the three variables, they can get as much performance as they want.

It doesn't mean their approaches are lame (although most management fads are). "Different" isn't "wrong" -- I would hope someone at Google would be intellectually agile enough to recognize that.

Your whole blog entry reminds me of trying to get my daughter to eat broccoli. She had never tried it, but from the look and smell she knew she would not like it. Never mind it has fiber, and vitamins, and so on. She had convinced herself that she would not like it and no amount of my logic was going to change her opinion.

I see why he changed his blog title from "Drunken" blog to "Whining" blog. He's regressed from eating too much free ice cream at Google, so like your daughter whining comes naturally.

To be fair to his readers and advertisers, I think "Stevey's Drunk (on Google Koolaid) and Whining Blog" best represents the recent content. I much preferred when Amazon stress drove him to real drink, and writing interesting things about Lisp and Emacs.

Listening to highly paid consultants will discolor any idea for hands on hackers. Stevey, stop listening to those hacks and get back to hacking. The top 'agile' people that I know spend most of their time coding cool stuff and solving interesting customer problems, not writing articles or speaking at conferences. They don't generally bother trying to sell others on 'agile' because most people just won't get it -- and never will. One good programmer will always outcode 100 hacks in the long run, no matter how good of a process or IDE you give them.

Would you by any chance be willing to chat about how the "pile of work" to do system works? I'd love to give that a try. Sounds like it would work for a professional investor as well. Do any non-proprietary pieces of software emulate it?

Your experience:there are managers, sort of, but most of them code at least half-time, making them more like tech leads.

My experience: one manager of mine never coded (and barely had enough time to see me once a month), and another manager of mine probably coded 25% of the time, but never on my project.

Your experience:developers can switch teams and/or projects any time they want, no questions asked; just say the word and the movers will show up the next day to put you in your new office with your new team.

My experience: Unless you were considered a "star", you couldn't switch teams until your manager gave the OK, and even then only if you were switching to a more important project. Maybe the MTV office has more constrains than the Kirkland office, I dunno.

Your experience:Google has a philosophy of not ever telling developers what to work on, and they take it pretty seriously.

My experience: Huh? I was always told what to work on and given no choice in the matter. In fact, I was moved off a project because it had too many engineers, not because I asked. Not that I asked to be on the project in the first place.

Your experience:developers are strongly encouraged to spend 20% of their time (and I mean their M-F, 8-5 time, not weekends or personal time) working on whatever they want, as long as it's not their main project.

My experience: Sure, as long as the 20% time didn't interfere with your main project, which, despite your experience, did have deadlines that were expected to be met. And, given the fact that after code reviews, slow unit tests, excruciating builds, etc., are done, you really don't have any time left unless you work until 9-10pm each day.

Your experience:there aren't very many meetings. I'd say an average developer attends perhaps 3 meetings a week, including their 1:1 with their lead.

My experience: Similar. However, you're expected to do much of the tasks normally assigned to a project manager, such as keeping on top of writers, UI folks, etc., to get their stuff done so you can write a design document, or finish the code, or fix a bug, or get QA to look at it. So yeah, not many meetings, but still a bunch of administrative overhead.

Sorry, but there's a large part of software development at Google that is extremely frustrating, much more so than eBay where I worked before. What's worse, is most everyone I encountered knew this, but it's too large of a problem to fix in 20% time. Google needs fewer "smart people" and more people who know how to code and build Java software in an modular (dare I say OO) way.

You are indeed lucky that your experience is so different than mine. Enjoy!

I don't mean that as a slam. I read the whole "bad agile" and thought, "he's gotta be pokin' fun at something for some reason. Who'd even call his description of 'bad agile' Agile?"

I've had the XP or not discussion a lot of times. The most memorable one I had was with a group of all senior level programmers who walked away from the experiment convinced that it really was a faster way to program.

I was a lead programmer for a startup with two senior and two junior programmers (right out of college). I can tell you right now that if I had to manage that project again. We'd definitely do paired. Most kids right out of school don't have a clue and paired can give them that clue faster than any other process I know.

The problem with most "methodologies" -- including Agile -- is that they are best understood as a descriptions of what functional projects tend to look like. Good people, good teams, good incentives, good management and so on drive project dynamics in such ways that the dynamics can be best described by methods such as Agile. Just because I can describe a functional project however, doesn't necessarily mean that I can force a project to be functional. I can whip Agile on any team I want, but if the mix of people stink or any one of a number of project-killing factors exist then all bets are off, Good Agile, Bad Agile or Ugly Agile.

Remember, the Poseidon missle project was one Hell of a successful very large scale software development project. Also remember that the Poseidon missle project was also a waterfall project. All conditions in many dimensions of project dynamics contributed to its success, not just the fact that it was a well-run waterfall method.

In the end, a project's success is much more a function of the quality of people up and down the entire food chain.

Pair programming is invariably promoted by people using the implicit threat that resistors aren't 'team players' with the goal of imposing futher social coercion, in other words, just like high school.

hobbittS1witty post steve.pity that some of the people you have offended have such short attention spans...

perhaps that is why they like -ologies.

i started programming before there were any -ologies around. we designed and used our own METHODS for managing the various aspects of software development. i still use method to keep me from the madness of the development environment.

methodologies came about because a) someone wanted to make some money and b) because managers wanted to find a quick way to make ineffectual developers into effective developers. get real! there is no shortcut. you need a wide range of skills; you need to apply them diligently; you need to be creative; you need to share and be supportive; you need to communicate; i could go on.

But most software developers are actuallly more like well-paid mechanics, not pioneering researchers.

To wit:For most businesses, IT is a "utility" function- like the phone or janitorial service. Just something to build and keep a system running that allows them to do their REAL job- selling cars, insurance, pork bellies, medical care, etc.

People look to engineers mostly to MAINTAIN systems, not reinvent them- the same way we look to our mechanics to keep our cars running, not re-engineer them whenever we bring them in for an oil change.

Do I want a mechanic who is an ARTIST at changing my oil, but can't guarantee (or even estimate) when my car will be ready? "The oil will be changed when it's changed, and not before."

No. There is a definite element of pragmatism and scheduling incumbent on most mechanics, just as there is on most software engineers.

For those mechanics (or mechanical engineers) that work at research labs, or the R&D department of GM, perhaps they can spend years envisioning the next generation model car; but those jobs are few (as is the demand for them).

For the rest of us:PM and software development team methodologies in general exist (IMHO) to elevate less disciplined (and talented?) team members to a median standard. Unfortunately, the overhead involved in this also tends to lower the productivity of the team "superstars".

Most business have far more mediocre programmers than superstars, so for most orgs, adopting strict(er) development methods will, in theory, at least, be a beneficial trade-off.

Even a small SWAT Team of superstar coders can be amazingly productive. Oh, and they get bored easily, too. So they need fun and challenging projects coming down the pike constantly- and/or oodles of money to keep them around.

Businesses large and small walk a constant tight rope between attracting and keeping superstar coders, and keeping the business systems humming along without the tremendous upheaval of systems change.

People who are content to maintain insurance systems, for example, are probably not the world-class coders that Google attracts. But they are needed, just the same. And some method(s) are needed to get the team pulling in the same direction to meet delivery schedules.

Most businesses just don't have revenue models like Google's, and that's OK. In fact, some (myself included) would say that Google's business model isn't very sound, relying almost exclusively on ad placement, and the recent purchase of YouTube.com (another no-money-maker) doesn't seem like it helps much.

So, who's to say that Google's vaunted relaxed and lavishly rewarded development cycle won't be one cause of their undoing?

I tend to agree that XP is a joke, and I find it hard to believe that any decent programmer would prefer the pain of non-stop pair programming to independent work most of the time. I think people who enjoy it aren't very productive, and would prefer to socialize.However, others have raised valid points about the nature of Google's business and others; dates are a hard fact that cannot be ignored. Google sounds great, but long term they will have to start actually producing in order to keep Wall Street interested and competitors away. Google's search results are extremely weak, but there's much less competition than during the dot-com boom. At some point, Google may wake up and find another major player with better search results, and boom - suddenly time matters. But your points are well taken, it does seem like Google tries hard to make a solid working culture.

I've just came back from JAOO conference where Jeff Sutherland claimed that development teams in Google were using SCRUM... bu that's not my point.

I am a consultant (shame on me) and my company is both mentoring customer development processes (and the customer might be a banck, an insurace company as wee as a software house, this doesn't matter) and developing in house software . We can stick to the customer development process, coach a new one or propose ours. My default choice looks like agile, but it's always a context driven trade-off.

Google environment looks shiny - and you have all of my genuine envy for working in such a paradise - but its model cannot be applied elsewhere. One reason above all: Google is well known for hiring the best available people, which normally helps you deliver regardless of the process (that's the same reason probably the first XP project succeeded...). If you have to deliver and take the best from a given team, then you'll need some sort of process. As a team leader I tend to be strong on code convention, bug tracking - sounds like your queue - integration and testing. Are these agile or not? I don't care. But really what you define as "bad agile" - that sounds scary to me too - looks like some West Point trainee, told to read the book and make it "the rule".

This is clearly a misunderstanding, after all Agile folks are consultants - as you said - so they'll never come up with a "rule" but only with some (brilliant) suggestions...

Reading the comments I'm actually quite intrigued to work in a more Agile way now. I was unaware of the Manifesto or the principles, but they do sort of reflect much of what I do anyway and I can see how becoming more agile could help. My main role is to support software that tests hardware, so most of the time I have dates to work towards for new hardware, or have to give my own target dates for bug-fixes/yield enhancements. We don't have index cards, but most of the work I do is recorded in our bugzilla db. I meet with my manager and the people to whom I deliver software regularly and I make (sometimes) frequent, incremental releases corresponding to the changes they request.

Incidentally, the project I enjoyed the most recently was outside that format, but even though the person who requested the software had been asking for it for months before my manager told me to do it, he has shown very little interest since I asked for the requirements and he gave me a rather simplistic flowchart. Yes it was frustrating working with a sparse spec and minimal customer input, but I got to do my own thing for a couple of weeks. Whether that program is ever used, or is even useful, I may never know as I don't really want to see it again if my customer doesn't care about it. But at least I enjoyed work for a short time.

On the agile manifesto, though, I would say that while working code is more important than documentation (because the code should do most of its own documentation), the volume of documentation should be proportional to the complexity of the software.

any company(well atleast good companies who would care abt thier employees) would like to keep the employees happy. perks is one of them. but how do u do it? u reduce the salary little bit. thats what exactly google does. Im a fresh grad and have many friends in google as well as other major s/w companies. Atleast for fresh grads, in general the salary seemed to be a 5k less. Not that few thousands extra in salary is a big deal, but it gives room for perks. Imagine instead of giving Xk bonus every year to you, if the company gave u (x-1)k, but also a free round trip to a ski resort with stay. I think the expense for the company is same or maybe even less, but it seems cooler and the human mind becomes happy with such perks and feels more valuable than 1k. Im not saying Google is paying less or tries to be evil applying this principle. Overall, the pay including the perks in Google might equal to just the pay in any other biggies. Its definitley a wise choice as in overall you pay enough to the employees(market value as MS,Y!,Oracle,Cisco I know of), but base is slightly less and load instead with perks like free food(which motivates them to them to stay longer or come early to have free breakfast/dinner) etc

I have a question. What happens in Super Happy Fun Land once you've made a couple of bad hires, and there are many unproductive people running around?

Does everyone at Google pull their weight? And if they don't, will their peer reviews reveal their lack of performance?

Will people continue to be productive when the company is 10 years old, and some decidedly poor performers have been rewarded because they are good at politics?

Organizations that are young have the advantage of very little history. Its easy to view Google as Happy Fun Land because you haven't got to the point where the skilled people-operators have infiltrated your structure.

Read about the way Intel and Sony used to be. Look at the way they are now i.e. horrible bureaucracies. You know they used to be awesome engineering havens too. What went wrong? What went wrong is that they lasted long enough to get heavily embedded with the type of douchebags that don't make things, but who spend their time thinking about how to use human organizations for their own advancement.

It's inevitable. Eventually the fairytale will end, and you'll have sweet-talking douches running wild up in your house too. The talents that these people have are amazing. When used for good, they make excellent sales people. When used for evil they upset the delicate structure of incentives in the company.

I've already know a few people from Google that I would never hire for the start-up I'm at currently. Its not that they aren't smart, because they are. It's that they are better interviewers than they are producers.

And Google doesn't fire people or have lay-offs, even if you could identify them in the organization. So good luck working with these people for the next twenty years.

Back in the early 90s, I was one of three developers in the Macintosh team at Sierra Online (remember Leisure Suit Larry?). We ported DOS games to the Mac - some ports would take a short ime, others would take a long time with lots of changes to the interpreter. We didn't know what a process or a schedule was. We just worked as hard as we could to ship games. We shipped over 15 ports in 3 years, along the way rewriting the most of the Mac game engine, including the graphics engine, memory and resource managers, and parts of the virtual machine. improving performance 8-fold, and seeing Mac revenues jump from < 2% to 8% of total company revenues as a result. We didn't work off specs, we didn't have a project manager. The reason no one bugged us for schedules or dates was that they didn't care about a product line that accounted for less than 2% of the company's revenues. Of course, they sat up and took notice when sales started improving hugely. What was our incentive to make significant innovations and ship frequently? All three of us loved games, had a sense of pride in what we did, and felt we could make a difference. Every time I went to the stores, I could point at atleast 4-5 titles with my name in the credits. What's more, Sierra had a great royalty-sharing scheme and for every box that sold, we would get a cut of the revenue. I would get royalties equivalent to up to 20 - 25% of my salary every year. I do believe that if you have smart, motivated people who are left alone and given sufficient incentives, great things can be achieved on a consistent basis. Unfortunately, most of the software industry doesn't operate that way - not any more.