My experience leaving mainstream game development to go indie, and the adventure that ensues.

Monday, April 15, 2013

If You Agree to Crunch, You're Part of the Problem

Recently, Ben Kuchera called to light a eulogy for LucasArts, drawing particular attention to the sacrifices made by employees in enduring crunch. In his words, crunch is a "monstrosity" not to be romanticized, and that's actually a pretty good way to put it. For not only is it a wholly destructive force unleashed on those who work in the industry, it also never seems to die, no matter how many times we try to kill it.

And as developers, it's our fault.

What is crunch?

Before I get into the hows and whys, though, let's quickly establish some definitions.

Overtime, which is a component of crunch, is defined as working more than 8 hours in a day, and more than 40 hours in a week. Some places vary in those numbers (e.g. Alberta, Canada uses 44 hours for their week), but it's roughly in that ballpark for most of the world.

Crunch is the "sustained" application of overtime. Definitions of what "sustained" means here vary, but I think all will agree "months" qualifies without hesitation. "Weeks" has a pretty good shot of qualifying as well. "Days" is probably arguable, and starts to depend on how often within a month it happens.

Crunch is, basically, leaning on overtime too heavily to solve problems. And as it turns out, there are some more concrete definitions of how much overtime is "too much."

Why do we crunch?

What is good about crunch? Why do we do it?

Ostensibly, crunch is meant to increase productivity of a given set of developers within a given time frame. Evan Robinson actually addresses this question in his Why Crunch Mode Doesn't Work: Six Lessons:

Management wants to achieve maximal output from employees — they want to produce a (good) product as cheaply as possible. They also want to avoid hiring extra resources that increase the cost of the finished goods unless absolutely necessary. On the surface of it, Crunch Mode looks like the most obvious and logical way to reconcile these two desires.

He also points out that this is often related to a looming project deadline. He then expresses that assumption as a simple, linear function:

O = R * t

Where O is the team's total output, R is the team's rate of output per hour, and t is the number of hours worked.

However, as Evan's extensive research illustrates, R is not constant over time. R actually varies over the course of a normal work day, peaking in the first 4-6 hours. In fact, R approaches and even drops below zero after enough hours, resulting in negative output for workers.

Effectively, an employee starts damaging the product they work on (bugs, quality issues) if left to work for too long continuously. (For reference, Evan cites 21 as the number of continuous hours of work one needs to reach a mental state equivalent to being legally drunk.) Conversely, working 8-hour days was shown to produce a 16-20% productivity boost over working 9-hour days.

What's more, prolonged periods of long work days start to produce a cumulative fatigue, reducing the average daily rate R by more each subsequent day. In The Revay Report, experimenters found that 60-hour weeks boost productivity for a few days, but productivity noticeably dropped after a week, cancelling out the boost at 2 months, and continuing to worsen over time after that. (see figs. 2, 3a, 3b, page 2)

Evan goes on to point out that every single other industry has spent the past 100 years optimizing this formula, and has arrived at the 8-hour day, 40-hour week. In Evan's words:

Any way you look at it, Crunch Mode used as a long-term strategy is economically indefensible. Longer hours do not increase output except in the short term. Crunch does not make the product ship sooner — it makes the product ready later . Crunch does not make the product better — it makes the product worse.

I don't think I could put it any clearer than that, except to point out that this is only how crunch negatively impacts the product. For the other drawbacks, I certainly can't put it any better than those whoalreadyhave. In short, crunch ruins lives, products, and sometimes companies, with no up-side.

How is this our fault?

So how is crunch our fault? I mean, developers are the victims, aren't they?

It's true that developers bear the brunt of the pain in crunch. And it might be comforting to imagine that someone "up the pay scale" is responsible for botching things enough to warrant crunch. It's that last phrase, though, that's the problem: "enough to warrant crunch."

How can a project ever warrant crunch? We've already shown that crunch has a purely negative impact except in the shortest of intervals. So how can we agree to do it? Whether for selfish reasons or love for our product, how can we say it's "ok" to begin working in such a way that damages the product, ourselves, and our coworkers?

Nothing warrants crunch. As we've just seen, industry has been wrestling with the pros and cons of crunch for a century now, and crunch has been shown to be ineffective. Crunch is not a rational option. And our guilt is enabling crunch to happen in the first place.

If you agree to crunch, you are incrementally making the whole industry worse for everyone, yourself included.

Take a moment to let that sink in. Everyone is worse off for your agreeing to crunch. You have to work more, and deal with mounting stress and diminishing rest. Your coworkers have to agree or appear unreasonable in the face of your compliance. The lives of those around you and your coworkers are affected. And your company is forced to absorb the drop in product quality, along with a workforce that's losing competence, morale, and loyalty by the day.

Crunch isn't helping anyone. And by agreeing to it, you're making it easier for crunch to exist.

What if your manager asks you to crunch?

In short, tell them "no." Be polite about it, of course, but don't let politeness give way to bad business.

Look at it this way. By asking you to crunch, your manager has simultaneously just done two things:

Indicated that they have failed to manage the project appropriately thus far.

Indicated that they have no clue how to continue going forward.

Now, if you like this manager, it might be a good time to pull them aside and direct them to this post, or Evan's piece, or any of theresearchsourcesEvanused. If they didn't know, fair enough. Now they do.

If they still insist on crunch after learning about its negative effects, you might want to say "I'll think about it" and speak to their manager. Again, arming yourself with polite concern, and some established research on the topic, hopefully you can impress your point on someone with more experience and reverence for historical data.

If you find yourself talking to more than 2-3 managers in this way without progress (or run out of managers), you might want to consider another option: finding a new employer. Because any company willing to impose crunch on their employees at that many levels of management has a deep-seated philosophical problem, and it may take more than common sense to fix it.

What if your manager doesn't ask you directly?

What if you're in a place where crunch is the elephant in the room? I.e. nobody explicitly asks you to crunch, but it's implied that overtime is expected? Perhaps that's just "the way it's always been," and sticking to normal working hours seems like an invitation to be fired.

Again, this is a topic to be broached with managers. Give them the benefit of the doubt. Bring it up, and see where things go. As above, if you can't make any progress after 2-3 rungs on the company ladder, that may be a sign you're in a place that's unhealthy.

What if they fire me?

What if they do?

Ask yourself that, seriously. If the company you work for sees your concern about unhealthy work-life balance as cause for termination, I think you can safely write them off as an employer you want a relationship with any longer.

You may have heard (or even been told at some creepier companies) that there are 100 others waiting to take your job if you leave, so it's no skin off the company's teeth if you go. That's not true, though.

Good companies recognize the true cost of finding and training good workers. It takes time and resources to find a good worker, and more to train them in your company's workflow. If your company doesn't recognize this, it's probably not a good company. They'd be doing you a favor by firing you.

What if I'm compensated for crunch?

In light of the aforementioned effects of crunch, this should be irrelevant. The focus should be on removing crunch entirely, not putting a band-aid on it.

If employees were compensated for crunch, we might see a change in crunch habits in the industry, but that's just conjecture. As it is, employees are rarely compensated fairly for their crunch, either in dollars or days off.

California may be the one place where this is addressed by law, promising anyone under $83k annual salary ($39.90/hr) time-and-a-half pay for any and all overtime (Labor Code Section 515.5(a)(3)). They also define several exceptions and exemptions, which mandate overtime for entry-level or in-training software fields, film and television artists, and writers. How this impacts software development in California remains to be seen, though.

However, whether you're compensated for extra work or not doesn't address the bigger issue of whether your management knows what they're doing or not.

The bottom line

The point to remember here is that a century of industrial research and work hours testing has resulted in the nominal working hours of 8 per day, and 40-44 per week. This combination produces the highest sustainable productivity possible.

In short bursts, overtime can produce a slight boost in productivity. But the boost wanes after only a few days, and actually cumulatively grows worse each day after that.

If your employer is suggesting (or fostering) an environment where overtime is sustained longer than a few days, start considering whether they're causing more harm than good. See if you can open a dialogue with managers or HR to address the problem in other ways. And be prepared to say "no."

If you like your employer, reward them accordingly. Give them diligence, timeliness, hard work, honesty, creativity, accountability, loyalty, and all the other things you're paid for. In exchange, expect them to provide a healthy working environment, tools and support, intelligent planning, and monetary compensation for your efforts.

10 comments:

I think having automated smoke tests was a good way to keep us more honest. I remember the few times we did short crunches for hard deadlines that those tests would start to fail more and more often as time went on. My boss sure didn't like it... and he (and his boss) was able to convince upper management to dial it back.

I was actually wondering about that as I composed the post. Surely with all the bug-tracking software being used, one could do some number-crunching to compare rate of bug creation (and discovery) as compared to working schedule.

Only if your QA dept isn't quota or metrics based. If they are, then you're not going to measure anything other than what the targets are.

I think we were blessed with having a high quality dedicated in-house team. Elsewhere I've worked... not so much - not to say they weren't professional - but let's just say they had different priorities.

Re: promoting the behavior you measure, this is true. I feel like the QA folks I've worked with were more intrinsically motivated, so that probably helped promote useful info rather than busywork.

Evan's article did make it a point, though, to be aware that our line of work is difficult to measure. It wasn't like we could measure widgets on an assembly line, so it was hard to prove when things were working (or more importantly, not).

Great post! I think as the industry matures, this view will spread and will result in better games and happier people. Some studios have already had success with going "crunch-free" (which is why I find it funny when a studio claims to be the first one to do it, and haven't shipped anything yet).

Sadly work weeks over 40 hours seem to be increasingly common in other industries too (while not called "crunch" per se, some people's workloads are simply impossible to accomplish without overtime).

Personally I don't mind the short bursts like you mentioned when a hiccough occurs, to keep things on track as long as they're minor. But planning crunch into a schedule and expecting free work is definitely the wrong way to go about it.

First of all, as a project manager, I take offense to this. the first thing you should be addressing is the fact that sales often hands us an impossible deadline to begin with - because the job of sales is to say yes to get someone to sign on the dotted line.

Now, moving along:

Even in the most optimal sales situations, anything can happen on a project that causes delays. For example, I had one project where BOTH of my developers just quit. The client doesn't give a shit what happened on our side, we have to deliver on the date we planned. So I had to ramp up two new developers on this project, and manage to make the date original promised. To do that, we had to do two weeks of crunch at the end even though I consistently build in 3 weeks overflow into all my projects. The project turned out fine.

I also had another project where the client was completely crazy and we had to do two weeks of crunch locked in a conference room.

I get what you're saying about extended months of this, but weeks? It happens, and a lot of time it's not because the project manager doesn't know how to manage the project.

No offense intended. My aim is to point out that sustained overtime is often the first go-to strategy PMs use to solve productivity crises (especially in my industry, video game development), but is actually one of the least effective means to do so.

The key element to consider is "sustained." If one believes the Revay Report (linked above), a few days of overtime is actually fine. It works, even. In fact, a few weeks might even garner a net boost. But beyond that, one attains diminishing returns as the employees burn out, eventually crossing a threshold where even the short boost in productivity is eaten by mistakes, sluggishness, and fatigue. And employees need recovery time after any overtime push, to restore normal productivity again.

Re: "sales hands us an impossible deadline to begin with," I agree that that is the first thing that should be addressed. Any organization that expects its project teams to do the impossible to pay the bills is going to have a hard time. If sales has the kind of power to promise the impossible, then one of two things needs to happen:

1 - sales needs to be reigned-in in to promise only what can be delivered.2 - the development team needs to be bolstered in some way so that they can deliver what is being promised.

Re: "anything can happen," you're right. The unforseen can play havoc with a project. Some projects have the luxury of renegotiating delivery dates, and if available, I think this will produce the best overall product (and employee morale).

However, in cases like you describe, the date is fixed, and the development house has to eat the cost of incidentals. Two weeks of crunch is undesireable, but nowhere near the 6-12 month crunch cycles that inspired this blog post. Even I would probably agree to a 2-week crunch in the circumstances you describe, provided the company usually treated me well, and that we'd get a break afterward (a small bonus would not go underappreciated, either).

The key point I'm trying to make is that *sustained* overtime is not a viable tool for solving project management issues. Short bursts of extra work can have a positive effect on timelines, but are probably best if paired with comparable breaks afterward. And whatever the case, happy workers are more engaged workers.

The problem about crunching without transparency when sales gives out impossible deadlines is that you'll get the response 'Hey, you obviously had the bandwidth/buffer.' Which you did, risk and quality assessments aside. You need to approach higher up the management chain and have sales account for the internal overtime costs when they're doing their own performance metrics*. Otherwise they're silently debiting against the company's goodwill and the HR hiring budget.

Inevitably, it comes down to whether the company thinks that goodwill is worth protecting or whether they can live with the dev churn. Realistically speaking, some places and markets are ripe for the dev-churn strategy hence why it exist.

Indeed, sales may be aiming for a performance metric that ignores damage done to the shop's sustainability. Approaching management with that caution might be a bit scary, but I think I'd prefer that risk to grinding my life away at a shop with a precarious future.