Monthly Archives: October 2018

In one form or another, that question has been asked to me in 90% of all the job interviews I’ve ever had and my answer has probably changed with every response I’ve ever given.

Overtime working.

At the time I was first asked the question, I had probably never stopped to fully comprehend the nature of the 40 hour week. In primary school I’d been cursorily told about the Magna Carta, English Feudalism and the Lord’s day, and in my teenage years I’d probably watched Fight Club, Easy Rider and Jerry Maguire, while quietly thinking to myself, “I’m never gonna be a wage-slave to the Man, man!”. I’d certainly never heard of the 8-8-8 movement. The theories of Henry Ford or Karl Marx on the matter. The origins of the perfect beuracracy…

Over…

Time…

Working…

What a question to ask in an interview! I thought this was a tech job interview, not a philosophy jam. Where are the questions on how many tennis balls can fit in a school bus? Overtime working is such a complex issue and one that games and other tech industries have so endemically failed to get to grips with, that the culture of persistently working beyond contracted hours has spawned it’s own term – Crunch.

// The #RyseFacts Twitter Storm

(Oh what, you thought this was going to be about R*? Nah, I started writing this article 5 years ago.)

#RyseFacts was an interesting moment in games development. To recap the #RyseFacts campaign was to form part of the pre-release publicity for the Xbox One launch title Ryse: Son of Rome, revealing behind the scenes snippets about the game’s development. One of the campaign’s first tweets was as follows:

By the time #Ryse ships for #XboxOne, we will have served the crunching team more than 11,500 dinners throughout development. #RyseFacts

The resulting outpouring of commentary formed a twitterstorm far beyond anything the game’s community team might have expected from such a simple factoid. The debate quickly became heated, instantly focussing on the devisive subjects of deadlines, ambitious specifications, overtime and developers’ health. All of which was widely reported within the gaming press [1][2][3]. The outcome for the developer was a toxic hashtag and a hastily dropped PR campaign.

Some Industry members came out in condemnation of Crunch. Cliff Bleszinski, former Design Director at Epic Games on Gears of War and Simon Unger, Animation Director on IO Interactive’s Hitman: Absolution tweeted:

Simon Unger (@Simonunger)
Bragging about how many overtime meals your team has had is like bragging about how many times you've crashed your car.
11:05 PM - 15 10 Oct 13

However the condemnation was not unanimous, there were plenty of people willing to declare their support for overtime working:

John @JonVoy
@therealcliffyb no dishonor in working hard and having passion about what you do ,CRYTEK is a amazing company #RyseFacts
03:49 PM - 16 Oct 13

I reflected previously that one of the aspects that made the #RyseFacts tweet so unusual, was the fact that a developer itself had revealed such detailed information about the management of a project under active development and the long hours being worked by its team. While previous public discussions of long working hours in the games industry have raised the physical and mental issues relating to long working hours on employees and their support networks – notably the EA Spouses and Rockstar Wives (oh wait, maybe this article *is* about R*…) open letters – it is highly unusual for official comment on overtime working.

Usually the debate is conducted from individual to individual. In June 2014, Corrinne Yu, Vice President of Engineering at General Motors, a former graphics programmer at Naughty Dog and former Prinicpal Engine Programmer at 343 Industries on Halo 4 caused a stir with her comments on Twitter.

We are coding hard over the weekends with our Naughty Dog graphics team to push The Last of Us past 60 fps.

On occasion games company directors have spoken about overtime working, with Cliff Bleszinski’s former colleague Mark Rein, Vice President at Fortnite developer, Epic having said in July 2012:

‘I’m sure we crunch just like everyone else. But passionate people will want to ship the best product they can.’

– Mark Rein

// Does Crunch Make Things Better?

It is clear that the subject of Crunch is a deeply personal issue on both sides of the debate and the culture of Crunch would never be sustainable without the support of the developers who undertake it.

However, the question remains, does working longer hours demonstrate passion, commitment and result in an overall improvement in quality, or is it passion theatre that hides underlying failings in management? To remove the emotive language, is there a causal link between high quality output and high intensity working, or is it a matter for personal preference? Should a conscientious manager encourage developers who show extra commitment to working hours or insist they stop work at 5 o’clock?

Well actually, for the amount of debate on the subject, it’s quite hard to say for sure. The anecdotal data is insufficient to make a solid judgement and measurable data is quite hard to come by. Indeed, when I first began writing this article in October 2013, the most striking aspect about working hours was the prevalence of political motivations in forming the 40 hour work week.

In Das Kapital, 1867, Karl Marx wrote, “By extending the working day, therefore, capitalist production…not only produces a deterioration of human labour power by robbing it of its normal moral and physical conditions of development and activity, but also produces the premature exhaustion and death of this labour power itself.”

By the beginning of the 20th century, the 40 hour work week was championed by revolutionary socialists and free market capitalists alike. Henry Ford was one of the first major industrialists to embrace the 8-hour day, 5-day week. In an interview with his biographer in 1922 he stated, “The country is ready for the five day week. It is bound to come through all industry. In adopting it ourselves, we are putting it into effect in about fifty industries, for we are coal miners, iron miners, lumbermen, and so on. The short week is bound to come, because without it the country will not be able to absorb its production and stay prosperous.

“Just as the eight hour day opened our way to prosperity, so the five day week will open our way to a still greater prosperity.”

In modern times, a 2003 report for the UK Department of Trade and Industry was commissioned to perform a review of the research literature and secondary analysis about the effectiveness of long hours working (defined as more than 48 hours a week), particularly with respect to organisational performance and increasing productivity.

The review looked at working time patterns in the UK and made comparisons with the EU and other developed countries, with a view to explaining why the UK workforce had some of the longest working hours in Europe.

“There are considerable theoretical and methodological difficulties in measuring the impact of long hours working on organisational performance. Overall, however, on the basis of the current evidence, it is not possible to establish conclusively whether long hours working has beneficial, detrimental or neutral overall effects.”

While generally inconclusive, the review did highlight specific concerns with shift working and health and safety issues caused by fatigue.

“The review of the research literature shows clear grounds for concern about the adverse effect of long hours working and (the frequency of) health and safety incidents. However, most of this research focuses upon specific occupations (e.g. long distance lorry drivers, the medical professions), which precludes more general conclusions being drawn.”

While many of the hazards in heavy industry that may cause severe injury or loss of life are not present in the digital workplace, the concept of industrial accidents may be repurposed for the modern creative industries in the form of digital accidents that could cause loss of earnings or endangerment of employment. Such accidents could take the form of irretrievable data loss, security vulnerabilities that cause disclosure of personal private information or more simply a collection of bugs and poor user experiences that cause users to avoid your game.

This overall lack of data led me to shelve this article. However, in December 2014, the Game Outcomes Project published their research on the factors that make or break game development projects. It was the fourth part of this summary, “Crunch Makes Games Worse”, that really piqued my interest.

The summary focuses on two hypotheses relating to the effectiveness of Crunch:

The “Crunch Salvage” Hypothesis – the idea that crunch is more likely to be used on projects in trouble, and that this “trouble” is itself the cause of the poorer project outcomes, and that when crunch is used in this way, it leads to outcomes that are less poor than would otherwise be the case.

In both cases the authors found strong correlations to refute the hypotheses that Crunch was beneficial to projects but, in fact, was clearly detrimental to a project’s economic and critical outcomes.

// So Why Crunch?

There are only three things that you are working with as a development manager: resources (people and money), features and the schedule. Changing one has an impact on at least one other axis, usually two.

In traditional software development, the launch day is the single most expensive day in the project. All the development costs have been spent and marketing committed, yet no revenues have been generated. There is only so much capacity in the supply chain: manufacturing, warehousing, distribution, retail space, billboards, advert breaks, website banner ads, feature articles, endorsements and digital store front page promotions must all be booked in and paid for in advance. To add to those pressures, gaming is a highly competitive sector.

In the case of a normal sales period timing is a major factor, if Grand Theft Auto shipped last month, Batman this month and Call of Duty the month after, the developer must hit the deadline or face the very real possibility that their game just won’t sell enough to recoup the development costs. Worse still, if that game is an exclusive, launch day title for a brand new console in a multi-billion dollar industry then the deadline is everything, it will not move.

If the timing cannot be changed, another option is to try and reduce the amount of work required by cutting features from the specification. However, if the goal is to make an astounding, new gameplay experience to sell an expensive, new entertainment device to consumers, it is highly unlikely that the stakeholders will accept reducing the feature set around which all the marketing and sales pitches have already been made.

So if time & features are intractable and the project is behind, the only option is to look at resources. How about scaling up?

There are only so many developers in the world and integrating new people into a team is extremely difficult due to problems with the time it takes for new recruits to bed in to the team, with the increased complexity of communication that adding more people causes. Once you’ve scaled up to ship the project, what do you do with all those extra people when the project is over? Have people sitting around doing hobby projects? Roll the entire team straight into full development of another major project? Let people go again?

Contractors are a potential solution but due to the high demand for such highly skilled individuals and the time constraints on the project deadline they can come with a high price tag.

So the only option to managers appears to be to increase the amount of output from the existing resources and if/when that is insufficient they add more overtime. A former colleague of mine who’d worked in production, once described AAA games as ‘crunch driven, miracle based software development’ and for the most part, I wouldn’t disagree with that.

When the schedule slippage is recognized, the natural (and traditional) response is to add manpower. Like dousing a fire with gasoline, this makes matters worse, much worse. More fire requires more gasoline, and thus begins a regenerative cycle which ends in disaster.

// How to Avoid Crunch

Firstly a large amount of emphasis should be put on the organisational effectiveness of the development team. In the analysis of the 2014 survey results, The Game Outcomes Project indicated three positive correlations that makes crunch more likely and one negative correlation that makes crunch less likely:

+0.51: “There was a lot of turnover on this project.”

+0.50: “Team members would often work for weeks at a time without receiving feedback from project leads or managers.”

+0.49: “The team’s leads and managers did not have a respectful relationship with the team’s developers.”

-0.49: “The development plan for the game was clear and well-communicated to the team.”

As such, it is the duty of managers to take a hold of their projects and ensure good communication within their teams and with external stakeholders, whether they are corporate clients or end consumers.

It is fundamental too, that the risks in a project are well understood and mitigated, while the scope of work is kept in check. (see Jim McCarthy’s, 21 Rules of Thumb for Shipping Great Software on Time for more information on this topic). Shifting the risk out of the manufacturing, distribution & warehousing can give a team a huge advantage here. The advent of cloud computing and digital distribution has revolutionised the iteration times and confidence with which a team can deliver a new version of a digital product to the end user.

The incredible success of Minecraft has demonstrated the value of generating income early and often by releasing Alpha builds to the paying public, while also maintaining user interest through keeping the ideas simple and the production values down by using systemic, procedural content and allowing users the freedom to customise and modify the product to do more of the things they want over time.

The lessons of Minecraft’s early access model have been feeding back into the games industry far and wide, when Valve introduced Greenlight into their long established Steam digital distribution platform, it allowed smaller developers to reach an audience early without the expense of traditional marketing channels. A model that has since evolved into Steam Direct.

Valve have been the masters of this new development model, ever since the release of Half Life in 1998. Taking focused core gaming experiences and pairing them with community features, that provide great player retention over time. With the release of Steam in 2003 and Half Life 2 in 2005, Valve began the radical shift to a fully digital distribution model.

With Valve’s more recent games, their digital model has evolved yet again to take on aspects of the hugely successful and far-reaching Free-to-Play market, as typified by Zynga, while trying not to alienate their core audience.

The latest iteration, as seen in games such as Dota2 and Counter Strike Global Offensive, focuses on a well defined core game experience before adding player value incrementally over time. The success of this model can be seen in the analysis of Steve Gaffney, a former Director at Splash Damage and now Program Manager at Google DeepMind:

Is it any wonder that having seen the incredible successes of Valve’s transition, Mark Rein and Epic should follow that trajectory to utilise their Unreal Engine licensing as a platform, away from deadline fueled, crunch heavy development, into releasing Fortnite as a service? Not really.

// Passion Proganda

(Okay I’m going to have to talk about R* now aren’t I?)

Yet here we see Rockstar, a world spanning, mega-developer – with a successful games as a service business already in operation in the form of GTA Online – bragging about how much crunch their team are doing, when:

They have seen others getting into hot water over it

They have a record of getting into hot water over it themselves

They have a history of utilising hype to gain front page media coverage

Do they have a game out next week by any chance?

// So… What *Do* I Think About Overtime Working?

I guess I can’t avoid it any longer, I’m going to have to answer the overtime question once and for all.

The Graduate me answer:

“Whatever overtime is required to get the job done!”

The Experienced me answer:

“It’s best not to work overtime but sometimes there are exceptional circumstances where some extra effort is required.”

The Director me answer:

“Development is always a trade off between Time, Resources and Features. The requirement for overtime is indicative of a failure of management. Tired people do bad work, while being horrible to each other and I have high quality standards in both code and people. Actually, you know, this is a complicated issue. Here, I wrote a blog about it…”