Wednesday, July 15, 2015

Life as an Indie Dev - 1 Year Later

Just over a year ago I left my "cushy" job at Microsoft as a software engineer in the Office division to follow my passion: video games. Technically I've been at Flying Helmet Games for a year and 2 weeks, but who's counting? So far it's been an absolute blast, an absolute slog, and absolutely worth every moment.

The Vancouver Studio

As a quick refresher, I've been doing the work from home thing while the rest of the studio generally resides in Vancouver, BC. For most of the project, I've been working on a team of 3 programmers, along with a number of other engineers (sound, animation), artists (concept, environment, creature, cinematic, etc.), designers (level, system, writer, etc.), QA, and more.

What We've Been up To

Over the past few months we've put out a trailer, put up a Steam Greenlight campaign (and got greenlit in 6.5 days!). Our producer took the game to E3 at the Media.Indie.Exchange (MIX), an indie after party for E3. We've been doing a little streaming on Twitch.TV--Wes, our media/QA guy, managed to get the couch, the game, and 2 phones streaming simultaneously, which is really cool. The game is looking really good, our systems are all generally in place and now we're into the polish and bug fixing phase.

It's definitely been an interesting ride. I've learned so much about game development in general--though not all of it is flattering to the game industry, but that'll be a post for another day. However, that being said my coworkers are very smart and talented folks, and just working with them has allowed me to pick up all sorts of knowledge. We've made some missteps--as any project generally does--but damn have we put together some awesome stuff in just over a year of production time (they started a couple months before I was brought on board).At the beginning of April our lead programmer left to work for a large company, though I don't blame her one bit. With a family to support, and a new born child, it's quite difficult to justify working on a startup. But with her departure, that elevated me to lead programmer, and I have been ruling the code base with a velvet glove and an iron fist. Actually, probably mostly the iron fist.

Crunch

As we've gotten closer to release, the pressure has ratcheted up significantly. The dreaded "crunch"--massive overtime--reared its ugly head severely. For the period of April through May I worked at least 80 hours a week. It wasn't pretty, but there was a lot of fairly mechanical work to get through that didn't necessarily require a lot of brainpower.A while back, Kotaku ran an article about crunch time in the game industry, and our producer/director/Idon'tactuallyknowhistitle Ed Douglas gave them some pretty candid information about the why and how we got into the situation we had found ourselves in:

Early
in a game cycle the skies are blue and anything is possible. As you get
closer to final everything you have to trim, simplify, remove, feels
like failure. You also have an optimistic perspective on what can be
done in any given schedule, so you over-scope. What they call ‘technical
debt’ builds up, and you have more and more features to maintain and
fix when it comes to the end.

In
a perfect world a game is built on a ‘Minimal Viable Product’ model,
where each feature works nicely on its own, and is enhanced by features
around it. Although we give internal lip service to this method, to be
honest to ourselves we do not do it. We’re making a complex (for us)
RPG, with a huge number of moving parts. No feature works in isolation,
and nothing can be cut without massive ripple effects throughout other
systems. Here, we were overconfident, didn’t scope properly, and now
have a huge number of features with lots of bugs to fix and a hard
deadline coming up.

We’re
completely independent which means we set our own release date, but it
also means that when we’re out of money, that’s it. We’re not EA where
we can shift resources from studio A to help studio B for the last few
months. The only way to finish is to crunch. To ask people who did not
commit to the features in the game to spend the evenings, their weekend,
their family time, digging us out of this ‘technical debt’ we’ve
incurred. Now they’re critical path in getting a game worth shipping,
and if they don’t do it they know the game could fail. It’s the absolute
worst position to put someone in and it’s shameful that I did it.

As Ed suggests, we've had to fix some of the technical debt we've incurred (which reminds me, he told us he'd give us that whole letter since Kotaku only printed part of it). I'm certainly not the only employee affected by the technical debt, but for me, that's been the entirety of June and July.

Some of that was setting up an automated build system--we haven't an actual build engineer, so we've had QA/IT run builds manually. That couldn't stand, so I sat down, re-learned bash scripting after not touching it for a decade, and put together some build scripts. We still build our handsets manually, so there's some work left to be done and it's a far cry from an actual fully automated build system, but we're in much better shape these days and don't need to babysit and manually perform the steps to build the game anymore.

Another few pieces have been required features for shipping that we just hadn't gotten around to yet, like crash logging and analytics. Requirements for shipping, certainly, and not something we can ignore, but they weren't critical for unblocking designers, artists, or other engineers. Leaf features, making them naturally fall to the bottom of the pile, which honestly is how we got into this mess of required features not being done in the first place--not giving engineering enough time set aside to build the infrastructure, rather having us all support design directly through most of the project.Finally, we've had some major features that required complete re-writes--part of that technical debt. When we built our handset reconnect logic and our save system, the ideas behind them were good, but in practice turned out to have some intractable technical issues. And when I say intractable, I
mean by redoing it, I increased the speed at which reconnect and save
would occur by upwards of 30,000% in some scenarios. No, that's not a
typo.

We hadn't put in the QA time nor the engineering time to really dig into those systems until recently, and I spent half of June re-writing reconnect, and half of July re-writing save. But again, to a certain extent, those systems were leaf systems. Not having them working didn't really block anybody from doing their job, so the priority on them got pushed down.That being said, I've cut my hours down to something much closer to a normal 40 hours per week. There's absolutely no way I could ship features as complex as reconnect and save at the required quality if my brain was fried and I was just burnt out in general. And taking that time for myself has paid off, because those features are a lot less buggy than any of the work I did during the crunch period. It's also kept my enthusiasm up a fair bit. I'd say I'm sorry to my coworkers, but I'm really not sorry at all. I did what I needed to do to ship awesome code.

There'll be a time when we need to do a postmortem, and figure out where exactly we goofed, but I don't think any of us are under any illusions about what some of those goofs are. And we're all guilty to an extent. Ed puts all the blame on himself--and to be fair, he deserves a fair chunk--our individual departments could have pushed back a lot harder than we did. We were the ones handing out estimates, and telling him what was possible, what might be possible, and what was highly improbable.

It goes back to nobody wanting to ship a crappy game, so instead we hiked up our enthusiasm and went at it. The perils of working in a somewhat creative field. It didn't sink us, though, and that I attribute to a lot of hard work from a dedicated team that believes in what we're building.Working RemotelyThere's the upside to being indie and being remote. And the new lead programmer. I have the absolute freedom to do my job. Yeah, okay, crunch is complete balls, but if I really don't want to do it, I just don't do it (like I'm not doing it right now as we speak). But I'm getting the job done and done well if I say so myself--and no Ed, I'm not fishing for compliments, nor apologies for that matter :P

Being remote means I'm out of sight, out of mind for both parties. I don't get into what I heard called "sympathy crunch"--where I'd crunch because others are crunching--and it also means I can do what I need to keep myself healthy and happy. Let's just say there's been a few times where I've napped in the afternoon rather than work, though I made up that time later in the evening because, well, I really want this project to succeed. On the other hand, not having AC these past couple of weeks has been awful, so the folks in the Vancouver studio had it lucky on that aspect. Coding in 95F is pretty horrendous. On the other other hand, nobody's around, so pants are optional. I have visited the Vancouver studio a few times, so it's been good to see people in person when normally they're faces on Google Hangouts and text in Slack. Sadly, visiting now is difficult because my work laptop's hard drive died so I'm tethered to a desktop machine. Still, there's a lot to be said about meeting folks in person.

The group playing our E3 build.

Visiting the studio was good when I became lead, as well. One of the issues in my mind with our release is that while most of our studio was localized, the two senior programmers (the old lead and myself) were elsewhere, either across the country or across the border. Our junior programmer basically was left to the devices of the designers and artists. I feel like I wasn't able to provide enough guidance because of that distance once I became lead. Though, all things considered, he did a good job. I've no complaints there, I just feel it might've been even better had I actually been on site.

Also, I'm going absolutely bat-shit insane because of the lack of meaningful contact. People suggested going to coffee shops but that's just background noise--annoying at best, distracting at worst. I need folks around who I can talk about the game, about the craft of programming and design. The team's been pretty great about discussions in Slack so I get some of the office chatter, but it's still no substitute for actually being there. On the other hand, I get way more work done at home then when I visit. Open floor plan, blech.ConclusionWe're wrapping things up in some aspects on our side, and still ploughing ahead in others. I think we have more convention plans upcoming, so look out for that, and our plans on Steam are full steam ahead; look out for something there soonish, too!It's been great, and is going to continue being great. I've a good feeling!#Personal, #IndieDev, #EonAltar