Microsoft attacked for five-figure Xbox 360 “patch fee” (updated)

UPDATE, 7/20/12: Microsoft provided the following statement to Ars Technica after this story was originally published: "Polytron and their investor, Trapdoor, made the decision not to work on an additional title update for Fez. Microsoft Studios chose to support this decision based on the belief that Polytron/Trapdoor were in the best position to determine what the acceptable quality level is for their game. While we do not disclose the cost of Title Updates, we did offer to work with Trapdoor to make sure that wasn’t a blocking issue. We remain huge fans of Fez." We have yet to hear back from Fish for further comment on the story.

ORIGINAL STORY:

Developer Phil Fish knows there's a problem preventing some people from enjoying his Xbox 360 puzzle platformer Fez as intended. But he's not going to fix it, thanks to what he says is an exorbitant fee of "tens of thousands of dollars" that Microsoft would charge to recertify the game after a needed patch.

The issue started on June 22, when Fish released a patch intended to fix some outstanding gameplay and performance issues with Fez. That patch gave rise to new problems for some players, though, by causing their save files to appear as corrupted, in effect erasing their progress through the game.

Microsoft pulled the initial patch for the game mere hours after it first went up, to prevent the bug it contained from spreading too far. But even though Fish initially described the issue as "fairly widespread," he now says he estimates the progress-destroying problem only affects the less than one percent of players that have already completed the game (or come very close) before installing the patch.

So rather than pay Microsoft "such a large sum of money" to approve a new patch that would fix the small issue (Double Fine's Tim Schaefer pegged the cost of submitting an Xbox 360 patch at $40,000 in an interview with Hookshot Inc. earlier this year), Fish has decided to put the initial, slightly error-prone patch back up on Microsoft's servers.

This entire saga would be a non-issue if Fez wasn't exclusive to Microsoft's system, Fish said. "Had Fez been released on Steam instead of [Xbox Live Arcade], the game would have been fixed two weeks after release, at no cost to us," Phil writes in a recent post on the Polytron blog. "And if there was an issue with that patch, we could have fixed that right away too!"

Fish offered his sincere apologies to the small number of dedicated players he says are "screwed" by the first patch, saying it broke his heart to leave them with a broken game. But paying what Fish calls "a ton of money" to fix such a minor issue "just doesn't make any sense, especially when Microsoft itself considers the current patch "good enough," according to Fish.

"We already owe Microsoft a LOT of money for the privilege of being on their platform," he said. "People often mistakenly believe that we got paid by Microsoft for being exclusive to their platform. Nothing could be further from the truth. WE pay THEM."

Microsoft didn't immediately respond to our request for comment, but it's hard to see why developers should be faced with such hefty costs to make their games better after release. Yes, Microsoft has an interest in making sure games on its hardware aren't going to cause permanent hardware or security issues. But the company would also do well to ensure its developers can provide the best version of their games to its customers without having to incur extra costs or jump through too many hoops.

Apple manages to review and approve thousands of iOS app updates for similar issues without charging developers extra money for each distinct update. Sure, iOS developers often complain about delays and headaches caused by Apple's approval process, but Xbox 360 developers have been known to make similar complaints about Xbox 360 certification. And Microsoft's process can sometimes miss major problems with title updates, despite the added monetary and time costs the developer incurs.

Then again, there's nothing physically stopping Fish from ironing out the game-breaking issue his own patch is causing for some players. For all his talk about Steam's more open update system, Fish presumably knew the roadblocks inherent in Microsoft's certification process when he signed up to make Fez an exclusive Xbox 360 release. You could argue that Fish owes it to his most devoted players to fix the game he himself broke with the earlier patch. (Fish also hasn't immediately responded to a request for comment.)

In the end, no one is served by a bureaucratic process that ends up letting a game with known, fixable issues stay broken.

Kyle Orland
Kyle is the Senior Gaming Editor at Ars Technica, specializing in video game hardware and software. He has journalism and computer science degrees from University of Maryland. He is based in the Washington, DC area. Emailkyle.orland@arstechnica.com//Twitter@KyleOrl

Where's that cost come from exactly? Is it Microsoft charging a fee? Is it lost revenue because it takes a team of ten people a month to write the patch? Is it lost sales due to having a buggy game on the console? It's not clear from this article or the original how this $40K is calculated.

It seems to imply that it's a Microsoft surcharge of some sort but I'd like clarifications.

Comparing to Apple is like comparing apples to oranges. XBox is a closed system and it requires lots of $'s and a contract to get a dev kit in the first place. The approval process is a lot more involved than submitting something to the Apple app store, (or even to MS's WP7 app store). The costs involved for submitting patches were probably spelled out in detail the developer signed when obtaining their XBox360 dev kit.

It's the price developers have to pay if they want to release games to the huge captive audience that consoles provide. The large re-cert fees are also an incentive to get things right the first time. (Or abandon your users and suffer the slings and arrows).

Where's that cost come from exactly? Is it Microsoft charging a fee? Is it lost revenue because it takes a team of ten people a month to write the patch? Is it lost sales due to having a buggy game on the console? It's not clear from this article or the original how this $40K is calculated.

It seems to imply that it's a Microsoft surcharge of some sort but I'd like clarifications.

My understanding is that MS does the work to test and certify your patch, and then bills you for the cost it incurred to do so.

Double Fine had exactly the same problem with a game on XBLA, and Fish couldn't have not known about this issue before he decided to put the game exclusively on MS's service. Unless he's literally living in a cardboard box, it's reprehensible not to pay for the patch.

It seems to me that the fee is a way to encourage more thorough testing before the initial release, and do it right the first time. However, trying to realistically envision this is impossible, as you simply can't account for every possibility out there.

Microsoft owes it to their Xbox Live subscribers to provide them with the most stable platform possible. That much I understand. This means it is up to Microsoft and the developers of the applications available on their platform to create stable content. Yes, you can encourage the developers to have as few bugs to begin with, but they inevitably do get through, and by putting up such a barrier for the developers to improve the stability of their applications, we as the subscribers now begin to suffer.

Microsoft needs to accept that bugs get through, and instead, find a better way to work with developers to encourage them through other means to improve their code. Perhaps, a discount for the developers on the royalties they have to pay Microsoft for selling applications via the Xbox Live Marketplace, and the discount improves based on your track record of fewer application crashes, bug reports, etc.

It seems to me that the fee is a way to encourage more thorough testing before the initial release, and do it right the first time. However, trying to realistically envision this is impossible, as you simply can't account for every possibility out there.

The fee is not for this, the fee is because Microsoft (wheither rightly or wrongly) believes that they must ensure that each product that gets sold to XBLA is up to a lot of strict standards (including bugs). In order to do this they must put each patch through rigourous testing, which costs Microsoft a LOT of money. Microsoft then passes this charge onto the developer.

The fee is probably reasonable, there's a lot of localization and testing required for an XBLA game, and that doesn't come free. That said, pretty much the entire software world except MS has loosened things up over the last few years with the idea that faster is better, even if it's sometimes a little buggy. Devs can quickly push buggy patches out over Steam, but they can also quickly push out fixes to those buggy patches. Going through a full AAA QA process for every patch is wasteful and not really necessary.

Fish is known as a drama queen, so this isn't really surprising, even though he certainly knew about the costs and requirements going in, and decided having his game on XBLA was worth it.

So the guy messed up with the first patch and by all rights should have been aware of these costs beforehand.

This does not change the fact that $40K is a RIDICULOUS sum of money to have a few of Microsoft's QA people test a small patch for a $10 video game. Are they sifting through the code line by line over the span of a month?!? No, they're running a series of tests that are at least partially automated. If they can't manage to do this without cutting into a huge chunk of the developers profits, it's going to drive devs away from XBLA to another platform.

The rationalization of setting up a harsh punishment to make sure developers get everything right the first time is also quite flimsy. Do you really think $40K is going to be a much bigger deterrent than, say, $10K? Mistakes happen, perhaps too often these days with big studios that exert a huge amount of pressure to have the game out the door by a certain day, but alienating smaller studios is not the right thing to do.

"It’s a shitty numbers game to be playing for sure, but as a small independent, paying so much money for patches makes NO SENSE AT ALL. especially when you consider the alternative. Had FEZ been released on steam instead of XBLA, the game would have been fixed two weeks after release, at no cost to us. And if there was an issue with that patch, we could have fixed that right away too!

"but HEY! only a few months left to our XBLA exclusivity!"

Fish (two months ago) wrote:

"Fez is a console game, not a PC game. It’s made to be played with a controller, on a couch, on a Saturday morning. To me, that matters; that’s part of the medium. I get so many comments shouting at me that I’m an idiot for not making a PC version. ‘You’d make so much more money! Can’t you see? Meatboy sold more on Steam!’ Good for them. But this matters more to me than sales or revenue. It’s a console game on a console. End of story"

This goes inline with Microsoft's double-dipping policy. They double dip live gold subscribers (yearly fee + ads), and they double dip developers (pay to be in the program, and pay per patch/release). Microsoft is following Cable company scams to the letter.

On a side note, yes, this guy has a history of being a real dill-hole. No sympathy for this creep.

If MS charges this kind of fees for QA, then how did the bug get through in the first place? Isn't it their fault as well? To be honest, I'm not even sure that the developer owns enough money to pay the fee.

I'd like for everyone bitching about "Oh, he released buggy code!" to name me one piece of modern software that is COMPLETELY bug free. I'll eat my hat if you have a legitimate example. Mostly bug free doesn't count - it can't have one single bug in it.

Exactly, there isn't one.

That being said, it was extremely poor planning on this guy's part for going after the XBLA. All codes have bugs. Thinking you won't have to patch it is pure foolishness. Knowing that you can't afford the $40,000 to re-certify the code after spending more time and money fixing it, and counting on not having to do so, is pure idiocy in this day and age.

If the XBLA was the only platform on which to distribute his game, then that's one thing. But he even knew using Steam wouldn't involve this sort of issue. I'm pretty sure also that Microsoft didn't go and hide this little tidbit of information in paragraph 487b.4 section 6.

Moral of the story - actually read a contract before signing it and make sure you agree with the terms. Don't ignore it then bitch about it later.

I'd like for everyone bitching about "Oh, he released buggy code!" to name me one piece of modern software that is COMPLETELY bug free. I'll eat my hat if you have a legitimate example. Mostly bug free doesn't count - it can't have one single bug in it.

Exactly, there isn't one.

That being said, it was extremely poor planning on this guy's part for going after the XBLA. All codes have bugs. Thinking you won't have to patch it is pure foolishness. Knowing that you can't afford the $40,000 to re-certify the code after spending more time and money fixing it, and counting on not having to do so, is pure idiocy in this day and age.

I the XBLA was the only platform on which to distribute his game, then that's one thing. But he even knew using Steam wouldn't involve this sort of issue. I'm pretty sure also that Microsoft didn't go and hide this little tidbit of information in paragraph 487b.4 section 6.

Moral of the story - actually read a contract before signing it and make sure you agree with the terms.

Might want to try mainframe programming if you think all codes have bugs. In an environment where you have control of every aspect of the environment (including dev, test, and prod), yes, it is possible, and have been done, to write bug free code/applications. Mainframe environments are well known for that.

As someone who QA'd games for a job, i can say you dont always catch critical bugs like this. Sounds like it required you to be passed a certain point before the bug could be triggered. If you get a patch that says we fixed a b and c, then thats what you test the patch for, along with general playthrough. It sounds like microsoft is saying they need to recertify the game since it caused saved data to be corrupted.

Lots of negative comments by people here. I'd take a bet that not one was from an actual software developer.

Most people outside the games industry are completely clueless about the financial side of the business, and make all sorts of foolish and ignorant comments as a result.

There's a saying about not criticizing someone until you have walked a mile in their shoes.

Which is a rather massive logical fallacy.

I'm perfectly capable of criticizing someone who chooses a specific platform for a game, releases it in a known-buggy state, releases a game-destroying patch, then blames the willingly-selected platform. I don't have to know the ins-and-outs to recognize the problems here. If he chose to put out a buggy piece of code, EA-style, then he was making the decision that fixing the code post-release would be more cost-effective then fixing the code beforehand; that this decision whipped around to bite him is his own fault.

Might want to try mainframe programming if you think all codes have bugs. In an environment where you have control of every aspect of the environment (including dev, test, and prod), yes, it is possible, and have been done, to write bug free code/applications. Mainframe environments are well known for that.

Hat = eaten.

Yes I was being a bit too broad when I said any kind of modern software. I was more aiming at applications produced on windows / android / iOS / apple / etc. where the programmer doesn't have that extent of control. I could write a "Hello World" program that's bug free too, but not quite what I meant

I don't believe the $40,000 figure for a minute. It's surprising how many little stories I've seen on this, but none of the journalists/bloggers covering it thinks to contact Microsoft and confirm a figure for the developer cost of resubmitting a patch.

1) It costs money to be "exclusive" on Xbox3602) Certifying a patch costs about $40K

This doesn't look like good business sense for an Indie game developer. $40K is equivalent to a year's salary for a lot of people, and unless the game makes more money than a middle-class living -- and many indie games on Xbox Live do not even do that well: http://arstechnica.com/gaming/2011/07/xblig-feature/ -- I can see why Mr. Fish doesn't want to pay the fee.

But for a big developer, $40K is probably peanuts as part of the software maintenance budget. Not so much for an indie-dev. I hope this is a business lesson learned for Mr. Fish.

I don't believe the $40,000 figure for a minute. It's surprising how many little stories I've seen on this, but none of the journalists/bloggers covering it thinks to contact Microsoft and confirm a figure for the developer cost of resubmitting a patch.

The dude didn't adequately TEST his patch. He should have anticipated (via testing) the 'corruption' HIS software would cause.

This is in NO way Microsoft's fault. What they did was DAMAGE CONTROL for the guy's crappy code.

Yet, if they do indeed charge $40K to test his patch, then he got ripped off. After all, they released the patch and in the process, upset his users. Yes, it's his bug, so he deserves some blame. But what is the $40k for, if MS didn't even find the bug?

Those who are saying he didn't do adequate testing...there is no such thing. Bugs occur and no amount of testing can guarantee there are no bugs. You simply cannot anticipate every way a program will be used. Personally, I write my code very carefully and try to think of every possible bug. But I have colleagues who write code at twice the rate that I do, and they fix bugs as customers find them. Which is "best" depends on your goal, I guess.

For comparison, I'm a junior developer in a start-up and I get paid about that much /per year/. Now I am massively underpaid because of my relative lack of hard experience (coding in my bedroom at high school doesn't count, working for a company does), but it does help put it into perspective. (Also note that I don't live in the US, so I am unaware of the cost of living differences, it may be cheaper to live here than there so...)

Software bugs get through, and its bad enough trying to test and fix a patch for a complex subsystem that is affected by innumerable variables fast enough to limit damage. Add on a cost just for getting your game re-certified, then I can see why its not an appealing option.

To all of the people saying "He should just pay it": This is an indie game, he's probably using the money from the game for selfish things like rent and food. The company I work for is ~2 years old, and now has a revenue ~$1500, so about enough to pay me and another developer for a week.

I'm sure he knew the costs involved with publishing on XBLA ahead of time, but that doesn't change the fact that $40,000 per patch is RIDICULOUSLY high--even if Microsoft does do some testing on each patch. How much testing could they possibly be doing, anyway? They missed the bugs in the original release and in the patch, so whatever they're doing is probably basic and mostly automated.

Lets see... a couple hours of testing work, probably outsourced, around $30? Someone's gotta take the patch and actually make it available... another $30? Maybe factor in a little bandwidth delivery costs, another $30? So, cost per patch, around $100? That's still too high if you ask me, and other digital stores (Steam, Google Play, Apple Apps) prove it by not charging anything.

You're almost guaranteed to have a few bugs in the first release of your game. It's next to impossible to program something that complex and have it be be absolutely flawless--no matter how much testing you do. You are going to need to release patches. Microsoft knows this, and instead of allowing you to release them freely, they're cashing in on it at the expense of developers (or in this case consumers).