Quick Links

Club PA 2.0 has arrived! If you'd like to access some extra PA content and help support the forums, check it out at patreon.com/ClubPA

The image size limit has been raised to 1mb! Anything larger than that should be linked to. This is a HARD limit, please do not abuse it.

Our new Indie Games subforum is now open for business in G&T. Go and check it out, you might land a code for a free game. If you're developing an indie game and want to post about it, follow these directions. If you don't, he'll break your legs! Hahaha! Seriously though.

Our rules have been updated and given their own forum. Go and look at them! They are nice, and there may be new ones that you didn't know about! Hooray for rules! Hooray for The System! Hooray for Conforming!

Trenches comic: Thursday Dec. 1, 2011

“Has anyone tried maxing out all 4 players’ coins and killing them all simultaneously?”

The team that I’d just joined shook their heads.

“Not yet. Dude, do you have any idea how long that would take?”

We were testing a newly-added multi-player game mode where players collected coins, and—should they die—dumped those coins into the environment for other players to steal. I’d just been added to the team, but the rest had been pulling 12+ hour shifts for months now. They were tired, ground down by the monotony, out of ideas.

“Do we have something better to do?”

It was 10 AM. We started right away.

We spent the next 12 hours collecting coins on 3 characters, getting two players up to 9999 coins and a 3rd up to roughly 7000 before the tedium overwhelmed us. We placed those three characters together and, using the 4th character as a “camera,” we set off a bomb that killed all three on the screen.

A deluge of coins dumped into the environment non-stop for 20 or 30 seconds. At first, everything seemed okay. But then, the frame-rates began to dip, then the screen began to visibly chug, then the game froze momentarily, and then… blue screen. A hard lockup.

We roared a cheer and began to congratulate each other..

“What did you do?!?”

The assistant coordinator approached us from his desk and we explained what we had done and how long it had taken to set up.

“Alright,” he said, turning, already heading to sit back down at his desk, “Reproduce it.”

I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.
I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

Not 10k, closer to 1k. And actually if it took 12 hours both times for all 5 people it would be 1,200.00

I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.
I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

Not 10k, closer to 1k. And actually if it took 12 hours both times for all 5 people it would be 1,200.00

Probably more like 2K, once you factor in taxes and so forth, but still, a terrible waste of money for a pretty standard hook. That said, some testers only make minimum wage.

I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

Adding in cheat codes that work properly and don't otherwise mess up the normal play of the game would require more QA, so you have to take the cost benefit of do I spend $1200 to code in a cheat, QA it, and then another $50 to do the testing, or do I just let them test without the cheat code for only $1200. Sure, the cost-benefit may break out for some bigger/harder to reproduce bugs, but I imagine for a lot of bugs it just isn't worth it.

Not to mention that you have to take those codes out/turn them off before you ship, which requires more QA (and thus more money).

Not cheat codes, but a developer console. They've been a natural part of development and QA since the dawn of time, and except for the weird cases like Id or Bethesda, they are never intended to make it into the final product.

I was also shocked at the idea that the devs wouldn't include some cheat codes for QA during development. This is simply inexcusable. They spent 50 man hours on a task that should have taken 1. Assuming your lowly QA person is making a paltry $10/hr (that might be generous, I don't know), they just spent $10k to reproduce a "bug" that will likely get filed in the "will not fix" category. Some people just have no business sense.

I would imagine that most of the submitted stories are true, if not embellished a tad, but the stupidity of some managers astonishes me.

Adding in cheat codes that work properly and don't otherwise mess up the normal play of the game would require more QA, so you have to take the cost benefit of do I spend $1200 to code in a cheat, QA it, and then another $50 to do the testing, or do I just let them test without the cheat code for only $1200. Sure, the cost-benefit may break out for some bigger/harder to reproduce bugs, but I imagine for a lot of bugs it just isn't worth it.

Not to mention that you have to take those codes out/turn them off before you ship, which requires more QA (and thus more money).

I really don't see that breaking things. I mean, I suppose if things are coded poorly, maybe. But ultimately it's probably just an easily-modified counter. Unless the game does a whole ton of other shit when you collect a coin besides just incrementing the counter, it shouldn't be hard to do that programmatically.

That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

What the assistant coordinator said was nothing compared to what the developers would have said, they would have said something along the lines of, "Reproduce it until you can tell us just how many coins it takes to cause the crash."

ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.

Yes a game like that NEEDS a dev console. Although dev consoles can introduce bugs, or hide bugs by allowing testers shortcut past things. I knew of a bug in a game where if you left a door open in one of the starting levels, then went back to the level near the end of the game, when all the levels were opened, the door would start closed (it's default position) but could not be opened because the saved data said it was already open. No one found it because they tested the end game by skipping to it with the dev codes so the door was always closed. That's just a small sample. But just adding coins to a player shouldn't be a big issue.

Zazu Yen on December 2011

ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.

That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

I think that you're severely underestimating how much people like doing shit like that in games. That said, I think a player actually would use a mod or a cheat to pull it off, a la

I would think putting in some kind of limit on the number of coins that a player can drop would solve the issue. Or really, is it necessary to even allow them to pick up 9999 to begin with? That seems really excessive. Even Mario rolled over at 100 ;)

See, I figured it was supposed to be a kilt, but I didn't understand why it looked like it was cut into big strips. But I guess you can actually see some tan color between them, so it just must be the folds.

That's such an edge case I wouldn't even worry about testing something like that. Sure its a test, but not a valuable test considering the setup required in addition to the chance that would actually happen in the real world. I'd think maybe .001% of the population would actually spend 12 hours grinding through to max out the coins just to see what would happen. Past that, the end user isn't normally going to enter that state so maybe running some boundary cases where you could get a large ammount of coins (in a reasonalbe amount of time based off of normal game play) and see if that situation (all dying at the same time) resulted in the same performance hit.

I tend to agree. Here, that would come under the rare category of a high severity, low priority defect; it's really serious, as it causes a hard lock up and kills the user experience. On the other hand, the reproduction path is so ridiculous that only people *looking* to crash the game are going to find it and get their game crashed. SO it would go on the backlog.

That said, not working game QA, I'm finding some of these stories a bit bizarre - do people really have the spare time to spend on setting up an edge case like this? I suspect if I set my team doing something similar for a day, I would receive several angry emails from a righteously peeved PM.

Is this amount of dead time common over in the games industry? Seems like we've had a few stories about it...

It's not so much dead time as they're probably on a near final build and aren't finding any bugs and are desperate to do something they haven't tried yet to not only break up the tedium but potentially find a bug. They potentially could have found many bugs on the road to creating this one so it wasn't like all the time was wasted. Sometimes some really important bugs are discovered due to that yearning to try something different.

I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

They were asked to test dropping 9999x4 coins on the screen.

I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

Savant Way
Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

Smart Way
Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

Ballsy Way
Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

Ghetto Way
Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

Spiteful Way
Make one person do the tedium and copy the characters.

It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

They weren't tasked to do it, they decided to do it. Generally games testing isn't limited by following test cases and a test plan due to how many working components they have so they're given a generic "test multiplayer" sort of task and they decide what that means and how to do it.

I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

They were asked to test dropping 9999x4 coins on the screen.

I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

Savant Way
Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

Smart Way
Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

Ballsy Way
Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

Ghetto Way
Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

Spiteful Way
Make one person do the tedium and copy the characters.

It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

I hadn't seen any really silly gooses on this forum 'til now. I guess it's nice to know there's at least one.

Simply, addressing the writer's story in any of the ways you described would have resulted in no story. He may have actually done one of the things you describe, and made up the story anyway, 'cause people do that. Probably, there are specifics of the story he left out, because who cares?

Like someone said in a prior comic thread, the extreme anonymity of these stories (90% of the time we don't even know the game -- or type of game -- they're talking about) turns most of them into Cool Stories Bro.

Like someone said in a prior comic thread, the extreme anonymity of these stories (90% of the time we don't even know the game -- or type of game -- they're talking about) turns most of them into Cool Stories Bro.

Do people fear reprisals? Or is it just not important to the story? Trash talking past employers is always bad form. Exceptions might be made for specific heinous deeds that can be backed up with facts and you still managed to take the high road in the situation.

I can’t say what this task was hoping to accomplish, I can only assume they were attempting to test the graphical limitations.

They were asked to test dropping 9999x4 coins on the screen.

I can tell you right now that you don’t need to get 9999x4 to test performance issues. I’d bet dollars to coins that you could have accomplished that crash with just a few hundred coins.

If the test case was to SPECIFICALLY test WHAT would happen if 9999x4 coins are dropped at once, this team failed miserably. In fact, they didn’t even kill all 4 players, they only killed 3 of them and the 4 was standing off to the side. I can only assume it was the guy with 7000 coins and was probably napping most of the time. If I were to go to my producer and tell him, well, we got 9999x3, you’re damn sure he’d tell me to go back and do it again the right f-ing way. I don’t about you but here we don’t get paid to half-ass our tasks.

So incompetent testers aside, there are a myriad of methods you unimaginative peons could have used to save yourself some time.

Savant Way
Debug the coins to yourself. I don’t know what kind of sadistic programmer wouldn’t have a debug command to give coins.

Smart Way
Create a save file. This seems like the most basic step. Make a save file you can reload. You know, so you don’t have to spend another 12 hours reproducing nothing. Load the file with you maxed out, have everyone die, did you crash? Yes.

Ballsy Way
Ask the programmers to add a test creature that drops 9999 coins. Trust me, they can do it in about 5 minutes.

Ghetto Way
Grab a rubber band. Wrap up that controller and go to lunch. Come back and save yourself some time.

Spiteful Way
Make one person do the tedium and copy the characters.

It’s no wonder that this team has been pulling 12 hour shifts for “months” if they’re off performing meaningless tasks like this. Look, I know QA is all about performing monotonous tasks but what separates the kings from the peasants is the ability to perform tasks quickly and efficiently.

I hadn't seen any really silly gooses on this forum 'til now. I guess it's nice to know there's at least one.

Simply, addressing the writer's story in any of the ways you described would have resulted in no story. He may have actually done one of the things you describe, and made up the story anyway, 'cause people do that. Probably, there are specifics of the story he left out, because who cares?

Wow, JPandarelli manages to be condescending, snarky and wrong all in one go. Well done!

A. They were not tasked with this, they chose to do this themselves because they had nothing else to do. This isn't their fault, it's a resource management FAIL. You would be a TERRIBLE boss for pink slipping guys who made their own work when their managers weren't making sure they were kept busy. Even if you don't value what they chose to do, it's not their fault they were directionless.

B. All of your "ways" assume these guys had access to dev tools and/or the developers themselves or that the game allowed saving mid-stream. I think it's pretty obvious that none of those things were available. This sounds like some kind of consol Mario/Sonic knockoff (if not an original) most of which dont allow saving mid level (I'm not aware of one that does, actually). Since it's in multiplayer I'm going to go ahead and say there's no saving.

C. If this is a big studio project it's quite likely the QA wasn't in the same building, city or even state as the actual programmers AND if done correctly any additional dev tool added to a product (particularly one late in development) would have to go through proper channels or the dev who did it could get in real trouble. Particularly if it caused problems or made it into production.

D. Also, since this was a late addition to the game it's quite likely all the dev tools were already cut out, or they were disabled for multiplayer to prevent cheaters.

E. The original Sonic could get over 100 rings on screen without crashing. Frame rate drop, sure, but actual death, no. I'm pretty sure anything made in the last decade targeted at anything more powerful than a handheld could get upwards of 1000 coins before destabilizing. (There are a LOT of factors involved in this, naturally, so your game milage may vary)

Achievement Unlocked: Snark Backfire

ExistentialExistenceException: Your thread encountered a NULL pointer and entered a state of non-being.

I think the original Sonic (and all the Genesis ones) actually had a cap on how many rings would fly out when hit, and it was well below 100. Sure, you had 100 rings, and you dropped to 0 when hit, but there were only maybe 40 rings bouncing around.