I'm working on an HTML5/webgame and am considering utilizing Mechanical Turk as a source of alpha/beta testers. Something like "$1 to create an account and beat the first 3 levels" and then require feedback like "General Response" "Annoyances about the game" "Enjoyable parts of the game" etc.

Has anyone attempted something similar, and was the feedback you received useful or largely garbage?

Heh so this is probably the last thing you want to here. Got caught up in life, lost motivation and interest on that particular video-game, etc, etc. Have had a few ideas since then but never gotten them far enough to consider beta testing.

Thanks for the reminder though... maybe I should go back and work on that game again :)

Sure, this might not be the best explanation, but I'll try to tailor it towards the idea we're discussing in this thread.

To start off with a quick definition from Wikipedia:

"The name Mechanical Turk comes from "The Turk," a chess-playing automaton of the 18th century, which was made by Wolfgang von Kempelen. It toured Europe, beating the likes of Napoleon Bonaparte and Benjamin Franklin. It was later revealed that this "machine" was not an automaton at all, but was in fact a chess master hidden in a special compartment controlling its operations. Likewise, the Mechanical Turk web service allows humans to help the machines of today perform tasks for which they are not suited."

Wow. That wasn't quick at all. TL;DR, Mechanical Turk is a site designed to take tasks that we think computers should be able to do, but, can't and have human do them for a small amount of money (usually $0.01-1.00)

The way MTurk works for a worker like me, is I'll log in and be greeted with a tremendously large menu of "Human Intelligence Tasks" (HITs) to do. I select one, I can preview what exactly it requires me to do, and then accept it. I am given a time limit. After I complete the task, I submit it for review by the person who requested the HIT. If you've ever played Final Fantasy Tactics Advance, it's basically like using a virtual job board. Uplink is also decently analogous.

For a requester, basically, you set up the requirements to be fulfilled, and then ask people to accept your hit and complete it. Afterwards you'll be able to see things like the amount of time it took them to complete the HIT and what they actually accomplished. Examining the site might help you here, but there are a lot of techniques that can be used to make sure your workers are doing quality testing. If there's interest, I can explain some of the things I've seen.

As far as payment, from my understanding you set the value you want to pay, and then deposit an amount into amazon payments. Everytime you approve someone, it subtracts that amount from your account.

If there are any other questions anyone has let me know, and I'll be happy to answer them.

Also, there are excellent subreddits in /r/mturk and /r/HITsWorthTurkingFor. These are oriented towards workers, but they may be able to help you out as well.

That's actually pretty neat. Thanks for the great explanation. How much would you say you earn doing this? (If you don't mind me asking) And what kinds of tasks do you usually complete? (Are there more profitable/easy/more fulfilling ones?)

I use /r/HITsWorthTurkingFor and the unofficial forums to find good HITs, which are mostly psychological studies from universities, or marketing surveys. I usually will do anything for a short amount of time, but as the tasks get longer the more money it takes to entice me

I've made I think around $50-60 right now, but I'm a pretty infrequent worker as a university student with another part time job.

I used to use Mechanical Turk for a while. If you're getting consistently good HITs you'll get at least 10 cents per minute or 6 dollars per hour. It's a pretty good wage for just sitting in front of your computer for a few hours.

I'll definitely get back to you once I have it set up. My game is still pre-alpha, and it'll probably be a couple of months before I start paid testing. What do you think is a fair amount for something like this? Each level could take someone a couple of minutes, but I'm worried about how to handle people who are really bad at it. I don't want them wasting a ton of time for no reward.

FGL actually uses mTurk to do at least some of their first impressions. I found this out when I ordered a video review and saw in the player's browser url that he was playing on mTurk. Then I did a search and found one of their mTurk postings... for every $1 review you buy from them, they pay the tester $0.20. They require players to first complete a video game review qualification before they can start reviewing (no idea how hard or easy that is).

I've found their reviews to be very hit and miss. Sometimes you get amazing paragraphs of feedback, but most of the time you just get a couple short comments. If the review is completely useless they will get you a new one.

Since you can put html5 games on FGL, this could be an easy way for you to get mTurk reviews without having to do any of the approving / admin stuff yourself.

I think if you really can't find testers that are willing to fill out a survey, then this might be a solution. But it's likely not going to be your target audience and they'll be there to make a buck, rather than play the game.

You can reject hits (they don't get paid) if your requirements weren't met. I'm actually going to require them to create a user/pass as part of the process, and I'll require them to enter that on the hit. I can check which levels have been played by which users in the database.

I think it would be better to buy or volunteer more detailed and in depth QA. This is not a trivial thing to do, it's not something like "translate this phrase into spanish" or "identify the objects in this photo," it's much more objective. You'll want quite detailed information from your playtesters, more than someone would be willing to do for $1. I like the idea of mturk, but it's really suited to monotonous tasks that require a minimum of effort and can be massively parallelized. This just isn't one of those tasks.

It's worth a shot though, you won't be spending too much to see what results you get. And of course this might have the opposite effect, a scattershot of opinions rather than a few people's focused opinions.

I'd be cautious, I've previously dropped money on "QA groups" from companies and I ended up paying 100 bucks for a pile of people making really shallow observations of the graphics on the start menu.

Honestly when testing out the first couple minutes of a game, I had MUCH, MUCH superior results from going on forums about free games, primarily visited by younger audiences. I had over 50% of them respond with meaningful comments. There was a lot of suggestions I didn't really worry about reading but it gave me a much better return on the cost :P

Any specific forums you'd recommend? Also, are any of them receptive to someone who joins and says "Hey, check out my game!" If they aren't, I don't want to be the jerk who joins and suddenly posts self-advertising without a history on teh message board.

I see no reason not to ask them for their Worker ID on your own system and avoid breaching their ToS altogether. Amazon won't likely budge on it, from my experience.

Good luck with the email and your project though.

If you want quality work done for your project, you should pay at least $6/hr ($0.10 per minute of work), and be fair. Explain everything properly, and test the systems the workers would be using to do their job before you post your HIT.

Workers rate requesters and the quality workers will refuse to do your work if you have a bad reputation.

Sorry if I seem a bit rough, but I've used MTurk to help pay bills, as do many others who are worse off than me. I'd like to take the chance to help people who post HITs there do them properly so everyone wins.

Oh! One more thing. The requester interface will try and push Masters qualification onto you. In general, this is a bad idea. No one knows what Amazon uses to rate someone good enough for that qualification, and I've seen HITs go untouched that would've normally been completed because it required it.

The best way to still reach a larger number of workers and still have quality work is by restricting your HITs by both approved HIT count and approval rating. For example, a lot of requesters require their workers to have at least 1,000 HITs approved and a 95% HIT approval rating.

The most common approval counts seem to be: 50, 100, 500, 1000, 5000, and 10000. I would personally say either 500 or 1000.

Don't worry about that. I know a lot of people are going to do the bare minimum to move on to the next HIT, because who wouldn't when its a struggle to make anything close to $6/hr, but I'm going to leave a nice optional open response for anyone who enjoys the game, or just in general wants to provide extra feedback.

If you want to do some early testing let me know, I can send you a message with the URL and such. It's playable right now, just not complete so far as how I intend the game to play out.

That would be cool. Send me the link and I'll go ahead and check it out. What's the concept you are going for? And honestly I've seen some good feedback come from /r/gaming but only if it's ready for a beta test and you have the concept clear.

I use Mechanical Turk as a requester. Set your constraints to %95 approval and US based location only. They make it pretty easy to use. Pilot the play testing. Spend 10 bucks; look at the results; modify the questions/instructions and I suspect you will be successful.