Random thoughts of a computer scientist who is working behind the enemy lines; and lately turned into a double agent.

Tuesday, October 12, 2010

Be a Top Mechanical Turk Worker: You Need $5 and 5 Minutes

The current reputation system on Mechanical Turk is simply inadequate. The only built-in reputation metrics are the number of completed HITs and the approval rate.

Some people believe that they are adequate as a basic filtering mechanism. They are not.

For example, ask for all workers in your HITs to have 1000 completed HITs and 99% approval rate. You believe that you will only get high quality workers? You are wrong!

I tried to filter workers using just these metrics. I failed. Spammers got me again. (And once in, spammers submit a lot of crap. It costs nothing.) I questioned why. How can it be? And then I realized: It is trivial to beat these metrics.

Let's see the effort it takes to beat the system.

The mission: Become a top Turker, 100% approval rate and 1000 completed HITs.

Step 1: Login as a requester. Post a task, with 1000 HITs. Each HIT pays 1 cent. Total cost: $15. Out of these, $10 go to the worker, $5 go to Amazon. The title of the HIT: "Write a 500 word review". No sane worker will touch these HITs. Done. Logout.

Step 2: Login as worker, using a different email. Complete and submit the 1000 HITs created in Step 1. You just need to click submit 1000 times. Bored? iMacro and Greasemonkey can help. Done. Logout.

About Me

Panos Ipeirotis is an Associate Professor and George A. Kellner Faculty Fellow at the Department of Information, Operations, and Management Sciences at Leonard N. Stern School of Business of New York University.