Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

University IRBs review and approve proposals for scientific research, much of which gets done on AMT. Several times, researchers (sometimes unwittingly) caused damage and pain in Turkland, even with IRB approval. After a recent research mishap on Turkopticon, a group of Turkers got together to draft guidelines for academic/research requesters working with Turkers. You can find the draft here: wiki.wearedynamo.org

We want your feedback to make sure these guidelines make sense for most Turkers.

Creating these guidelines sets expectations for IRBs, who hold researchers accountable for the ethics of their research methods. These guidelines cover issues such as the basics of how to be a good requester, how to pay fairly, and what Turkers can do if HITs are questionable.

We are inviting all Turkers to give feedback on the draft. The draft is a wiki, so you can make edits directly or post on this thread (you need to sign in first). The legitimacy of this guideline with requesters and IRB boards eventually lies in Turkers broadly supporting it. To edit the wiki sign into Dynamo and click on ‘Get login to contribute’.

In a week, after everyone has had a chance to comment, we hope to get Turkers and requesters to sign these guidelines. An editor of crowdresearch.org has invited us to announce and post the guidelines there so they can easily be found and referenced among researchers and their IRBs.

Hi there -
I’m actually in a bit of a unique position where I do work on mturk and also have been a requester (hooray grad school), and I think you could tighten up some of the things you have there (I disagree with some of it, though I think for the most part you’re clear in telling requesters - do what you want, but be up front about what you’re doing if it may bother people, which is a good rule).
I especially noted this comment: It seems many universities currently exempt online surveys from many or all IRB requirements.
This isn’t exactly true. Many online studies are given “exempt” status from the IRB. This is because they contain no information that can trace responses to the study to the research subject. The same would be true, for example, if a requester went to the mall and paid people, as long as their names were not also on the survey they took. Also, an exemption from the IRB is more of a procedural term. Phone polls are something else that is granted the same status as most of what is done on mturk.
None of these studies are truly exempt from review - they are reviewed and then granted exempt status. What that means is that the studies, as shown to the IRB, are found to have minimal harm to subjects. This typically means that, if there is any deception, that deception should be noted in a debriefing statement (at least, for studies on mturk, this would always be true) and that potentially disturbing content has a warning label. I can tell you from personal experience across multiple universities that IRB standards for “minimal harm” are quite low. Conducting a study without an IRB exempt status, at most universities, would be unusually cumbersome on mturk, since most places will then require a signed (or e-signed) consent form to go along with that study. At my university, this always requires a collection of names and contact information for study participants.
There are some academics who are, for lack of a better word, assholes. They aren’t people who will listen to these guidelines. For the most part though, we’re a well meaning bunch. Most of my colleagues who have conducted studies on mturk are very careful to pay a fair wage and do the right thing, and correct mistakes quickly.
I think one thing worth adding for you on your FAQ is for workers to contact the requester DIRECTLY before contacting a university’s IRB. 99% of the mistakes are just that, and academic requesters are eager to correct them. I think some of the issues people may have with giving out there names arise from turkers who are too quick to jump at the IRB for any little thing. It’s a bureaucratic mess once that happens. I think it needs to be made clear that trust and respect goes both ways, and workers have an obligation to make academic requesters feel comfortable that their information will not be used to damage their career.
Anyway, sorry for this novel of a message, I just figured I could give you an unique perspective here. This may be an especially useful tool, once finished, for a lot of academic requesters who are starting out using mturk. Of course, the big problems are with non-academic requesters, who can essentially do as they please when collecting data, since it’s not being done for publication purposes.

Another is on responding to messages. As a requester, you get… a LOT of messages. Obviously, there are some that require a response. There are a lot that don’t - if someone says “I forgot my code” in an email, and then I see a code box that says “I forgot my code - here is what I did in the study….”, I approve that one and don’t respond to the message. People also need to be a bit more careful. I had a survey I had to repost to get another 100 or so responses, and clearly posted a list of worker IDs. I had to reject about 5 people, and 3 of them sent messages asking me to overturn the rejection but not pay them (to their credit, they all admitted they weren’t careful and deserved a rejection). There’s not any way to do this - I think it’s something people may ask a lot, judging from that experience. One of them got a little heated with me, which was pretty annoying, and I just stopped responding to the messages.

I guess the final thing is on collection of email addresses. I think it’s totally ok to do that, especially since the mturk messaging system can be a bit cumbersome. Obviously, it’s important in that case to make it clear in the description you will be collecting emails. One thing that may be worth noting to workers is to create a “workerID@gmail.com” account to maintain anonymity. Collecting emails to send out mass invites on a bcc saves requesters a lot of time over the mturk messaging system.

I guess the final thing is on collection of email addresses. I think it’s totally ok to do that, especially since the mturk messaging system can be a bit cumbersome. Obviously, it’s important in that case to make it clear in the description you will be collecting emails. One thing that may be worth noting to workers is to create a “workerID@gmail.com” account to maintain anonymity. Collecting emails to send out mass invites on a bcc saves requesters a lot of time over the mturk messaging system.

It doesn’t matter if you think it’s cumbersome, collecting emails is not permitted on mTurk.

“You may not use Amazon Mechanical Turk for illegal or objectionable activities. Here are some examples of prohibited activities:
- collecting personal identifiable information”

https://requester.mturk.com/help/faq#restrictions_use_mturk

Amazon itself considers an email personally identifying information. I was working with a requester who wanted to get email addresses so he could notify workers when more work was posted. His HITs were pulled as violating the TOS. I asked Amazon directly why, and they said it was collection of emails. He then tried to ask people to sign up to follow his Twitter for the same reason, and the HITs were pulled again.

Not asking for emails isn’t about Turkers’ needs per se, it’s about not breaking Amazon’s rules.

I think some of the issues people may have with giving out there names arise from turkers who are too quick to jump at the IRB for any little thing. It’s a bureaucratic mess once that happens. I think it needs to be made clear that trust and respect goes both ways, and workers have an obligation to make academic requesters feel comfortable that their information will not be used to damage their career.

I’m a researcher and know the two months it can take to get a protocol approved. It is a bureaucratic pain. I also feel strongly that IRBs are institutions set up to protect “human subjects.” Any language that discourages workers from exercising this right seems against the very spirit of human subjects protections.

I disagree that Turkers should have to make academics feel comfortable about career damage. How can a Turker judge that? And what makes a researcher’s career more important than the basic rights of those in the experiment? Why doesn’t the requester have to make the Turker feel comfortable that there won’t be damage to their career?

long_cattle, are you talking about researchers who are such precarious workers that an IRB engagement will mean the loss of their jobs and reputability? Are there any cases of someone’s career actually getting into trouble from a situation where the IRB has to intervene? I’ve not heard of one personally that wasn’t in the medical sciences but I haven’t looked in depth. It’s not like these researchers are treating people with experimental drugs. It is just a headache for the researcher, but it is the only recourse for workers.

From the somewhat limited examples of IRB policies/procedures/guidelines currently available online regarding online surveys / Internet research in general, and MTurk in particular, I found there appears to be quite a bit of variation in how they handle these things (there are differences of opinion on how to interpret some terminology and policies that have been in place since before the Internet), and I think trying to spell out the many permutations of university-specific intricacies is beyond the scope of this project. We hope that the guidelines produced by this project can help both individual requesters/researchers and IRBs better adapt their policies to the realities of MTurk (e.g. understanding that they shouldn’t require researchers to collect MTurk workers’ names). There will always be some ‘assholes who won’t listen’, but all we can do is try to raise awareness among those who do care to listen.

Regarding the onerousness of responding to all the messages you may receive as a requester, I have added a paragraph to Basics of how to be a good requester: Communicate with workers promptly and politely which more thoroughly addresses this. In short, the more you follow the ‘basics of how to be a good requester’, the fewer reasons workers will have to bother spending unpaid time and revealing their identity to email you.

Regarding the inconvenience of the MTurk messaging system, if you are using the one-at-a-time method of the Requester GUI to contact a large amount of workers, it is obviously not ideal. But with one of the many open-source HIT management tools available for requesters to access the Requester API, a requester can send a message through MTurk to up to 100 workers at a time. Once you (or in some cases a more technically-minded colleague) have set up and familiarized yourself with a tool of your choice, it shouldn’t take much more work than sending a mass email, and with no risk of accidentally revealing the workers’ email addresses by neglecting to bcc (which has happened to some requesters).

I’m a researcher and know the two months it can take to get a protocol approved. It is a bureaucratic pain. I also feel strongly that IRBs are institutions set up to protect “human subjects.” Any language that discourages workers from exercising this right seems against the very spirit of human subjects protections. ... It is just a headache for the researcher, but it is the only recourse for workers.

I’d like to add that the ‘what to do in case of violations’ thread has posts regarding why many workers may already be intrinsically reluctant/unlikely to contact IRBs themselves, and an example of a worker who had to spend a lot of time and effort going “back and forth for hours and 63 emails” with the requester and some unspecified higher-level contacts at the requester’s university to reach resolution. The vast majority of workers have plenty of other ways they’d prefer to or need to spend their time; that’s not most people’s idea of fun.

It doesn’t matter if you think it’s cumbersome, collecting emails is not permitted on mTurk.

“You may not use Amazon Mechanical Turk for illegal or objectionable activities. Here are some examples of prohibited activities:
- collecting personal identifiable information”

https://requester.mturk.com/help/faq#restrictions_use_mturk

Amazon itself considers an email personally identifying information. I was working with a requester who wanted to get email addresses so he could notify workers when more work was posted. His HITs were pulled as violating the TOS. I asked Amazon directly why, and they said it was collection of emails. He then tried to ask people to sign up to follow his Twitter for the same reason, and the HITs were pulled again.

Not asking for emails isn’t about Turkers’ needs per se, it’s about not breaking Amazon’s rules.

But every snowflake is different. Not all ToS violations are created equal, either. Different workers disagree on how far they are willing to go. A recent CMU study https://docs.google.com/viewer?url=https%3A%2F%2Fwww.andrew.cmu.edu%2Fuser%2Fnicolasc%2Fpublications%2FCEVG-FC11.pdf paid workers to download software, but more than half wouldn’t do it, no matter how much money was offered. Some workers may download a program because they’re clueless and hungry, others (less than 2% in the CMU study) may set up a virtual machine ‘sandbox’ and believe their safety is ensured. A few reject it as categorical moral hazard.

But the ToS are defined by Amazon, not by us. At the IRB level, we are dealing (or ought to be dealing) with carefully prescribed research protocols. Part of this protocol is to obey the law of the land, and part of the protocol is the ‘law’ of Amazon, which requesters must subscribe to before they can even be called requesters.

I hope the bête noire counterexample for ethical requesters to mock and critique will always be the University of Minnesota experiment, whose “exogenous manipulation” of Turkopticon’s database probably inspired our current effort (and certainly provided a spur). If Minnesota had enforced its own code of ethics on use of its data center, their project could not have occurred as it did; bureaucratic complacency made that code of ethics a dead letter. Minnesota’s IRB was equally negligent of the relevant state laws prohibiting unauthorized use of other people’s computers and databases.

An amorality like that practiced by University of Minnesota does not need additional fig leaves, sanctioned by this project, in the name of “the workers” or anybody else.

Are workers in a Zyklon/B factory responsible for the final use of the product? Perhaps not … from the perspective of atomized individuals feeding their families, without recourse to collective action.

This project is a collective action. No matter what we may say, no ethical researcher can consciously violate Amazon’s ToS. I’m gratified to see attempts at educating requesters. Some researchers will violate ToS, and some workers will perform their HITs. But today we’re in the business of describing ethical conduct for the benefit of IRBs, not providing a bathroom hall pass for offenders.

A introductory reminder of the legal and ethical underpinnings of an IRB’s role might not be out of place.

I think the purpose of writing these few sentences was to let requesters know the other difficulties that Turkers have to face when they choose to use Inquisit. There are a number of solutions to your critique that I can think of.

We could remove this section, or rewrite it if you have any suggestions?

I favor removing it altogether from this document. But it is very good information. (I happen to agree with all of it). I don’t want to see it lost.

I think there needs to be a discrete project, addressing practical problems common to researchers’ interface with AMT workers. Part of that would be the more than half dozen ToS violations we commonly see from researchers whose “unique” and “original” insights demand (in their eyes) their very own special snowflake exception to the rules they already agreed to.

I’m mindful of the justifications Minnesota’s Dr. Aaron Sojourner offered for his damage to Turkopticon, implicitly staking the claim that whatever is not forbidden in three different ways somehow be somehow allowed through an uncovered loophole. The more we sweat this now, the less we bleed in the future.

Let’s focus on the ethical core ideas, with a view to an audience of academic IRBs as much as researchers. The spinoff project can deal more directly with the gritty technical details appropriate to an audience of researchers with assorted, highly specific, needs.

Elsewhere, I wrote that Inquisit was the single most common of the ‘download’ violations. But in aggregate, we probably see a greater number of “download our app” HITs, many from academics. Other researchers download Visual Basic programs. Some use Java downloads other than Inquisit. As a worker, I’ve encountered a class of HITs which prompts for permission to bypass my personal security and run unspecified Java code; I always forbid it; and I am always able to complete the HIT successfully while providing valid data. Such anecdotes should be of interest to somebody, but maybe not to an IRB.

dark_bird_of_paradise wrote of a requester who “needed” Twitter logons – and had HITs pulled from AMT by Amazon’s episodic enforcement. Our current document mentions only Facebook, and only from a privacy perspective. I don’t want us to add more information about FB or Twitter to this document. Yet serious researchers deserve to know some of the other technical drawbacks that workers don’t care about directly, because of the effects on the AMT ecosystem of such things as (e.g.) workers who admit on TO to half a dozen fake FB accounts, gleefully outdodging the dodgers. In such cases, I don’t hold the researchers blameless: they created the moral hazard themselves in the first place. Somebody indeed ought to clue them in; I just don’t think it can be the current project that does that. We can simply identify the moral hazard, and agree that Yes, sometimes it’s complicated.

If others agree, I don’t think the other project needs to wait for this one. Inside that very long sentence are three or four excellent sentences struggling to find a good home.

I think this is an excellent insight, do you think we can start this off by creating a separate page on the Wiki?
If so let me know and I’ll create the page and add a link to the sidebar.

Something like: Practical and technical problems common to researchers’ interface with AMT workers?

Edit: To clarify, right now the wiki is only used for the academic guidelines, but in the future we can consider it being used for any document that Turkers want to collaboratively write. If you decide that you would like to add a new project to the wiki to write about the practical and technical troubles that Turkers and requesters face (and how to resolve them). We can create a new space for this in the wiki and differentiate it from the academic guideline and it’s subpages. If you don’t want to use the Wiki for this you can also post about these problems on any other forum or blog and add a link to them from the current guidelines.

I’m finding myself a bit torn here. IRBs concern themselves with ethics, not legality of research. If your research is illegal, it’s the problem for the lawyers of the university, not for the IRB. The IRB is there to make sure that people are respected.

In that sense, an ethical guideline has no place saying anything that relies on TOS: TOS is a legal document only. TOS can say that you can’t use vowels in your posts, and all vowel-ridden HITs get deleted, but there’s nothing inherently unethical about vowels. Inquisit and the no-download requirement is an excellent example: it is against TOS, but is it actually unethical? Personally identifiable information, on the other hand, is against TOS and generally a worry area for ethics and IRB boards as well. That one is much clearer.

The other concern about binding an ethical document to Amazon TOS is that Amazon can change its document at any time. Should we be bound to Amazon’s whims here, like if it decides that all of a sudden it’s OK to ask for personally identifiable info?

I’m finding myself a bit torn here. IRBs concern themselves with ethics, not legality of research. If your research is illegal, it’s the problem for the lawyers of the university, not for the IRB. The IRB is there to make sure that people are respected.

In that sense, an ethical guideline has no place saying anything that relies on TOS: TOS is a legal document only. TOS can say that you can’t use vowels in your posts, and all vowel-ridden HITs get deleted, but there’s nothing inherently unethical about vowels. Inquisit and the no-download requirement is an excellent example: it is against TOS, but is it actually unethical? Personally identifiable information, on the other hand, is against TOS and generally a worry area for ethics and IRB boards as well. That one is much clearer.

The other concern about binding an ethical document to Amazon TOS is that Amazon can change its document at any time. Should we be bound to Amazon’s whims here, like if it decides that all of a sudden it’s OK to ask for personally identifiable info?

Thanks, excited_iguana. My ethical problem is not ‘with’ the TOS itself. It is with the ethics of any researcher’s protocol which, having already agreed to certain constraints (ToS) on the research, chooses to disregard those constraints. (Except for the question of proportion, I felt the same when I executed a consent form from Harvard saying I’d be remunerated 0.30 for participation, when the underlying HIT on AMT paid only 0.25). Researchers who never promised to follow the ToS aren’t unethical when they ignore Amazon’s rules – but there can be no such researchers doing business on AMT, by definition.

I disagree with your contention that an IRB can bestow ethical sanction on acts prohibited by law. Thankfully, IRBs formally incorporate such a view already, even if they occasionally sanction a catastrophe (cough University of Minnesota cough), and even if they overconcentrate on those areas of law most intimately connected to funding.

“Should we be bound by Amazon’s whims?” For defining ToS itself? Absolutely, when work on AMT is involved. (The ‘A’ stands for Amazon).

That’s why I’m sorry to see the core document get excessively mired in technical detail in what purports to be a high-level document on ethics. I certainly hope this document sees fewer revisions than it will if it ties any of its own legs directly to the five-year-old “beta test” which Amazon calls “policy.”

That’s also why I’m pleased to see our core document raise such issues as “personally identifiable information” as our own concern; because, Yes, Amazon might in theory suddenly say it’s okay, and No, for us it still isn’t.

That’s why I’m sorry to see the core document get excessively mired in technical detail in what purports to be a high-level document on ethics. I certainly hope this document sees fewer revisions than it will if it ties any of its own legs directly to the five-year-old “beta test” which Amazon calls “policy.”

That’s also why I’m pleased to see our core document raise such issues as “personally identifiable information” as our own concern; because, Yes, Amazon might in theory suddenly say it’s okay, and No, for us it still isn’t.

Woo. So let me transform this into a concrete proposal. Simply state in the document that we expect requesters to abide by the Terms of Service of the platform they are using. (Doesn’t matter the platform, really.) Remind them of the basic requirements. Then we focus the document only on issues that are ethical issues we would hold on to regardless of what the TOS says.

Woo. So let me transform this into a concrete proposal. Simply state in the document that we expect requesters to abide by the Terms of Service of the platform they are using. (Doesn’t matter the platform, really.) Remind them of the basic requirements. Then we focus the document only on issues that are ethical issues we would hold on to regardless of what the TOS says.

Thank you, excited_iguana! I like that very much. (Others and I have made the simplifying assumption all along that we were looking only at AMT as our platform. But it is stronger the way you present it here – more focused on the ethical issues, while no less valid technically).

When we remind people of the basic requirements (under Amazon Requester ToS), we can frequently link a clause of Amazon’s policy to one of our own ethical concerns, and I think it’s productive if we do. Many of the links are in place already, or nearly so; the author of much of the draft material took a comprehensive view.

As a practical matter, I don’t expect Amazon to change the main corporate butt-covering clauses in ToS, the same ones of greatest concern to us. But if they were to do that, we do not lose any credibility: We consistently insist that the agreement be kept, whatever its terms may be, and dropping (or adding) a linked clause still allows us to stand on solid ground of our own.

Inquisit and the no-download requirement is an excellent example: it is against TOS, but is it actually unethical?

(I didn’t answer this before):

If Amazon were to allow downloads as part of ToS, downloads (including Inquisit) would then only be “unethical” in the same relative sense that paying workers $4.50/hour is “unethical” by our lights as workers.

I don’t think excited_iguana would do some of the things I’ve seen workers do with downloads . But I don’t think it’s inherently unethical for a worker to run with scissors, and don’t think it’s inherently unethical for a requester to offer scissors to an ostensible adult – unless they already promised not to.

Woo. So let me transform this into a concrete proposal. Simply state in the document that we expect requesters to abide by the Terms of Service of the platform they are using. (Doesn’t matter the platform, really.) Remind them of the basic requirements. Then we focus the document only on issues that are ethical issues we would hold on to regardless of what the TOS says.

Trying to figure out concrete steps to take on this:

The wiki already has a section stating that we expect requesters to abide by AMT terms of service: Abide by AMT TOS

So cheerful_panda and excited_iguana, are you proposing that we remove all of the sections referring to technical issues on AMT (such as problems with inquisit and how to stop turkers from re-doing your HITs) from the guideline and adding them to a separate project?