Main menu

Tag Archives: research scandals

Thank you Facebook for doing what many of us have been trying to do for some time, and that is raising awareness of the important and largely un-held ethical debate regarding the use of personal information and data from the internet in research. The Facebook story bought together three important things:

A commercial organisation claiming that their terms and conditions cover them to do anything they wish when researching human subjects.

Authors who didn’t make it transparent whether their research had been reviewed at a university ethical committee, what issues emerged and how they were addressed in the study.

Journal publishers who seem to take leniency in the ethical nature of their papers rather than upholding high standards of research and review.

“participation wasn’t voluntary, it was enforced, and as a result people’s state of mind may have been manipulated, in my view, without consent” (The Reed Diaries).

Peter Reed rightly suggests that advertising companies have been manipulating us since the dawn of time, and Stephen Downes agrees that commercial companies deal and trade in consumer data to bring us the products and services we would like. Companies constantly experiment with consumer data and “ethics regulations are routinely ignored”.

All of the opinions this week miss other important points regarding the clearly unethical nature of the research conducted. And also, having worked in consumer understanding within a multinational blue-chip company, the statement that we routinely ignored ethics, as did many of my collaborators, for me, is entirely garbage.

“The term ‘research’ designates an activity designed to test an hypothesis, permit conclusions to be drawn, and thereby to develop or contribute to generalizable knowledge (expressed, for example, in theories, principles, and statements of relationships)”.

Researchers have an obligation to ensure:

“Persons are treated in an ethical manner not only by respecting their decisions and protecting them from harm, but also by making efforts to secure their well-being”.

How are research ethics governed?
When conducting research it is the responsibility of the investigators and ethics boards of universities or organisations (commercial or otherwise) to evaluate the risks and benefits of their work. This is an important step on many levels, to protect the researcher, to ensure the work is of the best quality, and to also provide the required documentation to ensure the participant is fully aware of the risks, hence ‘informed’.

How should these systems work? Regardless of the field of research, whether it is medical, psychological, biomedical, sociological, the responsibility of the researcher is to identify and comply with relevant ethical protocols, and in reality this usually means complying with internal protocols alongside codes of conduct from often more than one professional bodies and organisations.

When I worked in market research, we complied with the codes of practice from the UK Market Research Society. Those companies we worked with, such as those processing store card data, also followed research codes of practice. It has amused me the sweeping statements this week about ALL commercial companies being unethical. I would even go so far to say that for commercial companies, there is much more at stake than for an academic institution. Conducting good ethical research would be directly linked to reputation, and if a company is investing half its R&D budget on a product (the rest on packaging and even more on marketing), the research underpinning it needs to be robust. I suggest that with both universities and companies conducting human research, there is a wide range of ethical standards employed, but we cannot tar everyone with the same brush.

“Collecting information from a social networking service” and “any other collection of personal data in the online environment for the purpose of research”.

And therefore it follows:

“In accordance with the principle of voluntary informed participation, information identifying respondents (personal data) must not be collected from without their consent”, and “researchers conducting social media monitoring, text analytics or sentiment analysis should take steps to avoid collecting personal data. Researcher must ensure that any personal data is not further processed without consent”.

And ethical codes particularly govern working with children and vulnerable individuals:

“In accordance with the principle of voluntary informed participation, information identifying children must not be collected from forums, social networking sites, blogs, etc without their consent and without obtaining the consent of a parent or responsible adult”.

The flaw in the research published this week, and many other studies like it, is that the risks and benefits of the study were not fully weighed up. The methods of ‘informing’ the participants about such risks and the nature of the work was not in place, and there were no means of ensuring each participant was “truly informed?” Many people think that any information on the internet is up for grabs, and the ‘public private’ argument is a bit of a smoke screen. All internet research requires ethical consideration. Fact.

With research on Facebook, Twitter, MOOC platforms and other social networking services, the answer appears far to simple. The Terms and Conditions increasingly seem to out trump the need for any ethical discussion. This is a horrific situation, and if it persists, it gives anyone researching carte blanche to experiment with human subjects, including possibly minors or vulnerable people.

So what were the risks?So we have established that the study was a deliberate experiment with a hypothesis that had considered the recruitment of individuals into a study. In January 2012, “participants were randomly selected based on their User ID. The experiment manipulated the extent to which people (N = 689,003) were exposed to emotional expressions in their News Feed”. (Kramer et al 2014).

The outcome was they had manipulated people’s emotional thoughts through a social network, not a new finding, but a study done on a large scale none the less. So what might the risks have been to individuals?

The UK Office of National Statistics in their ‘Social Trends 41 for Health’ analysis in 2009/2010 shows that 1 in 10 adults (11%) of England were diagnosed with depression. That is 11% of adults within that year, not the prevalence over a lifetime which is much higher. Over the preceding decade, prescriptions of antidepressants increased by 334% in England. You will find similar figures on the internet elsewhere, so assuming therefore that a fair proportion of study participants might have been depressed, what might the impact on them have been, and what might the impact on non-depressed individuals have been over the research period? Well both negative thoughts, and levels of rumination and dwelling on negative thoughts are measured outcomes in studies of depression; research suggests negativity may or may not contribute to it, and work in this field is generally in its infancy (Pasyugina 2014, Takano 2014).

Within this population there would undoubtedly have been risks. Individuals were not fully informed of the implications and that they were participating in an experiment, with none of the affordances of being free to withdraw at any time. But it doesn’t matter because they gave their consent right?

What do the Facebook terms cover?The authors claim that signing up to a Facebook account constitutes informed consent for this research. At least they made passing mention of informed consent. In another interesting Facebook clinical study looking at using the service as an enhancement to therapy there was no mention of ethical approval within the publication that had conducted research on vulnerable participants.

So, searching through the “Full Data Use Policy” that they refer to, I couldn’t find any reference to the statement made by the authors in their paper, that creating an account constitutes informed consent. Maybe I just missed it, but this hardly makes for users of Facebook being fully or truly informed that research might be taking place?

In all of the terms I could only find ONE mention of the word ‘research’ and how Facebook uses your information:

“for internal operations, including troubleshooting, data analysis, testing, research and service improvement”.

It also states to what extent they share their data with others, and I’m assuming here that they shared their data with the researchers in order for them to analyse it and write up the study.

“While you are allowing us to use the information we receive about you, you always own all of your information. Your trust is important to us, which is why we don’t share information we receive about you with others unless we have:

received your permission; given you notice, such as by telling you about it in this policy; or removed your name and any other personally identifying information from it.”

I suspect the 600,000 odd participants were not invited to provide their permission.

Where does this leave us?Unfortunately we cannot stop social networks and corporations setting their terms and conditions to suit their commercial needs. I’m not sure whether they can legitimately ‘out trump’ the need for any ethical approval governing research it might undertake because these are legal terms, not philosophical arguments. Facebook can simply amend their terms now to cover research, and make their mechanisms of informed consent more explicit, but academics and publishers can still commit to good quality research? Surely, those researchers who published this week had some moral and ethical conscience, or were they gilded by the prospect of funding and reputation-boosting work?

Academic publishers should surely want to uphold the standards of the articles that they produce, and to uphold the rigours of the peer-review process? Some journals do send papers back that haven’t been ethically considered, and these stances are far more rigours in science and medicine than in other subject areas.

What do the public think?In all this, nobody knows what the public thinks. We know what opinion makers on social networks think, and we know what some researchers like myself might think. A few conversations with my friends reveals that it is just generally accepted that everyone gathers your personal data anyway. What they didn’t understand were the implications of the research that might have led to risks to the participants.

The situation is that some less ethical corporations allow their socio-ethical boundaries to gradually creep forwards, and it is only when we look back at how far they have been stretched that we get a perspective of the distance of travel. My personal view is if academic researchers and academic publishers do not start contributing to the debate and to govern the quality of research, then who will?

One friend drew parallels with the ‘frog in the pot’ experiment. Don’t worry, no frogs were harmed in the making of the video.

Here we are again with the “Research Quality’s Scandals of 2014“. Those of you awake out there might notice that George White performed his “scandals” or reviews on an annual basis, you will note that on my blog there is ample evidence to write about “research scandals” much more frequently.

Why today?

Well I was anticipating a nice morning writing an abstract for a forthcoming conference (Open Ed 14). As usual, once I started to delve into the literature, practically none of it was accessible at all. Maybe this was a typically bad day, but not really. This experience was after more or less two hours of research and finding numerous interesting papers, and then not being able to openly access a single one of them. Rather ironically the subject was “open learning”.

This really is so bad, but why?

There are many people who dedicate their life to education. Ha. I wouldn’t be sitting here on a Bank Holiday corresponding with students and doing research if I wasn’t one of them. Few of us actually have time to conduct good quality research, and even fewer have the luxury of any funding these days to support their research.

So I find this so deeply annoying because research is so sparse and so precious, particularly anything of good quality, and when you do find it you cannot access it! I do not want to pay $39 and my institution doesn’t subscribe to much of it. Come on, a paper from 1970 surely could be released into the public domain now couldn’t it? Apart from the immorality of it all, that tax payer and public money has funded the work and the livelihoods of the researchers, and then it becomes a commodity to sell on.

And what if you don’t pay for it?

Publication bias is huge – not just from the predominance of publishers and authors only writing about “positive” results, but increasingly these days, people will only write about what they can access for free. So all that stuff hidden behind paywalls is excluded from the discussions and thinking.

Ultimately we are wasting time and re-inventing the wheel, and we are always blaming ourselves for this! I think it is criminal to not learn from previous knowledge. I really do want to know what Roger Lewis wrote about open learning (“What is open learning“?) because there were discussions in the 1970’s about distance learning and the advent of open learning that are important for today’s online learners. This was one of many papers that I found this morning, and I can’t read any of them.

Will the future look brighter?

Well policy is moving slowly in support of open access but that isn’t going to help the back catalogue of papers that are closed off to those that don’t have the money to buy them.

In a recent presentation I talked about changes to the UK research assessment exercise and funding body specifications toward open access.

Richard wrote a great article on the BMJ blog reflecting on a previous editorial by statistician Doug Altman 20 years ago, and how little had changed in the quality of medical research. I often ponder this myself, not just within science and medicine, but education research where I tend to hang out these days.

Altman’s original reasoning was that researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them. Ethics committees, who approve research, were ill equipped to detect scientific flaws, and many journals lacked statistical skills and published misleading research. He suggested back then that we needed less but better quality research.

Richard observes that 20 years on, Altman’s article could be published again unchanged, with the observations that poor methodological quality and researchers who chase careers and are under high pressure to publish, are in fact part of the problem today .

I agree with all of that, but also think this.

It is pretty safe to say, if you set a target, then people will work towards that target. Research excellence frameworks are geared around publications, and there is no getting away from the fact that papers are the academic currency for research returns and for promotion. Anyone interviewed for an academic post will be asked if they are REFable? I think the judgment that we are all career-crazed is a bit harsh; most researchers I know are just very passionate (and slightly mad) about their area of work and are just keen to share the results by publishing. Saying that I do know folk who withhold their papers from publishing to meet REF targets.

What else might be part of the problem?

I can understand publishers wanting articles that make great discoveries, but this biases the literature toward positive results (part of publication bias to which we all contribute); but just as important are the things that don’t work. I can think of one of my own which didn’t advance our understanding of fish oil as a therapy for inflammatory bowel disease in children, but did apply novel methods of fluorescence confocal microscopy for studying human intestinal biospies structurally and functionally (based on Naftalin’s work that we adapted for human). The paper was “not interesting enough”.

I’d be the last to advocate yet another series of measures or frameworks, but there is no quality assurance for research methodologies within universities or hospitals, and that is not the function of the ethical committee I feel. Lordy, I have already known NHS ethics to take up to two years to be granted in some cases, so we don’t want to add to the process. As a researcher if you are lucky, you can be allied to a team where much of your methodology learning can take place, but I can’t remember at any time in my PhD or PostDocs being taught about experimental design or the correct statistics. Some of my best learning has been through journal peer-review – that is, receiving comments from accomplished reviewers, and you soon learn to be a hard nut to take it on the chin.

What about education research?

The problem for me as a novice moving from science to education research is that many of the really informative papers are locked behind paywalls. I am trying to write a research methods paper with bioscience colleagues at the moment, and a number of seminal references are not openly available. Surely, items so fundamentally important to the quality of “our business” should be openly available to be accessed by all? Who in reality these days is inclined to wait 2-3 weeks for an article to be purchased through intra-library loans? We just don’t do it, so our learning is not made easy.

What do we teach to science and medical undergraduates? On my Medical Science degree at De Montfort Univeristy I had modules on “Health Informatics” that looked at data retrieval, search strategies, and appraisal of medical research. A second module on “Evidence Based Medicine” covered clinical trial design, publishing industry, critical appraisal, systematic review and meta analysis. Come to think of it, Medical Sciences also had a “Medical Statistics” module and this culminated in a final year project. I have never come across another EBM module taught to undergraduate scientists.

There is good news for medical research!

The good news for medical research is at least there is a reasonable number of good quality studies that are published. If you were in the field of education research, that would be a different matter. In my recent systematic review of how massive online open courses (MOOCs) support the student experience, I started with 141 potential articles that had investigated the subject, of which only 25 were empirical studies, of which only 1 had a control group so could be deemed of any reasonable quality (Rolfe 2013).

If this were a medical systematic review, none of these studies would have passed the quality checks, although it is acceptable to broaden the entry gates for education research to encompass the practicalities of the classroom (Evans and Benefield 2001), and many other articles have discussed this. In another systematic review by Douglas Gray and myself, 176 studies were identified through the keyword search and were whittled down to just 38 empirical studies of which 21 were excluded because of having no control group, or due to having crucial data being missing. The concept that papers publish even without including numbers of participants, or results of statistical tests, is quite amazing!

Just to convince myself I’m not going mad, here is one further systematic review led by Barbara Means in 2010 at the US Department of Education that looked at literature to compare blended learning with face-to-face approaches, and were there benefits to the learner?

The article states:

The most unexpected finding was that an extensive initial search of the published literature from 1996 through 2006 found no experimental or controlled quasi-experimental studies that both compared the learning effectiveness of online and face-to-face instruction for K–12 students and provided sufficient data for inclusion in a meta-analysis. A subsequent search extended the time frame for studies.

What can we do about it?

“My confidence that things can only get better has largely drained away” says Richard. I don’t think I’m quite at that point although the water might be starting to swirl toward the plug hole!

In our Bioscience community (which will never close), the Higher Education Academy runs research workshops and events that are always well attended. For our subject there is an active community on email. A group of us are working on a guide to getting started in research, although our reviewers suggest we aim for a higher academic and theoretical starting point than originally intended, which is exactly what we didn’t want to do as picking up the language of research is often a difficult starting point.

“OPEN” is the way forward I am sure, with more:

Open access articles for basic methodology papers

Open educational resources instructing researchers in how to get started

Open educational resources and research method modules for our students

On-line communities working together

Now there is an idea! I might have to put my money where my mouth is now.

And for some of the real George White’s Scandals of 1935, go to YouTube! Now HERE is what I call a stage entrance.