Consider seniority of authors when criticizing published work?

I’ve written my fair share of letters to the editor and commentaries over the years, most of them languishing in the file drawer. It used to be impossible to get them published. The situation has improved a bit, but not enough.

In any case, I never think about the sex of the author(s). Or the race, for that matter. I do consider the seniority of the authors. When the lead author is junior — a graduate student, a post doc, a young assistant professor — I much prefer to bring problems to his or her attention using private e-mail. It’s more collegial and constructive. I know that many younger people in psychology have not had good statistical training at their colleges and universities. I like to attach relevant references of which he or she may not be be aware and so on. I’ve also done this with authors who are more senior. Interestingly, I have found that the junior authors are more likely to reply, more likely to initiate a corrigendum, etc. The senior authors are more likely to ignore my e-mail message.

The situation gets sticky when the lead author is a graduate student but the other author(s) are professors, and you know that said professors engage in a lot of questionable research practices or even research misconduct. Not too long ago, I discovered that an article published in Psychological Science re-used data from an article published earlier in another journal without so stating. In psychology, this is considered unethical. Of course, I had to alert the editor to this data re-use, but I worried a lot about the impact on the lead author (then a graduate student, now a post doc) and her career if the editor decided to retract the article. (He decided not to retract it, but to issue a corrigendum, which I think was the wrong decision.) Nick and I have had a lot of discussions about this dilemma. He thinks (a) that the student should know better and (b) that it is better for the student to have something like this happen earlier in his or her career, when recovery from such a setback is easier, than to have it happen later. I understand and have some sympathy with this view, but it still makes me feel terrible to go after a junior person.

I wonder if other people worry about this sort of thing. You could write a blog article about this sometime, Andrew, if you run out of other topics.

My reply: Let me break this up into three parts.

First, the question of contacting the author directly. I understand the appeal of this, but I usually don’t. Why not?

a. I don’t like conflict. It’s been my experience that when I do contact authors directly, they don’t like to admit error, and the conversation becomes awkward. When people send me their papers directly, I have no problem sending back by praise and criticism. But unsolicited criticism—and even unsolicited questions that are open-ended enough that they might imply criticism—that doesn’t always work so well.

b. There’s also a principle here, which is that published articles are . . . public. That’s what publishing means! So even if I can reach the authors directly, not everyone can, or will. I think there’s value in public comment on public papers. If people really don’t want their papers criticized in public, they shouldn’t go around publishing them and publicizing them.

Similarly, if you disagree with something I post here, please say so in a comment: that way others will see your reaction. It won’t just be me you’re talking to.

Second, the question of going easy on younger researchers. I don’t know about this. I published a false theorem when I was 28! I want that sort of thing corrected as soon as possible. One can also flip it around and ask, is it appropriate to be less nice to people, just cos they’re older?

I guess I’d like to think that it’s not mean to point out flaws in published work. Again, I’m happy for people to do this service for me, whether privately or publicly. I was grateful when someone sent me an airmail letter (!) with a counterexample to my false theorem, and I was grateful when someone wrote an angry blog post pointing out implausible estimates that I’d produced. In both cases, I would’ve been even happier if I’d done things right in the first place—but, conditional on the error, I appreciated the corrections.

I know that lots of you disagree with me on this issue, so by posting this I’m setting myself up for some criticism, but that’s fair. It’s good to get others’ perspectives.

41 Comments

A former student of a professor whose plagiarism I publicly exposed once suggested to me that I should have contacted him (the professor) about it privately first. My response was similar to yours, Andrew; published work should be publicly criticized. With plagiarism, perhaps, its a bit different since it’s an ethics issue too. In fact, that was another reason for not approaching him privately first. I didn’t want to give the appearance that I was somehow blackmailing him, saying, “I know what you did last summer,” and waiting for him to tell me what he could do for me to sweep it under the rug. I’ve never had that happen, but I have been in those frustrating situations of pointing out a mistake to someone privately that they then don’t correct publicly. Now you’ve got to not only correct and error but blow the whistle! So I’ve settled on your approach, more or less. Just do everything in the light of day.

Everyone makes mistakes so I always find it strange that commenting on or admitting to mistakes is such a big deal in academia, especially since I would hope we are all smart people who like to gather valid knowledge. But perhaps that is my naivety as a junior researcher speaking.
That being said, I can imagine I would feel pretty bad and embarrassed if someone publicly commented on my work. However, should someone send me an e-mail about some mistake privately, I would still want to correct it in the publication and my mistake would still become public. So then I would say that commenting publicly is preferable, especially if you explain the mistake and attach relevant references etc. This might even help other researchers avoid certain pitfalls, because everything is out in the open. More importantly, if everyone would do this, it might actually become more accepted to point out and correct mistakes, which is the way academia should be.

I reported a case of suspected data falsification to a journal earlier this year. Normally when misconduct is suspected, it’s not a good idea to contact the authors directly; I absolutely agree with Andrew that the conversation can quickly become awkward, extremely awkward. Furthermore, the authors can start to cover their tracks, delete data, etc.

But on the other hand, if you don’t contact the authors, I have found that they can try to score some sympathy points with the editors by stating that “this is the first time we have heard of these allegations. So-and-so never contacted us first.” That is, they can try to frame themselves as victims of a witch-hunt.

I tried to balance these two by contacting the authors without raising the issue of data falsification. I showed them a problem with their result even in the absence of data falsification and politely asked for their data. They declined to share the data, and that was good enough for me to show the editor that I had contacted them in good faith. Or so I hope.

Contacting authors or immediately going to editors can both be pretty dangerous for researchers. I’ve gone both routes and both have had very unpleasant consequences. The first approach led to unpleasant interactions, threatening statements about my personal safety and future ability go get published, and even defamatory statements about me to my Dean and Provost. The second approach led to legal threats when the editor shared my e-mails to him with the authors in question. Ideally, mentioning something anonymously on social media or pubpeer.com should work because most journals are signatories to the COPE ethical standards that state that editors should investigate concerns that are raised anonymously but at least some editors appear to be unaware of the ethical standards that they’ve signed up to.http://retractionwatch.com/2016/07/13/management-journal-changes-its-policy-to-avoid-fraud-qa-with-editor-patrick-wright/#more-41160

A long time ago, a colleague and I were trying reproduce some analyses that was published in a psychiatric journal. We didn’t have the raw data but could approximate the raw data from their graphs and weren’t getting anywhere close to the published results. We decided to formally write up our concerns as a Letter to the Editor. The Editor sent it to the original authors and they acknowledged they had made a coding error in the analyses and that our analyses were correct. However, they also added some post-hoc additional analyses trying to defend their final conclusions in spite of their flawed original analyses. The Editor published our letter and their response. We were not allowed to rebut their response but at least the original analyses’ errors were documented. I don’t think we would have handled things any differently if the first author had been a young researcher.

I must admit I misread ‘Consider Senility of authors’ initially.
However, the point raised is real, especially when senior authors may have a long history of questionable methods and refusing to share data.
Do you really want to give them time to circle the wagons.
Anonymous commenting has a valuable place, if those involved cannot be trusted not to try to apply unscientific pressures in the face of criticism.

I have always written criticisms to authors before contacting editors for several reasons. 1. I have the humility to recognize that my criticism could be based upon a misunderstanding of what the authors meant or did or perhaps an error on my own part. In such a case, direct correspondence with the authors can quickly resolve the issue. 2. The authors are more likely to recognize and positively respond to criticism if not as emotionally charged as an official complaint lodged through the journal. 3. It is far better for the authors and the larger community if the authors acknowledge their error rather than have a comment/response, most especially if the response attempts to support the original work by further (often weak) claims. 4. The direct correspondence is vastly more rapid than through the journal and so issues can be discussed to a natural conclusion and comment/reply ultimately published earlier. 5. While the authors are likely to be unhappy, even angry, with the authors of an attack on their prior work, I think that the obvious respect shown by an initial private discussion will help reduce long term animosity.

If I personally know and have respect one of the authors of a paper I am reviewing, I directly communicate my critique of a manuscript, even though editors generally do not like that. This can lead to a dialog that allows for disagreements to be clarified; sometimes by data or arguments not in the original manuscript. I hope that the authors, recognizing the experience of the reviewer, will take the criticism more seriously — too often authors assume that a negative review came from someone who lacks expertise instead of resulting from their own text lacking clarity.

Years ago, I wrote a comment on the statistical problems in an article that appeared in an environmental journal (even though I’m a psychologist). Subsequently I received a letter from the author castigating me for not having written to him first and asserting that this is the appropriate way to proceed. I had no idea of the truth of this assertion; certainly the editor who handled my comment did not tell me this. Possibly what’s considered appropriate varies across disciplines, then and now. and, now, of course, the advent of the social media may have changed any such traditions.

I’ve never had any trouble communicating directly with authors, but maybe that’s because I’ve never accused an author of doing anything wrong, other than confusing me. An instructive example came quite recently here, however, when my question about about an interpretation of an author’s result, which I sent directly to the author and here, got a very polite reply from the author and got me called an idiot here by an anonymous commenter. I’m not sure there are any rules here.

I tend to do just write a letter saying I’m having trouble understanding such and such in the paper and think perhaps there might be a problem and maybe even hint at what it is. What I don’t do is make an accusation. If you let them come to the truth of it then the record is more likely to get corrected since the power to do that is much more in their hands and that’s really the goal.

It also reduces conflict since about the worst they seem to do when this happens is stonewall. But I must admit that, as long as I’m not asking for data, that’s been a rare occurrence.

I simply would like to note that I have until now not received a single review / comment of an expert (within this field of research) who rebuts / refutes any of the findings of https://www.academia.edu/33827046 This report was released on 1 July 2016 (so not 1 July 2017). This report was distributed to a wide range of stakeholders. So the herculinian efforts to get reviews / comments from experts are until now unsuccessful. It therefore seems reasonable to conclude that such experts (within this field of research) do not exist.

(b) that it is better for the student to have something like this happen earlier in his or her career, when recovery from such a setback is easier, than to have it happen later.

I can see Carol’s point. Surely “recovering from such a setback” (of participation in research misconduct) is easier for senior high-ranked academics. If anything, high-ranked academics don’t suffer a setback at all, even when they are found to have co-authored work that was completely fraudulent. The case of Fredrickson & Losada comes to mind.

An additional thought: Yes, it seems that public criticism has almost no impact on the career of a senior author such as Barbara Fredrickson (as you mention), but it could have a detrimental impact on the career of a junior author, such as a graduate student, a post doc, or an untenured assistant professor. Would not hiring committees think twice before hiring a graduate student as an assistant professor whose work has been excoriated in a published comment or the social media?

A complication is that it is so often difficult to determine who is responsible for the flaws in an article. Nick Brown and I wrote a comment on a published article for which we had both the article and the master’s thesis on which the article was based. The master’s thesis was muddled, as master’s theses tend to be, but it seemed to me that the graduate student who wrote the thesis tried to do a good, conscientious, honest job. The published article based on the thesis was p-hacked to improve the results. My guess is that the p-hacking was instigated by the graduate student’s advisor, who is a full professor, and who often engages in questionable research practices (and worse). Should the graduate student suffer in any way for the sins of the professor? Nick’s position is that the graduate student should know better. I’m not so sure. Graduate students have little experience in these matters, and tend to trust what the professor tells them.

Actually, it was my colleague Nick Brown who argued that it is better for the student to have something like this happen earlier in his or her career, when recovery from such a setback is easier, than to have it happen later. Nick and I have been writing some critiques together, and so we’ve discussed this issue several times.

I am not asserting that articles authored or co-authored by graduate students and other junior people should not be critiqued, just that I prefer the gentler approach of first contacting the graduate student privately re possible statistical or other problems in the study. Graduate students are just learning, after all.

Perhaps, perhaps not. My experience over many years is that most critiques (either published, on the social media, or in private discussions or correspondence) are either ignored, or wrongly refuted, regardless of the tone of the critique. Authors rarely admit that they are wrong, even when it is easy to demonstrate that they are. Many examples of this have been discussed here on Andrew’s blog.

May I ask what your evidence is for your claim that the best way to retard learning is gentleness or politeness? One could argue that harsh attacks retard learning, because the person criticized becomes so defensive and determined to protect his or her work or thinking.

Someone told me the other day that Uri Simonsohn (Data Colada blog) always first contacts the authors of a flawed or suspicious article first, before publishing a comment. Does Uri participate here? It would be interesting to have his opinion.

> May I ask what your evidence is for your claim
May I (Keith) ask what your evidence is for your claim ;-)

The comment is based on my experiences in clinical research and most of it clinical rounds presentations – the recurring debate was whether to send email after the rounds or raise criticism in the rounds. Doing it in the rounds – firmly as opposed to harshly – did a lot more good for the presenter and especially others in the rounds. Yes, one clinical fellow wanted to escape this ordeal so much, they withdrew from the program. Folks have to learn to benefit from criticism (as a vitally important opportunity to get less wrong) – even if overly harsh.

In another area, training kickboxers, if one did firmly point their flaws, over and over again, they would go into the ring and get knocked out. Much much better for them if they necessary verbal and physical feed back in training. In fact, a sign of being a champion was getting hurt more in training that in the ring e.g. http://www.ikfkickboxing.com/PaulBiafore.htm

Recently a former head public health said that you could tell the really bright folks – when others raise criticisms and point out flaws in their work – they smile. They are recognizing an opportunity to become less wrong when it will cost them less.

Now science does need to NOT be exclusive and so overly/unnecessarily harsh is bad too. But failure to engage criticism meaningfully and publicly is, in my opinion, much worse.

As for letters to the editors, I give two instances. First, I always cc the authors when I submit the letter. In one, David Spiegelhalter immediately replied thanking me for providing an opportunity to re-think the work. That’s the ideal.

On the Data Colada blog (“Menschplaining: Three Ideas for Civil Criticism,” September 26, 2016), Uri Simonsohn wrote “Perhaps the single thing that has helped me improve the most in the civility department consists of our policy on this blog: contacting authors whose work we discuss before making things public. It is _amazing_ how much it helps.”

I certainly wish the editor of Perspectives on Psychological Science and the chair of the APS publications board had contacted me before publishing an article that lied about my work. But maybe it wouldn’t have helped. After all, I did point out the error to them after the article had been published and they showed no concern at all. I guess I’d prefer that people care about getting things right and that they accept criticism when they’re wrong; it’s not so important to me to be contacted ahead of time.

My original question, though, was whether we should be gentler with junior people than senior people, with one possibility for “gentler” being contact (or first contact) being private (e.g. e-mail). But most of the comments have been about public vs. private criticism more generally.

I don’t think that you ever said or even implied that the entire field (of psychology? of social psychology?) is inept or misguided, certainly not in the 2014 blog post of yours that Fiske cited. You and Uli Schimmack were cited; I think it only fair that the editor of PPS (is it Robert Sterberg?) and the chair of the APS publications board (dunno who that is) should be allowed a letter to the editor or some such in response.

One important thing to remember when dealing with people is that different people are different. For example, you seem to be good at responding to things quickly; not everyone is. So contacting someone ahead of time is a courtesy/kindness especially appreciated by people who are not good at thinking on their feet — and it’s an easy courtesy to implement.

It seems to me that events intended to be educational, such as the clinical rounds that you described, or brownbag research seminars in psychology, are different from published articles. In these events, students should expect public criticism, and such criticism is likely to be beneficial to all participants.

But take the situation where a graduate student’s published article makes Andrew’s “the difference between ‘significant’ and ‘not significant’ is not itself statistically significant” error. (See also Nieuwenhuis et al., 2011.) I ran across just this error in an article first-authored by a graduate student. I sent him private e-mail about it; he responded that he had never learned about this.
I have no doubt that this is so. Statistics courses in psychology departments are often not very good. In the meantime, unknown to me, there was a lot of discussion of this error on the social media, especially twitter.

If the situation were to occur again, I would still probably send private e-mail about the error, especially if the person making the error were a graduate student. I don’t see any value in ripping a graduate student to shreds on twitter or the other social media, although I do acknowledge that the twitter discussion may have brought this (not uncommon) error to the attention of people who might not otherwise have known about it.

My views come from many years of providing statistical help to people, including graduate students; to observing how different people deliver criticism; and to observing how people react to criticism delivered in different ways.

When I write a comment for publication, I usually do send a courtesy copy to the first author of the article, regardless of the status of the first author. I don’t do so if I believe that something dishonest is going on, or if I know the author(s) will be unresponsive.

If there is no response, or an unsatisfactory response, to the private communication, then I would consider a public critique.

Thanks for the thoughtful response. I add one comment: I think the venue matters. A public criticism on Andrew’s blog would almost receive more readers than anything else I’ve written (with perhaps one exception), and could easily remain more visible for many years. Moreover, unless I can understand the mistake very very quickly, it is likely that the conversation about the mistake will happen without any substantive response from me. I don’t really want this to happen, even though I certainly make mistakes, I’m happy when people take the time to mention them to me, and I am pretty fast to acknowledge mistakes (and certainly go out of my way to point out weaknesses in my own published work, even when nothing is “wrong” or inaccurate). In other words: when Andrew publishes a blog post about making some silly careless error, it is a footnote in an amazing career. If he pointed out the same error in one of my papers, that would be the best-known fact about my work. There is some asymmetry here even if the work is identical.

I don’t propose any sort of hard rule here, but it seems like a good thing if high-profile critique is mostly limited to mistakes that are either high-profile (e.g. by a famous person like Deaton, or about an urgent subject like cancer care) or essentially dishonest (e.g. pizzagate,etc). Actually I think Andrew mostly follows this already.

Perhaps. I think two different issues have gotten mixed up in the discussion here and on twitter. One issue is whether criticism of published articles should be public (a published comment or letter to the editor, or a discussion on the social media) or private (at least initially). The other issue is whether criticism of work by junior people (graduate students, post docs, and perhaps untenured assistant professors) should be gentler than criticism of work by senior people. The reason they have gotten mixed up is that private criticism is usually a gentler approach than public criticism (especially public criticism on the social media), although of course the contents of a private e-mail could be harsh.

I was more interested in people’s opinions about the second issue. A couple of my e-mail correspondents agreed that we should be more forgiving of junior people and hold the feet of the senior people to the fire. But apparently we are in the minority.

Another thought on the question of whether criticism of junior researchers ought to be gentler than that of senior researchers:

A junior researcher is likely to react to private criticism by asking a senior researcher how to respond to it. This can muddy the waters. So I think it is especially important when criticizing a junior researcher (and a good idea in any event) to give a clear explanation of what the the problem is (and why it is a problem), including references to further explanation as feasible, and perhaps even suggesting possible means for correcting/amending/retracting/qualifying the research in question. These might help the junior researcher respond professionally if a senior researcher they consult advises “ignore it” (or worse).

I think our different opinions may come down to how we trade off the harms of embarrassing/discouraging graduate students and new researchers versus common persistence of serious errors in learning from observations. Having little to no evidence for what is the least harmful trade off, I have to acknowledge I could be wronger than you.

Keith > Now science does need to NOT be exclusive and so overly/unnecessarily harsh is bad too. But failure to engage criticism meaningfully and publicly is, in my opinion, much worse.

Carol > I do acknowledge that the twitter discussion may have brought this (not uncommon) error to the attention of people who might not otherwise have known about it.

Thanks for your thoughts. I don’t know which is better, either, which is why I raised the question.

Embarassing/discouraging the junior people is not the only issue involved. Public criticism is likely to be much more devastating to the career of a junior person than to the career of a senior person, for example. Also, I think questionable research practices, or worse, are more likely to be the (intentional) doing of senior people. Junior people often just don’t know better.

In private e-mail, a colleague suggested always starting with private criticism (these days, this is likely to be via e-mail)
regardless of the status of the author(s), and proceeding to public criticism only if there is no response to the private criticism or that response is unsatisfactory. This allows to authors to correct or retract the flawed article themselves.
I have no idea of the success of this approach. Apparently this is what Uri Simonsohn and the other Data Colada blog people
do.

I agree that one seems to see the same statistical and methodological errors over and over again, and I don’t know the solution to that. It would be great if the set of reviewers for every manuscript included at least one statistically savvy reviewer, but there aren’t enough such reviewers to go around.