Plan 9 from PPNAS

“the study results showed that one daily 6- to 8-ounce serving increment of 100% fruit juice was associated with a small .003 unit increase in body mass index over one year in children of all ages.”

No confidence intervals but obviously this finding is very worrisome. Children shouldn’t be gaining weight.

Meir continues:

Of course it’s not a coincidence that it’s weird. I send you a very unrepresentative sample of the stuff I read. I mostly don’t send you ordinary schlock but rather things that are really weird – like a “0.003 unit increase in BMI” which is not only statistically insignificant but even if it was able to be substantiated would be of 0 health consequences.

I really enjoy seeing things like this, they are so ridiculous they are like those cult movies that are so bad they’re good.

P.S. Yeah, yeah, I know that this particular piece of junk science didn’t appear in PPNAS. But until PPNAS apologizes for wasting the world’s time with air rage, himmicanes, ages ending in 9, etc., I think we have the moral right to continue to use them as shorthand for this sort of thing.

20 Comments

Obviously, believing in or getting worked up about a BMI change of 0.003 is inane, and points to the usual problems of “binary” thinking (an effect exists or not) and not understanding noise. It’s great that you’re writing about this.

However, your title and P.S. have the same flaws. Here, for kicks, are the titles of about 1/3 of the articles in this week’s PNAS. (I could copy and paste more…) What fraction do you think are “junk?” What’s your threshold for maligning the whole journal, or using it as a cute shorthand for crap? This isn’t an issue of social vs. natural sciences. (In fact, one of the worst articles I’ve ever read was a physics paper in PNAS, last year.) Nor is is that I particularly care about PNAS. It’s the same issue as raised above, and I think your excellent points in this and other posts aren’t well-served by the “PPNAS” schtick.

My problem with PPNAS is not that it publishes crap. Every journal publishes crap; it’s unavoidable for a journal that publishes 3000 papers a year. My problem with PNAS is the hype. Part of this is the fault of journalists who think that anything associated with the National Academy of Sciences must be solid research. But part of this is the fault of PNAS itself, given that on its webpage it advertises: “PNAS publishes only the highest quality scientific research.” That’s what they claim, so I hold them to this high standard. If you think it’s unfair to “malign the whole journal,” just because they publish some really bad stuff, you might want to take it up with whoever is standing by the slogan, “PNAS publishes only the highest quality scientific research.”

Or, to put it another way, my problem with journals is not that they publish bad papers; it’s that they’re part of a system that minimizes or even denies error, a system under which virtually no correction is enough for authors to accept that their conclusions might be mistaken. I pick on PPNAS because PPNAS contributes to and benefits from this system: they contribute via their “PNAS publishes only the highest quality scientific research” and their unwillingness to apologize for himmicanes, air rage, etc., and the benefit from all the typically uncritical media exposure given to their papers. The National Academy of Sciences can and should do better.

Or, to put it another way, my problem with this administration is not that they make mistakes; it’s that they’re part of a system that minimizes or even denies error, a system under which virtually no correction is enough for authors to accept that their conclusions might be mistaken. I pick on the White House because the White House contributes to and benefits from this system: they contribute via their “Make America Great Again” and their unwillingness to apologize for incorrect Tweets, false statements by press secretaries, etc., and the benefit from all the typically uncritical media exposure given to their events. The United States can and should do better.

i think it’s a good thing that we have media outlets holding the Republicans’ feet to the fire, and other media outlets doing the same for the Democrats. It’s not necessary for every journalist to offer both these perspectives; it’s just good for the criticism and responses to be based on facts and clear reasoning.

PNAS is a little different because they’re not part of a two-party system. There’s no natural “opposition” to the National Academy of Sciences. Also, when criticizing PNAS, I’m not criticizing the National Academy of Sciences as an institution, I’m just saying they can do better by being more open to criticism.

The political story is different, in that people are criticizing the Trump administration (and others were similarly criticizing the Obama administration, and before that the Bush administration, etc.) not merely for lack of openness to criticism but for particular bad or dangerous decisions. So I don’t think your analogy works so well.

“i think it’s a good thing that we have media outlets holding the Republicans’ feet to the fire, and other media outlets doing the same for the Democrats.”

GS: An alternative view is that the media fits nicely into the façade that “the government” is kept in check by either the media or the voters in some MEANINGFUL way – that is, in a fashion commensurate with problem. Put simply, but somewhat non-technically, all the self-congratulatory back-patting that goes on surrounding the supposed superiority of democracy and the role of a free-press etc., fits nicely into the overall aims of the propaganda arm of the controllers (the ultra-rich and the ultra-rich politicians that collude with them); *we* busy ourselves with the status quo while we slip inexorably toward totalitarianism. This is manifested in the Trump era; Trump’s antics are not really “the story” here. The real story is that he is President when he is so clumsy vis-à-vis the Grand Façade. Politicians are appalled that he is so blatant in his reverse-Robin Hood redistribution of the wealth. That was supposed to be discrete enough to be below “the threshold.” Government has but one, two-part function, 1.) keep the rich rich and 2.) keep the poor from killing the rich. As long as the correct balance is struck, the system can continue to slide toward totalitarianism without a hitch. Anyone that thinks that electing a mainstream republican or democrat significantly (in the real sense of “significantly”) alters the trajectory towards totalitarianism is, IMO, just as gullible as the Trumpies. Again, though, to get back to the point about the press in case it isn’t clear: The “free-press” is the real “opiate of the masses” because it legitimizes the non-Trump status quo which is, in fact, characterized by the inexorable flow towards totalitarianism just as much as Trump’s clumsy rampage.

That’s a fair point. It would be exhausting, though, to apply this standard consistently. I’d bet that Columbia makes laughable statements about how excellent it is, and that it benefits from a “system” of elite university status, but we don’t feel compelled to bring up its name whenever any university over-hypes itself.

Even sticking to journals: the non-tabloid Physical Review Letters, for example, states that it “is the world’s premier physics letter journal. It publishes short, high quality reports of significant and notable results.” There’s certainly crap in PRL, and crap that gets media coverage. It would be perfectly fair to call them out on it, just as it’s fair, constructive, and admirable that you call out crap that’s in PNAS. However, I still maintain that it would be unfair to title a post about an awful physics paper published elsewhere “junk from PRL,” just because PRL benefits from “the system.” The same holds for a lot of journals.

I think PPNAS should remove its statement, “PNAS publishes only the highest quality scientific research,” as it is evidently false.

Your PRL statement doesn’t seem so bad: they say the publish “short, high quality reports.” That’s a lot better than saying “highest quality” and it’s a lot better than saying “only.” I don’t see any grounds for criticizing the PRL statement, and I similarly would be fine if PPNAS wrote that they “publish high quality scientific research” or even that they “aim to publish the highest quality” etc.

If Columbia University says that it offers an excellent education, that would not bother me. If Columbia said that they admit only the highest quality students, or that they employ only the highest quality researchers, then, yes, that would bother me.

The impression I had even before I encountered Andrew’s PPNAS ranting was that PNAS has always (or at least for a long time) been in some ways like a vanity press — where, for example, a NAS member is almost always assured of getting his/her student’s work published, and often in a more timely manner than in other journals.

Could be. What’s weird is that PPNAS seems to have such high status among journalists. Go back a bunch of years ago, and I’d never heard of PNAS. I don’t really know what happened to transform PNAS from an obscure, archival journal of record to a high-profile, third only to Science and Nature in its ability to attract publicity.

Why is it not considered a requirement that news articles written about published studies provide links to those studies? They link to everything else they can think of, no matter how tangentially related to the topic at hand. But linking to the paper that the article is about? Who wants that?

I know I can google it, I just find this really annoying. Anyway, here it is:

“RESULTS: Of the 4657 articles screened, 8 prospective cohort studies (n = 34 470 individual children) met the inclusion criteria. Controlling for total energy intake, 1 daily 6- to 8-oz serving increment of 100% fruit juice was associated with a 0.003 (95% CI: 0.001 to 0.004) unit increase in BMI z score over 1 year in children of all ages (0% increase in BMI percentile). In children ages 1 to 6 years, 1 serving increment was associated with a 0.087 (95% confidence interval: 0.008 to 0.167) unit increase in BMI z score (4% increase in BMI percentile). 100% fruit juice consumption was not associated with BMI z score increase in children ages 7 to 18 years.

CONCLUSIONS: Consumption of 100% fruit juice is associated with a small amount of weight gain in children ages 1 to 6 years that is not clinically significant, and is not associated with weight gain in children ages 7 to 18 years. More studies are needed in children ages 1 to 6 years.”

So the 0.003 is a z-score increase, and the CI doesn’t contain zero! But only if you exclude children over 6. Therefore future research should exclude children over 6. And this research needs to be performed, but by the way the effect is too small to be of interest.

This is one of those times where since we know the nutritional content of fruit juice is crap, it reduces confidence in the study that the effect is so small. Of course, the normal practical limitations on researching whether a single food item makes a child gain weight more than they are already bound to gain makes it seem unlikely you’d ever get much more than noise unless there are unobserved variables that correlate positively with both weight gain and fruit juice intake (I suspect there aren’t).