The billionaire waging a war against bad science

At 14, he started his first business. At 23, he began making millions for Enron. At 28, he launched his own hedge fund. At 33, he became the youngest billionaire in America. At 38, he retired. Now, his mission is to challenge poor scientific practise

Brian Nosek had pretty much given up on finding a funder. For two years he had sent out grant proposals for his software project. And for two years they had been rejected again and again - which was, by 2011, discouraging but not all that surprising to the 38-year-old scientist. An associate professor at the University of Virginia, Nosek had made a name for himself in a hot sub-field of social psychology, studying people's unconscious biases. But that's not what this project was about. At least, not exactly.

Like a number of up-and-coming researchers in his generation, Nosek was troubled by mounting evidence that science itself - through its systems of publication, funding and advancement - had become biased towards generating a certain kind of finding: novel, attention-grabbing, but ultimately unreliable. The incentives to produce positive results were so great, Nosek and others worried, that some scientists were simply locking their inconvenient data away.

Advertisement

The problem even had a name: the file-drawer effect. And Nosek's project was an attempt to head it off at the pass. He and a graduate student were developing an online system that would allow researchers to keep a public log of the experiments they were running, where they could register their hypotheses, methods, workflows and data as they worked. That way, it would be harder for them to go back and cherry-pick their sexiest data after the fact - and easier for other researchers to replicate the experiment.

Nosek was so taken with the importance of redoing old experiments that he had also rallied more than 50 like-minded researchers across the country to participate in something he called the Reproducibility Project. The aim was to redo about 50 studies from three psychology journals to establish an estimate of how often modern psychology turns up false-positive results.

Read next

You can plot exactly where your delayed train is on these mesmerising junction maps

ByMatt Burgess

It was little wonder, then, that funders didn't come running to support Nosek: he wasn't promising novel findings, he was promising to question them. So he ran his projects on a shoestring budget, self-financing them with his own earnings from corporate-speaking engagements on his research.

But in July 2012, Nosek received an email from an institution whose name he didn't recognise: the Laura and John Arnold Foundation. A Google search told him that the Arnolds were a young billionaire couple in Houston, Texas. John, Nosek learned, had made his first millions as a wunderkind natural-gas trader at Enron, the infamous energy company, and he'd managed to walk away from Enron's 2001 collapse with a seven-figure bonus and no accusations of wrongdoing attached to his name. After that, Arnold started his own hedge fund, Centaurus Energy, where he became, in the words of one hedge-fund competitor, "The best trader that ever lived, full stop." Then Arnold abruptly retired at the ripe age of 38 to focus full-time on philanthropy.

Advertisement

As Nosek tells it, John Arnold had read about the Reproducibility Project in The Chronicle of Higher Education and wanted to talk. By the following year, Nosek was co-founding an institution called the Center for Open Science with an initial $5.25 million (£4.3m) grant from the Arnold Foundation. More than $10 million more in Arnold Foundation grants have come since. "It transformed what we could imagine doing," Nosek says. Projects that Nosek had once envisioned as modest efforts carried out in his lab were now being conducted on an entirely different scale at the centre's startup-like offices in downtown Charlottesville, with some 70 employees and interns churning out code and poring over research. The skeletal software behind the data-sharing project became a slick cloud-based platform, which has now been used by more than 30,000 researchers.

The Reproducibility Project, meanwhile, swelled to include more than 270 researchers working to reproduce 100 psychology experiments - and in August 2015, Nosek revealed its results. Ultimately his army of volunteers could verify the findings of only about 40 per cent of the studies. Media reports declared the field of psychology, if not all of science, to be in a state of crisis. It became one of the biggest science stories of the year.

But as it happens, Nosek is just one of many researchers who have received unsolicited emails from the Arnold Foundation in the past few years -researchers involved in similar rounds of soul-searching and critique in their own fields, who have loosely amounted to a movement to fix science.

Read next

Infoporn: UK government tops tables for open data access

ByRizzoli Denis

John Ioannidis was put in touch with the Arnolds in 2013. A childhood maths prodigy turned medical researcher, Ioannidis became a kind of godfather to the science reform crowd in 2005, when he published two devastating papers - one of them titled simply Why Most Published Research Findings Are False. Now, with a $6 million initial grant from the Arnold Foundation, Ioannidis and his colleague Steven Goodman are trying to turn the study of scientific practice, known as meta-research, into a field in its own right, with a research centre at Stanford.

Advertisement

"If Arnold decided he wanted to beat hunger, I wouldn’t want to bet on hunger."John D’Agosto, former head of strategy of the New York Mercantile Exchange

British doctor Ben Goldacre also received an email from the Arnold Foundation in 2013. Known as a scourge of "bad science", Goldacre spent years building up a case that pharmaceutical companies, by refusing to reveal all their data, have essentially deceived the public into paying for worthless therapies.

Now, with multiple grants from the Arnolds, he is leading an effort to build an open, searchable database that will link all publicly available information on every clinical trial in the world. A number of the Arnolds' reform efforts have focused on fixing nutrition science. In 2011, science journalist Gary Taubes received an email from Arnold himself. Having spent more than a decade picking apart nutrition science, Taubes soon found himself co-founding an organisation with a substantial grant from the Arnold Foundation to rebuild the study of obesity from the ground up. And in 2015 the Arnold Foundation paid journalist Nina Teicholz to investigate the scientific review process that informs the US Dietary Guidelines. Just weeks before the federal guidelines were due for an update, Teicholz's blistering report appeared in the prominent medical journal BMJ, charging that the government's panel of scientists had failed to consider evidence that would have done away with long-held worries about eating saturated fat.

And those are just a few of the people who are calling out iffy science with Arnold funding. Laura and John Arnold didn't start the movement to reform science, but they have done more than anyone else to amplify its capabilities - typically by approaching researchers and asking whether they might be able to do more with more money. "The Arnold Foundation has been the Medici of meta-research," Ioannidis says. All told, the foundation's Research Integrity initiative has given more than $80 million to science critics and reformers in the past five years alone.

Read next

Thanks to Care.data, your secrets are no longer safe with your GP

ByAsher Wolf

Not surprisingly, researchers who don't see a crisis in science have started to fight back. In a 2014 tweet, Harvard psychologist Daniel Gilbert referred to researchers who had tried and failed to replicate the findings of a senior lecturer at the University of Cambridge as "shameless little bullies". After Nosek published the results of his reproducibility initiative, four social scientists, including Gilbert, published a critique of the project, claiming, among other things, that it had failed to accurately replicate many of the original studies. The BMJ investigation, in turn, met with angry denunciations from nutrition experts who had worked on the US Dietary Guidelines; a petition asking the journal to retract Teicholz's work was signed by more than 180 credited professionals. (After an external and internal review, BMJ published a correction but chose not to retract the investigation.)

Nina Teicholz, who investigated scientific reviews of US dietary guidelines on behalf of the Arnold corporation in 2015. Her report charged the government panel of scientists with failing to consider evidence that would have discredited concerns over the consumption of saturated fats.

Vince Talotta/Toronto Star via Getty Images

The backlash against Teicholz also furnished one of the few occasions when anyone has raised an eyebrow at the Arnolds' funding of science critics. On the morning of October 7, 2015, the US House Agriculture Committee convened a hearing on the controversy surrounding the dietary guidelines, fuelled by the BMJ article. For two and a half hours, a roomful of testy representatives asked why certain nutrition studies had been privileged over others. But about an hour in, Massachusetts representative Jim McGovern leaned into his microphone. Aiming to defend the science behind the guidelines, McGovern suggested that the doubts that had been cast over nutrition science were driven by a "former Enron executive". "I don't know what Enron knows about dietary guidelines," McGovern said. But "powerful special interests" are "trying to question science".

McGovern's quip about Enron, a company that hasn't existed for 15 years, was a bit of a potshot. But given the long history of deep-pocketed business interests sowing doubt in research, his underlying question was a fair one: who is John Arnold, and why is he spending so much money to raise questions about science?

Fortune magazine once described Arnold as "one of the least-known billionaires in the US". His profile in the public consciousness is almost non-existent, and he rarely gives interviews. But among hedge funders and energy traders, Arnold is a legend. John D'Agostino, former head of strategy of the New York Mercantile Exchange, says that in Arnold's heyday, people in the industry would discuss him in "hushed and reverent tones". In 2006, Centaurus reportedly saw returns of more than 300 per cent; the next year Arnold became the youngest billionaire in the country. "If Arnold decided he wanted to beat hunger," D'Agostino says, "I wouldn't want to bet on hunger."

Read next

Open data platform reveals complex corporate structures of banks

ByOlivia Solon

For all the swagger of that description, Arnold himself has virtually none. He is universally described as quiet and introspective. At Enron, a company famous for its brash, testosterone-laced cowboy culture, the perennially boyish-looking trader was reportedly so softly spoken that his colleagues had to gather in close to hear him at restaurants. "People would read into it, and they would say he's just being cagey," D'Agostino says. "And then, after a couple of years, people were like, 'Oh, no, he's actually like that.'"

Arnold is still quiet. "Usually the division of labour in most of our work is that I talk," Laura Arnold says in a phone interview. By all accounts, Laura, who attended Harvard College and Yale Law School and worked as an oil executive, has been equally influential in setting the direction for the foundation. But when WIRED visits the Arnold Foundation's Houston headquarters in June, Laura has been called away on a family emergency, leaving John to do the talking. Arnold is 175 centimetres tall, trim and conventionally good looking, his youthful appearance now somewhat concealed by a salt-and-pepper beard.

Arnold grew up in Dallas. His mother was an accountant (she would later help manage the books at his hedge fund). His father, who died when Arnold was 18, was a lawyer. By preschool, Arnold's talent for mathematics was apparent. "I think I was just born with a natural gift for seeing numbers in a special way," he says. Gregg Fleisher, who taught him calculus in high school, recalls an occasion when Arnold instantly solved a maths puzzle that had been known to stump PhDs. But he also stood out for his scepticism. "He questioned everything," Fleisher says.

" The Arnold Foundation has been the Medici of meta-research."John Ioannidis

By the time he was 14, Arnold was running his first company, selling collectible sports cards across state lines. Those were the early days of the internet, and he managed to gain access to an online bulletin board intended only for card dealers. The listings let him see that the same cards were sold at different prices in different parts of the country - which presented an opportunity for arbitrage. "Hockey cards didn't have much of a market in Texas," he tells me. "I would buy up all the premium hockey cards and send them to Canada or upstate New York." He called the company Blue Chip Cards. Arnold estimates that he made $50,000 before he finished high school.

Read next

TomTom launches assault on open mapping data

ByDuncan Geere

Arnold graduated from Vanderbilt University in 1995, taking only three years to finish his degree. He started working at Enron four days later. A year after that, at age 22, he was overseeing Enron's Texas natural-gas trading desk, one of it's core businesses.

Arnold's work at Enron - seeking to capitalise on seasonal price differences in natural gas - wasn't all that different from what he'd done as a teenager selling sports cards. In Hedge Hogs, a 2013 book about hedge-fund traders, Jeff Shankman, another star trader at Enron, is quoted describing Arnold as "the most thoughtful, deliberate and inquisitive person" he worked with. But Shankman recognised that he and Arnold were different in one key respect: Arnold had a greater appetite for risk. On some days at Enron, Arnold would trade more than a billion dollars' worth of gas contracts. In 2001, even as Enron was collapsing amid an accounting scandal that covered up billions in debt, he was reported to have earned $750 million for the company. A former executive at Salomon Brothers later told The New York Times that there were very few incidents in the history of Wall Street comparable to Arnold's success.

As Enron neared bankruptcy, executives scrambled to hold its operation together, offering bonuses to keep traders on board. Arnold was given $8 million, the biggest payout of all, just days before Enron filed for bankruptcy. He started Centaurus the following year, bringing along a small group of former Enron traders who worked out of a single large room.

Arnold says he wasn't sure if he could match the success he'd enjoyed as a futures trader at Enron. As a pipeline company, Enron had a direct view on many of the factors that influence gas prices. Now he'd have to rely purely on his prowess with data. By law, natural-gas pipelines had to make much of their information public, and around the time Centaurus was forming, more of that information began to appear online. "A lot of people didn't know it was out there," Arnold says. "People who did, didn't know how to clean it up and analyse it as well as we did."

It wasn't long before Arnold had the answer to his doubts. In 2006, Centaurus reportedly generated a 317 per cent return overall, after taking the opposite side of a risky bet that another hedge fund, Amaranth, had made on fluctuations in natural gas prices. Amaranth, which was gambling with money from large pension funds, suffered a $6 billion loss and collapsed. By 2009, Centaurus was managing more than $5 billion and had more than 70 employees. In its first seven years, according to Fortune, the fund never returned less than 50 per cent.

Read next

25 big ideas for 2012: The open-data economy

ByClive Thompson

But Arnold had to come down to earth eventually. In 2010, Centaurus experienced its first annual loss. And though the fund bounced back the next year, tighter regulations on trading and a far less volatile market made it unlikely that Arnold would again see the astonishing returns of only a few years earlier. And so, at the age of 38, Arnold walked away from it all. He announced that he was closing Centaurus in a letter to investors: "After 17 years as an energy trader, I feel that it is time to pursue other interests."

Chris Seto, left, and Lyndsy Simon, right, software developers at the Center for Open Science

Bill O'Leary/The Washington Post via Getty Images

Arnold tells me that he had lost some of his passion for trading. At the time, his net worth was estimated to be around $3 billion. In 2010 the Arnolds had signed the Giving Pledge, promising to give away at least half their wealth - and he wanted to be as strategic about that goal as he had once been about trading. Arnold has said that the first phase of his life was "100 per cent trying to make money" and that it's now "100 per cent trying to do good". As the Wall Street Journal noted, in "US history, there may have never been a self-made individual with so much money who devoted himself to philanthropy at such a young age".

The Arnolds had been dabbling in philanthropy for years, supporting a few handpicked programmes in education, criminal justice reform and other areas that were important to them. But now, with their stepped-up ambitions, the couple entered a new realm. Arnold had always been ready to make huge bets, but it was his hunger for reliable data that made him a brilliant trader. That same hunger would make large-scale philanthropy more challenging than he had anticipated.

In a glass conference room at the Arnold Foundation's offices - which occupy the same space as the old Centaurus trading floor - Arnold explains that his and Laura's initial plan had been to simply locate the most effective organisations and write them cheques. But figuring out which organisations were most effective turned out to be vexing. Nonprofits are good at reporting their success rates, but dig into their claims and you find that they often omit relevant context or confuse correlation with causation. "The more you read the research, the less you know," Arnold says.

Read next

Creative Commons 101: An introduction to CC licences

ByOlivia Solon

"It became frustrating." Then, one day in November 2011, he was listening to the podcast EconTalk, hosted by libertarian economist Russ Roberts. The guest that day was science journalist Gary Taubes, and he was talking about how the prevailing dietary wisdom of the past 40 years - that eating too much fat leads to obesity and heart disease - arose from the flimsiest of scientific evidence. The foundational studies, Taubes said, looked at the diets and disease rates in various countries, then essentially guessed at which items in the diet were responsible for the country's good or bad health statistics. Worse yet, whenever evidence came along that contradicted the consensus about the dangers of eating fat - often evidence that was much stronger than the evidence for the dangers - it was ignored or not even published. Hardly anyone in the world of nutrition science seemed willing to question the science behind the low-fat diet, even after westerners grew fat and diabetic in record numbers. The picture Taubes painted wasn't of a flawed study here or there but of a fundamentally broken scientific culture.

During the podcast, Taubes mentioned that he was raising money in the hope of funding experiments that might deepen our understanding of the root causes of obesity. Not long after the podcast went online, he received a five-line email from Arnold. "From the little I know about the science of nutrition, your study makes a lot of sense," Arnold wrote. Like Nosek, Taubes had to Google Arnold to learn who he was. Six months later, the Arnold Foundation made a $4.7 million seed grant to the Nutrition Science Initiative (NuSI), the nonprofit Taubes co-founded to support fundamental research on diet and health. The next year the Arnolds promised another $35.5 million. Arnold is careful not to lump all researchers together when he talks about the problems in science. But he tells WIRED that listening to Taubes and reading his book, Good Calories, Bad Calories, had been an "aha moment" for him. "Science is like a building," Arnold says. "One floor on top of the next." In nutrition, "the whole foundation of the research had been flawed. All these things we thought we knew - when we look at the evidence base - it's just not there."

Arnold says that now, unless he trusts a researcher's work, he no longer believes the findings of any scientific study until he or someone on the staff carefully vets the paper.

Together with Taubes' work, Arnold was also reading Ioannidis's and Goldacre's equally devastating analyses. These critiques of science amounted to a deep philosophical quandary for the Arnolds, philanthropists who had dedicated their lives to a data-based approach to giving. "In everything they do, they want to be evidence-driven," says Stuart Buck, vice-president of research integrity at the Arnold Foundation. But if you look at the studies that can't be reproduced and other issues facing science, "you start to think: 'What is evidence? What do we actually know?'"

The Arnolds had already decided that, with decades of life ahead of them and almost unlimited resources, they had the time and money to evaluate charitable programmes properly, even when that meant paying for expensive randomised controlled trials that could take years to complete. But now they were widening their scope. If they wanted to embark on truly "transformational change", as their foundation literature states, it wouldn't be enough to properly evaluate this or that education or criminal justice programme. They would have to take on a far more ambitious project: they would have to try and fix science itself.

Read next

Rebooting Britain: Set government data radically free

"…The whole foundation of the research had been flawed. All these things we thought we knew - when we look at the evidence base - it's just not there.”John Arnold, co-founder of the Arnold Foundation

In their philanthropy, the Arnolds like to say, they follow data where it leads rather than let themselves be guided by ideology. And it's true that, when it comes to political leanings, they are somewhat hard to pin down. The Arnolds identify as Democrats and were major financial supporters of Barack Obama. In 2013 they donated $10 million to keep Head Start, the early-childhood education programme for low-income kids, running through the federal government shutdown, and many of the issues they've taken on, from criminal justice reform to making prescription drugs more affordable, are decidedly progressive. Yet the foundation is also focused on reforming what the Arnolds see as a broken public-pension system - a project that usually means cutting payments to retirees, raising retirement ages and switching new workers to 401(k)-style plans. That focus led Rolling Stone to call Arnold a "young right-wing kingmaker with designs on becoming the next generation's Koch brothers."

If John Arnold does have an identifiable ideology, it is that of a lifelong trader and quantitative analyst: unsentimental, metrics-focused, interventionist. He is unapologetic about having worked at Enron, and he can be defensive about the moral standing of Wall Street in the public mind. In 2015, after a cancer researcher was found to have falsified research data and defrauded the government out of millions of dollars, Arnold complained on Twitter that the penalty, a five-year funding restriction, was too light. Had something similar happened on Wall Street, he tweeted, the perpetrator would have been sentenced to ten years in jail and the bank would have been fined a billion dollars. "Is there something special about frauds in the securities biz that they should be penalised infinitely more harshly than other business frauds?" he went on. "Or is Wall Street an easy target while cancer researchers and universities are not?"

So it's no surprise that, in practice, the Arnolds' approach to giving has a lot in common with John Arnold's approach to investing. Laura says she sees her husband's appetite for risk - an appetite she says she shares - as the most obvious link between his approach to trading and philanthropy. Once the foundation has identified areas where they believe they can make the biggest difference, they go all in. "We're not looking to create an organisation of safe success," she says. "We're looking to create an organisation of thoughtful failure and fantastic success."

Arnold is, in at least one respect, trying to make science a little more like finance. In recent decades, maths and science whizzes such as Arnold have invaded Wall Street, bringing a level of scientific precision to trading and often making fortunes in the process. And good traders, as Arnold sees it, naturally come to appreciate something that researchers too often miss: it's very easy to be fooled by your own data. They internalise the risk of mistaking correlation for causation - not because they're smarter than scientists but because they have money riding on the outcome. "As a general rule, the incentives related to quantitative research are very different in the social sciences and in financial practice," says James Owen Weatherall, author of The Physics of Wall Street. "In the sciences, one is mostly incentivised to publish journal articles, and especially to publish the sorts of attention-grabbing and controversial articles that get widely cited and picked up by the popular media. The articles have to appear methodologically sound, but this is generally a lower standard than being completely convincing. In finance, meanwhile, at least when one is trading with one's own money, there are strong incentives to work to that stronger standard. One is literally betting on one's research."

Read next

25 ideas for 2010: Transparency mania

ByRussell M Davies

Brian Nosek, co-founder of The Center for Open Science, in his offices

Bill O'Leary/The Washington Post via Getty Images

In conversations with Arnold and his grantees, the word incentives seems to come up more than any other. The problem, they claim, isn't that scientists don't want to do the right thing. On the contrary, Arnold says he believes that most researchers go into their work with the best of intentions, only to be led astray by a system that rewards the wrong behaviours. "Scientists really do want to discover things that make a difference in people's lives," says Goodman. "In a sense, that's the strongest weapon that we have. We can feed off that." Figuring out exactly what rewards work best and how to simultaneously change the incentives for researchers, institutions, journals and funders is now a key area of interest for Goodman and Ioannidis.

At the Center for Open Science, Nosek has already begun to experiment with new incentives for scientists. Because investigating and replicating research begins with having the data and materials necessary to do so, he is focused on making science more transparent. In 2014 he partnered with the journal Psychological Science to offer "Open Data" and "Open Materials" badges for papers that met specific criteria for sharing. A 2016 study to determine their effectiveness showed that the number of articles that reported publicly available data had increased tenfold. "It's a stupid little badge," Nosek says, but it works.

Nosek is also still campaigning to convince researchers to preregister what they plan to analyse and report in a study, so that they can't adjust their experiment on the fly or hide poor results - a problem that Goldacre is also tackling. To promote pre-registration, the Center for Open Science offered the first 1,000 scientists who pre-register their studies with the organisation $1,000 each. Nosek says that the cash offers were Arnold's idea.

Denis Calabrese, the Arnold Foundation's president, says they don't expect immediate results. The Arnolds have a "multiple-decade timeline to work on problems". Yet the most remarkable thing about the Arnold Foundation's research integrity projects is that they already appear to be paying off. For one thing, the problems plaguing scientific research are now increasingly well known. Of 1,576 researchers who responded to a recent online survey from Nature, more than half agreed there is "a significant crisis" of reproducibility. The comedian John Oliver spent 20 prime-time minutes on HBO last May mocking the reign of terrible science on TV news shows and in public debate: "After a certain point, all that ridiculous information can make you wonder: 'Is science bullshit?' To which the answer is clearly no, but there's a lot of bullshit masquerading as science." (Some of the background footage in the segment came from the Arnold Foundation.)

Advertisement

Ioannidis, whose name is almost synonymous with scientific scepticism, says he has seen immense progress in recent years. The journals Science and Nature have started bringing in statisticians to review their papers. The National Institutes of Health (NIH) is moving forward with new requirements for data sharing; starting as early as this year, all NIH-funded training programmes must include plans for teaching researchers the principles of reproducibility. "Now everybody says we need replication; we need reproducibility," Ioannidis says. "Otherwise our field is built on thin air."

The Center for Open Science's next undertaking is another reproducibility project - this one for cancer studies. In 2012, the former head of cancer research at the biotech firm Amgen revealed the results of the company's effort to replicate 53 "landmark" papers in hematology and oncology; only six studies' findings could be confirmed. So there is already widespread concern about reproducibility in the field. The centre's replication efforts, in turn, have inspired economists and even tropical ecologists to plan reproducibility projects of their own.

Whether all this momentum will lead to transformational change decades from now is impossible to know. Arnold figures that some of his specific grants might not work out as planned. (The foundation's funding of the Nutrition Science Initiative is now scheduled to end in November.) More generally, it may not be possible to truly reform a system where the incentives are already so deeply embedded. "It's probably too big a lift for us to expect we're going to change researchers who have been around for decades," he says. Plus, systems of prestige and advancement die hard. "You don't shift a culture overnight," Nosek says. But as many Wall Street veterans can testify, betting against John Arnold is usually a bad idea.

This article was first published in the March 2017 issue of WIRED magazine