About this Author

College chemistry, 1983

The 2002 Model

After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases.
To contact Derek email him directly: derekb.lowe@gmail.com
Twitter: Dereklowe

November 21, 2011

Of Drug Research and Moneyball

Posted by Derek

This piece on Michael Lewis and Billy Beane is nice to read, even if you haven't read Moneyball. (And if you haven't, consider doing so - it's not perfect, but it's well worth the time). Several thoughts occurred to me while revisiting all this, some of them actually relevant to drug discovery.

First off, a quick peaen to Bill James. I read his Baseball Abstract books every year back in the 1980s, and found them exhilarating. And that's not just because I was following baseball closely. I was in grad school, and was up to my earlobes in day-to-day scientific research for the first time, and here was someone who applied the same worldview to a sport. Baseball had long been full of slogans and sayings, folk wisdom and beliefs, and James was willing to dig through the numbers to see which of these things were true and which weren't. His willingness to point out those latter cases, and the level of evidence he brought to those takedowns, was wonderful to see. I still have a lot of James' thoughts in my head; his books may well have changed my life a bit. I was already inclined that way, but his example of fearlessly questioning Stuff That Everybody Knows really strengthened my resolve to try to do the same.

A lot of people feel that way, I've found - there are James fans all over the place, people were were influenced the same way, at the same time, by the same books. It took a while for that attitude to penetrate the sport that those books were written about, though, as that article linked to above details. And its success once it did was part of a broader trend:

Innovation hurts. After Beane began using numbers to find players, the A’s’ scouts lost their lifelong purpose. In the movie, one of them protests to Pitt: “You are discarding what scouts have done for 150 years.” That was exactly right. Similar fates had been befalling all sorts of lesser-educated American men for years, though the process is more noticeable now than it was in 2003 when Moneyball first appeared. The book, Lewis agrees, is partly “about the intellectualisation of a previously not intellectual job. This has happened in other spheres of American life. I think the reason I saw the story so quickly is, this is exactly what happened on Wall Street while I was there. . .”

(That would be during the time of Liar's Poker, which still a fun and interesting book to read, although it describes a time that's much longer ago than the calendar would indicate). And I think that the point is a good one. I'd add that the process has also been driven by the availability of computing power. When you had to bash the numbers by hand, with a pencil, there was only so much you could do. Spreadsheets and statistical software, graphing programs and databases - these have allowed people to extract meaning from numbers without having to haul up every shovelful by hand. And it's given power to those people who are adept at extracting that meaning (or at least, to the people willing to act on their conclusions).

The article quotes Beane as saying that Lewis understood what he was doing within minutes: "You’re arbitraging the mispricing of baseball players". And I don't think that it can be put in fewer words: that's exactly what someone with a Wall Street background would make of it, and it's exactly right. Now to our own business. Can you think of an industry whose assets are mispriced more grievously, and more routinely, than drug research?

Think about it. All those preclinical programs that never quite work out. All those targets that don't turn out to be the right target when you get to Phase II. All those compounds that blow up in Phase III because of unexpected toxicity. By working on them, by putting time and effort and money into them, we're pricing them. And too much of the time, we're getting that price wrong, terribly wrong.

That's what struck me when I read Moneyball several years ago. The problem is, drug research is not baseball, circa 1985. We're already full of statisticians, computational wizards, and sharp-eyed people who are used to challenging the evidence and weighing the facts. And even with that, this is the state we're in. The history of drug research is one attempt after another to find some edge, some understanding, that can be used to correct that constant mispricing of our assets. What to do? If the salt has lost its savour, wherewith shall it be salted?

The major difference I can think of is that baseball players were still being evaluated based on their past performance. Sure, the method of evaluating that past performance was improved by the use of statistics, but the question that was being answered remained, "How well has this player been playing?"

It's a question that's remarkably easier to answer than "How well is this player going to play in the future", although of course there is a correlation. You can use the former question as a proxy for the latter, but that only works as long as the players in question *have* been playing in the past. If you pick a bunch of kids nobody's ever put on the baseball field before, the same method won't work as well.

I remember looking into a second career in licensing a few years ago and was a bit shocked to find that there was little emphasis on a potential candidate's scientific background or ability to evaluate in-licensing opportunities for their scientific merit. Rather, experience in "doing deals" or a "deal sheet" seemed to be the most valued criterion. I know that many in the field have very strong technical backgrounds, but overall this did not seem to be an over-riding factor in deciding what to license and how much to pay. Perhaps now would be a good time to rethink how in-licensing should be conducted?

another major difference is that a statistical analysis for finding contributors to a baseball team could work when a great hitter has a "success rate" of 1/3 or less, and an average hitter is about 1/4. Great teams win at a rate of about 6/10, and the worse teams win at about 4/10. Fine tuning methodologies with these small differences could make for better averages, and put a better team on the field, improving revenue with lower costs. But did moneyball ever win a world series? I don't think it did, but some argue the methodology contributed to the Red Sox.

In any case, in Pharma, we don't get ticket and hot dog revenue for our efforts prior to our world series wins, and modest improvements in our batting averages (screening efficiency, other work flow improvements) or win percentage (achieving portal metrics, upping Ph1 entrants) have only associated costs prior to market success (smallcos might be able to realize gain here). In fact, as many have argued here, gaming this system and working hard to "value" these efforts seems only to stifle innovation and reward individuals who meet the internal metrics and play by the rules.

So, I don't really think our assets are mispriced, or at least don't see that as the big issue. Instead, the parallel I like better is that we value the wrong players, those that operate well in the system of drug discovery today, those that survive in political environments, and we actively work against innovators and the innovative process, in part by demanding to know more about future performance of molecules than is knowable.

Sabermetrics has its value, but as was stated above, can't necessarily predict how a player is going to perform on any given year. It can help spot trends or potential, but it can't take into account outside influences that could affect one's statistics. A nasty divorce, a child being seriously ill, someone not training as hard in the off-season after signing a big contract, etc...

Beane is living off of his legend at this point. He is extremely over-rated as a GM at this stage of the game, IMHO.

"The history of drug research is one attempt after another to find some edge, some understanding, that can be used to correct that constant mispricing of our assets. What to do? If the salt has lost its savour, wherewith shall it be salted?"

I think there is such an edge, such an understanding, and, speaking as a relatively naive and uninformed outsider, it seems to me to be severely underappreciated and seldom employed.

Namely, the natural history of living things and their interactions as seen from an evolutionary point of view, focused specifically on the chemicals they deploy to effect their strategies.

For example, we are apparently in the midst of a recrudescence of bedbugs, and beyond the revulsion we all feel, there are some very intriguing aspects of their adaptation to feeding on us that may well have pharmaceutical significance, in my opinion.

Despite lots of looking, no one has yet found a pathogen they transmit. This is in stark contrast to other blood sucking insects such as fleas, mosquitoes, and assassin bugs, and is particularly surprising given how easily they seem to spread.

It's almost as if doing as little harm as possible is a part of their arsenal, and that might imply that they are highly adapted to us and aim for long term infestations as a default, so that keeping us in good shape is in their best interests.

That doesn't mean that they aren't manipulating us, however.

The redness and swelling some people develop at bite sites show they are injecting something. Bloodsuckers are known to favor anti-coagulants, but that may not be all in the case of bedbugs.

A bedbug would seem to derive unqualified benefit from rapid and deep sleep on the part of their intended victims, and a bedbug which could somehow make this more likely would have a very strong selective advantage over bedbugs which couldn't.

So if any organism has developed a minimally harmful chemical capable of putting us to sleep, I'd think it might be bedbugs, and that we should look at them as a potential source of sleep aids.

This post is very timely for me. I have been on a Michael Lewis kick reading a handful of his books including Moneyball and Liar's Poker.

I agree with the above posts that Sabermetrics is not applicable to pharma. In fact, one of the lessons from Moneyball is to withhold judging a recruit until they have played college ball and accumulated reliable statistics. Prior to Billy Beane, too many scouts were judging a high school recruit by his size, swing, and speed without regard to stats. This is analogous to judging a compound's value without having in vivo human data. Unfortunately, pharmaceutical companies don't have reliable statistics until phase III trials, which doesn't help with the original problem.

@4 There definitely is a mispricing problem. Two examples that jump out are Sirna and Sirtris. Every scientist knew that Merck and GSK were getting ripped off. I think most of the mispricing comes from focusing on possible future market size. An exec starts counting the zeros after the dollar sign and ignores the underlying science.

If Billy Beane focused his recruiting on on-base percentage, I think pharmaceutical companies should focus on limiting side-effects. Side effects are normally the reason most drugs fail clinical trials. If the drug can get through clinical trials should we really care if the drug brings in $10 million or $10 billion, it's better than zero? Now the question is how can we get these statistics earlier than phase III?

I don't think that sabremetrics attempts to predict future performance, per se. In fact, one of the tenets is that future performance can be somewhat reliably predicted based on past performance if the right measures and factors (age, park effects, etc.) are added in. Rather the point of sabremetrics,IMO, is to use alternative verifiably better measures to value performance. Basically, it tries to identify those statistical measures that materially effect the outcome of a game or a season as opposed to those stats that don't translate well into wins. The classic example being batting average as opposed to slugging percentage with the former having a much lower correlation with wins and losses versus the latter.

What Derek is saying is that if we apply that to drug discovery we need better measures of value for drug candidates. In some ways that's what the oft maligned Lapinski guidelines tried to do, isn't it? Of course, maybe that's analogous to batting average and what we need is something more like on-base % and slugging % to be truly better to assess value.

Statistics + drug industry makes me think of Lipinski. Thinking of Lipinski makes me angry. Just Read The Goddamn Paper. If you made everyone in pharma read that damned thing (especially anyone in a mangement role) you would have come a long way.
Hell I'll summarize it: out of the ORALLY active drugs on the market in 1997, 2/3 of them only violated 1 out 4 rules imposing a limit on their complexity and hydrophilicity.
Anyone disagree then correct me. 6 years since I read it.

To clarify, I definitely think assets are mispriced in deals, as noted above, but I took Derek's comment to mean something else altogether. I meant only that our "pricing" of pipeline assets through the time and effort we apply is not a central problem, more it is the culture and processes of pharma that are the core problem, which includes looking for the bigger fool. We must invest in research, without knowing we have created value, until late in our game. Trying to refine discovery efforts through business process efficiency exercises has been quite disruptive. The silly deals that get done, or overpaying for a company don't change this fundamental failing of pharma internal effort in my opinion.

Drug discovery is so hard because, unlike baseball, we don't fully understand the system we're trying to affect (cellular biology, molecular biology, organismal biology). My guess is that we're not even close to understanding it. We don't know the targets we want to hit, and we don't fully understand what the drugs we have are actually doing.

Luysii has it exactly correct. To cast it in Taleb's "Black Swan" terms, baseball is a "Mediocristan" problem while drug design is an "Extremistan" problem. If you know the history of a baseball player, you can do a really good job of predicting what he'll do next. The infield shift works for Ortiz, but wouldn't work for Ichiro (the classic spray hitter).

Now, a really interesting question is whether we will EVER be able to collect enough information for drug design to become truly predictive. Theoretically, yes. But to follow up on Derek's sentiments, I ask, "How long, O Lord?"

"We're already full of statisticians, computational wizards, and sharp-eyed people who are used to challenging the evidence and weighing the facts"

Although this is true, it isn't exactly the same. It is reasonably easy to evaluate a single baseball player's history and deduce how well he might do this year, like it is easy to look at a single drug's past sales and predict their present sales. But how does this apply to research? And where in baseball would you get the theory that, since you evaluated 50 walk-on players to find one promising player, all you would need to do is sample 250 random people to find another 5, which is the philosophy that drove so much of our combinatorial chemistry for so long.

The background to Billy Beane is illuminating: "denied the conventional college path (he) had that hunger. It's a huge advantage to him that he has some slight anxiety". (Contrast this with the "incurious people (who) will go to Princeton, Harvard, Yale, Stanford and come out and think they know everything").

For many years now, Pharma has had the money to recruit "the best" people from the best Universities.... maybe it was a mistake to stop asking about other qualities, such as curiousity and character?