Tag Archives: Medicine

The Catholic Church used to sell “indulgences”; you gave them cash and they gave you the assurance that God would let you sin without punishment. If you are at all suspicious about whether this church can actually deliver on their claim, this seems a bad deal. You give them something tangible and clearly valuable, and they give you a vague promise on something you can’t see, and can’t even check if anyone has ever received.

We make similar bad “bargains” with a few kinds of workers, to whom we grant extraordinary privileges of “self-regulation.” That is, we let certain “professionals” run their own organizations which tell us how their job their job is to be done, and who can do it. In some areas, such as with doctors, these judgements are enforced by law: you can only buy medical services approved by doctors, and can only buy such services from those who the official medical organizations labels “doctors.” In other areas, such as with academics, these judgements are more enforced by our strong eagerness to associate with high prestige professionals: most everyone just accepts the word of key academic organizations on who is a good academic.

There is a literature which frames this as a “grand bargain”. The philosopher Donald Schön says:

In return for access to their extraordinary knowledge in matters of great human importance, society has granted them [professionals] a mandate for social control in their fields of specialization, a high degree of autonomy in their practice, and a license to determine who shall assume the mantle of professional authority.

In acknowledgement of and in return for their expertise, experience, and judgement, which they are expected to apply in delivering affordable, accessible, up-to-date, reassuring, and reliable services, and on the understanding that they will curate and update their knowledge and methods, train their members, set and enforce standards for the quality of their work, and that they will only admit appropriately qualified individuals into their ranks, and that they will always act honestly, in good faith, putting the interests of clients ahead of their own, we (society) place our trust in the professions in granting them exclusivity over a wide range of socially significant services and activities, by paying them a fair wage, by conferring upon them independence, autonomy, rights of self-determination, and by according them respect and status.

Notice how in this supposed bargain, what we give the professionals is concrete and clearly valuable, while what they give us (over what we’d get without the deal) is vague and very hard for us to check. Like an indulgence. The Susskinds claim that while this bargain has been a good deal so far, we will soon cancel it:

We predict that increasingly capable machines, operating on their own or with non-specialist users, will take on many of the tasks that have been the historic preserve of the professions. We anticipate an ‘incremental transformation’ in the way that we produce and distribute expertise in society. This will lead eventually to a dismantling of the traditional professions.

This seems seriously mistaken to me. There is actually no bargain, there is just the rest of us submitting to professionals’ prestige. Cheaper yet outcome-effective substitutes to expensive professionals have long been physically available, and yet we have mostly not chosen those substitutes due to our eagerness to affiliate with prestigious professionals. We don’t choose nurses who can do primary care as well as doctors, and we don’t watch videos of the best professors from which we could learn as much as from attending typical lectures in person. And we aren’t interested in outcome track records for our lawyers. The existence of even more such future substitutes won’t change this situation much.

It is usually bad for people to die, and so good for them to keep living. Overall in our society, people who weigh more for their age and gender tend to die more, and so many are concerned about an “obesity epidemic”, and seek ways to reduce people’s weight, such as by getting them to consume fewer calories. Such as from drinking sugary soda.

TIME magazine says that evil soda firms, like evil tobacco firms before them, are lying about science to distract us from their evil:

You may not have noticed it yet, but sodamakers are working hard to get you off your couch. On Aug. 9, a New York Times article revealed that Coca-Cola was quietly funding a group of scientists called Global Energy Balance Network that emphasizes the role of exercise, as opposed to diet, in fighting obesity. … This has some nutrition and obesity experts charging soda companies, whose sales of carbonated soft drinks have hit a 20-year low, with cherry-picking science to make its products more appealing. … Indeed, there isn’t strong evidence to show that exercise alone … can help people shed pounds and keep them off. … It’s not the first time science has been used to sway public perceptions about the health effects of certain behaviors; the tobacco industry famously promoted messaging passed on studies that claimed to prove that “light” or “low-tar” cigarettes were less harmful that regular ones. (more)

Yes, it is true that the literature usually suggests that for most people exercise won’t do much to change their weight. However, another consistent result in the literature (e.g., here, here) is that when we predict health using both weight and exercise, it is mostly exercise that matters. It seems that the main reason that heavy people are less healthy is that they exercise less. Obesity is mainly unhealthy as a sign of a lack of exercise.

So if we cared mainly about people’s health, we should cheer this effort by soda forms to push people to exercise. Even if that also causes people to cut down less on soda. A population that exercises more doesn’t weight much less, but it lives much longer. In fact, exercise seems to be one of the biggest ways we know of by which an individual can influence their health. (Much bigger than medicine, for example.)

I suspect, however, that what bothers most people most about fat people isn’t that they’ll die younger, its instead that they look ugly and low status, and so make them also look low status by association. So we don’t want people near us to look fat. All else equal we might also want them to live longer, but that altruistic motive can’t compete much with our status motive.

So boo soda firms if you want your associates to not seem low status. But yay soda firms if you want people to live and not die (sooner).

… Global Energy Balance Network, which promotes the argument that weight-conscious Americans are overly fixated on how much they eat and drink while not paying enough attention to exercise. Health experts say this message is misleading …

Actually that message seems exactly right to me, and not at all misleading.

In one of my first blog posts back in 2006 I said people overestimate the social value of joining a helping profession, like doctors:

Yes, if you choose to be a doctor, you will spend your time providing services that people perceive to have value, sometimes enormous value. However, you cannot take full credit for this value. (more)

Now 80,000 Hours’ Rob Wiblin, who once blogged here, says “If you want to save lives, should you study medicine? Probably not”:

Most people skilled enough to make it in a field as challenging as medicine could have a bigger social impact through an alternative career. The best research suggests that doctors do much less to improve the health of their patients than you might naturally expect. Health is more determined by lifestyle factors, and most of the treatments that work particularly well could be delivered with a smaller number of doctors than already work in the UK or USA. (more)

In contrast, 80,000 Hours is quite bullish on getting an Economics PhD.

At her Rationally Speaking podcast, Julia Galef talked to me about signaling as a broad theory of human behavior.

Julia is smart and thoughtful, and fully engaged the idea. Even so, I’m not sure I convinced her. I might have had a better chance if we’d dived quickly into a detailed summaries of related datums. Instead we more talked more abstractly about her concern that signaling seems a complex theory, and shouldn’t we look to simpler theories first. For example, on the datums that we see little correlation between medicine and health, and that people show little interest in private info on medicine effectiveness, Julia said:

Like the fact that humans are bad at probability and are pretty scope insensitive, and don’t really feel the difference between a 5% chance of failure versus an 8% chance of failure. Also the fact that humans are superstitious thinkers, that on some level, it feels like if we don’t think about risks, they can’t hurt us, or something like that. … It feels like that I would have put a significant amount of weigh even in the absence of signaling caring, that people would fail to purchase that useful information.

Yes, the fact that we follow heuristics does predict that our actions deviate from those of perfect rationality agents. It predicts that instead of spending just the right amount on something like medicine, we may spend too much or too little. Similarly, it predicts we might get too much or too little info on medical quality.

But by itself that doesn’t predict that we will spend too much on medicine, and too little on medical quality info. In fact, we see a great many other kinds of areas, such as buying more energy efficient light bulbs, where people seem to spend too little. And we see a great many other areas were people seem too eager to gain and apply quality info; we eagerly consume news media full of info with little practical application.

As I said in the podcast, but perhaps didn’t explain well enough, we are often tempted to explain otherwise-puzzling behaviors in terms of simple error theories; the world is complex so people just can’t get it right. This won’t explain why we tend to do the same things as others who are socially near, but that we often like to explain as social copying and conformity; we try to do what others do so we won’t look weird, and maybe others know something.

But even conformity, by itself, won’t explain the particular choices that a group of socially adjacent people make. It doesn’t predict that elderly women in Miami tend to spend too much on medicine, for example. It is these patterns across space, time, group, industry, etc. that I try to explain via signaling. For example, relative to other products and services, people have consistently spent too much on medicine all through history, especially in rich societies, and for women and the elderly.

I’ve offered a signaling story to try to simultaneously explain these and many other details, and yes it takes a few pages to explain. That may sound more complex than “its all just random mistakes”, but to explain any specific dataset of choices, that basic error story must be augmented with a great many specific ad hoc hypotheses of the form “and in this case, the particular mistake these people tend to make happens to be this.”

The combination of “its just error” and all those specific hypotheses is what makes that total hypothesis actually a lot more complex and a priori unlikely than the sorts of signaling stories that I offer. Which is why I’d say such signaling hypotheses are favored more by the data, at least when they fit reasonably well and are generated by a relatively small set of core hypotheses.

Years ago I was being surprised to learn that patients usually can’t pick docs based on track records of previous patient outcomes. Because, people say, that would invade privacy and make bad incentives for docs picking patients. They suggest instead relying on personal impressions, wait times, “bedside” manner, and prestige of doc med school or hospital. (Yeah, those couldn’t possibly make bad incentives.) Few ever study if such cues correlate with patient outcomes, and we actively prevent the collection of patient satisfaction track records.

For lawyers, most trials are in the public record, so privacy shouldn’t be an obstacle to getting track records. So people pick lawyers based on track records, right? Actually no. People who askarerepeatedlytold: no practically you can’t get lawyer track records, so just pick lawyers based on personal impressions or the prestige of their law firm or school. (Few study if those correlate with client outcomes.)

Despite being public record, court data is surprisingly inaccessible in bulk, nor is there a unified system to access it, outside of the Federal Courts. Clerks of courts refused Premonition requests for case data. Resolved to go about it the hard way, Unwin … wrote a web crawler to mine courthouse web sites for the data, read it, then analyze it in a database. …

Many publications run “Top Lawyer” lists, people who are recognized by their peers as being “the best”. Premonition analyzed the win rates of these attorneys, it turned out most were average. The only way that they stood out was a disproportionate number of appealed and re-opened cases, i.e. they were good at dragging out litigation. They discovered that even the law firms themselves were poor at picking litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between win rates and re-hiring rates, i.e. a barrister 20% better than their peers was actually 2% less likely to be re-hired! … Premonition was formed in March 2014 and expected to find a fertile market for their services amongst the big law firms. They found little appetite and much opposition. …

The system found an attorney with 22 straight wins before the judge – the next person down was 7. A bit of checking revealed the lawyer was actually a criminal defense specialist who operated out of a strip mall. … The firm claims such outliers are far from rare. Their web site … shows an example of an attorney with 32 straight wins before a judge in Orange County, Florida. (more)

As a society we supposedly coordinate in many ways to make medicine and law more effective, such as via funding med research, licensing professionals, and publishing legal precedents. Yet we don’t bother to coordinate to create track records for docs or lawyers, and in fact our public representatives tend to actively block such things. And strikingly: customers don’t much care. A politician who proposed to dump professional licensing would face outrage, and lose. A politician who proposed to post public track records would instead lose by being too boring.

On reflection, these examples are part of a larger pattern. For example, I’ve mentioned before that a media firm had a project to collect track records of media pundits, but then abandoned the project once it realized that this would reduce reader demand for pundits. Readers are instead told to pick pundits based on their wit, fame, and publication prestige. If readers really wanted pundit track records, some publication would offer them, but readers don’t much care.

Attempts to publish track records of school teachers based on students outcomes have produced mostly opposition. Parents are instead encouraged to rely on personal impressions and the prestige of where the person teaches or went to school. No one even considers doing this for college teachers, we at most just survey student satisfaction just after a class ends (and don’t even do that right).

Regarding student evaluations, we coordinate greatly to make standard widely accessible tests for deciding who to admit to schools. But we have almost no such measures of students when they leave school for work. Instead of showing employers a standard measure of what students have learned, we tell employers to rely on personal impressions and the prestige of the school from which the student came. Some have suggested making standard what-I-learned tests, but few are interested, including employers.

For researchers like myself, publications and job position are measures of endorsements by prestigious authorities. Citations are a better measure of the long term impact of research on intellectual progress, but citations get much less attention in evaluations of researchers. Academics don’t put their citation count on their vita (= resume), and when a reporter decides which researcher to call, or a department decides who to hire, they don’t look much at citations. (Yes, I look better by citations than by publications or jobs, and my prestige is based more on the later.)

Related is the phenomenon of people being more interested in others said to have the potential to achieve X, than in people who have actually achieved X. Related also is the phenomenon of firms being reluctant to use formulaic measures of employee performance that aren’t mediated mostly by subjective boss evaluations.

It seems to me that there are striking common patterns here, and I have in mind a common explanation for them. But I’ll wait to explain that in my next post. Till then, how do you explain these patterns? And what other data do we have on how we treat track records elsewhere?

Added 22Mar: Real estate sales are also technically in the public record, and yet it is hard for customers to collect comparable sales track records for real estate agents, and few seem to care enough to ask for them.

I’ve suggested that the main function of medicine is to show that we care. I’ve suggested that we spend a lot on medicine to signal our care, and that this can explain the placebo effect, wherein the mere appearance of care increases health. Some apparently confirming evidence:

Parkinson’s Disease patients secretly treated with a placebo instead of their regular medication performed better when told they were receiving a more expensive version of the “drug,” … While most people think of a placebo as a sugar pill that replaces a real medication, the impact more commonly comes from “the engagement between patients and clinicians,” in particular the way doctors create expectations that their efforts will help, Kaptchuk said. That includes a good relationship between doctor and patient; certain medical rituals, such as taking blood pressure and a medical history; and the “color, shape, number and cost” of the placebo drug. (more; the study)

Now this study is hardly definitive – it had only twelve subjects, and the placebo difference is only significant at the 3.4% level. But I guess that it will be verified in larger trials.

I don’t post on medicine much lately, because my attention has been elsewhere. But this looks too important not to mention:

In 1999, the Institute of Medicine published the famous “To Err Is Human” report, … reporting that up to 98,000 people a year die because of mistakes in hospitals. The number was initially disputed, but is now widely accepted by doctors and hospital officials — and quoted ubiquitously in the media. In 2010, the Office of Inspector General for Health and Human Services said that bad hospital care contributed to the deaths of 180,000 patients in Medicare alone in a given year.

Now comes a study in the current issue of the Journal of Patient Safety that says the numbers may be much higher — between 210,000 and 440,000 patients each year who go to the hospital for care suffer some type of preventable harm that contributes to their death, the study says.

That would make medical errors the third-leading cause of death in America, behind heart disease, which is the first, and cancer, which is second. …

James based his estimates on the findings of four recent studies that identified preventable harm suffered by patients – known as “adverse events” in the medical vernacular – using use a screening method called the Global Trigger Tool, which guides reviewers through medical records, searching for signs of infection, injury or error. Medical records flagged during the initial screening are reviewed by a doctor, who determines the extent of the harm.

In the four studies, which examined records of more than 4,200 patients hospitalized between 2002 and 2008, researchers found serious adverse events in as many as 21 percent of cases reviewed and rates of lethal adverse events as high as 1.4 percent of cases.

By combining the findings and extrapolating across 34 million hospitalizations in 2007, James concluded that preventable errors contribute to the deaths of 210,000 hospital patients annually.

That is the baseline. The actual number more than doubles, James reasoned, because the trigger tool doesn’t catch errors in which treatment should have been provided but wasn’t, because it’s known that medical records are missing some evidence of harm, and because diagnostic errors aren’t captured.

An estimate of 440,000 deaths from care in hospitals “is roughly one-sixth of all deaths that occur in the United States each year.” (more; source)

From ’97 to ’99 I was a RWJF Health Policy Scholar (at UC Berkeley), and my final project and presentation was on what I called “treatment futures”, i.e., the idea of using decision markets to forecast treatment-conditional health outcomes for individual patients. I proposed:

At major treatment decision point, post sanitized medical record & options to web.

I also posted on this in ’07. Yesterday I learned that a new startup, CrowdMed, is spending $1.1M to try a related idea. They will have ordinary people “bet” on particular patient diagnoses. I put “bet” in quotes because they only bet donations, and they don’t tell users how individual predictions, individual winnings, and consensus estimates on patients are related. That is apparently part of their patented secret sauce – you’ll just have to trust them.

A patient pays $200 to post their problem, and promises to eventually declare a “correct” diagnosis. Each player is given $5 to start, and can only spend winnings on donating to Watsi patients. So if after several years hard work, you do much better than average, and end up with $20, you might donate that much – woo hoo! Player incentives to diagnose correctly are diluted further by the fact that they only predict what the patient will say is their diagnosis, not the true diagnosis. And players don’t get to look at a full medical history, just a few paragraphs of description.

Patients mainly pay for possible diagnoses to suggest to their doctor to consider, diagnoses that players believe might find supporting evidence, if only the patient’s doctor would consider them. So patients have to believe that their doctor will believe that these volunteer amateur detectives have useful diagnosis suggestions to pursue, ones the doctor would not have otherwise considered. Seems a pretty high bar to me.

My conditional forecasting concept could help patients even if patient doctors don’t believe in it, but it does require players to wait longer to find out if they win. And I think that players deserve a much higher fraction of the patient payments than this startup seems willing to give them — I expect CrowdMed incentives are way too weak. Many seem to have decided that the big idea in “crowd-sourcing” is getting amateurs to do for free what you’d otherwise have to pay professionals to do. Me, I think you usually need to pay good money to get good info, even when you do it right.

Added 3p: The CrowdMed founder replies in the comments; I respond also.

What many people like about being religious is being part of a community built on the idea of being and doing good. They can meet and discuss how to be and do good, share practical tips and sometimes just do good together. That sure can feel great.

What many people dislike about other people being religious is their habit of presuming that if you aren’t religious in their way, you aren’t being or doing good; you are bad. Religious people often prefer similarly religious people to be their teachers, grocers, leaders, etc., because they can’t trust bad people in such roles and shouldn’t support bad people even if they can.

Many non- or otherly-religous folks say they have nothing against doing good, but say it is laughable to presume that people who are religious in your way are actually much better than others. Most religions do little to actually sort people by how much good they are or do; they mostly sort by loyalty, conformity, impressiveness, and local social status. Religions could sort people better if they spent lots of time together doing things most everyone agrees are clearly good, like healing the sick, but that is pretty rare.

My ex-co-blogger Eliezer Yudkowsy left this blog in 2009 to start the Less Wrong (LW) blog, which helped seed a growing community that sees itself self-consciously as “rationalists”. They meet online and in person and often discuss how to be more rational. Which is a fine goal. I’ve supported it by listing recent LW posts on the sidebar of this blog, and I’ve attended many LW-based social events. Some high status members of that community now offer (not-free) workshops where they teach you how to be more rational.

As with religion, the main problem comes when a self-described rationalist community starts to believe that they are in fact much more rational than outsiders, and thus should greatly prefer the beliefs of insiders. This happens today with academia, which generally refuses to consider non-academic beliefs as evidence of anything, and with political ideologies that consider themselves more “reality-based.”

Similarly, I’ve noticed a substantial tendency of folks in this rationalist community to prefer beliefs by insiders, even when those claims are quite contrarian to most outsiders. Some say that since most outsiders are quite irrational, one should mostly ignore their beliefs. They also sometimes refer to the fact that high status insiders tend to have high IQ and math skills. Now I happen to share some of their contrarian beliefs, but disagree with many others, so overall I think they are too willing to believe their insiders, at least for the goal of belief accuracy. For the more common goal of acceptance within a community, their beliefs can be more reasonable.

Some high status members of this rationalist community (Peter Thiel, Jaan Tallin, Zvi Mowshowitz, Michael Vassar) have a new medical startup, MetaMed, endorsed by other high status members (Eliezer Yudkowsky, Michael Anissimov). (See also this coverage.) You tell MetaMed your troubles, give them your data, and pay them $5000 or $200/hour for their time (I can’t find any prices at the MetaMed site, but those are numbers mentioned in other coverage). MetaMed will then do “personalized research,” summarize the literature, and give you “actionable options.” Presumably they somehow try to stop just short of the line of recommending treatments, as only doctors are legally allowed to do that. But I’d guess you’ll be able to read between the lines.

Of course that is usually what you pay doctors to do – study your charts and recommend treatment. And if you didn’t trust your main doctor, you could always get a second or third opinion. So why use MetaMed instead? The main evidence offered at the MetaMed site is data on high rates of misdiagnosis and mistreatment in medicine. Which of course means there is room for improvement via second and third opinions. But it doesn’t tell you that MetaMed is a relatively cost effective source of such opinions.

I wrote this post because I know several of the folks involved, and they asked me to write a post endorsing MetaMed. And I can certainly endorse the general idea of second opinions; the high rate and cost of errors justifies a lot more checking and caution. But on what basis could I recommend MetaMed in particular? Many in the rationalist community think you should trust MetaMed more because they are inside the community, and therefore should be presumed to be more rational.

But any effect of this sort is likely to be pretty weak, I think. Whatever are the social pressures than tend to corrupt the usual medical authorities, I expect them to eventually corrupt successful new medical firms as well. I can’t see that being self-avowed rationalists offers much protection there. Even so, I would very much like to see a much stronger habit of getting second opinions, and a much larger industry to support that habit. I thus hope that MetaMed succeeds.

Investigating your condition in depth, in the context of your entire medical history, genetic data, and personal priorities, may well turn up opportunities to do better than the standardized medical guidelines which at best maximize average health outcomes. That’s basically MetaMed’s raison d’etre. … Fundamentally the thing we claim to be able to do is give you finer-grained information than your doctor will. …

Robin Hanson seems to be implying that MetaMed is claiming to be useful only because we’re members of the “rationalist community.” This isn’t true. We think we’re useful because we give our clients personalized attention, because we’re more statistically literate than most doctors, because we don’t have some of the misaligned incentives that the medical profession does (e.g. we don’t have an incentive to talk up the benefits of procedures/drugs that are reimbursable by insurance), because we have a variety of experts and specialists on our team, etc. (more)

I was asking why pick MetaMed over ordinary medical specialists. I expect most doctors will disagree strongly with the claims that they don’t give patients personalized attention, only improve average health outcomes, and don’t offer the finest-grain advice available. But they could be wrong, and it would be great if MetaMed could show that somehow. On misaligned incentives, a reason to ask a different ordinary doctor for a second opinion is exactly that they can know they won’t get paid for any treatments they recommend.

Two years ago I posted on evidence that called into question the effectiveness of bike helmet laws. A new NBER paper confirms this skepticism:

Using hospital-level panel data and triple difference models. … We consider the effects of the [US bike helmet] laws directly on [’91-’08 US] bicycle related head injuries, bicycle related non-head injuries, and injuries as a result of participating in other wheeled sports (primarily skateboarding, roller skates and scooters). For 5-19 year olds, we find the helmet laws are associated with a 13 percent reduction in bicycle head injuries, but the laws are also associated with a 9 percent reduction in non-head bicycle related injuries and an 11 percent increase in all types of injuries from the wheeled sports. ..

The estimated reduction in head injuries resulting from helmet laws is robust to changes in the definition of the control group, to changes in the type of fixed effects included (state versus hospital), and to changes in the samples of states and hospitals evaluated. … Considering the different offsetting results, we run our preferred specification on injury counts for 1) all head injuries and 2) total (all head and body) injuries arising from cycling and wheeled sports. The net effects of the helmet laws are small and are not statistically different from zero. (more)