Sabermetric Research

Thursday, March 26, 2009

Cricket: the "nightwatchman" strategy

Cricket fans reading this, please correct me if I'm wrong, and forgive my use of the wrong terminology. (For instance, when a cricketer bats, can you also say he "hits"? I hope so.)

The idea, I think, is this:

In cricket, a batter will hit until he makes an out, and which point he is replaced and will not bat for the remainder of the innings. Batters hit in pairs. Once ten of the eleven men are out, that leaves only one, who can't bat alone, and the innings ends.

Outs can be infrequent; typically, a batter can hit for 25 runs or more before making out, and good batters occasionally hit for 100 runs or more. Therefore, an innings (or game) can go on for several days.

The best batters normally come to bat first. Sometimes, one of the better batters will go out late in the day. When that happens, the team will sometimes send up a worse batter to end the day. That batter is called the "nightwatchman."

Why would they do this? According to Wikipedia, the idea is that the end of the day is a bad period in which to hit – the next batter may be tired, or the light may not be good. Also, if they do send up the good batter, and he is quickly put out, the psychological effect might hurt the team.

And so, they sometimes put in an inferior batter, who can waste some time between now and dusk, so the better batter can be saved until tomorrow.

Now, if all this is correct, what would be the strategic advantage? Every batter has to hit eventually, and there is no inherent benefit of putting good batters together as in baseball, because every batter comes up in the same situation (the equivalent of "bases empty"). And the psychological rationale seems weak to me.

That leaves the "hard to hit in the dark" hypothesis. If the dim light causes all players drop by the same percentage, then it makes sense to put in the batter who normally bats for 10 runs than the one who normally bats for 35 runs. Better to lose X percent of 10 then X percent of 35. But isn't it also possible that it's the other way around? Maybe the better the batter, the more able he is to handle the adverse conditions.

Also, you have to keep in mind that every batter gets the same chance to bat, except the one who's left after ten men have gone out. The longer you wait before putting in your best batters, the greater the chance it'll be one of those good ones who doesn't get to finish. So, generally, you'd want your better batters first.

So which is the better strategy? This seems like a good problem for cricket sabermetrics. The original article points to a study by Charles Davis, who (I get the impression) is cricket's foremost sabermetrician.

In that study, Davis finds that teams who used the nightwatchman strategy (late in the day after two men had gone out) undershot expectations by 25 runs over teams who didn't. It wasn't because the nightwatchmen didn't do well – they did about the same as their career average lower in the "batting order." So it must have been ... what? Maybe stranding a better batter after the last out? That still seems like a lot; the difference between a good batter and a bad batter might be ... what, 50 runs? And there are still 8 outs (wickets) left in the match. So the fraction 25/50 seems too large under the circumstances.

But look at Davis's graph: an increase of 100 runs scored in the first two wickets leads to a final score only about 35 runs higher. That shouldn't be the case, should it? Wickets are independent except for the identities of the players involved. Consider a baseball analogy: if the Houston Astros score three runs in the first two innings, wouldn't you expect their final score to be three runs higher than if they scored zero runs in the first two innings? Why isn't that happening in Davis's study? The only thing I can think of is that if you score more runs in the first two wickets, it's because you've used up your very best batters, and all that's left is your weaker ones. In that case, it means that team strategy is a huge factor in the distribution of scoring. And so, when you divide innings into "nightwatchman" and "non-nightwatchman," you can't assume the two groups are identical, as Davis did.

Friday, March 13, 2009

The Gini coefficient

Note: non-sports post.

-------------

Income inequality is rising in the United States. If you measure inequality by the Gini Coefficient, which I guess is as good as a measure as any, you get a fairly steady increase from 0.399 in 1967 to 0.466 in 2001. (A higher number means a more unequal distribution; 0.000 means everyone has the same income; 1.000 means one person has all the income.)

Inequality has increased in Canada, too; in fact, what got me interested in this topic was a recent newspaper article complaining about the rise in Canadian income dispersion. These kinds of articles come around all the time, in discussions about how the rich are getting richer.

The stated or unstated assumption is that more inequality is worse, and more equality is better. I don't think that's true. Rather, I think that rising inequality can indeed be worse, but it can also be neutral, or it might actually be a sign of improvement for everyone.

The problem isn't with the measurement itself; as I said above, the Gini Coefficient is a reasonable way to measure what it's supposed to measure. The problem is with the interpretation of movement in the measurement. Even if you think that, all things being equal, more equality is better than less equality, it doesn't necessarily follow that a higher Gini is bad, -- because all things are never equal.

A baseball analogy, perhaps a mediocre one, might be stolen bases. Yes, stolen bases are good for an offense; every additional steal creates about 0.2 runs. But it doesn't follow that an increase in the number of steals is good for offense (in the 60s, steals were up but offense was down.). It doesn't follow that bad offensive teams should try to steal more (they'll just get caught stealing more and score even fewer runs). And it doesn't mean that teams with more steals are somehow better than teams with fewer steals (stolen bases don't correlate well to winning).

The number of steals is just a measurement of – the number of steals. If you want to go beyond that, and draw conclusions from changes in the stolen base rate, you have to justify those conclusions. The naive view -- steals are good so it's bad when teams steal less! -- just isn't true.

It's the same for measures of income inequality. There are dozens of reasons why an increase in income inequality can be a neutral thing, or even a good thing, and you don't have to think very hard to come up with them. I have thirty of them here. I could probably think of ten more. (Some of them, of course, have been thought of before; this Wikipedia page includes several.)

1. First, absolute income matters a lot more than relative income. Would you rather live in the USA where the Gini is 0.46, or in a country where the Gini is zero but everyone earns exactly $5? That's a contrived example, but there are real-life examples too. Albania has a Gini of 0.27, but a per-capita income only one-eighth of the USA. Would you be willing to sacrifice 87% of your pay in order to be more equal to everyone else?

Obviously, other factors can be more important than income inequality. If you think it would be bad for the USA's Gini to go from 0.46 to (say) 0.48, would you change your mind if a 10% increase in income came with it? How about 20%, or 50%?

The Gini rose from 0.42 in 1991 to 0.47 in 2001. But per-capita income, in inflation-adjusted 2006 dollars, rose 23%, from $21,102 to $26,024. Is that a fair trade-off? If you think it is -- and even if you think it isn't -- then complaining about income inequality without mentioning the trade-off isn't very reasonable.

2. The above argument wouldn't be all that persuasive if there weren't a link between the level of equality and the level of income. And there is. If you want to lower the Gini, you have to take income from the rich and give it to the poor. (You could raise income to the poor in other ways, but if those ways were easy, the poor would be doing it themselves already.) Taking income from the rich reduces the incentive to get rich, which in turn reduces overall wealth. Economists could probably tell you what the trade-off actually is -- how much an increase in upper-income tax rates would slow down economic growth. But I think they'd all tell you that there *is* a significant trade-off. More on this later.

3. Suppose that tomorrow, someone comes up with a cure for an untreatable cancer. He sells a million cures at $1,000 each, and makes a billion dollars. Inequality goes up. But everyone is better off! Everyone has just as much income as before, but now we have the opportunity to buy a cure for cancer -- should we ever get it -- for only $1000.

So it's good when inequality goes up for reasons like this ... and I don't think it's that contrived an example. Bill Gates got rich creating Windows ... which costs only, what, about $40 a copy? I get way more than my $40 worth, and I'm happy that Microsoft succeeded and increased inequality on my behalf. The same is true for most of the products I use. I hope that there are more breakthroughs that make other people rich, even if I don't get richer at all.

4. As a general principle, the way you earn a high income is by doing things that benefit other people. You can cure cancer or create an operating system, but you can also perform appendectomies or write computer programs quickly or give a better haircut. A higher Gini might just mean that more people are figuring out better ways to benefit more other people. Or, it could just be a matter of population growth. Fifty years ago, movies played to only millions of people. Now, a movie of the same quality would be seen by billions. That means directors who would have merely been millionnaires in 1970 might be billionnaires now. That's a good thing, not a bad thing, that those who enrich others' lives have many more "others" to benefit. I'm glad that someone in China can now enjoy "Airplane!", the greatest movie ever made, even if it means that the Zucker brothers get a little richer and the US gets a little more unequal.

5. What's the right value for the Gini coefficient? Is .46 too high? Too low? What should it be?

We know that for incomes, higher is better. We know that for golf scores, lower is better. We know that for room temperature, the closer to 72 degrees Fahrenheit, the better.

What's ideal for the Gini? It's not zero; if everyone had to have equal incomes, that would be more Marxist than Marx -- nobody would want to work, and everyone's income would be very low. It's not 1 -- that means one person has all the income, and the rest of us starve.

So what's the point we're shooting for? Nobody knows. And if nobody knows, how can you draw any conclusions at all about whether 0.46 is too high or too low?

In fairness, there is a counterargument here: that we don't know what the Gini should be, but we know there's too much inequality, so we need to go lower. To which I have two answers: first, suppose we reduce inequality. How will you know when we're done? If there's a way to tell, why not tell us now? And, second, some people may *not* think there's too much inequality now. To those people, the fact that the Gini is rising is irrelevant -- you have to convince them that more equality is essential, in order that they see the rise as a bad thing.

The issue shouldn't be that inequality is rising -- the issue should be that inequality is too high. Is it?

6. The Gini doesn't consider that people could be equal over a lifetime, but temporarily unequal due to age differences. Even if everyone had exactly equal income patterns -- say, zero for the first three years of adulthood while they're in school, then escalating equal salaries until retirement, then equal pensions in old age -- the Gini would be non-zero, because, in any given year, you'd be looking at some zeroes, some low-income, some high-income,and some retirees.

Part of the Gini is simply the effect of differing lifetime income among any given individual. If it turns out that people are voluntarily changing their work patterns to have fewer low-income years and more high-income years, the Gini increases even if inequality hasn't.

7. There's a trend to more years of higher education, which would cause part of that increase. If the long-term trend is for more schooling (meaning more years at zero income) and higher incomes after, that would increase within-lifetime inequality, and therefore the Gini, even if everyone remains exactly equal over their lifetime.

Look at it this way: suppose everyone has three years of school at $0, then forty years of work at $50K. They're all equal over their lifetime. But what happens any given year? Of every 43 people, then, three have $0 and forty have $50K, which looks unequal.

Now, suppose everyone decides to go to school for five years to make more money: now you have five people at $0 and thirty-eight people at $70K. This is more unequal, resulting in a higher Gini -- but it's actually better for everyone.

8. People are living longer. If the trend is to make lots of money between 25 and 65, and take a low pension later, then adding more low-pension years will appear to reduce inequality, even if everyone is still the same.

To oversimplify: the trend used to be that you start work at 18, then earn a fairly low salary until you die (because you can't afford a retirement pension). Now, the trend is: make no money for five years while you're at school. Then make good money until 65. Then get a lower pension. The old way, everyone is fairly equal each year. The new way, everyone is still fairly equal over a lifetime, but income varies considerably each year. The new way is less equal when comparing individuals during any given year, but just as equal over a lifetime. And, of course, more desirable.

9. People have different propensities to work; that's just human nature.

Co-workers Bob and Joe are each offered 50 hours of overtime work. Bob accepts, and makes $60K that year. Joe declines, preferring more time with his family, and makes $50K that year. Inequality of income has increased, but that's because inequality of WORK has increased. That's perfectly fair and desirable.

Shouldn't inequality of income be fair to match the inequality of work or effort that created it? And isn't it good that Bob and Joe both have more choices, even though the Gini goes up?

10. People have different ambitions. Ann and Cathy both make $40K a year and are of equal proficiency at their job. Ann makes an extra effort to climb the corporate ladder into management. Cathy doesn't -- she doesn't like kissing butt and prefers to avoid the hassles of supervising other people. Ann jumps to $60K a year; Cathy stays at $40K. Both are happy. Inequality of income goes up, but, again, this is a good thing, as both Ann and Cathy got what they wanted.

11. People have different propensities for risk-taking. Right now, T-bills are paying less than 1%, while other, riskier investments yield 10% and more. Suppose Tom and Jerry have equal jobs and salaries, but Tom is more conservative than Jerry. Tom invests in a CD paying 1.5%, while Jerry invests in real estate yielding 10%. Their incomes suddenly become unequal, but, again, both Tom and Jerry got what they wanted, so the increase in inequality is again a good thing.

12. Even when people have the same propensity for risk-taking, they might just get different results. Jerry and Kevin might be the same in every other respect, but if Jerry invests in mutual fund A, and Kevin invests in mutual fund B, their returns will be different, and their incomes will start to become unequal. Why is that necessarily bad?

Or, take lottery tickets. Everyone who buys a lottery ticket knows that the distribution of incomes among ticket-buyers will be unequal. But they don't mind, and they do it anyway. Is it a bad thing when lotteries increase the Gini? Ticket-buyers probably don't think so.

13. There are other ways to buy security. I work in the IT field, and several of my co-workers, who were on contract, voluntarily took a large pay cut, in some cases close to 50%, to become government employees. Other of my co-workers did not. That created income inequality between the two groups. But did it really make us unequal? Government jobs are very secure. Isn't it reasonable to assume that my friends who took the pay cut simply bought their security, and we're just as equal as we were before? Half of us take our pay in money, and the other half take it in security.

The moral: you can't just consider monetary income; you have to consider things that money can "buy" in the non-traditional sense.

14. There are lots of examples of non-traditional income-substitutes. Suppose you have two identical couples who are neighbors. Helen works, earns $40,000 a year, and takes her kids to day care. Iris quit her $40,000 job to take care of her kids at home. Again, is this unequal? I don't think so. It looks unequal to the Gini, which only measures monetary income. But if Iris chose to stay home and forgo her $40,000 a year, she's obviously benefiting by at least $40,000 -- or she wouldn't stay home! To me, Helen and Iris are exactly equal, despite what the paychecks say.

15. Here's another one: home ownership. Kevin and Laura both make $50,000 a year, but Kevin owns his home, and Laura pays $1,000 a month to rent an identical one. They are equal in money income, but, really, Kevin has more total income -- he "earns" an extra $12,000 a year by having the use of his home rent-free. In this particular Kevin-and-Laura case, the Gini actually *underestimates* inequality, unlike all the previous examples, where it *overestimates* it. However, for the country as a whole, it might again overestimate it, if low-income earners are more likely to own their homes outright. And retirees are likely to have paid-off mortgages.

16. Here's still another one: children. Oscar has two kids, and has to accept a lower income in order to stay in their town and raise them in a stable environment. Paul has no kids, and can move around looking for higher-paying jobs. Their incomes are unequal, but are they really? Oscar has chosen to have kids and pay the price; he values being a father more than he values the bit of extra money he could earn if he were childless. While it's politically incorrect to compare children to money, the fact remains that children are hugely important to people, and they're not free -- there are real costs and opportunity costs to having children. It seems weird to feel sorry for Oscar because he chose children over money, just as it would be weird to feel sorry for someone who was poorer because they chose to buy a fancier car. You have to count everything that affects income, not just the actual income.

These cases of trading money for non-money might be more frequent now then ever, because, as we get richer, they become more affordable -- it's easier to stay home with the kids when your spouse makes $80K a year as a computer programmer in 2009 than when she made $40K (inflation-adjusted) as a programmer back in 1974. As overall wealth increases, the ability to choose to earn less increases. In turn, the diversity of choices increases, and the distribution of monetary income gets wider. Again, this is a most excellent development, in my opinion, and I hope it continues.

17. I quit full-time work a couple of years ago. Right now, I'm not working much, just occasional jobs. If I go back to work, I will earn an above-average income and increase the Gini coefficient. Is that really a bad thing?

18. Small-business owners have incomes that fluctuate from year to year. This year, Al might make $100K and Bruce $20K, but the next year it's reversed. Since the Gini is calculated per year, it looks like Al and Bruce are unequal, even though they had exactly the same income over a two-year span.

19. Part of what's measured by "income" is capital gains. But the stock market has up years and down years. If inequality fell last year because of all the capital losses realized in the stock-market collapse, is that a good thing? Should we rejoice because Warren Buffett is poorer, even if we're poorer too? If that's not worth celebrating, then why is it worth complaining when the reverse happens, when Warren Buffett gets richer but inequality goes up?

20. Do the measurements of income inequality take taxes into account? I don't think they do. That defeats the purpose, doesn't it? A large part of what we expect government to do is help out the poor who need it. And they do that by disproportionately taking money from the rich.

So if the point of the Gini is to measure how much money people actually have to spend -- which is the number that actually means something -- you certainly have to adjust for taxes! It's very possible that the pre-tax Gini is up but the after-tax Gini is down.

21. It's not just cash benefits that we get from government; it's services too. Suppose it costs $10 million to run the public library, and that works out to $10 per person. Since everyone has equal access to the library, shouldn't we add $10 to everyone's (after-tax) income before we calculate the Gini? I think we should. (You could argue that the poor don't benefit as much from the library as the rich (or that hungry poor people can't eat books!), but I'm not sure that's the case. Because, if it were, we'd give the library tax to the poor, and pay for the library with membership fees. That would make everyone better off. Since we don't, it's fair to assume that giving the poor free access to the library benefits them more.)

It's not just libraries -- it's police, and fire protection, too -- the poor are served just as well as the rich by police protection. And don't forget public transit (which certainly *does* benefit the poor more than the rich), and all the other services that government provides.

Part of the reason we pay for these services out of taxes is that they're so essential that we want to make sure even the poorest members of society benefit from them. But in that case, we need to figure them into the calculation.

My perception is that the level government services has exploded over the last few decades, while the Gini has increased only moderately. Figure in the value of those services -- and taxation, if you haven't already done so -- and the increase would be substantially lower.

22. It's spending that matters, not income. If I gave you an income of $1,000,000, but told you you could never spend it, it would be useless. And if I let you have a million dollars worth of merchandise (of your choice), instead of cash, the merchandise would be just as good.

And it's a principle of economics that people work to smooth out their consumption (spending) over their lifetime. That means they borrow money when they're young, to pay for cars and TVs and houses, and pay it back when they're older. And so, consumption is much less unequal than income. (It's theoretically possible for everyone to consume exactly the same, despite differing incomes over time; imagine an insurance company that offers you $50,000 worth of consumption every year in exchange for your salary every year. That would make everyone equal. Then, imagine doing this yourself -- you borrow in years where you're below $50,000, and pay back the debt in years when you're above $50,000.)

I remember seeing a few articles that mentioned that low-income people actually spend a lot more than their income, almost twice as much in some cases. (This might be through tax credits, or welfare, or such.) In that case, measuring income inequality is a pretty crappy way of measuring the differences in how people actually live. We should measure *consumption* inequality.

23. And if we DO measure inequality of consumption, the number goes way, way down. The way the Gini index is constructed, the more money you make, the more effect you have on the Gini. But suppose a highest-income earner makes $1 billion per year. There's no way that person can, or would, spend that much money. If he's really good at spending, he might consume (say) $10 million. (I couldn't consume anything close to $10 million in a year, but let's be conservative.) So if you want to measure spending, rather than income, the Gini is going to be overestimated by the effect of that $990 million.

As the very-rich get very-richer, they have the potential to stretch the Gini way out of proportion; but, in the most meaningful sense, consumption, inequality won't have changed much at all.

24. The super-rich, the ones that account for so much of the Gini, are also the biggest charitable donors. Bill Gates gives a lot of money to philanthropic causes. Even if they're in Africa, rather than the US, that actually increases worldwide equality, doesn't it? If you gave a million dollars to 100 random people, they'd spend it on themselves. If you give $100 million to Bill Gates, he'd spend it on some of the poorest people in the world. So greater inequality among the super-rich becomes greater equality overall!

25. Bernie Madoff lowered the Gini a little bit last year.

26. A substantial portion of income comes from savings and investments. And, because of the power of compounding, a small increase in savings today can add up to a huge difference in income in the future.

Ruth orders a pizza every week for a year, at $1,000. Sarah, having seen those commercials about saving money and living better, buys her pizza at Wal-Mart, for $400. She invests the $600 difference at 5% after inflation.

Forty years later, her $600 has grown into about $4200. She now pulls in an extra $300 or so in income from that $4200. The Gini rises, but there was never any inequality there, just differences in saving patterns.

And differences in savings are HUGE. I know older couples who made incomes I couldn't live on myself, who managed to save enough that they make more in retirement than they did when they were working. And I know people with six-figure incomes who are deep in debt and can't afford to pay their bills. The difference is not inequality of opportunity to earn income -- it's inequality of savings rates.

27. As we get richer and richer, it becomes easier to save. A 20" (tube) color TV now costs $100; I bought one thirteen years ago for my Dad at $500. In terms of TVs, you can save $400 with no change of lifestyle from 1996.

Or, of course, you could use the $400 to upgrade to a flat panel LCD, which many people do. But there's a choice there now that wasn't there only a few years ago. And because people are different, they make different choices. More choices means more diversity. More diversity means more dispersion in savings rates. More dispersion in savings rates means higher income inequality.

I think all this is a good thing.

28. Again because of the power of compounding, the effect of savings grows the more years you can save. So as life expectancies rise, you'd expect income inequality to rise. Imagine saving $1000 at age 25, again at 5%. By 65, you'll have $7,040. But if you can live to 95, you'll have $30,426. So it follows that the Gini should rise as people live longer and save longer.

If people lived to 200, inequality would be absolutely huge: save $1000 at 25 and you'll have $1.5 million by age 175. It wouldn't mean that society was unequal, just that it naturally takes time to build wealth. We don't worry about "inequality of knowledge" between a 25-year-old doctor and a 65-year-old doctor -- why should it be any different for savings?

Or, looked at another way: a 65-year-old has worked, over his lifetime, 40 times as much as a 25-year-old. Why would you expect their incomes to be equal?

29. It's pretty much accepted that we could create more equal incomes if we wanted to, by increasing tax rates on the rich. And it's also accepted that it would slow down economic growth -- say, by reducing it from 3% a year to 1.5% a year -- because of reduced incentive to work or take risks.

Now, suppose that country A decides to follow that route, and grow by 1.5% a year. Country B decides to let the inequality be, and grow by 3% a year.

100 years later, average income in country A has grown from $30,000 a year to $132,961. In country B, income has grown from $30,000 to $576,559.

So B is four times richer than A.

If you were to combine country A and B into one country, you'd almost certainly measure more inequality than in B alone! Now, if there was a moral obligation in A to increase equality even at the expense of total wealth, then is there also a moral obligation to increase equality *between A and B*? If there is, how would you do it? By force? And wouldn't that be unfair to B, whose population decided that the trade-off favored higher incomes over equal incomes?

30. Suppose that we -- the US and Canada -- had decided to increase equality back in 1908, and raised tax rates on the rich. And suppose that had lowered our growth rate by 1.5 percentage points over the last century. Then, we'd have only 23% as much income as we have now.

Would it have been worth it? Would you be willing, in retrospect, to take a 75% pay cut (and a 75% cut in government services, and probably a huge cut in medical advances) so that previous generations would have been more equal?

I wouldn't, and I doubt if anyone else would.

So then, don't you also have to think it would be a bad thing to make your great-great-grandchildren take a 75% pay cut 100 years into the future? Because that's the trade-off. At 1.5 percentage points difference, we'd seriously be costing the next generations hundreds of thousands of dollars each.

Now, 1.5 percentage points might be a bit high. If we assume only 1.0 percentage points, then instead of a 75% pay cut, it's a 60% pay cut. If we assume 0.5 percentage points, it's a 39% pay cut. At what point are we happy with the trade-off?

It's a legitimate question, but I don't think the Gini coefficient factors into the answer very easily. At least not unless you have an argument about what the Gini *should* be, and how much it's worth paying to get it there.

Absent that, the Gini is not very useful at all.

-------

UPDATE

Brian Burke reminds me in the comments of another problem with the Gini, an important one.

31. New immigrants to Canada and the US tend to be poorer than average, if only because most countries in the world are poorer than we are. This distorts the Gini. To paraphrase Brian's theatre analogy in the comments:

Suppose that there are five people in the country, with incomes of $10K, $20K, $30K, $40K, and $50K. Over the next 10 years, everyone's salary rises $10K, substantially reducing inequality. But a new immigrant arrives, who earns $10K.

What do we now have? Six people, earning $10K, $20K, $30K, $40K, $50K, and $60K. This is a higher Gini than before -- but only because of the new immigrant! Within the country, equality is actually increasing.

This is huge, and I'm kicking myself for forgetting to include it. I remember several economists mentioning it in the context of income -- that the reason lower-quintile income hasn't seemed to increase is simply because new immigrants replace the lower-income people who moved up and out of the low-income group. (Here's a post by Arnold Kling, as an example.)

--------

UPDATE: I'm just going to add more as I think of them.

32. Immigration, which mostly brings in people who start out at low incomes, increases the Gini, as I noted above. However, it *reduces* inequality for the world as a whole -- immigrants usually do much better here than they would in their home countries. Doesn't that suggest that looking at the Gini for a particular country, in isolation, could be misleading?

33. Activists decry the low wages American firms pay employees in poor countries. But if those US companies paid higher wages, they would be well above-market for those countries, increasing inequality there. Does that mean a higher Gini in a poorer country is OK?

Saturday, March 07, 2009

The "Verducci Effect" revisited

That's the forecasting principle, invented by Sports Illustrated writer Tom Verducci, that when a young pitcher throws 30 more innings than in his previous season, he's due for a comedown next season.

But as I said before, I think the effect is simply regression to the mean. When a pitcher throws more innings than before, it's usually because he had a better year (since they don't normally let lousy pitchers throw a lot of innings). And when a pitcher has a better year, it's usually because he's somewhat lucky. And so he'll slide back to his normal level of performance the next year.

While I have no argument with the truth of Verducci's finding, I think it's not a matter of the innings, but, rather, a matter of the good performance.

In fact, I think that all things being equal, a pitcher with more innings is LESS likely to regress. Consider two 23-year-old pitchers: each has a career ERA of 4.50, and each pitched 100 innings in 2007. In 2008, both pitchers improved to 4.00. But pitcher A threw 105 innings, and pitcher B threw 150 innings.

According to Verducci, pitcher B is due for a comedown, while pitcher A is not. I disagree. I think pitcher A is more likely to drop back to his 4.50 career average. That's because there's less luck in pitcher B's record, and so his improvement from 4.50 to 4.00 is more likely to have been real.

Thursday, March 05, 2009

Why hasn't foul shooting improved?

Free-throw shooting percentages haven't changed much over the past 50 years, according to this New York Times article. Between 1950 and 1970, the conversion rate was around 72 percent. Since 1970, it's fluctuated between 72 and 77%.

Here's the NYT graph:

So it looks like free throwing hasn't really improved over the decades. That makes foul shooting an anomaly, because most other skills have improved: marathon times are better, football kicking is better, and "swimming records seemingly fall at each international event."

Why hasn't foul shooting improved? According to the article:

Ray Stefani, a professor emeritus at California State University, Long Beach, is an expert in the statistical analysis of sports. Widespread improvement over time in any sport, he said, depends on a combination of four factors: physiology (the size and fitness of athletes, perhaps aided by performance-enhancing drugs), technology or innovation (things like the advent of rowing machines to train rowers, and the Fosbury Flop in high jumping), coaching (changes in strategy) and equipment (like the clap skate in speedskating or fiberglass poles in pole vaulting). ...

“There are not a lot of those four things that would help in free-throw shooting,” Stefani said.

And that's fair enough. But what about, say, bowling? The article says explicitly that "bowling a 300 game is not as unlikely as it once was," and there are strong similarities between bowling and foul shooting. Physiology doesn't seem like it would help either way; technology and innovation don't seem like issues; and it's hard to see how coaching would be of more help in bowling than in foul shooting.

I'd propose another explanation: foul shooting is an ancillary skill in basketball – players are chosen for their overall ability, not just their free-throw potential. And so "natural selection" won't weed out mediocre shooters or reward the best shooters, at least not very much compared to other skills.

Compare this to other sports: bowling strikes is the primary goal of the game, the most important skill of all. And, in football, field-goal kickers are chosen for one thing: their ability to kick field goals. Any kicker below average in accuracy is out of the league instantly. But any NBA player who can't hit free throws can make it up in other aspects of the game (like Shaq). (A version of this argument was also made in the first comment of a discussion on Tango's blog, here). And coaches don't force their players to shoot underhand, which would make many players more accurate; that provides support for the idea that the NBA thinks free throw percentage doesn't matter that much.

If you want to *really* see if the skill is improving, don't look to NBA players, who may not be the best in the world at the skill. You'd have to look at free-throw specialists. I Googled "free throw shooting contest results," and got a link to an Iowa State contest where the winner made 49 out of 50 throws. That's 98%, and about 4 standard deviations away from the NBA average of 75%. Even considering that the contest had 72 entries, that's pretty significant.

And here's another argument: if foul shooting isn't considered a major skill, young players won't practice it as much, and it stands to reason that you won't get as much improvement over time if there's not as much energy expended to get better at it.

One last point: if you consider the graph's increase from 71 to 77 percent to be real, then that's actually pretty good evidence of an increase in skill. When you're already at a 71% level, it's harder to improve than if you start from, say, a 34% level (as field-goal percentage did). In 1950, players were missing 29% of their foul shots. In 2008, they were missing only 23%. That means that over the past 58 years, players learned to convert 20% of their misses into hits. That's pretty good. The field goal percentage improvement, from 34% to 46%, looks more impressive, but results from converting 18% of misses into hits – almost an identical improvement (although they probably shouldn't be compared directly, because field goals are influenced by where they're taken from, and the quality of the defense).

In summary:

-- there are good reasons you wouldn’t expect foul-shooting to improve as much as other skills over time;-- if you look at the numbers more closely, there actually *is* a significant amount of improvement.