Wednesday, July 24, 2013

Luck, careers, and income inequality

According to this (anonymous) blog post from "The Economist," increased income inequality is partly due to random chance becoming more important in picking winners and losers. The post (and accompanying article) cites a recent paper on "herd effects," where a song or book becomes popular kind of randomly, because people tend to flock to whatever other people have already flocked to. So, you can have two songs of equal merit, but one is a smash hit just because, for random reasons, its early adopters had better social networks.Furthermore, there's experimental evidence. In a recent academic study, researchers sent job applications to two different employers. The resumes were identical except that, in one of them, the applicant said he had been unemployed for several years, while in the other, the applicant said he had a current job. Overall, the "out of work" applicants got around half the callbacks of the currently employed applicants. Presumably, the employers believed that those who were out of work were less likely to be suitable candidates than those who weren't. The author writes,

" ... the implication is troubling: someone who ends up unemployed through bad luck, and for some idiosyncratic reason doesn't quickly land a job, finds his chances of reemployment diminish until he’s part of the long-term unemployed. ...

"... this research opens a new and troubling dimension on inequality. Unlike deficiencies of skill, it’s hard to tell society's losers they should go back to school to become luckier. "

------Certainly, there a huge, huge amount of luck involved in career success. There has to be: there's already luck in hockey, golf, and chess, and real life is infinitely more random and complicated than those.For starters, there's when and where you were born, what kind of schools you went to, what books you happened to read, what kind of influences you had from your friends and parents, how good your teachers were in various subjects, chance remarks from strangers about what careers are likely to be lucrative, and many more.Even in the more concrete sense, there are many, many small random things that can make a big difference. What company you choose to join for your first job, what your first manager thinks of you, how your bosses think of your performance amidst the noise of team accomplishments, whether your co-workers give you credit or blame behind your back ... and you can probably think of hundreds more.Basically, where you wind up in your life, in your career or any other sense, is overwhelmingly the result of so many random events that it's basically ... well, it's kind of a miracle that we are where we are now instead of somewhere else. If we had to live our lives over again, from any given point, we'd probably be in different jobs, doing different work, with a different spouse, maybe living in a different city, earning a different income. In that light, I find it a bit perplexing that the authors chose these two specific examples as evidence of how luck factors into it. They seem pretty minor, don't they? I mean, the unemployed applicants still got half as many calls as the employed ones ... that seems like a pretty small career stopper as opposed to, say, if their first manager didn't like them and put them into a dead-end position and never promoted them, among hundreds of other possible events.As for the "herding" argument ... wouldn't it be the case that there's LESS herding luck now, with the internet? It used to be that you only knew songs you heard on the radio. Now, you can find whatever you want on YouTube, on demand. There's more opportunity to find a niche. Isn't that established wisdom? Isn't that what "The Long Tail" is about, that, these days, there are many different markets that never existed before?In terms of "herding," I'd bet that luck is much LESS a factor than it used to be. -------Also: as much as luck is a huge, huge part of where any individual winds up, I don't think it's a big factor in the overall distribution.In baseball, luck is, what, around 30 percent of the variation in team W-L percentage? Now, suppose you expanded the major leagues to include 30 little league teams. Now, luck is a much smaller factor, isn't it? The "real" teams always beat the kids, so, now, whether or not you're over .500 is completely a matter of talent. Luck is still important within your group, but, overall, I'd bet the r-squared drops from 30 percent to ... I dunno, maybe 5 percent. Something similar happens with careers. Two accountants, Andy and Beth, graduate from the same school with the same talent. Andy winds up in a job where the culture matches his talents, while Beth does not. Just by that random happening, Andy eventually winds up a CFO making $1 million a year, and Beth winds up in some middle-manager type position, making $70,000. Yes, Andy outearns Beth because of a lucky break. But, the difference is contained within the accountants' income distribution, in the same way luck in our expanded MLB is contained within the teams in a single age level.Luck matters more, obviously, when the differences in talent are small. Within the subgroup of accountants, luck is going to matter a lot. Within the subgroup of burger-flippers, luck is going to matter a lot. But within the COMBINED group of accountants and burger-flippers, luck isn't that big a deal. You're not going to have many burger flippers, with average talent, getting lucky and outearning a typical accountant.You can probably see this for yourself with a little introspection. For my own part, I can see my career, and life, turning out radically differently based on a few key random events and decisions. But ... I don't think I'd be poor. I don't think I'd be making minimum wage. I like programming, and I'd find a programming job somewhere, and even with the worst luck, I'd be making a decent living. Also, I don't think I'd be ultra-rich. Even with extreme good luck -- say, 3 or 4 SDs above average -- I doubt I'd be a charismatic, celebrated CEO of a big company. That requires skills and personality traits that I just don't have.There's a lot of luck in individual baseball batting performance, but that doesn't mean you'll see Ozzie Smith hitting 60 home runs. And there's a lot of luck in career outcomes too, but that doesn't mean you'll see lots of Bill Gateses working minimum wage.At least, not because of the kind of luck the authors are talking about here. ------But, fundamentally ... I wonder about the entire premise of the argument, that the prevalence of luck causes inequality to increase. I wonder if it's the other way around.A "system that rewards ability", as the author puts it, is an important and desirable goal. A job, or promotion, or salary, or other reward, should go to the person who does the job best, not the one with the right resume, or the right skin color, or the right DNA.Nor, by extension, to the one who happened to obtain the best roll of the dice. But, why does it follow that a system that rewards ability must also be more equal? I think that's the wrong conclusion. It's understandable ... we see that some people get rewarded for being lucky, and we think, well, if only those people didn't get to be so rich, randomly, we'd all be more equal! But, I think that's backwards, at least in the hiring context. The more merit matters, the *bigger* the differences in salary. Because, an employer is only going to pay you the expected value of your production. Let's use a simplified sports analogy. A team is trying to sign players, who are $100 on average ... but, some are worth more, perhaps much more, and some less. Suppose you have no scouts. So, you have no idea of the players' relative talent (assume, also, that neither do the other teams, or the players themselves). That means you have to hire randomly. What are you going to pay? Obviously, $100 per player. You have complete income equality among your players, even though the guys on the team got there by 100% luck.Later, you hire some mediocre scouts. You're able to tell the better players from the worse ones, to a certain extent, based on "whether they look like ballplayers". So, you hire some at $130, and some at $70. Less equality.Finally, you hire some sabermetricians and some expert scouts, and now you can estimate each player's value within $10. You hire the superstar at $400, and a mediocre mop-up man at $5. Even less equality.Isn't that a better description of how real life works? You can't be a well-paid employee unless you can prove to an employer that you're worth it. The more merit-based the system, the better chance you have of getting full value for what you're worth.If you don't like that analogy ... try this. There are a hundred brand new Honda Civics on the lot, and they all cost the same. But, now, suppose you have more information about their merit -- somehow, it becomes possible to predict reliability. That blue Civic over there will run trouble-free for 20 years. But that red one ... the transmission is going to blow at 80,000 miles. Now, the top 1% of Civics will sell for a lot more than the bottom 1%. Right?-------In general: the closer you look for merit, the more differences you find. The more differences you find, the wider the distribution of the sum of those differences. So, the more you hire by merit, the more spread you get in salaries.Is that a problem? Not for me. I'm strongly in favor of hiring by merit ... and I don't think income inequality is necessarily always a bad thing. So, to me, there's no serious trade-off. But, if, you like both merit and equality, I think you have to choose which is the lesser evil, in this context.As a society, I think we've made our consensus trade-off ... because, there are many, many unspoken ways in which we deliberately ignore merit. For instance, we pay our teachers by seniority and experience, rather than by how good teachers they are. We try to avoid situations where long-term employees get demoted or has their pay cut, even when their performance drops. We discourage comparisons of which employees are better workers than their colleagues in the same position.We give lip service to the idea that we should be paid and rewarded by merit, but we don't actually believe it as much as we say we do. We claim we can't have nepotism because then we're being unfair to candidates with more talent, but then we believe we CAN favor older, worse teachers, without being unfair to younger teachers with more merit. Like a lot of other things in our society, we start with the outcome we want and work backwards to rationalize it. We hate nepotism, and we hate firing mediocre teachers. Merit is just an incidental consideration.

11 Comments:

I think you're overlooking a couple of pretty huge factors. One is the first and probably largest way luck enters our situation. It's very very likely that you weren't born into a poor family in lowland South Carolina, because very few of those people manage to get move "up the ladder" enough to do things like write blogs about sabermetrics. Even less likely for someone born into poverty in Haiti or Bangladesh.

Secondly, there are more than a few areas where it's genuinely difficult to judge "merit", and you spend quite a while on the one that probably affects the most people. I don't think there's anything like a consensus on what makes up a "good teacher", much less how to measure it. People jump to things like test scores because, well, it's something they can measure.

It's a typical bias of people who have been the beneficiaries of chance to minimize its effect in their lives. I'm not saying you did it intentionally, but if you thought a bit more outside the box, I think there are a lot of things that you don't even think of as chance events that are conspicuous in your life because they *didn't* happen. Most of us went through childhood with an intact family and don't think of that as "good luck", but it is an essentially chance event if your father didn't have a heart attack at 42 and seriously changed the economic fortunes of your family.

As Mike said, the biggest counterargument is the subjectivity of "merit". Life does not imitate the World Series - outside the constricted playing field/rules of baseball, it is much harder to objectively analyze "merit". Even in programming. Who has more merit? Someone who solved difficult problems quickly with code that is difficult to maintain or someone who takes a bit longer but whose code is elegant and long-lasting? But beyond that obvious point, luck is regressive. Steve Ballmer graduates a year or two before Bill Gates arrives. Gates, getting lost on the way to sell IBM on DOS, doesn't show for the meeting. Microsoft doesn't happen. There go 3 billionaires and 12,000 millionaires. It's not just whether you impressed your hiring manager with your clearly superior capabilities, it's whether your hiring manager was having a good day, whether she impressed her supervisors, etc. The cascade of things beyond your control is long. Last, this all may be wrong. But it would be good to see some way of quantifying your argument. This piece could come from Ken Harrelson :).

There's another problem with the baseball analogy: baseball is zero-sum, winner-take-all.

You would rather win with a score of 1-0 than lose with a score of 10-11, even though the latter involves more 'production'. That's why they pay superstars the big bucks: because any incremental advantage you get over the other team is worth it in a winner-take-all situation. This is how a guy like A-Rod made 20 times the amount of other third basemen, even though he obviously wasn't 20 times more talented (what would that even mean?). He may have been 10% more talented, but that 10% edge makes a huge different in a winner-take-all game.

Real life isn't like this. In fact, life would be nasty, brutish, and short if it were winner-take-all.

In our economy, merit appears to be tied one-on-one to how much money you can generate. Impact on financials seem to be the only metric we have to assess merit. What an inadequate and bankrupt metric.

Does a high school teacher have 'merit' if s/he produces the highest number of high-achieving students who go to top colleges and become investment bankers? Does a high school teacher have 'merit' if his/her students leave a class with more self-awareness and compassion towards humanity than they did going in?

As Mike and Tim pointed out, 'merit' is nearly impossible to assess and all too often, we end up falling back on financial indicators.

Capacity to generate money is such a shallow way of capturing 'merit'.

I don't think too many people complain about income inequality that is demonstrably based on merit. The disillusionment stems from situations where (a)notions of 'merit' are dubious and (b)the reward for any perceived merit is out of proportion to the difference in performance and (c) where the rewards and penalties are incongruent. Incompetent CEOs being 'fired' with rather extravagant packages is a case in point, as is the privatization of profit and socialization of risk. At a less rarified level, one could cite university administrators vs non-tenured faculty.

Saying merit can't be determined is a lame cop out. Sounds like an excuse for the mediocre to hide behind their mediocrity. As a student I could easily tell from whom I could learn easily and from whom it was difficult or painful. If you claim otherwise you're lying. As a supervisor and colleague I see wide ranges of productivity, talent, and value provision. Sometimes it's close and harder to say precisely, but as the author points out the bands are relatively easy to discern. Think about the last time you've received a service. Haven't you experienced wide ranges in satisfaction? Haven't you thought - wow, they're really good? Or, man, that was awful! Denying any of this result is purely a result of delusional ideology that blinds you to readily apparent truth and reality. Please check yourself.

Also the luck about being born into a poor family in lowland South Carolina is overstated. I and my brother were both born into a lower class family and have achieved upper middle class comfort through merit. I mean really in a sense it's all luck - genes, chance, etc. - but by and large if you have talent and drive - the definitions of merit - in this country you will do alright. I'm not aware of any evidence - because I don't believe it exists - that people with talent and drive (i.e., merit) don't do fairly well in this country, regardless of the region of the country in which their born, or their race.

I agree with the 'Haiti' argument, but come on man - this isn't the point he was making.

The statistics all up in here are wack, and most of your conclusions are pulled out of your ass (e.g. "I'd bet that luck is much LESS a factor than it used to be [with the internet]"). You don't even remotely explore what factors contribute to herding, and what the internet's impact on those factors would be.

And the statistics. Assuming luck accounts for 30% of W-L record among professional teams, this means that known factors account for 70% of the record. You then add little league teams. Assuming you add an equal number as professional teams (the value which minimizes random chance, and therefore maximizes your argument), assuming you can tell which team is little league, and assuming that the factor from luck is the same among little league teams, random chance now accounts for 15% of the W-L record, not the 5% you pulled out of your ass.

More generally, I am not getting the impression that your post understands luck, or even has a consistent model for how luck functions. Even less merit.

I am so sorry! I left the previous anonymous post, walk away, then 5 minutes later realize I was wrong! Silly me to scold you on your statistics when mine own are off.

The example with the baseball teams. If you had the teams, and then one day realized that half were little league, THEN the factor due to luck would lower to 15%. If you one day added a bunch of little leagues, the factor remains 30%. Too bad for your reasoning.

As several people have pointed out, merit is sometimes hard to quantify. Furthermore, rules about taking seniority into account protect against personal bias. For example, if the company has to cut workers, the boss may choose to keep his golf buddy and let you go, regardless of merit. Or she may choose to let you go because you have a higher salary than someone else. You may argue that the boss should be able to make the most cost-effective decision but don't pretend it's based on merit.

I like this post because it makes the problem with applying econometric logic to big intractable problems like income inequality so much more obvious.

Luck or randomness enters the picture long before the applicant pool exists. The logic here only applies if everyone starts at the same fair starting point with the same fair firing gun starting the race.

The merit argument is a classic false dilemma. Either we pay people what they deserve or communism. No one disagrees that merit should matter; that's not even the appropriate grounds for the argument. The argument is more substantive than the false dilemma implies. The argument is over what sorts of policies should we have that maximize the pool of people with the opportunity to realize their potential and get paid appropriately for it, and how do we develop policies that are culturally in seto and politically viable.