Sunday, March 06, 2011

Is "superstar bias" caused by Bayesian referees?

Would you rather have referees be more accurate, or less biased in favor of superstars?

In the NBA, a foul is called when the player with the ball makes significant contact with a defender while he's moving to take a shot. But which player is charged with the foul? Is it an illegal charge on the offensive player (running into a defender who's set and immobile), or is it an illegal block by the defensive player (who illegally gets in the way of a player in the act of shooting)?

"... the often-fractional difference between a charge and a block call is decided by a referee who has to determine, in a split second: a) were the defender's feet set, b) was he outside the court's semicircle, c) who initiated contact, and d) does the contact merit a call at all?"

It seems reasonable to assume that, in a lot of cases, the referee doesn't know for sure, and has to make an uncertain call. Maybe he's 80% sure it's a charge, or 70% sure it's a block, and makes the call according to that best guess. (Not that the ref necessarily thinks in terms of those numbers, but he might have an idea in his mind of what the chances are.)

Now, suppose there's a case where, to the referee's eyes, he sees a 60% chance it was a charge, and only a 40% chance it was a block. He's about to call the charge. But, now, he notices who the players are. Defensive player B ("bad guy") is known as a reckless defender, and gets called for blocks all the time. Offensive player G ("good guy") is known to be a very careful player with his head in the game, who doesn't charge very often at all.

Knowing the characteristics of the two players, the referee now guesses there's an 80% chance it's really a block. Instead of 60/40, the chance is now 20/80.

What should the ref do? Should he call the charge, as he originally would have if he hadn't known who the players were? Or should he take into account that G doesn't foul often, while B is a repeat offender, and call the block instead?

------

If the ref calls the foul on player B, he'll be right a lot more often than if he calls it on G. When the NBA office reviews referees on how accurate their calls are, he'll wind up looking pretty good. But, B gets the short end of the stick. He'll be called for a lot more fouls than he actually commits, because, any time there's enough doubt, he gets the blame.

On the other hand, if the ref calls the foul on G, he'll be wrong more often. But, at least there's no "profiling." G doesn't get credit for his clean reputation, and there's no prejudice against B because of his criminal past.

Still, one player gets the short end of the stick, either way. The first way, B gets called for too many fouls. The second way, G gets called for too many fouls. Either way, one group of players gets the shaft. Do we want it to be the good guys, or the bad guys?

Maybe you think it's better that the bad guys, the reckless players, get the unfair calls. If you do, you shouldn't be complaining about "superstar bias," the idea that the best players get favorable calls from referees. Because, I'd guess, superstars are more likely to be Gs than Bs. Tell me if I'm wrong, but here's my logic.

First, they're better players, so they can be effective without fouling, and probably are better at avoiding fouls. Second, because they're in the play so much more than their teammates, they have more opportunities to foul. If they were Bs, they'd foul out of games all the time; this gives them a strong incentive to be Gs. And, third, a superstar fouling out costs his team a lot more than a marginal player fouling out. So superstars have even more incentive to play clean.

So if superstar bias exists, it might not be subconscious, irrational bias on the part of referees. The refs might be completely rational. They might be deciding that, in the face of imperfect information on what happened, they're going to make the call that's most likely to be correct, given the identities and predilections of the players involved. And that happens to benefit the stars.

------

When I started writing this, I thought of it as a tradeoff: the ref can be as accurate as possible, or he can be unbiased -- but not both. But, now, as I write this, I see the referee *can't* be unbiased. If there's any doubt in his mind on any play, his choices are: act in a way in which there will be a bias against the Bs; or act in a way in which there will be a bias against the Gs.

Is there something wrong with my logic? If not, then I have two questions:

1. Which is more fair? Should the ref be as Bayesian as possible, and profile players to increase overall accuracy at the expense of the Bs? Or should the referee ignore the "profiling" information, and reduce his overall accuracy, at the expense of the Gs?

2. For you guys who actually follow basketball -- what do you think refs actually do in this situation?

10 Comments:

A similar bias might occur when the change in win probability relative to the start of the play is greater for one team than the other. On a long (football) pass with potential interference, for instance, the offense stands to gain more with the call than they would lose by an incomplete. Using advancednflstats.com WPA calculator, on 1st and ten from the 50 at the start of the 4th quarter, the team with the ball has a 0.59 WP. A complete pass/interference call at the 15 would give them 0.67 (+0.08), an incomplete pass would give them 0.55 (-0.04). (Perhaps the "correct" way to look at this is to say it's either +0.12 or -0.12 by comparing "Call" to "No Call", but a ref would probably keep in mind what the state was before the play.

Right ... that falls under the "omission bias" effect that the "Scorecasting" guys found. I bet you're right -- I bet you'd find that omission bias increases greatly when the effect of a wrong call is magnified.

I think the whole "omission bias" thing could yield some juicy results if you look at things like these. A fruitful topic for "Scorecasting II", perhaps!

If the ref is leaning 60/40 towards calling the charge, he should always call the charge. I don't quite follow your logic on calling the block against the "bad guy". His 80/20 bias against the bad guy, based purely on his reputation, does not make his call any more correct in my mind, as it is still a 60/40 charge call based on what actually happened. Yes, he is wrong 40% of the time if he calls the charge, but the alternative is being wrong 60% of the time. Perhaps the best call is a no-call.

1. The problem is that the ref can't call 60% of a foul, so it would be 100% of the time. If the same two players were involved all the time, I suppose the ref might be able to keep track and call the foul 60% of the time one way and 40% the other way, but that's impractical in real life.

2. OK. Imagine 40/60 to 60/40, then ... the point remains the same, I think.

In effect what I think you are saying is that for G:B your baysian prior would be 20:80 -- then if you see a 60:40 what do you call -- do you go with the prior or with what you see? Is that right?

However, I don't think the decision is in series. I don't think the ump would go ... gee, I thought G did the foul but I expected B to do the foul ... more realistically it he perception of the observed G:B is INFLUENCED by this prior. Therefore when he thinks he observes 60:40 he is actually seeing 80:20, if that makes sense.

I would imagine what he observes (after adjusting for prior) translates in to a call depending on range ie..

- above 60/40 foul to save- btwn 60/40 and 40/60 no call (unless such an obvious foul that he'd look silly for not calling, in which case call in the balance of probabilities).

Yes there is still bias but I think it manifest differntly to how you described.

How the Bayesian works is that you see a call that you'd think was 60:40 if you had no other information ... but with the information of who the players are, it changes to (for instance) 20:80.

For instance, suppose I toss a coin 50 times and it lands heads 44% of the time. The naive guess is that the coin is a 44%-heads coin. But you have additional information -- that almost all coins are very close to fair. So you change your guess, and you guess that the coin is a 50%-heads coin, not a 44%-heads coin.

Bayesian means just that the estimate is influenced by other information you may have.

The point is that refs can use that information (about who the players are), or not. They can use it any way they choose.

I'm not 100% sure in how I'm interpreting your comment, but if you're saying that sometimes a referee will need more than 60% to make a call, sure, that might be true. I kept it simple to make the point.

One of my comments (third comment above overall) said that I discovered Chris Moore's argument ... apparently I read it at the time it came out, since I commented on it there. So, consider this an extension of Chris's argument, now applied to fouls.

Bayesian - How accurate is the refs perception of the good or bad players reputation?

Non-Bayesian - If you don't account for reputation, then I think that you give the bad players too much of an opportunity to manipulate the refs perception of the foul play.

Basically, what I'm getting at is that I would prefer a Bayesian method that allows for a minor adjustment.

To be honest though, I don't think that superstars tend to fall under the good label as much as you hypothesize. But then again I'm a casual basketball fan, so my opinion in that regard should probably be ignored.