Sunday, January 3, 2010

You get what you measure - APD version

As Rafael Goyeneche, president of the New Orleans Metropolitan Crime Commission, told me in November, Pennington was besotted with COMSTAT, a crime data mapping system that he later brought to Atlanta. Basically, it’s a numbers tracker. By focusing on the numbers of crime, rather than crime itself, Pennington set a tone in the NOPD that encouraged district majors (what we call zone majors in Atlanta) to compete with each other for low crime stats. The result was cops underreporting crime. Unwittingly or not, employees in any institution tend to pick up on what matters to their boss, and they act accordingly.

I certainly think Ramage has a point about underreporting, but I don't think it may be as malicious intentional as she suggests. I want to stress 'may,' because I can't know how intentional these things are inside the APD.

In b-school, one phrase that has come up a few times is "you get what you measure." When you make someone's performance evaluation dependent on a measurement, that measurement will likely improve. What WON'T improve is anything else. You are, in essence, telling them that all you care about is the measurement. If a low reporting of crime numbers is what you want, that is what you will get - never mind if it impacts customer satisfaction, or if response times stay high, or other aspects of the job that officers aren't spending their time on. You will get what you measure, and that is all.

This is why I've been calling for more than "more cops," but an organizational overhaul of the Police Department and an audit of their management practices. Are the police using a holistic scorecard, or is there too much focusing on the reported crime numbers? I think you can still use metrics to asses performance, but lets use more than just the number of reported crimes.

My first thought at some metrics I'd love to see published every month or at least quarter, in simple CSV files so that anyone with excel can do their own analysis if they want to:

Reported crimes, by type, zone, beat, etc.

Crimes solved, and whether that person was convicted (do they even keep track of this?)

Number of calls responded to, location, and how many were reported as crimes (this would help measure Ramage's accusation)

Response times, by zone, beat, etc.

Call center wait times, blackout times

Some sort of customer satisfaction measure - something like a quarterly poll of city residents, or some other regular

The city has some of this stuff, but they happen in irregular audits. I'd love to see the APD set up to record this stuff regularly.

3 comments:

Where did I say it was "malicious"? In the column to which you refer, I actually compare Pennington to a cub reporter at the Red and Black who "was not evil" and "did not mean to hurt anyone." I never attributed malice to the APD, nor did I accuse the APD of data manipulation. I wrote about underreporting. That means there is no crime report to manipulate. So, I'm curious about your motive for distorting my column here on your blog. Do you have a need to spin me as someone who wants to attribute malice? --Stephanie Ramage

Stephanie - I honestly did not intend to spin you. My use of the word malicious, while perhaps sloppy, was based on reading your columns about the crime reporting problem for a while now.

When I read your columns about the crime stats and underreporting, I always come away with the feeling that manipulation is implied, and that top brass at APD is aware of this and doesn't care. That very well may be the case - I simply have no idea of knowing it.

“Manipulation of crime statistics starts when the police officer doesn’t make a report, or he discourages a citizen from making a report, or his commander interferes and has him change a report,” says Arcangeli.

You also wrote in the same piece:

"The problem, say experts in both cities, isn’t Pennington or COMSTAT, but the combination of Pennington’s numbers-oriented leadership style and COMSTAT’s vulnerability to manipulation.

You wrote another time about similarities between New Orleans and Atlanta crime stat programs:

Unfortunately, Central Records ultimately has final say-so on how an incident is classified.”

Central Records is firmly under the authority of Police Chief Richard Pennington.

Pennington was lauded in New Orleans for bringing down the number of that city’s reported incidents of crime. Yet shortly after Pennington left to become Atlanta’s chief in 2002, some high-ranking officers were fired for tampering with New Orleans’ crime reports.

Based on your previous writing, I'd say there is a strong implication of data manipulation going on at the APD. Again, this may be true. But you have a strong opinion on this matter which comes through in your writing, and I translated that as attributing ill-intent on Pennington and top brass for overlooking and/or encouraging data manipulation and/or underreporting (which you've quoted an expert as being equivalent to manipulation). Hence, the underreporting being done by APD as malicious, with ill intent.

I honestly did not intend this as spin or a personal slight. I enjoy your writing and your perspective, and I've also enjoyed the bits of professional correspondence we've had over the last year or so.

I hope you'll notice that at the end of the day, I largely agree with you about how the underreporting can happen - I just may attribute it more to poor management and incompetence than I do to willful neglect (on the part of Pennington). I also think that anyone with any management training would predict it would happen.

I've edited the post to remove the word malicious. It was a poor choice of words that did not accurately reflect what Ramage has written. Based on her previous writings, I'm comfortable with stating that she implies intent from top brass on the APD's possible underreporting.