How we did it: Denver Post and Pioneer Press launch local crime maps

Pins on a map. That’s what probably comes to most people’s minds when they think of what has come to be known as a “crime map,” an online tool for displaying crimes that have occurred over a specific timeframe in a geographic area, usually a city or a county.

But, pins on a map only gets you so far. It’s difficult to see trends, and while you can draw neighborhood boundaries on the same map, it still doesn’t tell you what all those pins mean beyond “a crime happened here.”

More than pins on a map
That was the challenge for the Thunderdome data team when we began to envision how this project would look and how someone might use it. Luckily for us, crime maps have evolved since Holovaty’s original. The Chicago Tribune’s own crime app is a good example, ranking neighborhoods by crime per capita, among other things. The Los Angeles Times focuses on trends in communities, using colors to indicate whether crime is edging up or down. These were some of the types of ideas we incorporated into our design.

The main page of our map aims to give users an overview of their town, showing the number of crimes by time or location and whether crime is up or down in particular neighborhoods. Visitors can drill down to the neighborhood level from there, showing – yes – pins on a map, as well as trends for that area.

This first iteration, we think, is fairly simple and easy to use. We’re considering other features to incorporate in later releases — more filtering options on the main page or the ability to search for a specific crime, for example. Much of that will be driven by feedback from site users and from our newsrooms. Let us know what you think and leave a comment on this post.

What’s under the hood?
On the technology side, we obviously used Google maps, as well as Google’s Visualization API, jQuery, a JavaScript templating library called Handlebars and a date library called date.js. All of the sorting, filtering and aggregation happens on the front end in JavaScript, almost all of which is custom-written for this app.

The app never touches a database. Initially, it loads data for all of the current month plus the previous month. It loads earlier data a month at a time as it needs it. This part of the project (inspired in part by a comment from NPR’s Brian Boyer that “servers are for chumps“) is somewhat of an experiment and may change in future versions. Databases are a common bottleneck for online apps, and this lets us avoid that. But these so-called “baked-out” files also can be a limitation. They’re fairly big, so the initial page load can be a little slow. They also may limit the amount of customization we can offer.

Where does the data come from?
Obviously, none of this would be possible without the crime data itself. For those Digital First Media properties (or anyone for that matter) who want something similar, the first place to start is with the data. It’s important to note that this data isn’t like the Uniform Crime Statistics that the FBI puts out every year. You really can’t build something like this from aggregate information. You need some sort of constantly updated feed of individual crimes with, at a minimum, an address or at least a block number, a crime type and a time of occurrence.

In the case of Denver, the information comes over as a comma-separated value file, with latitude and longitude included. That’s probably the best anyone could hope for. No scraping. No geocoding. Just pull and slurp into the table.

St. Paul’s is a bit more complicated. First, it’s posted in Excel files on the police department’s website. Second, it has to be geocoded. This is complicated further because the last digit of the number in the addresses has been redacted and the addresses are fairly freeform: some non-standard street-type abbreviations (La for Lane) and some streets whose names are really two words but come over without a space (Mississippiriver). We’ve tried to account for some of this on the back end when we process the data, but some addresses slip through with incorrect locations. So, we created a very rough tool that MaryJo Webster uses to correct the strays.

Partly because of this geocoding challenge, MaryJo used ArcMap to geocode historical data, back to the start of 2010, for the Pioneer Press map. She was able to resolve some of the data problems noted above before the geocoding to get better results.

One of the other challenges with the St. Paul data, which will likely be true in other jurisdictions, is that very few homicides show up. The data includes only certain crime types —primarily the big ones — but it’s common that the police department classifies a murder as a “death investigation” and that designation often doesn’t get adjusted in their computer system until a suspect is charged, typically long after the time the data shows up online.

How has St. Paul responded to the Pioneer Press crime map?
Reaction from Pioneer Press readers was swift and included both compliments and criticisms, although most were constructive in nature. One reader was excited to be able to zoom into her neighborhood and find the burglary she had reported to police a few weeks earlier.

The St. Paul mayor’s spokesperson sent a note to the Pioneer Press criticizing the way the initial view of the map color shades the neighborhoods as having crime “up” or “down” compared to the past. Like many readers, he misunderstood that the comparison of the current month was not to the previous month, but rather the average of the previous 12 months (and it only indicates a change if it’s at least one standard deviation up or down). He wanted us to change it to compare to the same month one year previously, which is how the police department looks at crime trends. That’s not really practical for this app, though, given that we allow users to select the beginning and end dates (up to 31 days’ worth) of data to display, we’d have to try to calculate quite a few “averages” to accomplish that. This may be something that we address in future versions, though.

Despite the criticism, the mayor’s spokesperson was mostly impressed, saying, “It’s a really cool and useful feature, and I think we should be doing more of this kind of thing at the city.”

Some older readers were frustrated that the map wasn’t included in the print edition (there was only a short story). MaryJo tried to explain that it simply wouldn’t be possible to recreate the full experience of the crime map in print. After hearing an explanation of what the map could do, one of them said she would venture out to the public library to see if somebody would help her navigate the Internet to find the map. Before hanging up, she asked if it would be online just that day, or “would it be okay to go tomorrow?”