Celebrating the Data Guru

Today marks a bittersweet day in Data Driven Detroit’s history as we celebrate the achievements of Detroit’s own data guru and D3’s founding director, Kurt Metzger.

This morning, Kurt was honored at a breakfast at Detroit’s Rattlesnake Club. Friends and colleagues praised his accomplishments and thanked him for his work using data to make nonprofits and governments more efficient and effective. Although Kurt has moved on to new endeavors, he has left an indelible imprint on the city and Data Driven Detroit.

Kurt has spent more than 35 years promoting data accessibility in Southeast Michigan through his work with the Census Bureau, Wayne State University, the United Way for Southeastern Michigan, and founding of Data Driven Detroit.

In 2008, Kurt was serving as the Research Director for the United Way when the philanthropic community in Detroit recognized the need for data to help focus investment and measure outcomes. City Connect was developing a concept for the Detroit-Area Community Information System (D-ACIS). They tapped Kurt to bring the concept to life and serve as director.

Kurt’s vision for D-ACIS was an independent, objective clearinghouse that would provide information to the community at large, working with organizations that held the same beliefs regarding collaboration and information-sharing. In 2010, D-ACIS became Data Driven Detroit (D3). For five years at D3, Kurt was our mentor, leading us through many challenges and opportunities along the way.

Erica Raleigh, D3’s current director, is proud to continue the work that Kurt began. “Kurt’s vision is the reason I wanted to work with him,” she said. “He gathered a talented group of people at D3 who are committed to ensuring the equitable distribution of unbiased and essential information. We will continue our mission of providing high-quality, accessible information to drive informed decision-making.”

Building the 2014 One D Scorecard with NiJeL: D3.js for D3

Data Driven Detroit (D3) launched the 2014 One D Scorecard in May (read more about that here). Today, we’re writing to share more about our process for making this interactive data tool through a Q & A with NiJeL, a team of data scientists and developers that D3 collaborated with to build out the Scorecard. But first, a little context about the project:

In its third iteration, the new Scorecard makes exciting strides in data management and presentation. Working with NiJeL, we focused our resources and energy on two key components of development: an administrative tool for data management on the back end; and an interface powered by interactive data visualizations. We also revamped our data by updating and curating indicators, creating indices for each of the five Priority Areas, and incorporating a data deep-dive using original Opportunity Mapping research from the Kirwan Institute.

Let’s dig into four questions with Lela Prashad and JD Godchaux of the NiJeL team.

—

D3: Way back when we first started working together, we warned you that we were managing the One D Scorecard data through Excel workbooks, a single workbook per indicator (over 50 workbooks at one time!). It was a bit of a data nightmare in a few ways, especially when it came time for annual updates or when we needed to compare individual indicators across geographies. How did you sift through our data and come to the new centralized solution we’re using today?

NiJeL: Good question! We realized early on that we needed an easy way for D3 staff to update indicators as these datasets are updated, rather than all at once, say on an annual basis. We also understood that we wanted an automated way to add these new data to the One D Scorecard website as soon as an indicator is updated, and the only reasonable way to make this happen was to build a database to house all these indicators. So, we built a MySQL database and modified Xataface, an open-source software designed to add a simple admin interface for a MySQL database.

Once we had honed in on using these tools, we went through each Excel workbook and added each indicator to the MySQL database, slightly reorienting the data from the Excel sheets to make it easier to use. We then wrote two scripts, one to simply pull all of the relevant indicators for each region and package them up all together so the website could create the visualizations that it does, and another to calculate the five Priority Area index values and the overall One D Index score for each region.

Now, the staff at D3 can update any specific indicator by uploading a CSV (comma-separated value) file with any new data they would like to add. Once these new data are added, the web app will update the site visualizations once there is a critical mass of the data from each year to make the index calculations meaningful. We’re hoping this will be a big improvement!

D3: While older iterations of the Scorecard ranked regions based on their performance in a single indicator, we took that a few steps further this year using indices. An index lets us roll up the individual indicators that comprise a Priority Area into a single summary score, and then roll up each of those five Priority Area scores to create a One D Index Score for each region, making comparisons comprehensive and straightforward. Our favorite feature is how an index calculates on the fly and smartly recognizes when too few indicators for a given index have been updated to update the index itself. Can you share a bit about the process for programming in these analytical features to the highly visual front end?

NiJeL: Of course! As you mentioned, we want to be smart about how we’re calculating the index scores for each of the five Priority Areas and the overall One D Index, and we want to do this in the context of new data being continually added to the database. To accomplish this goal, we programmatically look at the group of indicators in each Priority Area and determine if more than 50% of these indicators have data for the year in question. For instance, the Economy Priority Area has 7 indicators, so if 4 or more of these indicators have data available for 2012, then we would calculate an Economy Index for 2012.

However, for us to go ahead and calculate a One D Index and include the year in our visualizations, each Priority Area would need to surmount the 50% threshold. Once that occurs, the indicators and Priority Area indices are added to the visualizations and the data become available for download.

D3: The design of the new One D Scorecard comes from Kat Hartman who drew inspiration from a JavaScript library for interactive data visuals aptly named Data-Driven Documents, D3.js. What are some of the challenges of taking static wireframe designs to interactive visuals? And mostly, which chart was your favorite to build and why?

NiJeL: Well, we had the distinct advantage of working with two individuals, one being Ms. Hartman and the other being D3 Project Manager Jessica McInchak, who both were interested in web interactive design and building interactive visualizations. Both actually contributed to the codebase for the One D Scorecard, which is an extremely rare thing for a designer and a project manager to want to do, but both Ms. Hartman and Ms. McInchak were excited to have the opportunity which made for a fantastic working relationship.

Ms. Hartman’s design for many of the elements in the One D Scorecard were inspired by other designs live on other websites, and so when it came time to build these visualizations, we did have some examples to view, though most were written using other tools like Raphael. We decided to use D3.js mainly because of its flexibility — it allowed us to be able to build the visualizations as closely as possible to what Ms. Hartman designed. The most challenging aspect of building to static wireframes, like the ones Ms. Hartman designed, is understanding the intended interactions and transitions between states within a particular visualization. It’s challenging as a designer to draw out intended interactions and as a developer to follow through on those intentions, but our close collaboration with Ms. Hartman and Ms. McInchak, minimized any differences we had on building the interactions as intended.

It’s tough to pick our favorite chart to build, since they all had their challenges and rewards, but we’d have to say the “array of pinwheels’ visualization (where viewers can see each region’s pinwheel in an array of rows and columns) was our favorite to build.

In this visualization, the pinwheels load such that the region with the highest One D Index value is placed in the upper left corner and the remaining regions are placed in descending order from left to right and top to bottom. Visitors can reorder the pinwheels by selecting a specific priority area to view from the “Organize By” drop down menu. Building this chart required extensive use of D3.js transitions, which allowed us to be creative in how we moved from one state to another. When visitors select a different Priority Area (or a different year of data) to view, we effectively run three separate transitions on each regional pinwheel. First, we change the color of each pinwheel slice, setting the Priority Area selected to its full opacity and setting the opacity of the other slices to almost fully transparent. At the same time, we change the size of the pie slice if a visitor has selected to view a different year with the time slider. Finally, we reorder the pinwheels in descending order based on the index chosen, but that transition only occurs after a 1 second delay to allow the first set of transitions to complete. Building these transitions in an attempt to clearly compare the differences across regions and indices was the most challenging and fun part of the development.

D3: The 2014 One D Scorecard presented a lot of opportunities to collaborate. Not only did we work with your team around development, but we also partnered with the Kirwan Institute to integrate their Southeast Michigan Regional Opportunity Mapping initiative. Kirwan’s original research was presented through static maps of the overall index scores. What was your motivation and method for interactively mapping both the index and individual indicators?

NiJeL: The work that the Kirwan Institute contributed to mapping the Regional Opportunity Index for the Detroit region was fantastic, and fairly straightforward to map. However, we wanted to build a way for visitors to view the Opportunity Index in the context of the component variables that were used to calculate the index. To this end, we modified an open-source Javascript library, Crosslet, that combines the power of D3.js, Crossfilter.js, a Javascript library that allows for multi-dimensional filtering of large datasets in a web browser, and Leaflet.js, an open-source mapping library.

Crosslet is particularly designed to allow visitors to explore one or multiple variables to see how each is connected, and to see representations of those connections on a map and a simple frequency distribution bar chart. For instance, if a visitor selected the median household income variable, and then selected the income range of $0 – $50,000, they would see only the geographies (census tracts) that have median household incomes below $50,001. They would also see the frequency distribution of the Opportunity Index, high school completion rates, and vacant property rates. Clearly, the distribution of each of the other variable is skewed toward the negative end of the spectrum when we select that income range. However, if we click on the selected range and drag it toward a higher income range, we can see the frequency distributions of the other variables shift toward the more positive end of the spectrum along with income, and on the map we can see which census tracts specifically fit these new criteria. One can also select a range of any other variable on the map to further filter these data. We think that gives visitors a great entry point to exploring these data and drawing their own conclusions about the drivers behind opportunity in the Detroit metro region.

—

Thanks, NiJeL!

The new One D Scorecard is a powerful tool for its users to access data through visualizations, but it’s also a powerful data management system for D3 to maintain and scale these datasets into the future. And we couldn’t have built it without the awesome team at NiJeL. We’re already counting down the months until the newest annual data indicators are released so we can do our first update!

The Historic Resource Survey: Community Data Collection in Action

In November 2013, the Motor City Mapping project moved forward at a dizzying pace, with the goal of identifying every blighted structure and empty lot in the city of Detroit. Loveland Technologies was in the process of finalizing the Blexting application for the city-wide parcel survey, and the Michigan Nonprofit Association hired more than 100 community surveyors. Meanwhile, in these early stages, Data Driven Detroit (D3) focused its surveying expertise on developing and testing the questions that would drive the survey. During this time, D3 received a number of requests from community groups to integrate additional questions into the survey process, ranging from assessing historic relevance to estimating rehabilitation costs. Unfortunately, with the speed of the project and the complexity of the existing survey, the Motor City Mapping team was forced to delay such modifications until future phases of the project. However, in one of these instances, a stunning example of grass-roots organizing and collaboration between the Michigan Historic Preservation Network, Local Data, and D3 resulted in the Historic Resource Survey, one of the richest datasets that would emerge from the Motor City Mapping project.

Background for the Survey

The early weeks of Motor City Mapping focused on surveying properties located within six areas that the city of Detroit identified for disbursement of the $52 million in Hardest-Hit (HHF) funds. These funds are required to be used for targeted demolitions in stronger-market areas, and the city faced a short timeframe in which to disburse the awarded resources. Due to these caveats – the emphasis on demolition, and the quick turnaround required by the conditions under which the grants were awarded – some groups expressed concerns that the HHF deployment would result in demolitions of structures with considerable historic significance. The Michigan State Historic Preservation Office received determination from the U.S. Department of Treasury that the HHF program was not eligible for preservation oversight, accentuating these reservations.

In response to these worries, the Michigan State Housing Development Authority – the state department charged with disbursing the HHF dollars – exempted designated historic districts from demolition efforts. However, eligible but not yet designated historic districts did not receive this protection. Concerned about the potential loss of historic structures in these areas, the Michigan Historic Preservation Network (MHPN) partnered with Preservation Detroit and the Detroit Land Bank Authority to advocate for bringing a preservation perspective to the process of identifying structures for demolition.

Bringing the Survey to Life

Though every organization involved in Motor City Mapping supported the aims of MHPN and Preservation Detroit, the initial requests to add a historic component to the survey faced the same obstacles as other issues that were brought to the team’s attention. By this time, the survey was in full motion, and adding additional questions into the Blexting application was virtually impossible. Not to be dissuaded, Emilie Evans from MHPN and the National Trust for Historic Preservation reached out to Data Driven Detroit to ask if there was any potential way that the Historic Resource Survey could still be completed. In response, D3 offered up use of its Local Data license. This provided MHPN with access to a mobile surveying application similar to the Blexting platform used by the wider Motor City Mapping survey. D3 and MHPN collaborated further, identifying parcels for the survey based on two criteria – location in an eligible or proposed historic district, and location in one of the six designated HHF deployment areas. Once D3 had determined the survey geographies and delivered them to Local Data, MHPN was ready to commence the Historic Resource Survey…

…from a technical standpoint, that is. MHPN still needed surveyors to collect data on each of the nearly 18,000 properties that were located within the targeted areas. A call for volunteers was met with a tremendous response – nearly 55 individuals from dozens of organizations throughout the city. Offering their evenings and weekends to the project, these volunteers used Local Data to survey 17,500 properties across the City of Detroit in only two weeks.

The Results

The Historic Resource Survey answered three questions for each property, focusing on its architectural integrity, how in-keeping it was with neighborhood character, and how well its block remained intact. These questions were then aggregated into an easily-digestible Historic Preservation Score – Very Important, Important, Less Important, or Not Historic – that identified how much of a preservation priority a particular property was for a neighborhood. The survey data makes possible data-driven decisions about both building removal and restoration. Funding sources tied to demolitions, such as HHF, can be focused on properties that are labeled as less important or not historic, while preservation efforts can focus on properties of greater importance – not only measured in their architectural significance, but also by their place as an anchor of some of Detroit’s most vibrant communities.