Now that the Senate has confirmed Ben Carson to lead the U.S. Department of Housing and Urban Development, it is worth considering one question that must be on Carson’s mind: Is HUD working?

There can be no question that HUD has enormous impact. With an annual budget of more than US$30 billion, it provides financial assistance or direct housing to more than four million Americans. We’d argue that HUD works well in helping low-income households find places to live. A HUD subsidy helps these Americans make ends meet and provides a lifeline that prevents people from homelessness.

But what kind of impact does HUD make on American cities? HUD’s budget also includes roughly $4 billion a year devoted to cities, largely through the Community Development Block Grant (CDBG) program. Local governments and community organizations use these funds for emergency repairs, street work, parks and infrastructure.

Carson may want to consider whether this money is being used as it is intended. In the past several decades, HUD has handed over much of the responsibility for the identification of urban problems to states and cities. Credit and blame for whether those investments worked to solve those problems also fall to states and cities. Evaluating HUD’s success in this arena then becomes quite a challenge and requires looking at more than just federal investments, but also at the ways those investments have been directed on the ground by local and state agencies.

What we found is that listening to the social media posts of millions of urban residents can help weave a nuanced story of how this federal agency of 8,400 employees really works – or doesn’t.

Urban social listening

We began by listening to the “digital crumbs” generated by our collective internet searches and postings to social media like Twitter. We developed an approach called “urban social listening,” which offers a systematic, rigorous collection and analysis of these “crumbs.” This kind of careful listening can offer useful insights into understanding cities and the government’s role in addressing urban problems.

Research shows a solid correlation between positive sentiment and health indicators and outcomes. One study of every county in America showed that Twitter sentiment was a better predictor of cardiovascular disease than any of the conventional countywide indicators like smoking or socioeconomic characteristics. Our own work in New York City found that neighborhoods with the lowest sentiment (as expressed by the use of negative words like “sad,” “angry” and “upset”) also had the highest diabetes rates, self-reported rates of high anxiety and overall poor mental health. While public health researchers have understood these relationships for decades, the availability of social media data from areas like neighborhoods, or even smaller blocks, can help cities address and manage problems and evaluate interventions.

Low cost data dump

Digital crumbs are an attractive data source for the student of cities. They offer unparalleled volumes of data at a relatively low cost. Using millions of tweets from cities across the U.S., we have been creating rankings of the general attitudes and opinions expressed by the people who live in each city. We used a lexicon-based analysis of all words containing some form of sentiment.

For instance, the word “elated” generated a positive score, while the word “disgruntled” generated a negative one.

We used a lexicon developed by data mining researcher Finn Årup Nielsen at the Technical University of Denmark that includes more than 3,000 words precoded for sentiment on a scale of -5 to +5. After totaling up all scores, we ranked the cities according to the degree to which the people in each city were expressing positive or negative sentiments.

One obvious limitation is that such an analysis relies on Twitter users. And Twitter users tend to be younger, more highly educated, urban and have higher earnings than the general public. Twitter users are also slightly more likely to be male and are disproportionately comprised of African-Americans and Hispanics. The data we collected will likely overrepresent the opinions of members of those groups.

We selected a sample of 65 medium-sized cities in the Northeast and Midwest U.S. that met certain characteristics, including having populations between 30,000 and 250,000 people. We found that people in these cities are, generally speaking, expressing positive sentiment on Twitter. For the average city in this group, the ratio of positive tweet scores to negative was more than three to one.

The highest score was 11 in Minnetonka, Minnesota. The lowest was three in Reading, Pennsylvania. That means that in Minnetonka’s Twitter posts, we found a ratio of 11 positive words to every negative one. In Reading, there were three positive words for every negative one. While imperfect, analyzing sentiment words provides a window into a city’s overall happiness or quality of life. Although such an approach has limits, it’s certainly no worse than other more conventional indicators we tend to use like income or wealth.

Any nudge from HUD?

Turning back to HUD, we next asked how might we measure the impact of HUD’s programs on these cities? Most of HUD’s Community Development Block Grant spending goes to cities with the poorest, youngest, least educated and employed residents. Examples include Altoona, Pennsylvania and Elmira, New York. Using total CDBG dollars spent in each city, we looked at what impact that spending has on people’s sentiment online. In other words, do residents of cities where HUD spends more money send more happy tweets?

The answer is not so much.

Twitter sentiment is least positive in the highest poverty cities. In this limited data analysis, we didn’t see an effect of CDBG funding levels on people’s mood. In future research, we could get a more sophisticated understanding of that relationship with better longitudinal data and more elaborate statistical analysis.

In Carson’s confirmation hearing to lead HUD, he said “I believe we need to ensure that the help we provide families is efficient and effective.”

One way for Carson’s new HUD to monitor how its funding impacts the lives of people is to do urban social listening. This kind of receptiveness would be a good start for Carson’s plans to put the billions of dollars spent on HUD to better use.