The traditional method used by social workers to help the homeless is a conveyor belt model, which moves people methodically from street, to shelter, to permanent housing. Few service providers ever questioned this basic model. Yet any experienced social worker would concede that there is a so-called service-resistant population: homeless individuals who refuse shelters. Faced with this reality, in 2001 a nonprofit organization called Common Ground (now known as Breaking Ground) conducted an experiment: a winter count of the homeless individuals in New York City’s Times Square. The Common Ground team went out to the field. They suspended judgment, postponed analysis, and drew conclusions based solely on up-close observations and careful listening.

This idea of a human-centric design — developing solutions through intensive observation and gathering of small data, those tiny clues of human behavior — is especially profound today. The rise of smart machines often leads observers to paint an apocalyptic outlook for the traditional workforce. At their most extreme, predictions foretell the majority of white-collar jobs being automated, leading to a jobless future. What often gets left out of this narrative is that even as artificial intelligence (AI) and other emerging technologies are transforming the way routine tasks are done, they are also freeing humans from a sort of white-collar drudgery. This in turn allows people to do what they are best at: creative problem solving.

In the midst of predictive analytics and machine learning, with big data sweeping across sectors and industries, the importance of small data cannot be overstated. Big data and machine learning concern themselves with correlation, not causation. Computers excel at precision, rigor, and consistency, but they are not designed to integrate social interactions across domains, or traverse data online and offline, or come up with a working hypothesis that explains human behaviors. In other words, AI will augment professionals, offering them expertise and assistance. But it still takes creativity to ask the right question, the question that guides one toward a desired outcome — whether it’s improved corporate performance or sheltering society’s most vulnerable on a cold winter night.

Understanding through Empathy

With sharp facial features and shoulder-length blond hair, Rosanne Haggerty, the founder of Common Ground, speaks like a high-powered business executive. She is polite, measured, and assertive, carefully choosing her words — not the gregarious social activist one might expect. She was no stranger to the problem of homelessness, having developed dozens of affordable housing units in New York City herself in the 1990s. But she grew increasingly restless about the perpetual homeless population outside her housing units.

Computers excel at precision and consistency, but they are not designed to come up with a working hypothesis that explains human behaviors.

That predicament was what led her team to canvass the Times Square neighborhood block by block beginning in January 2001, noting each homeless individual they encountered. At 5 a.m. every day for four consecutive weeks, Haggerty and her team canvassed the streets and created a roster of names with individual photos of all those who slept within the 20 blocks surrounding Times Square. Of the 55 individuals the team counted there that winter, only 18 slept on the street regularly. The others were there intermittently and were not consistently homeless. In other words, those 18 were the only chronically homeless people in the neighborhood.

It’s hard to overstate the significance of this finding. The term chronic homelessness had not yet entered the public works lexicon at the time. And most service providers were treating every homeless individual very much the same way. The team also learned that many homeless people were averse to shelters. They hated them. Featuring dank halls, the odor of urine, alcohol, drugs, and strangers in various states of coherence, shelters offered little respite from the dangers of the street. On the street, however, a community of people knew where to find food, where to get a weekly shower, and how to eke out a living.

Many in the homeless state were further trapped because of existing bureaucracies. To qualify for subsidized housing, for instance, one had to produce copies of one’s birth certificate, evidence of income, a credit standing document, or proof of sobriety for six months — a near-impossible feat for anyone who had been living on the streets for years. In fact, the 18 individuals Common Ground identified had been living on the street for an astonishing 14 years on average. They were known by practically every Times Square business operator, police officer, and service agency. They all had drug, alcohol, and medical problems, and had been contacted by at least one outreach team previously. Yet no one wanted to handle, or had so far successfully handled, this most difficult cohort.

Haggerty speculated that targeting these consistently homeless individuals and giving them exactly what they wanted — housing, not shelters — could persuade others who had been on the street for shorter periods to also seek help, thereby changing the neighborhood from the ground up. Her team focused on finding new ways to work through the existing system, navigating the bureaucratic labyrinth. They streamlined the application process whenever possible and fought against many rules and regulations that often discriminated against the very people they should have helped.

One of Haggerty’s next steps did not sit well with her staff. She mandated that Common Ground’s housing operation become accountable for serving a quota of chronically homeless people. Managers resisted her idea on the grounds that these potential tenants would be likely to miss rent payments, or, worse, vandalize the housing facilities — a nightmarish scenario, as each housing manager was accountable for his or her annual budget. To allay fears and minimize resistance, Haggerty agreed to keep track of the impact as new individuals were steadily accepted into the buildings, and to line up additional resources as a precaution.

None of the managers’ fears materialized after implementation of Haggerty’s initiative. And in fact, the positive effects proved striking. The team saw a 75 percent reduction in homelessness in Times Square over a year in 2006, and then another 50 percent the following year, resulting in a total sustained reduction in homelessness of 87 percent — as well as a 43 percent decline in homelessness in the 20 surrounding blocks.

Codifying Knowledge

By November 2012, what had started as an early morning count in Times Square had turned into a national movement that has since transformed the lives of 100,000 people. The key to this expansion has been Common Ground’s development of a vulnerability index, which embodies expert knowledge and empowers less experienced service providers across the nation. Haggerty and her team figured out a way to disseminate expert judgment across the wider community by codifying their knowledge into an easy-to-follow playbook.

The vulnerability index identifies those at extreme risk of dying on the street by looking at factors such as hospitalizations, ER visits, disease, and history of frostbite, among other metrics. The index is a person-by-person record, creating a sort of health registry for any given community. It’s the FICO score equivalent for homelessness. It no longer takes a well-seasoned social worker with extreme tenacity to identify the chronically homeless who need help the most.

With the support of Common Ground, Los Angeles County and City launched Project 50; Washington, D.C., adopted its Housing First program; and Phoenix implemented Project H3. Collectively, these newly mobilized communities showed that individuals could be moved from the streets into housing in an average of 10 days, demonstrating that what had started in crowded New York City was transferable elsewhere, even to less dense neighborhoods. The index spurred cities and housing bureaus to act quickly for the most vulnerable first.

Haggerty and her team went on to partner with Palantir Technologies, a computer software and services company in Silicon Valley, to develop an online platform that automatically matched the results from the vulnerability index to the available housing in the area. It not only identified the most vulnerable but also automated the once time-consuming housing matching process. That’s not to discount the importance of human observation, which is still critical to populating the index. But now the back-end processes could move more quickly and seamlessly, freeing up communities and experts to innovate.

And that’s just what Haggerty has done. She continued to investigate even thornier issues in more challenging environments, such as the Brownsville neighborhood in Brooklyn, N.Y. Despite having one of the largest concentrations of public housing in the country, Brownsville also had historically high rates of family homelessness. The spin-off organization that Haggerty founded, Community Solutions, created the Brownsville Partnership to help vulnerable people remain stably housed as family units and to improve their lives in the places where they lived. At the time of this writing, more than 1,200 residents have been connected to employers who participated in a job placement campaign.

Creativity Prevails

The story of Rosanne Haggerty and Common Ground is a helpful way to think about interactions among humans and machines. Recent advances in artificial intelligence undoubtedly have made expert systems far more powerful in accelerating knowledge transfer. Still, a human is required to decide where and how AI is deployed, and which key decisions are turned over to AI. Although automation has made the process of turning data into useful information faster than ever, it hasn’t done away with the human task of endowing information with meaning, and acting on that information creatively.

Many companies building AI systems have also found that humans must play an active role in both developing and running these systems. In fact, an investigation by Boston University economist James Bessen has suggested that jobs and automation often grow hand in hand, that automation creates more jobs than it destroys. My own perspective is that the question of whether automation will complement or replace human labor will ultimately come down to managerial choice: Automate as many routine cognitive tasks as possible, and in so doing, enable and empower your people to apply their creativity to the next big problem.

The innate desire to develop solutions, and to answer the question why, was what drove Haggerty when she tried to pry open New York City’s homelessness problem. And that very quality of persisting in the struggle of sense-making about the human condition will be people’s biggest advantage in the age of smart machines.

Author Profile:

Howard Yu is the LEGO Professor of Management and Innovation at IMD Business School in Lausanne, Switzerland.

Follow strategy+business

Articles published in strategy+business do not necessarily represent the views of the member firms of the PwC network. Reviews and mentions of publications, products, or services do not constitute endorsement or recommendation for purchase.

strategy+business is published by certain member firms of the PwC network.