ANN ARBOR—Researchers, hospitals, companies, consumers and government agencies are drowning in data that they can't fully capitalize on.

Now, a team from the University of Michigan has received $1.6 million from the Defense Advanced Research Projects Agency to help develop a toolkit so that even nondata scientists can use that data to possibly answer questions and ultimately speed up the process of discovery.

The Michigan project seeks to develop algorithms that draw on techniques like machine learning for applications such as understanding video. It is one of 24 projects selected from around the country. DARPA intends to combine techniques from the projects into a central repository.

"This is a visionary idea to help create a system that is intelligent about how it selects which algorithms to apply to a specific data set," Laura Balzano, an investigator on the project and U-M assistant professor of electrical engineering and computer science, said of DARPA's overarching project, known as Data-Driven Discovery of Models.

While the Data-Driven Discovery of Models system is expected to be able to tackle nearly any question that can be answered with a data set, one of the early problems it will address is agricultural. Given data from a farm covered in cameras and other sensors, can it predict how to optimize plant growth while limiting resources such as water and pesticides?

The Michigan project is called SPIDER: Subspace Primitives that are Interpretable and DivERse. The team, led by Jason Corso, U-M associate professor of electrical engineering and computer science, is developing new techniques to extract meaning from different types of data sets. Growing out of Corso's and Balzano's expertise in image processing and computer vision, the project will focus on breaking down data that has, in theory, a huge capacity for variation by identifying the features that are much less variable.

For instance, a 128-by-128 image of a face contains 16,384 pixels, but the pixels don't vary independently from one another. In fact, the expected variations can be described by about 10 or 20 dimensions, said Corso—down from 16,384 assuming that each pixel is independent of the others. By looking at "subspaces" like this, he and Balzano simplify the problem of interpreting images and other arrays of data.

SPIDER's contribution

It is difficult for computers to make sense of data that comes in multiple, connected flavors. For instance, video is more than just moving pictures. It's also audio, which is crucial to understanding the context. Corso has developed a technique that converts video into text. The text is not mere captioning—rather, the system identifies what's happening in the video and provides a written summary.

Already, the team has developed algorithms that analyze videos of car crashes uploaded to YouTube—deducing information such as how fast the cars were going, rates of deceleration, and forces at play—to create a data set that could one day train autonomous vehicles to plan evasive maneuvers for crash prevention.

Computer vision also stumbles when images or actions differ from what it has learned. One of Corso's new projects is looking at video from body cameras, trying to understand what goes right and wrong when police and citizens interact. His team wants to identify signs of agitation, indicating that the tension is escalating. Speculating, Corso suggested that agitated people might wave their arms, but different people make these gestures at different speeds. The algorithm needs to recognize the action of arm-waving, ignoring the rate of the movement.

Finally, the Michigan team aims to develop new "clustering" techniques to segment large data sets into meaningful categories. Balzano has used clustering for face recognition, looking at how one pixel relates to others in the image. This technique can identify a face in bright or shadowed conditions, or even when part of it is hidden.

Balzano's algorithms take advice from humans, similar to photo-organizing software. The human is asked to compare two faces to determine if they are the same, and this information goes into the system. Balzano's group enabled the computer to correctly find all matches after recording 100 human comparisons, down from 1,000.

DARPA's overarching project

The techniques that the DARPA-sponsored teams develop will go into a central repository, open to researchers. This repository will assemble these algorithms into models that use vastly different types of data sets to make predictions and draw conclusions. Already, a few thousand methods from currently available software systems are being added to the repository.

"You always need new algorithms and new ways of modeling data," Balzano said. "This project puts them into a system."

Once all of the algorithms are in the database, an automated software system will try them out to see which work best on a given problem. Then, they will use machine learning techniques to assemble them into models that can propose solutions, such as whether two faces are the same or not.

Finally, humans will be called in to interact with these assembling systems to provide insight and hints about the information in order to get the best outcomes.

"The insights of the human expert in the loop are invaluable," Corso said. "Humans have a great way of building bridges automatically."