The Numbers Don’t Lie: Self-Driving Cars Are Getting Good

Share

The Numbers Don’t Lie: Self-Driving Cars Are Getting Good

Google

It's report card time for the automakers and Silicon Valley denizens studying the tricky problem of making cars drive themselves, and everyone is passing.

The California DMV just released its annual slate of "disengagement reports," documents provided by the 11 companies that received state permits to test autonomous vehicles by the end of 2015. The results, summarized below, reveal how often humans had to wrest control away from the computer, and why (sort of).

Although the reports are an imperfect measure of how the technology performs, they do reveal rapid progress toward the day when you are no longer needed behind the wheel. Google and General Motors are leading the class with cars capable of driving hundreds of miles at a stretch without trouble. But even those who don't make the honor roll show impressive gains. Nissan's robocars, for example, needed human intervention once every 247 miles, compared to once every 14 miles in 2015.

The reports, which cover December 2015 to November 2016, leave a lot to be desired—more on that in a moment—but do offer interesting insights. Google's program, now called Waymo and gearing up for commercial applications, continues outpacing the competition. The company's cars drove 636,000 miles with just 124 disengagements, a 19 percent drop from 2015. Its fleet logged nearly all those miles on the quiet, suburban streets of Mountain View and its environs, and most of the interventions followed hardware or software discrepancies, when, say, the car's lidar and camera reported slightly different data.

Cruise, the startup leading GM's autonomous driving efforts, did all its testing in San Francisco, where it ramped up from five miles in June 2015, to 400 in June 2016. By late last year, it was clocking hundreds of miles without a hitch.

Most of Delphi's trouble came while changing lanes in heavy traffic. Ford's two autonomous cars in California only drive on the highway, during the day, with fine weather and road conditions, which explains why it only needed human help three times in 590 miles. (It has a larger test fleet in Michigan, which doesn't require any reporting.) The little testing Tesla reported (just 550 miles) was part of its preparation to launch a revamped Autopilot system. It gets most of its data from its customers, driving in the real world.

Like the state's requirement that companies publicly report any crashes involving their robo-rides, the point of these reports is to create accountability for the new technology. In a field reliant upon complex software, artificial intelligence, and sophisticated hardware, disengagements provide a metric the average person can understand. They reveal how often the car screws up so badly that the human inside had to take over.

"Disengagements are not a scientific measure of the complexity and operating characteristics of these vehicles," says Bryan Reimer, who studies autonomous driving at MIT. "They're just one very interesting data point."

What makes them interesting is they provide a picture of how each autonomous program is developing. Companies fess up to how many miles they've covered on public roads, and a careful reader can extrapolate general info about where each company is testing, and in what conditions.

They're unscientific because each disengagement involves all sorts of variables, which the reports log inconsistently. They don't reveal the impact of weather, or where exactly these "problems" occurred. They don't note if the cars are following detailed maps, or exploring an area for the first time. They don't account for the proclivities of human operators, who likely have different thresholds for when they'll take over. (Blame California law, which requires that the report include certain details but doesn't specify how they should be presented, or in what context.)

Automation experts warn that these shortcomings, and the fact individual reports offer a unique blend of information, means you can't fairly use them to compare one company's progress against another. A spike in disengagements might indicate a major problem—or reveal a company is experimenting in more challenging situations.

Disengagements are not a scientific measure of the complexity and operating characteristics of these vehicles.

Bryan Reimer, MIT

These reports would be more helpful if they were more rigorous, says Raj Rajkumar, who studies autonomous technology at Carnegie Mellon University. Qualitative comments would add context. For example, an automaker could explain its engagements are up because it is exploring city driving for the first time. Even consistent formatting and defined terms would help. The 11 reports come in 11 formats, and teem with vague language like "technology evaluation management" (Mercedes-Benz), "follower output invalid" (Tesla), and "planned test to address planning" (Cruise/General Motors).

However imperfect, the reports are better than nothing, especially because other states with self-driving regulations—including Nevada, Michigan, and Florida—don't demand any public disclosures. (That's why no one knows much about Uber's testing.) California's law may be flawed, but it keeps the public, and public officials, in the loop, and should help build trust as these systems move toward commercial use.

Plus, these are early days, says Huei Peng, who directs the University of Michigan's Mobility Transformation Center. California could update its requirements, and other states following its lead could improve on what it has started. Until then, here's a look at the hard squidgy numbers.

[google_table url="https://docs.google.com/spreadsheets/d/1nX_qdFmclPDUApSKoz7jV7NVlG7suvCpfngRHPYCVXQ/pubhtml" title="California Autonomous Testing Disengagements"]While experts say these reports should be taken with a shaker of salt, they do reveal interesting information. N/A indicates an automaker did not provide any relevant information, because it did no testing during the relevant time period.[/google_table]

Related Video

Auto

Meet the Blind Man Who Convinced Google Its Self-Driving Car Is Finally Ready

Google is getting serious about self-driving cars. So serious that it put a legally blind man in one that drove him around safely on his own. The successful trip means that the tech giant can now launch its own self-driving car company, which it's calling Waymo.