Here's How Tesla Solves A Self-Driving Crash Dilemma

With very rare exceptions, automakers are famously coy about crash dilemmas. They don’t want to answer questions about how their self-driving cars would respond to weird, no-win emergencies. This is understandable, since any answer can be criticized—there’s no obvious solution to a true dilemma, so why play that losing game?

But we can divine how an automaker approaches these hypothetical problems, which tell us something about the normal cases. We can look at patent filings, actual behavior in related situations, and other clues. A recent lawsuit filed against Tesla reveals a critical key to understanding how its autopiloted cars would handle the iconic “trolley problem” in ethics.

Do you remember that day when you lost your mind? You aimed your car at five random people down the road. By the time you realized what you were doing, it was too late to brake. Thankfully, your autonomous car saved their lives by grabbing the wheel from you and swerving to the right. Too bad for the one unlucky person standing on that path, struck and killed by your car. Did your robot car make the right decision?

Either action here can be defended, and no answer will satisfy everyone. By programming the car to retake control and swerve, the automaker is trading a big accident for a smaller accident, and minimizing harm seems very reasonable; more people get to live. But doing nothing and letting the five pedestrians die isn’t totally crazy, either.

By allowing the driver to continue forward, the automaker might fail to prevent that big accident, but it at least has no responsibility for creating an accident, as it would if it swerved into the unlucky person who otherwise would have lived. It may fail to save the five people, but—as many ethicists and lawyers agree—there’s a greater duty not to kill.

1. Ok, what does this have to do with Tesla?

The class-action lawsuit filed in December 2016 was not about trolley problems, but it was about Tesla’s decision to not use its Automatic Emergency Braking (AEB) system when a human driver is pressing on the accelerator pedal. This decision was blamed for preventable accidents, such as driving into concrete walls. From the lawsuit’s complaint:

Tesla equips all its Model X vehicles, and has equipped its Model S vehicles since March 2015, with Automatic Emergency Braking whereby the vehicle computer will use the forward looking camera and the radar sensor to determine the distance from objects in front of the vehicle. When a frontal collision is considered unavoidable, Automatic Emergency Braking is designed to automatically apply the brakes to reduce the severity of the impact. But Tesla has programmed the system to deactivate when it receives instructions from the accelerator pedal to drive full speed into a fixed object. Tesla confirmed that when it stated that Automatic Emergency Braking will operates only when driving between 5 mph (8 km/h) and 85 mph (140 km/h) but that the vehicle will not automatically apply the brakes, or will stop applying the brakes, “in situations where you are taking action to avoid a potential collision. For example:

• You turn the steering wheel sharply.• You press the accelerator pedal.• You press and release the brake pedal.• A vehicle, motorcycle, bicycle, or pedestrian, is no longer detected ahead.”

You can also find these specifications in Tesla’s owner manuals; see page 86 of 187.† What they suggest is that Tesla wants to minimize user annoyance from false positives, as well as to not second-guess the driver’s actions in the middle of an emergency. This makes sense, if we think robots should defer to human judgments.

But isn’t this the point of autonomous cars in the first place: to take humans out of the equation, because we’re such poor drivers?

Law professor Bryant Walker Smith said, “Am I concerned about self-driving cars? Yes. But I'm terrified about today’s drivers.” And Dr. Mark Rosekind, administrator of the US National Highway Traffic Safety Administration (NHTSA), compares the 35,000+ road fatalities in the US every year to a fully loaded 747 plane crash every week.

Science-fiction writer Isaac Asimov might also reject Tesla's design, recognizing that we can't always privilege human judgment. According to his Laws of Robotics:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2. A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

In other words, following human orders should take a backseat to preventing harm to people; and crashing a robot car into a wall can injure the passengers inside, even if there are no people around on the outside.

There’s a mismatch, then, between the safety case for self-driving vehicles—the primary reason why we want them—and Tesla’s design decision to not intervene in a collision that a human is actively causing, such as stepping on the accelerator regardless of who or what is in front of the car, apparently.

To be sure, if the car is in Autopilot or Traffic-Aware Cruise Control mode with no human input, it will (or should) automatically brake for pedestrians, and the AEB system has reportedly saved lives this way already. But this doesn't seem to be the case when the human has control of the car, as Tesla suggests in previous comments:

AEB does not engage when an alternative collision avoidance strategy (e.g., driver steering) remains viable. Instead, when a collision threat is detected, forward collision warning alerts the driver to encourage them to take appropriate evasive action.

Back to our trolley-type crash dilemma, if that's correct, it means that a Tesla car would not retake control of the wheel and swerve away from a group of people (or even brake), if the driver were deliberately driving into them. Again, this isn’t an unreasonable stance, but it is at odds with a goal to minimize deaths and injuries—in tension with the company’s frequent claims of “safety first.”

But it’s really a “damned if you do, damned if you don’t” problem. If the car were programmed instead to retake control and swerve (as it can do), it’d create considerable legal liability for Tesla if this action caused a different accident, even one that’s less serious. This accident wouldn’t have occurred if the car did not retake control; so the company seems causally responsible for it, at least.

And Tesla would have some explaining to do: why would it override human judgment if a pedestrian were in front of the car, but not if a concrete wall were in front of it? Is the driver’s safety less important here, because the pedestrian is more vulnerable to injuries?

Such a design principle isn’t unreasonable either, but it raises further questions about whether a car might ever sacrifice its owner’s safety. That's to say, how exactly is Tesla thinking about the value of different lives: are they all worth the same, or are some prioritized more than others? The answer is not publicly known.

Tesla did not do what no manufacturer has ever done—“develop and implement computer algorithms that would eliminate the danger of full throttle acceleration into fixed objects” even if it is caused by human error…Tesla disputes that there is a legal duty to design a failsafe car.

First, let's note that Tesla already and often does “what no manufacturer has ever done.” The company is clearly an innovator with many industry-first achievements; thus, a lack of precedent can’t be the real reason for its decision to not activate AEB in some situations.

Second, it’s unclear what a “failsafe” car means. If it means a “perfect” car that can avoid all accidents, sure, that’s an impossible standard that no automaker can meet. But if it means “overriding harmful human actions”, then that’s debatable, and we’ll have to see how the lawsuit plays out. The ethics argument might go something like this:

If you have the capacity to prevent something bad from happening—such as an intentional car crash with a wall or people—and you can do this without sacrificing anything important, then you arguably have a moral obligation to intervene. You may also have a legal obligation, such as if you were in a jurisdiction with Good Samaritan laws that require you to rescue or provide aid in perilous situations.

And Tesla has the capacity to intervene. With its AEB and advanced sensors that can detect objects, motorists, and humans, it has the capacity to detect and avoid collisions under normal circumstances. As any comic book fan can tell you, with great power comes great responsibility. Superman may be faulted for not stopping a bullet, but you can't be; and robot cars are superheroes, compared to ordinary cars.

So, if an automated car had to decide between overriding its driver’s actions and crashing into a concrete wall, then this argument suggests that it can and should choose the former. If anything is lost or sacrificed, it’s merely the driver’s autonomy, assuming that action was even intentional; and driving into one’s own house is usually unintentional.

But if the decision were to override its driver’s actions and foreseeably run over one person, or crash into a crowd of people, then it gets tricky. The car could prevent this mass-murder, but not without sacrificing anything important (the one person). In this no-win scenario, either choice could be defended, and none will satisfy everyone.

3. And the point is…?

This article isn't really about Tesla but about the larger autonomous driving industry. Tesla just happens to be the highest profile company here with production cars on the road, so there's more public information about them than others. The other automakers also face similar design choices and therefore dilemmas.

The point here is not that we should halt work on automated cars before solving oddball ethical dilemmas. But it’s to recognize that, by replacing the human driver with its AI driver, automakers are taking on a lot of moral and legal responsibility it never had before, just like someone who gained superpowers. The human driver used to be liable for accidents caused by her or his decisions; now that liability is shifting to the AI driver that's capable of saving lives.

If we think that this industry is as important as advertised, then it needs to be able to defend its design decisions in advance, not after the fact of an accident. Even if there are no perfect answers, had these issues been publicly explained earlier—that it's a feature, not a bug, that AEB doesn't second-guess human decisions, and here’s why—the lawsuit and its publicity may have been avoided.

As explained earlier this week, the fake set-up of these hypothetical crash dilemmas doesn’t matter, just like it doesn't matter that most science experiments would never occur in nature. The insights they generate still tell us a lot about everyday decisions by the automated car, such as how much priority it should give to its occupants versus other road-users. This priority determines how much room the car gives to passing trucks, bicyclists, and pedestrians, among other safety-critical decisions—such as whether it's ok to crash into walls.

Robot cars are coming our way, and that's fine if they can save lives, reduce emissions, and live up to other promises. But, for everyone’s sake, the industry needs to be more transparent about its design principles, so that we can better understand the risks and limits of the new technologies. NHTSA is already steering automakers in this direction by requesting safety assessment letters, which asks about ethical considerations and other things.

That isn't a lot to ask, since we’re sharing the road with these new and unfamiliar machines, entrusted with the lives of family and friends. Otherwise, as they say, expectations are just premeditated resentments; and if industry doesn't properly set expectations by explaining key design choices, they're on the path toward resentment or worse.

~~~

Acknowledgements: This work is supported by the US National Science Foundation, Stanford CARS, and California Polytechnic State University, San Luis Obispo. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the aforementioned organizations.

~~~

† Technical note:

Each one of Tesla's examples seems to be a sufficient, but not necessary, condition for AEB to deactivate. For example, several reports filed with NHTSA describe Tesla crashes into parked vehicles; if they're not cases of malfunction, they show that AEB can deactivate even if a vehicle or other listed object is detected ahead. User tests also suggest AEB will not prevent collisions with pedestrians in some circumstances; again, that's either a malfunction or by design.

The autonomous driving engineers I've contacted believe that AEB's deference to human control, even with a pedestrian in the way, is by design for the reasons discussed in this article. Other possible reasons include not wanting to trap a driver in an unsafe state, if AEB reacted to false positives (which have occurred) and didn't allow for manual override.

Without more elaboration from Tesla, the details of when AEB is supposed to work are still a bit hazy, even confusing its own customers. It could be that AEB initially activates even if the human is in full control, but AEB quickly backs off—that is, it “will stop applying the brakes”—if the driver keeps a foot on the accelerator; and this wouldn't change the analysis above.

Tesla has not yet responded to my request for clarification, and I will update this article if they do. But this may not matter: even if the AEB system is updated to intervene or second-guess a driver's action, ethical questions still arise, such as how it prioritizes the value of different lives, including that of its customers. Again, there's no obvious way to go here, and either choice will need defense; and the need for this technical note underscores the need for more communication by automakers, to reduce the speculation that naturally fills an information void.

I'm the director of the Ethics + Emerging Sciences Group, based at California Polytechnic State University, San Luis Obispo, where I'm a philosophy professor.⋯Other current and past appointments at: World Economic Forum's Global Future Councils; 100 Year Study on AI; Sta...