Why 'Toyota's Killer Firmware' May Not Have Been a Killer After All

As he delved deeper into the evidence, David Cummings says he did not find a convincing argument that Toyota's software was to blame for the accident.

As some readers may remember, in 2013 an Oklahoma jury found that Toyota's embedded software was to blame for unintended acceleration that resulted in a fatal accident. The embedded software expert who convinced the jury that software was to blame made his trial testimony and slides available to the public and invited us to "judge for ourselves." A flurry of articles in technical publications followed, most of which (to the best of my recollection) described the expert's testimony in favorable terms, accepting the conclusion that software caused the accident. Many of these articles had attention-grabbing titles such as Toyota's killer firmware and Toyota Case: Single Bit Flip That Killed.

Having developed embedded software for many years, and having written an opinion piece in the Los Angeles Times years before the trial about the possibility that Toyota's software was to blame for reported incidents of unintended acceleration, I read the expert's testimony with great interest and excitement. I was expecting to find a convincing argument based on the evidence that the software was indeed to blame for the accident in question.

As I delved deeper and deeper into the testimony, however, my excitement turned to disappointment. It became clear that there was no credible theory based on the evidence. I felt it was important to set the record straight, and so I submitted an article with my technical analysis to the IEEE Technology and Society Magazine. That article was peer reviewed and then accepted for publication in their most recent issue (December, 2016). (You can also find a pre-publication version of the article on my company's website, as well as two video interviews here and here providing additional context.)

In this column, I summarize some of my findings (please refer to the IEEE article for a more complete discussion that includes all the technical details).

As discussed in the IEEE article, the plaintiffs convinced the jury that Toyota's embedded software was responsible for the accident by employing the following approach:

First, they bombarded the non-technical jury with criticisms of the quality of Toyota's software from two different software experts. The first expert did not see any of Toyota's source code, but nonetheless his entire testimony, which is also publicly available (see Part 1 and Part 2), was directed toward criticizing the quality of Toyota's software. As anyone with extensive experience developing real-world software knows, software quality assessments can be highly subjective.

The second of the two experts, who did examine Toyota's source code, also criticized the quality of Toyota's software. Then he told the jury that "to a reasonable degree of engineering certainty, it was more likely than not" that the death of a task running on the engine control processor (referred to at trial as "Task X") was responsible for the accident, despite the fact that the evidence presented at trial did not support that conclusion.

The title of this article asserts that the "Toyota's Firmware Was Not a Killer". This is a fundamentally faulty logic, because the author cannot prove that the firmware didn't, in fact, kill people.

Also, as I remember correctly, the experts working for the plaintiff did offer a theory that the fault happened due to a stack overflow, which corrupted the operating system variables that happened to be conspicuously just below the stack. The experts further showed that Toyota's stack use analysis was faulty and the stack could overflow. Finally, they also showed that once the bit corresponding to TaskX was flipped in a real vehicle, all other fail-safes didn't prevent the unintended acceleration.

But this article raises an interesting issue of how much proof is sufficient to implicate complex firmware like this. Please note that error rate of just one per many millions of hours of operation is sufficient to reproduce the observed fatal accident rate. Yet, the author demands that expert witnesses reproduce such incredibly intermittent event at will, effectively performing debugging of the system for the car manufacturer. All this while receiving no support or very reluctant support from the actual developers of the firmware.

So, the question is: How bad is bad enough for a complex firmware to be?

I owned a Toyota when these incidents (there were several of them) were in the news. Regardless of any presumed computer failure, the drivers involved must have been pretty dumb. How hard is it to turn the ignition key? Toyotas (and most other cars) require that the key be pushed in to be moved from "accessory" to "lock", so there is little likelyhood of the steering wheel becoming locked. Moreover, power steering remains effective as long as the engine is turning over because it's in gear. Those too flummoxed to think of turning off the engine might at least shift into neutral.

The whole episode was shameful, but not necessarily the fault of the driver. Those who still maintain that a car can be braked to a stop from high speed with the engine at wide open throttle are deluding themselves. Try it sometime, hold the accelerator fully open while traveling at 70 MPH and step on the brake pedal. Do it sporadically as you might do when the car is acting strangely and will not stop. The brakes will fade pretty fast and although you may slow, soon the engine will overcome the fading brakes. You absolutely will not be able to slow the car as quickly as if the engine were not at full throttle.

The "simulations" that may have been done to prove that the brakes would work properly are just that, simulations, and are only as good as the assumptions. For a good idea of the usefulness of simulations in real world emergencies, go watch the movie "Sully" and pay attention to the section where they talk about the simulations showing he should have been able to land the plane on one of two alternate airports instead of the Hudson River.

A failure analyst that had reoccurring problems with his Toyota fly-by-wire accelerator did some failure analysis and found tin (Sn) dendrite growth between the pins of the IC on the pedal that could tell the computer that the pedal was fully depressed when it was not depressed. A summary of this was published in EE Times, "Toyota accelerations revisited-hanigng on by a (tin) whisker -2012-01-10"

The push-button start-stop switch requires the driver to depress the button for 3 seconds continuously to turn the engine off when it is in gear. At 70 MPH a car will cover over 300 feet in the time required to turn off the engine. That is a pretty scary scenario, expecting someone to hold that button down continuously for the time to cover the length of a football field in a panic situation. I know that I likely would not do it.

Toyota and other auto manufacturers are paying attention to this belated wakeup call to improve the software, firmware, and hardware of their cars. Autonomous vehicles will suffer the same challenges, no engineer or group of engineers, no matter how smart or wise, can anticipate all possible situations.

This whole episode was shameful. I don't think you pointed it out in the article, but even if the software had left the throttle open, it's been demonstrated that the brakes can still stop the car with no trouble. Apparently Toyota thought it was easier to settle and move on than to try to argue to juries that people had killed their families by confusing the gas and brake pedals.