Toyota Case: Inside Camryís Electronic Control Module

MADISON, Wis. -- As we continue to explore the Oklahoma court judgment against Toyota for unintended acceleration, EE Times readers have raised many astute engineering questions, ranging from the probabilities of bit-flip occurrence and safety standards applied to software and hardware to the safety system architecture built into Toyota cars.

Meanwhile, in a number of Toyota cases (including the Oklahoma one), one nagging question recurs among consumers: If there are software bugs in the system, why have millions of Toyota owners like ourselves never experienced unintended acceleration?

It turns out that Jean Bookout, the plaintiff in Bookout v. Toyota case, had driven her Toyota for several years and put 9,000 miles on the odometer without a problem -- until the crash that injured her and killed a passenger.

We would like to take you through how Michael Barr, CTO and co-founder of the Barr Group and an expert witness who testified in the case, concluded that a random hardware flaw -- combined with a software bug that's latent and lurking -- "can get through or knock down the fail-safes that are in place" under certain driving conditions on certain days.

Excerpt from the court transcript
EE Times is publishing a portion of the court transcript relevant to the Toyota Camry's electronic control module ECM. The following Q&A was carried out between Barr and Benjamin E. Baker Jr., an attorney representing the plaintiffs. This excerpt begins with Barr on the witness stand describing the ECM, which consists of two CPUs: a V850 supplied by NEC (which later became Renesas) and an ESP-B2 supplied by Denso acting as a second CPU (sometimes referred as a monitor CPU).

A. So this is a photograph of the ECM. And this ECM, or engine control modules, has two big chips on it. Has a bunch of other chips, capacitors, circuit tracers that you can see, and other things. This biggest one, the square one, is the main CPU. It is a type of a CPU or a model of CPU called a V850. That is kind of the equivalent of calling it a Pentium. V850 is the model number of that processor. Comes from a company, a supplier of Toyota that used to be called NEC. It has since changed its name.

Then there is a second rectangular chip here, and that chip is what has been referred to by various witnesses as the monitor CPU, the ESP-B2, and sometimes the sub-CPU.

Importantly, each of those is a processor with its own software. Then, of course, all together they comprise an embedded system.

Q. So the software that we're going to talk about is stored within components on this board?

A. Almost always when I'm talking about the software, I'm talking about the software on this main CPU, which performs the throttle control, the combustion, monitors the accelerator, and all those things, cruise control. But there is also software, and I will specifically call out when I'm talking about this monitor CPU and its software.

Q. This is from a 2008 Camry?

A. This particular photo is from 2008 Camry.

Q. Is the 2005 generally very similar to this?

A. The chips would be moved around a little bit, but in terms of the electronics of what is there, there is a V850 processor, there is an ESP-B2. From a substantial similarity point of view, they are very similar.

No. The brake applies the brake. In virtually every car, the transmission is a mechanical linkage, and it is still engaged in "Drive" when you are pressing the brake.

Assuming you don't apply throttle and brake at the same time (by driving 2 footed - eek!!) then the engine would be resisting the wheels by having a closed throttle.

That was the entire point of my initial post - if the engine is going crazy, for anyone of a million reasons, just push the gear selector forward to neutral, slow to a stop and shut the car off. **(you'll notice that you can just PUSH this lever forward and it stops in neutral for JUST this reason. It won't go into reverse without pressing the button so you can just push it forward HARD in an emergency and it stops at neutral!)

Engineers worked this detail out 50+ years ago to prevent just this thing. Unfortunatley, as the cars get smarter, the drivers get... worse.

There is no excuse for a lack of serious double and triple+ redundancies in life-critical safety systems. But there will ALWAYS be a condition that was untested that will require the driver to fall back on basic common sense.

"Not sure why so many people believe that driving a car is the most difficult thing imaginable."

Think. You probably avoided several accidents just this week by using your brain in ways a computer never can. Here are a couple of instances just from my driving last week:

(1) I see a dumptruck on the cross-street approaching the intersection where we will collide if they don't stop. I notice that the back of the truck is heaping overful with dirt. I also notice that the truck is REALLY old. I also hear the truck brakes making a not so good sound. I also see the driver's face, which is very alarmed. So I stop even though I have the green. A computer would not have noticed ANY of those details and I would now be dead.

(2) I get on 694 where there is road construction, narrowing the 4 lanes down to 2. The old lady in front of me gets confused by this and barrels on and scatters orange cones all over and she drives about 500 feet into the contruction zone before her front wheels shear off in a 2-foot deep hole cut into the concrete. Now why would my computer-driven car not follow her?

(3) It's the time of the month when folks are apartment-moving. Two guys have strapped a box spring to the roof of their car, with what looks like fishing line. I back off and change lanes. 20 seconds later, the box spring and matress go flying. A computer would not have noticed nor concluded that things were likely o go wrong and I might not be dead, but have had a comical driving experience with a mattress on the windshield. Good times!

(4) I'm driving in a part of town with lots of drinking establishments, at 2AM. A part of town where there are a lot of 1-way streets, merging at 45-degree angles, sometimes with 3 lanes across, narrowing to two between intersections. I see drivers making U-turns and barely making it. I slow down and use a lot of extra caution. Aha, here's a guy going the wrong way on a 1-way street. A computer would not make these value judgements and might get me into trouble.

(5) A plastic bag floats across the road ahead of me. I can tell by knowing the wind speed and the way the bag is moving, that it's an empty bag. To a computer's radar or laser or ultrasonic sensors, it would look just the same size as something to be stopped for- like a cement block or a baby. There is no way for a computer to judge, oh, yeah, that's an empty #2 bag from Target. A computer would have to decide whether to proceed over the obstacle or slam on the brakes. Slamming on the brakes during rush hour would get me badly rear-ended, all due to a plastic bag on the road.

(6) Again on 694, an aluminum step-ladder fell off a truck. It happened to land directly on the lane-dividing dashed lines and only about 5 degrees canted. I was able to identify the object, its position, and decide to proceed without swerving, all in under a second. I doubt of a computer could do anything other than sense "small object, do something?"

(7) Now on second thought, that ladder was mostly orange, could that have been a mostly-fiberglass ladder? Would car radar have picked up anything? How do ultrasonics bounce off fiberglass? What if the ladder had been fiberglass, had landed on its side, not exposing much reflective area? Hmmmmm......

See, lots of situations we see and handle every week. Sitiations where a computer would not have a clue what was hapennig or what to do.

A bit over two years ago I was in a case of the euphemistically called "unintended acceleration" (which they should term "Runaway Killer Car") and was lucky to walk away, with my family (who were in the same car) with minor concussions and scrapes, rather than the four funerals which looked likely, mostly by luck.

The Driver was astute enough to keep the car on the winding downslope road (rather than falling off 50+meter downon the outside of the road, where the car will pull to when accelerating) and to only mutilate raised flower beds, avoiding the concrete guards blockhouse at the bottom of the slope by a few inch at the most.

When the car, which had acelerated downhill all the while hit the flowerbeds, it became a ballistic missile, airborne at almost a mans height at the top of the Arc (at least that is what the guard in blockhouse claimed) and came down a fair distance on and slid onto a road, normally full of tucks carrying containers barreling down at (or above) the legal speed limit. It was empty when the car ended up there. Thank someone for that.

The Airbags deployed when the car re-engaged somewhat violently with Terra Firma. Lucky we all wore seatbelts and that at least someone in the Car had a guardian angle who was not on the phone or washing his/her hairor having a tall latte at Starbucks.

Now I drive (though not that day, perhaps for the best) and I design electronics and program and I have a few degrees. So I do know what we are talking about here. No system is foll proof (foolsare ingenious).

What I will not forget was the drivers face when the whole Fit hit the proverbial Shan.

He was just a real-estate agent picking us up to have a look at a place we were interested in. The car was on the company. And then it simply ran away and tried to kill the whole ruddy lot of us. He just had no control of the engine and the brakes seemed not to work either. He frantically operated the steering to keepus somewhere on the road, not off it.

There was no driver error, no buched up floor mats, the Car had just come back from it's first inspection (as we found out) and was almost brandnew (still had the new car smell). As no-one was seriusly hurt, Toyota just setteled for a new car and the guy still works at the agent (but refuses to get into the Toyota). Since then I have refused traveling in Toyota Car's. No point tempting the fates twiceor more.

Seeing the evidence presented in this case, I must say that I am totally appalled how cavalier the Toyota design team handled safety critical design and programming.

Not only did they not implement hardware safeties in truly critical areas, which any systems engineer will tell you is not just good practice but practically mandatory, their software design invites cascading failures on top of all of this!

I can only say that I sincerely hope that Toyota will be forced by regulators and governments around the world to fix ALL the cars they have sold that have this bad design AND that with those costs and further suits they will be forced into bankruptcy over this and never make cars again, as a salutary example to other car makers everywhere to take the health and lives of their customers more serious!

"The CAUSE was a driver who was not equipped with the knowledge, experience and ability to safely operate their car."

It could equally be argued that somewhere in all of this was a car company seemingly neither equipped with the knowledge, experience and ability to design electronic functional safety into their cars ( so that drivers would be able safely to operate them) nor blessed with the foresight and moral courage to warn drivers what to do in the event of a sudden acceleration.

"The complexity is in the decisions not in the speed. The enviroment of an engine control system is stable, i.e. from day to day it does not vary, but it does occur at high speed needing milli-second decisions to meet modern enviromental requirements."

That's just it. If it were stable as you say, the control could be managed manually. But it's far from stable, and the variables it needs to adjust to are way too many for a human to keep track of.

Think of the combination of ambient temperature, air pressure, engine temperature, power demand, engine revs, intake vacuum, impending detonation, oxygen in the exhaust, and on and on. The engine control system is managing a host of variables constantly, very much the same sort of thing a driver has to do. Only the engine management system does this consistently, accurately, without being distracted by conversations, texting, mental fog, panic attacks, drug abuse, or just plain stupidity.

@junko.yoshia, I can appreciate that the driver tried to stop the vehicle, but they simply didn't understand the operation of the vehicle they were responsible for piloting on public roads.

The bottom line is that they left the transmission engaged, keeping the drivetrain connected to an engine that was supposedly revving out of control. Simply shifting to neutral would have removed the out of control engine from the equation.

I would like to remind the readers that the brake handle (or extra foot lever in a minivan/suv) is a PARKING brake, not an EMERGENCY brake. It engages the rear brakes only via a simple mechanical linkage and is capable of producing maybe 5 or 10% of the braking the car is capable of with the brake pedal. In a true emergency, one is much better off shifting to low range gears on the transmission to slow the car instead of just applying the parking brake.

The malfunction of the car was a contributing factor to the accident, not the CAUSE of the accident. The CAUSE was a driver who was not equipped with the knowledge, experience and ability to safely operate their car.

@Bert22306. The complexity is in the decisions not in the speed. The enviroment of an engine control system is stable, i.e. from day to day it does not vary, but it does occur at high speed needing milli-second decisions to meet modern enviromental requirements. Self driving requirements are slower, fractional second, but filled with data, i.e. is that a vehicle, will it cross my path as I am travelling now?, will it cross my path on my intended change of path?, will a second, third, fourth vehicle cross his or my path and cause an incident? before you even make a decision to continue as currently or change your speed or direction.

It's interesting how MTBFs always sound better than failure probabilities. 10's of Kyears gives more of a warm fuzzy feeling than 10 to the minus some number.

If you've ever dealt with resynchronizers in digital logic when crossing an asynchronous boundary, the probabilities get very low very quickly -- and the MTBFs grow exponentially. It doesn't take very many flip flops to ensure that the MTBF of a resynchronizer failure (a metastable condition) is greater than the known age of the universe.

If only we could achieve those kinds of MTBFs in everything, then everyone would sleep a lot more soundly :)

"Even when the probabilities are extremely low, the enormous number of opportunities (3 trillion miles driver per year just in N. America) means that someday, somewhere, a failure might manifest itself."

True. In my unintended valve cycling case, we were sending commands, to many such valves, each set of valves on many different platforms, at something like 10 Hz. That's 24 hours a day, 365 days a year. So it's also a case of many, many opportunities for a failure.

With the valve controllers in question, which operated in such a way that a single discrete "open" command would cause the valve to cycle full open before the series of corrected "closed" commands could close it again, the initial error checking resulted in one likely fault every two weeks, per platform. Verified in real life. So my initial goal was to make the MTBF 35 years, the life of the platform, as an initial goal. And then we got something closer to 10s of Kyears, when the new error checking technique was finalized.

In this Toyota case, there is the advantage that a human driver is there to take over. It looks in the last testimony like the brakes actually did work throughout the problem period. What did not work is that braking didn't cut power. This is already a better situation than what I had thought at first.