One has to wonder what the engineers were thinking. Everyone in this industry faces twin pressures: to be fast and to go cheap. Deliver now, and cut engineering costs. I have no insight into how many person-hours went into the Toyota code, nor do I know the delivery schedule. But let’s look at that most recent $1.2B payout. How does that compare to the engineering effort? The NASA report talks about a code base of “more than 280,000 lines.” Mike Barr tells me there were “over a million lines of C source code.” For argument’s sake, let’s figure on a million.

The most expensive code ever written is that of the Space Shuttle, which ran about $1,000/LOC (according to 201 Principles of Software Development by Alan M. Davis). With just the most recent settlement, Toyota’s code cost them over $1,200 per line -- without accounting for any engineering effort. The difference is that the Space Shuttle’s code is the best ever written, averaging about one bug per 400KLOC, and Toyota’s has been intensely litigated. I am not suggesting that Shuttle development practices should be anyone’s goal.

Perhaps a better benchmark is avionics. It’s largely believed that no one has been killed by defective firmware in commercial aircraft, yet that code controls pretty much everything. Sure, the pilots can take over, but modern planes are fly-by-wire. The pilot flies a computer.

What does it cost to develop the fabulous software that mediates billions of passenger-miles per day in the air? Commercial avionics is done to a standard called DO-178B (supplanted recently by DO-178C). Level E applies to software that won’t impact operations in any significant way. Level A is for code that can lead to the loss of the aircraft.

How much does it cost to write code to level A? Who knows? Data is sparse and proprietary. However, most pundits figure it’s about twice the cost of typical commercial firmware. Others ("DO-178B Costs Versus Benefits" by Vance Hilderman), in this case based on data from some 150 avionics programs, claim code written to level A is 65% more expensive than that to level E. That figure includes both the engineering effort and the certification process.

I find it very hard (according to your analysis) that so many issues arise from firmware, when the process is so exhaustive.

Or, they did not follow any process at all. But they are required to, as far as I know. So I will assume a process was followed, but it [miserably] failed.

If this is the case, processes must be revised to minimize these things from happening. Although process is not itself a guarantee of high-quality, it is indeed a requirement for the high-quality to be achieved.

If there is a proper safety critical engineering process in place, do you think it is even possible for Toyota engineers like this Mr. Ikura to be guessing why vehicles are misbehaving on the road, after the fact, and that their guesswork involves vehicle electrical systems?

Here is a quote from a Toyota internal email that is almost surely is in the possession of the DOJ, one among perhaps a hundred similar emails:

"This is Ikura from 2SE-6G.

-Is it possible that the RPMs rise due to radio wave interference? And what level are the European standards?

(Previously, when I was in charge of Hilux in the Japan domestic service division, I experienced an engine stall malfunction due to radio wave interference from a nearby U.S. Naval Base in Yokohama. At that time I was told that it could absolutely never occur.)

→ Frankly, I (2SE) really do not understand this. At the very least, departments concerned with various electrical items must be gotten involved in a discussion."

****************

What could that "discussion" entail?

I also find it astonishing that poor Mr. Ikura was treated to the same mind-bending response from his colleagues--basically, their claim that he did not experience what he experienced--in a weird echo of the treatment reported by many hundreds, even thousands, of Toyota's customers.

*******************

I have drafted a letter to a U.S. Senator to request an oversight review of the DOJ investigation scope. Anyone who wants to sign this letter with me can be in touch via my blog -- betsybenjaminson.blogspot.co.il

Toyota's engineering documents on vehicle development and determining the causes of SUA (most of these were the same docs as the ones turned over to the DOJ) have been reviewed by experts who know those standards very well, and I can say with confidence they saw no evidence that Toyota was following the standards. One of them, a software expert who specializes in safety-critical software development, said that he was "shocked."

It bears noting that much of Toyota's code was written before ISO26262 was published.

Barr's testimony indicates that the MISRA-C standard was more relevant. But Toyota did not follow its rules either.

I saw many references to "Toyota Standards" but also instances where people inside Toyota were attempting to change test protocols or pass/fail criteria for the throttle or ECU (one such attempt came from a throttle supplier, and one such attempt came from Toyota to Denso), and these seemed to be based on cost considerations. "We lack sufficient budget, so can we please just test once instead of the prescribed X times?"