Posted
by
timothy
on Thursday September 24, 2015 @09:23AM
from the und-venn-did-he-know-it? dept.

Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."

Correction: "You can nail everyone that's in the official audit trail."

The people at the top that authorized it (or at least didn't condemn it) probably never actually sent a traceable e-mail to anyone. Nor did they touch any code. Nor do they appear in any meeting minutes. These sorts of discussions tend to happen over a drink in a bar somewhere, and for good reason.

seriously, if you're a dev and PHB asks you to do this then you make sure you get it in email form and then forward that email to a personal account or to a safe pst file you keep yourself. or you send an email to said PHB saying what you did. along with everyone on the testing and QA team.
there was a paralegal arrested in NYC recently who forged a judge's signature without his bosses knowing because he was swamped with work and in his mind it was the only way to do it knowing he would be cause and commit

I agree, but in real life things are often not this simple. You get requirements that you aren't sure are a good idea or the right thing to do, and you question them - but PHB assures you that it's all been approved and cleared by the people on higher floors, and you may even contact some of them and hear their agreement. You're a coder and now a lawyer, so...? You go ahead and write IF OBDIIPortHasSomethingConnectedToIt THEN EnterDiagnosticMode .

This case seems very egregious, but the truth of ethics in real life is often difficult to determine, and it's being thought about by a human whose livelihood may depend on the choice.

It's not hard to see how things like this happen. People will almost always act in (what they think is) their own best interest.

When someone three layers of food-chain above you tells you "do this", you don't get to refuse until you have it in writing (unless you already have a new job lined up - and even then, don't expect that one to go any differently).

Now, you can certainly try to get them on record - You can ask them to write up a quick spec for what they want; you can ask them to submit the Change Management request because you don't have the authority to approve this one; you can send emails asking for clarification; and as a last resort, you can just document the change as "at the request of Boss X". In the real world, however, we've all dealt with people who refuse to do anything except by phone or in person.

And at that point, it becomes your word against theirs. Guess who can afford the better lawyer? And even that assumes it completely blows up - If it remains an internal matter, you won't even get the chance to present your side of the situation, just pack your belongings up and GTFO.

I find it odd to hear how programmers seem so abused by PHBs. Maybe it's an American thing, but in the UK, I've always found that employers want to keep hold of skilled people like programmers, because new ones are hard to find and take a while to get up to speed. This means that saying no is always possible.

(Nothing to to with official engineer status and ethics. There's no general requirement for engineer certifications for programmers here.)

Employers here don't care about holding onto skilled programmers or other skilled people, because PHBs think they can just hire replacements on a whim.

Yes, in reality new ones are hard to find and take a while to get up to speed. The PHBs will even acknowledge this when they're trying to hire.

But once they have one employed, they don't care about keeping him happy, because they think they're al interchangeable cogs.

If you're seeing a giant disconnect here, yes, there is. This is how American corporations think; it makes no sense at all. I can't explain it. It's the same phenomenon where corporations will give a big salary offer to a new engineer, but once he's employed there, they'll just freeze his salary or give him paltry CoL raises, while giving new hires even bigger salaries, causing employees to switch jobs every 2-4 years (in Silicon Valley, it's 12-18 months).

It's fascinating to see how many posters here automatically assume that it must be the PHBs who pressured the engineers into this. Very few assume that the engineers saw an opportunity for a bonus or for the PHB to owe them one, and added the cheat function voluntarily. I've not seen any posts so far that suggest an engineer thought of the cheat and suggested it to a PHB.

A reminder that we tend to think of our peers as being much more ethical than "them" and look for reasons to think of them as victims of force or circumstances, and assume that "they" are only motivated by sheer callous greed. Whoever the "them" is.

And I find it hilarious that everyone here states that programmers and engineers work without requirements or documentation. I've worked places where the verbal meeting would have the engineer agreeing with everything, then when it's not written in the requirements document and signed off by 10+ people, it doesn't get built. Seems like all the programmers on Slashdot have never worked in a company larger than 10 people.

While it is likely this was a sin of commission it remains plausible that no one did this at all. My thinking is that if instead of being programmed explicity the computer program was allowed to train itself for it's emission and performance tuning that a very natural outcome would be for it to learn to minimize emissions during emission type testing. Then on the test track it would learn performance and handling. etc... and so you end up with something that cheats but no one told it to nor was anyone even trying to make it cheat. It's just the result of getting what you optimize for.

One reason that I like that theory is that if you consider the opposite, that it was a conspiracy, then this is not the sort of thing you can keep secret easily. You might succeed but that's pretty hard especially considering the time span and the inevitable entry of new personnel and suppliers into the supply chain. So I don't think this was intentional. The exception might be if if it's a conspiracy of one. that for some reason there was just one guy who could pull off everything. THen you would have a shot of keeping this secret.

Then you do the next best. You get it in writing from your immediate supervisor, playing as if you're verifying facts of the request. Someone that is used to receiving such requests from you. He/she, in turn, realizes what you are doing, and does the same with His/her immediate supervisor. This continues up the chain until it eventually reaches the upper management who requested it. If someone along the line doesn't do this, they become the scapegoat, not you.It's a complete CYA action.

As an American who has worked for two different large corporations, I am surprised by this sentiment. I once got an order to do something I felt was unethical from my dotted-line boss. I sent a note to my boss immediately and his response was, literally, "If you do that, I will fire you." There was no pressure to do anything that even remotely seemed like a Bad Thing (TM).

Now that I'm a little higher up the food chain, I get the occasional request for my team to do stuff that's not in keeping with ethical o

Not really. What I've done in very "interesting" situations (names will be withheld), is that I send an e-mail that would have the following structure (I'm putting the important part in bold):

As per our discussion/instructions today, we will implement X,Y,Z and that this has been cleared/authorized/whatever-adjective-you-see-fit, and that these steps are appropriate. (If there are any lingering concerns, I list them here.)

Please let me know if you have any questions or corrections. Otherwise, I will proceed with your approval.

Once I send that e-mail, I reply to it, to me, indicating the time and date the discussion/instruction took place.

The critical part is in bold. No reply to the contrary implies tacit approval. Not bullet proof, and I'm no lawyer, but I've gotten people to backtrack "strange" orders as soon as they get such an e-mail.

That's ridiculous. There are valid reasons for the code to have different operating modes. It is not inherently unethical to make software behave differently under different operating conditions.

Think about it, VW sells cars in multiple jurisdictions. They need to have different emission mitigation regimes for each of them. It's possible that the flags for the operating conditions just got screwed up. If they use the same shop/lab model to pass EU and USA testing, there must be a switch in there for "Detected_testing_regime == EPA." And they might have intended to turn on the same controls when "vehicle_sold_to == USA," but the guy responsible for that code screwed something up.

Even if it were an intentional deception, it would be trivial for management to divvy up the workload such that the individual programmers had no idea. "Jan, we need you to make a module that detects the following operating conditions and pass this parameter to the emissions controller, mm-kay?" or "Hans, when you see the disable_emission_control_for_emergency_use, make sure that the car switches to the high_power curve."

We really don't know. Until the results of the investigation are made public, and they are found guilty of wrong-doing, they are innocent. Is the legal equivalent of "assume good faith."

And even then, a programmer or his/her manager could claim that code was for testing purposes, it should have never made it into production. It could all be copped up to an accident. It will really depend on whether the company wants scapegoats or wants to cop it up to institutional incompetence. Unless they find documentation that says, "TODO: Remove this test code" or "Bwahahaha cheat the emissions test!" how are they going to prove it either way?

Keeping a job right now, when you've got children or other family to feed, or when finding a new job is very difficult, can be a strong incentive to keep your mouth shut. This is true even when there is a small risk of prison time.

> Where I work everything in code is documented with trouble ticket numbers,

For most such systems, I can pretty easily submit a code change as someone else. With many systems, i can submit a change as a "back door" modification that will only show up with the most meticulous monitoring of the changelogs. I have in fact done so to avoid a code review by someone who needed to not be informed of a security patch to reveal their misbehavior.

Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

In other words, the engineers who actually did the code may not have known the real purpose of what they were doing. Somebody knew, of course, but they may be harder to track down.

Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

Folks, you have to branch on emissions (and dyno) tests in the ECU solely because otherwise the safety side of things will bring everything to a halt. The most obvious reason (assume FWD) is that if the front wheels are going 65MPH and the rear wheels are going 0MPH, the traction control system is going to have a major freak out and say "HOLY CRAP WE ARE SKIDDING OUT OF CONTROL BRAKE FRONT UNTIL THE DIFFERENCE IS LESSENED". The procedure [hellcat.org] to enter dyno mode [bimmerforums.com] is not itself a secret.

So it's not a secret and it's not illegal for the ECU to detect and behave differently during the test, and everyone would know about this above-board feature. The secret-and-illegal part is modifying the behavior of things measured by the test while the test is running. That is a lot easier to keep secret and requires a lot less involvement from teams directly outside the module that is responsible for emissions.

[ Source: Tuner people that dyno fancy cars and have to solve these sort of issues. ]

Yes, but this might even have been sold as a compliance issue - we must by law make sure that the full emissions package is in place for any emissions test, even if the service tech turned it off for reason XYZ. What engineer would blink at that? Meanwhile, over in another department, the engineers are being told, these emissions packages must by default be off, as not all jurisdictions require them, and we are using an opt-in type system to turn them on when required. Again, who would blink at that? But by such stratagems, they could set up to that one person or a few people could flip a virtual switch, and the hack would be in place.

Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.

Somebody knew, somebody high up knew, but I rather doubt that everyone on the engineering bench knew, and that means that they had to be fed plausible stories along the way.

Spare me. The engineers were the ones that eventually spilled the beans on this. They weren't fooled by some clever management strategy. They knew exactly what they were doing and they knew or should have known that it was illegal. While maybe not every engineer involved knew, more than a few certainly did without question. The engineers at VW aren't dumb. I know a few personally. Please stop with the attempts to find clever ways to not have to blame the engineers who were guilty of helping to commit fraud. Management may have ordered the crime but the engineers were the ones that carried it out.

Is it not possible that years after these things were made by two different development groups as the GP suggested, these two groups happened to discuss the work they had done, blinked, and realized how those two things worked together. Or given this started back in 2009, there's probably been some turnover, no one in charge knew anymore this was happening and had to be kept secret, and one person was given full access to the code to add a few new snippets - and HE realized what it was actually doing.

From the article itself, (I know, I am not supposed to read the article), the traction control is mandated to be turned off when testing the emissions. So you already have a need to check that emissions testing is being done. Once that flag is set in the code other areas can use it and they may not even be aware of the ethical dilemma that arises. Perhaps someone was told to lower the fuel flow when the test_mode flag was on. Why would they feel that something does not "smell quite right" in that case? Someone else turned the flag on with a code change several years previously. There is plenty of other things being changed or turned off when in test_mode, so adjusting the fuel flow can fit right in with that. Unless they realized that test_mode was used to determine that emissions testing was being performed (it could be used for other tests also) and that the changes they are making will cheat the emissions test they might not even realize that something is wrong with what they are doing.

The engineers who coded the software for the engine computer? Why would they know what this does? The software is enabling a signal, hell the signal might even have a vague name, when a condition is met. The condition's name might not very clear.

Yes, a spec saying "when the car is undergoing a test then enabled the cheat mode to get past the test" would clearly implicate the developers.

But most likely it was: "when sig_x and sig_y and indicator_a are set, then set sig_Z to 1 in pattern P for n nanoseconds blah blah blah"

Someone knew what they were doing, and it probably went like this:

Engineer: We can't make this engine pass NOx tests.Message goes up the chain to a certain decision making level, possibly the board. Marketing chimes in: We can't have this, we're already sending out teasers about our new urea-less engine technology, etc, etc.Eventually a message comes down to fix it, in vague terms, entirely forgetting the original message that it's unfixable.Engineers: struggle for ages.In pub: Well, we could enable a special testing mode to pass the tests?In work: Shall we do this -> up the chain. Original context is half forgotten. Approved.Changes made. Software specs made. Timebomb implemented.

Ah, but that wasn't the situation. The engine could pass fine, which is what was happening when the software detected the test conditions. The problem is when running like that the engine didn't have as much power or fuel economy as when operating in dirty mode. This hack was to make the car more appealing to consumers by (in a virtual sense) selling one car to the public, and having the EPA test a different one. This was pure deceit.

All varieties of IC vehicles will be gone soon enough, NG, diesel and gasoline. Not because they pollute. Because they're enormously less efficient than electrical vehicles. The only benefit IC vehicles have right now is that gasoline, NG and diesel store larger amounts of energy by volume and weight than electrical storage does, and offer less maintainance of the energy supply for the vehicle. But those advantages continue to erode. Eventually the benefits will fall the other side of the line, and that'll be the end of that.

True.

One of the obvious consequences of this inevitability is that the very small amount of extra exhaust products generated by VW's software methodology in actual vehicle use isn't likely to make an actual difference to anyone in terms of pollution.

Non-sequiteur. Possibly you haven't appreciated that this isn't mainly about greenhouse gasses, but about pollutants that cause local heath problems in cities, and thus directly harm and even kill people. That most certainly *IS* a difference.

Yeah, I worked for a large bank. I was told my task was to implement tax calculation code in javascript, so it would update on the page immediately. While I balked at the request, it was made pretty clear that either I do it or I would get fired.
I was not given a javascript library that knew how to handle financial values.
Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors.
I happened to know about them but was not given any instructions that they were a problem.
I wasn't given any instructions on how to make sure they didn't cause issues.
As far as I know no more than basic testing was done on the code.
I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that.
You get fired now, or you implement something dubious - what do you choose?

You get fired now, or you implement something dubious - what do you choose?

You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

Even if you have to write it... "Per our conversation regarding _______ I am doing _______ at your request. Please let me know if you change your mind". In civil court, all you need is 50+%. It doesn't take much to get to 50+%. Self documentation is perfectly acceptable.

Yes and in the mean time you lose your house, car, and your kids go hungry. And the woman that would love you till death do you apart? She meant financial death. She will leave you for putting your family's financial livelihood in jeopardy to do some half-ass ethics play. You may get some money from the settlement, but the whole thing will now be part of public record. Good luck getting hired ANYWHERE else.

It's easy to be an idealist on paper, especially if you have nothing to lose. Real life is a tad

Actually it's quite simple in real life and in most sane countries in the world (the USA excluded by this statement) wrongful dismissal that needs to be settled is either settled very quickly (by tribunal) with financial compensation and re-employment or a buyout of the employee, or at the end of the court case you can retire with your new found wealth.

That's how it happens in real life in most of the western world where employees are protected by laws not bought by a corporation.

You get fired now, or you implement something dubious - what do you choose?

You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

Dubious and wrong aren't the same thing and the OP didn't explain further. He's a programmer and his employer asked him to write some tax calculation code. Either he can research the math and write some code or he can't. If he can't or refuses, getting fired isn't "wrongful termination", though it would be a dick move to fire him if he's simply not up to the task w/o affording him a chance to learn. Documenting his concerns is a good CYA idea in case anything goes south. Now, if his employer asks him to w

IANAL, but this is what I heard from an employment lawyer (paraphrasing): As long as you wrote and sent the e-mail, the onus is on the employer to show that they have replied to it in a reasonable fashion. Courts tend to throw out "I didn't see it" excuses and expect replies. And even if they replied verbally, oh well, too bad, they should've done so in writing.

One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

That worked on one of my tasks where I was asked to do something counter to standards. After I tried to argue them down for some time, my coworker (I wish I could take credit but I've made use of this since then) simply asked for them to send us an e-mail stating what we were being asked to do. At that point, the other person's manager told him to stop asking us to do it the wrong way.

" I was not given a javascript library that knew how to handle financial values."You're a programer so write one."Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors. "You're a programer so write one.Really just make a fixed point object or even a BCD object for the project.

I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that.

Nothing prevents you from printing emails of instructions to implement dubious decisions. I've done this from time to time just to protect myself when I worked at a large company.

You get fired now, or you implement something dubious - what do you choose?

If it is clearly illegal or will be very likely to cause major problems then you should seriously consider walking. If it isn't so clear then you get them to document their instructions to you and you keep a copy (print if you have to) for your records to cover your ass should it be a problem down the line. Make sure you document your objections and make it clear that you have taken every reasonable effort to ensure that what you are doing is legal. If the decision is merely dumb but legal, same thing but don't worry so much about ensuring legality.

No “they” didn’t. One team wrote code to detect an emissions test. One team wrote code for a new emissions heavy efficiency high mode. One team wrote code to switch between emissions tests. One dude with a beard added if( TRUE === emissionsTest ) { mode = MODE_X43_Y;} Thats all it took.

But frankly I have a VERY hard time believing that the engineers involved did not know that what they were doing would violate the law.

On the other hand, the code could be used to support a control test case, with the emissions controls switched on/off, to support development and how well the emissions controls were working on the same engine. This would allow the engine developers to test against a live performance benchmark. On the original hand, having that switch on/off automatically is quite hinky, so it's probably not (solely) for development testing.

Why would anyone want to hide evidence? All the people seem to operate under the impression that this is some kind of big revelation, when in fact everyone in the industry knows that this is happening all over the place. This wasn't actually big news that Volkswagen is cheating, the big news is that some authority is finally taking the rules seriously, when for years both the european and US authorities did their best to look the other way when the evidence was shoved in their face. And if no one cares, why would you want to hide it?

Definitely been going on for at least 17 years, and many other companies besides VW are getting caught.

"On October 22, 1998, the Department of Justice and the Environmental Protection Agency announced an $83.4 million total penalty against diesel manufacturers, the largest civil penalty ever for violation of environmental law...The seven companies sold 1.3 million heavy duty diesel engines containing illegal "defeat devices," which allow an engine to pass the EPA emissions test, but then turn off emission controls during highway driving. As a result, these engines emit up to three times the current level for NOx a harmful air pollutant."

Based on my experience with slimebag orgs and managers, it may have gone something like this:

A manager(s) asks for a software switch to deactivate the "clean" mode and also detect when smog testing is being done "in order to study and track resources devoted to environmental issues and make sure we understand and comply with the smog testing procedures."

Then a personal visit happens where key manager(s) ask the top-ranking technician to leave the bypass-on-test feature "on" in production. No paper trail. Experienced slimebags don't put such commands into writing.

When the IT lead later reveals "Executive X told me to in person", there's no written trail. It's one person's word against another's.

Sure, the IT lead is probably suspicious of the request, but when the big bosses tell you to do something, it's comply or hit the road.

I was once asked to cheat a client over database scalability. It was shortly after the dot-com crash, and knowing the market was really tight in Calif. and having a young family, it was a really difficult situation to be in. I won't go into the details here, but I was sick to my stomach over it. The experience made me more progressive.

If there is no audit trail, shouldn't the buck stop at the top? The roles, job titles, audits, change management controls, financial approvals, etc are all business processes designed to reallocate the default responsibility who is either one of the big Cs or the board of directors. Without the approvals, audits, or whatnot the responsibility goes to the person who can't push further down. And it can't be a he said, she said thing. Need documentation.

I *highly* doubt it was a single line of code. To toggle the car's "EPA Cheat" mode, maybe, but by all accounts, the system used a variety of inputs to detect artificial driving conditions (including, apparently, barometer data), as well as needing code to define what engine parameters to change once the mode was entered.

Sometimes but not always. This cheated the test both with and without a computer. It was instead detecting when it was on a rolling road. Emissions tests are always done with the car stationary but the wheels moving, and that'll be what the software was detecting.

From the sound it of it wasn't actually putting the car into a special 'mode'. It was turning on all the measures to reduce emissions. When the car was on a real road it was turning them off to get better performance and fuel consumption at the expense of emissions. So it sounds like the car does technically meet the regulations, but ignores them when it's on the road. So expect the recall to turn them on at all times, which'll mean you don't need the car/engine replaced, but will mean you pay more at the pump and see your car's less nippy than it was before.

Because VW are going to recall hundreds of thousands of cars, stick them permanently in test mode with emission stuff switched on. There will then be a massive class action as people who bought a VW expecting x horsepower and y MPG demand compensation for getting considerably lower performance. This will cost them billions, and that's before the multi-billion dollar fine piled on top.

It's been a while since I watched my car being tested but do they hook up the car to a computer terminal of some sort? Could those be used to trigger test mode?

The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.

Basically they wanted to avoid the cost of installing a urea injection system [wikipedia.org] so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.

The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.

Basically they wanted to avoid the cost of installing a urea injection system [wikipedia.org] so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.

As a diesel emissions engineer, I was always fascinated by how Volkswagen was able to do what they claimed. I had tried to make their technology work, it's extremely difficult. SCR is much simpler and more economical.

I had thought for a while that other companies had some secret information that my team was missing to get such good performance. Now I think other companies are just cheating.

AFAIK, in California, they do that for older cars, newer ones get a device that plugs into the OBD II port and they log from that.

Texas is similar. If the vehicle has an OBD II port, they plug in their reader, pull the values from that and call it done. The only time exhaust checking might be actually used is if it is obvious someone did a custom tune (the OBD II port showing clean on a coal roller truck, for example.)

Supposedly the "cheat mode" is an extension of the "testing mode", where the car knows it is running on a Dyno because one set of wheels is turning at a high RPM and the other set are stationary. For a car with traction control this is normally a freakout event so they have to check for it and make sure not to go crazy just because the machine is strapped into a test harness. Once you have the otherwise required detection code in there, adding a single line to fully open the EGR valve when in that mode would be a piece of cake.

Yes, but people run these cars on dynos all the time for performance figures, and they all saw the standard performance figures. They fixed it so that the cheat mode would kick in when EPA testing was done on a dyno, and only when EPA testing was done on a dyno. It's doable, thanks to the EPA's rigid testing protocol, but it wouldn't have been simple.

According to news reports (BBC etc.) it used a combination of inputs. Steering wheel position, barometric pressure variation over time, rate of acceleration, speed, g sensor stability etc.

Also, to enable the cheat mode the engine would have had to load a different set of operating parameters. Those parameters must be stored somewhere, and doubtless constitute more than a single line of code.

Also, to enable the cheat mode the engine would have had to load a different set of operating parameters. Those parameters must be stored somewhere, and doubtless constitute more than a single line of code.

Some thought must have gone into this fraud.

You don't just punch in a guess at those operating parameters either. Lots of data gathering and evaluation was done to get them to meet requirements. This was a decision at the program (I'm not talking software) level. A budget was approved to design, test, and implement this strategy. There was no lone gunman.

Most rolling roads don't spin the non powered wheels, so if the powered wheels are spinning and the others are stationary for any length of time its a good bet its having some sort of test. Obviously this isn't going to work with 4WD however.

"However, not all companies follow detailed auditing processes. The primary reason, Kaul said, is the speed at which software is being released to the marketplace. It necessitates an "agile approach," resulting in millions of lines of code being worked on and checked into production every minute."

That's true at a lot of software companies but it is almost certainly completely untrue when it comes to an embedded engine control unit at a major automotive manufacturer. That sort of development is typically slow, methodical, and rigorous, with extensive pre-release testing. And the team is probably pretty small, I doubt there are more than a dozen engineers working on that, and probably a handful of key guys write most of the code. And I can guarandamntee that their code churn is nothing like "millions

I wonder whether someone actually gave the order to implement a 'test defeat device' or they just started to optimize the engine to comply with regulation and to pass the test and then they went too far.

I wonder whether someone actually gave the order to implement a 'test defeat device' or they just started to optimize the engine to comply with regulation and to pass the test and then they went too far.

Someone in management made the decision to not install a urea injection system which is necessary to keep emissions to legal levels but costs a lot of money. Reportedly something like $400/vehicle. So it seems pretty clear that their "solution" to the problem was simply to cheat. This wasn't a case of optimization gone awry. They flat out knew what they were doing and went ahead with it anyway. As soon as they made the decision to not install urea injection, they effectively decided to cheat at that time because they were asking for the technologically impossible. There is no way they didn't know that their decision to leave off such a key piece of equipment would not result in unacceptable emissions. The engineers at VW aren't dumb. The decision was made for financial reasons (not surprising) but was aided and abetted by a bunch of engineers that should have known better.

The only real question seems to be who made the decision and who was responsible for executing it and covering it up.

Someone should have leaked this a looong time ago. Perhaps some dev, why not. Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln! Did VW really think it could get away with this indefinitely? Fucking corporate morons...

The way this was discovered (*1) was by an independent university group taking purchased vehicles and connecting it up with sensors and running it over real roads in real traffic conditions over long periods of time and comparing it to the rolling road test results. It's not an astronomical cost but if you are just looking for basical emission data then there are much simpler methods (namely a rolling road).

*1 - assuming this wasn't a case of parallel construction and the real road test data is just collecting evidence for what somebody already knew was happening.

I haven't found out if the normal driving emissions are actually"bad" or just fall foul of U.S. automaker protectionist lobbying.

No flame here, just wondering. In my travels to Europe I haven't found them to be any worse pollution wise than American cities.
Are these cars really that bad physically or are we talking goofy government crap?

While somewhat anecdotal, it occurs to me that any comparison of pollution/air quality in European vs American cities, and their respective emissions standards, should likely take into account that Americans tend to drive far more often than Europeans.

"Volkswagen’s rigging of emissions tests for 11m cars means they may be responsible for nearly 1m tonnes of air pollution every year, roughly the same as the UK’s combined emissions for all power stations, vehicles, industry and agriculture, a Guardian analysis suggests." http://www.theguardian.com/bus... [theguardian.com]

I have been teaching UML, modeling and systems architecting at several companies that directly supply to the German car industry ( especially to Volkswagen and BMW ). It is the car makers themselves that impose rigid rules and constraints upon software traceability and configuration management. So the idea of

"software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it"

is not that outlandish, and following such audit trails may well lead to (at least some of) the culprits.

I envision a scenario where as soon as the shit hit the fan on this, somebody really high up in the company who was in the know on this went into the necessary files and altered the documentation so that they were not incriminated.

Most of the actual codes that disabled the emission controls would have legitimate use during development cycle. Typically there will be a switch to put the engine in a baseline test mode, to gather baseline data. The test engineers would turn it all off, get a sanity check baseline runs made. Then they will turn on various switches and turn various knobs to fine tune the best values trading off power and efficiency to reduce emissions. So the actual code turning off emission control is not a big deal.

What would be a big deal however is the code that detects whether the car is in test bench or on the road. Apparently it uses steering input and other such details. So that code block is the interesting part. Proper audit of the code changes, pull request authorizations would nail the engineer who actually did the dirty deed. But would there be code review meeting/minutes, comments fingering higher management?

This scandal will have some salutary effect in engineers who manage code, they would refuse to merge or pull such cheating code changes because it would leave their fingerprints indelibly for ever. They might even add comments in code covering their tails fingering the actual perp in the higher management.

I worked for a Small software house that made SAP type ERP software before SAP ate the majority of the market. This was 1998 or so...

We had a customer come to us and ask for certain modifications. Then a few more. Then a few more.

Not unusual, we made a lot of money from change orders. So the first few were done. All were acceptable in the Generally Accepted Accounting Practices guidelines.

Somewhere along the line the GAAP accountant realized that this last modification set would, taken in combination with all the other mods, make a check disappear from the system and become untraceable.We refused to do it, and the customer dropped the product, saying we were too hard to deal with. A million+ of revenue were lost, no small amount for the company.

That customer? MCI Worldcom.

They clearly had picked apart the source code and found the edge case that triggered the behavior. I had left the company before MCI blew up, but my understanding is that they were called to give testimony/evidence in the trial.

This could be the same thing, a series of unrelated changes that trigger a diagnostic mode when hooked up to the test equipment.If so, it would be very hard to trace who made the ultimate decision to do this, as it might be spread across many teams working independently.

Yup. This is why you shouldn't be afraid to be a bastard with your supervisors over matters of ethics (eg: time reporting, license compliance, government regs, etc.). It's most likely your neck on the line, not theirs.

I work for a company that has paid out some 30 Billion Dollars worth of fines to the US government. Where does that money go? I think it goes directly into the pockets of well-placed individuals, because we never hear about where that money goes.

I work for a company that has paid out some 30 Billion Dollars worth of fines to the US government. Where does that money go? I think it goes directly into the pockets of well-placed individuals, because we never hear about where that money goes.