Posted
by
Soulskill
on Saturday July 28, 2012 @04:18AM
from the par-for-the-course dept.

McGruber writes "The Federal Times has the stunning (but not surprising) news that a new audit found six Defense Department modernization projects to be a combined $8 billion — or 110 percent — over budget. The projects are also suffering from years-long schedule delays. In 1998, work began on the Army's Logistics Modernization Program (LMP). In April 2010, the General Accounting Office issued a report titled 'Actions Needed to Improve Implementation of the Army Logistics Modernization Program' about the status of LMP. LMP is now scheduled to be fully deployed in September 2016, 12 years later than originally scheduled, and 18 years after development first began! (Development of the oft-maligned Duke Nukem Forever only took 15 years.)"

Call me a cynic, but I think everyone wised up to the fact that they weren't really buying solid gold toilet seats, so they had to find something else in the budget to fund all that black ops stuff...

The toilet seat thing was blown way out of proportion. It was a custom-molded plastic assembly for military aircraft use [yarchive.net], and as most people here know, when you do injection molding, the initial tooling costs are very high. High setup costs + low volume = seemingly outrageous per-unit price. It's not as if they were paying $700 for the same type of toilet seat you can buy at the local Home Depot.

Even if the specific example may have been blown way out of proportion, I actually see a lot of the plainly-worded outrage as a complete misunderstanding of how the whole DoD acquisition process [dau.mil] actually works.

That's how the company getting paid for it becomes a Fortune 500 company in the first place. What's that bunch of losers that ran over budget but never delivered for Police and FBI databases? The name escapes me, but they are in that 500 and have nothing but a record of being a failing leech on the taxpayer to get it.

That would be SAIC [saic.com], and the Virtual Case File program [ieee.org]. After that debacle, caused by evolving requirements AFTER contract award and a willingness to do whatever the customer asked for, as long as they paid for it, the final delivery was what you would expect, a steaming crock of bugs and crap.

SAIC used to pretty much OWN the FBI's data center contracts. They are pretty much gone now, and Lockheed-Martin now runs the show out at Clarksburg, WV. ..

SAP was around 18 years ago, but not well-known. And plus, being a German company, it probably would not have been the choice then for a US Defense Department project. Too many senators and congressmen would have howled about so much money being spent abroad, and not in their states and congressional districts.

Now, SAP is well-known . . . but I don`t think that many people really know what it actually does.

Are we so inured to this that we can't even speak the words, let alone call our congresspeople? Will we not even push people to ask whether the next president will be calling for the prosecution and imprisonment of the people responsible for creating a billion dollar, 18 year "army logistics" software development project?

Don't give me that "mistaking malice for incompetence" bullshit. That's exactly what's wrong with this country. Just because it's computers, don't tell me you can't tell a $100 toilet seat when you see one. A couple years late may be incompetence, but you should have the FBI given all necessary clearances and set them crawling all over it. At 8 years into a 4 year project, you fire the buy-side project managers and cancel the project, whether you uncovered fraud or not. Fail to maintain even these basic standards, and no estimate in time or money is ever real, and every contract becomes open season for treasury looters. Oh wait, like it is today.

There is no way on earth or heaven that a logistics system can cost this much or take this long to build. And I would say those so corrupt or negligent as the ones running implementation at the vendor or running procurement within the military should be behind bars. This is not a joke, people - this is keeping American troops in a decaying and ancient logistics system so that some weasel can steal your tax money.

Government procurement is one of the most frustrating processes known to man. This is true of both federal and state level purchasing divisions. That government contractors take advantage of the deep pockets shouldn't surprise anyone. The fault lies on the contract writer for overruns and timeline delays. It means the contract wasn't specific enough or realistic enough. It also means the budget wasn't properly vetted and that benefit cost analysis either wasn't done or was done improperly. Lastly, it means

I don't think you understand how defense programs typically go so wildly over budget and schedule. Project oversight is passed from person to person every two years or less. Each officer in charge spends the first six months or more learning what the project is, often from the contractors themselves. That's bad enough. But multi year tech development contracts suffer from continual scope creep as the state of the art advances on parallel to the project under development

Yes, but... that is a response to the miserable failure that the project already is. The fundamental problem, is "If there's one thing every junior consultant needs to have injected into their head with a heavy duty 2500 RPM DeWalt Drill, it's this: Customers Don't Know What They Want. Stop Expecting Customers to Know What They Want. It's just never going to happen. Get over it." --Joel Spolsky [joelonsoftware.com].

The customers, in the DOD's case, are combat leaders, not program managers. They don't know what they want. They

Thus causing the scope creep I mentioned. But there's also the undesirable effect that during that 6 months or more that they are getting up to speed on what the project is and its current status, the project is running without effective government oversight, so it might make no progress or develop in directions that don't meet the government's goals.

ALL government projects including military ones should be run by civilian project managers and every effort should be made to have personnel continuity start-

The ones I've seen have been directly run by govt. civilian program managers, with their military oversight person changing frequently as mentioned. These projects make slow and wasteful progress because the govie program mgrs seem to think it's a primary part of their job to jerk the contracting organizations around; make them suffer a little to make up for how good a thing they have going, it seems. So constant requirements churn and pressure to keep up, and evidently to justify your place on the contract

I wish they would. The problem with many folks on their side of the political isle, is that all gov't spending is bad except for the spending they actually like. DoD spending is spending they like, and somehow exempt from all their arguments about wasteful gov't spending "Because its in the constitution!"

Seriously, anyone who thinks DoD spending isn't wasteful gov't spending either:1) Is blissfully ignorant of how the whole industry operates2) Works in the industry, and is thus self-serving

Actually, just about every job the government does is something the private sector can't or won't do. Everything from building roads and bridges to educating our kids. You bitch about your taxes but in reality if you had to pay private sector prices for those items you would really be bitching. Imagine everyone having to pay tuition for their kids education, paying tolls on every street, paying for someone to test your food to make sure it was safe. You get the picture.

And what do you think happened here? It's maddening to work some of these long term implementation projects as government support staff is routinely changed causing delays in key decisions. Then requirements get changed along with the personnel changes. Duke Nukem might be a very apt description of what happened on these large projects. In addition to trying to do REALLY hard things with technology that just isn't quite there yet at on a scale most don't really understand.

Sadly true... I actually got grief for coming in *under* budget on a project. It was actually a repeat of a previous task, and the gvmt couldn't understand why this one was so much cheaper.

The first one was a completely new item, so I had to draw it up, work out the design, write the tech docs, etc. Took a while but the end result was exactly what the customer wanted. When they asked for a second (identical) item, they assumed the same price. I was able to use all of my drawings and docs from the previo

if you specify the actual cost during the planning phase, then they wouldn't be started in the first place. So people make best case estimates and then reality strikes, the actual cost exceed the allocated budget.

Six is a bit long, but not really outside the norm compared to how a typical large company would do it. One place I've worked, it took three years to fully roll out our Microsoft Exchange transition. And that's just desktop-oriented software for just regular employee usage. IT projects relating to anything more complex or business-critical could take more years. There are still mainframes operating at some places, and decade-long projects to replace them that haven't finished.

Not surprised...in the early 90's I was with a small computer biz and we won a small contract to build 50 PCs for the navy specced out to run off the shelf CAD software. We built them, shipped them and the navy promptly stuck them in a warehouse for a year. When they finally pulled them out the software no longer preformed well on the PCs and every one was shipped back to be upgraded to run the newer version of the software. Shame we were to small a companyt to really rape the government for big bucks.

I joined the Army in 03 just in time for Emilpo to be released. I was the pivot class(42F) not trained on the old and not having any training on the new but we had AIT for just as long for no reason.
In 05 I started hearing about its replacement coming down the line and when I finally left in 09 it was still "just going to be released soon"

No, they originally thought they could implement it in 2 to 3 years, with part of it in place after 1 year. Took 'em 5 years to get the first supply command (CECOM) up and running, but without most of the Finance part of the system. They implemented the second command at the 9 year mark, and the remaining three Commands *all at the same time* at the 10 year mark (talk about a data migration nightmare!).

The debacle occurred for reasons of ignorance and, perhaps, a fair measure of arrogance.

The people at HQ Army Materiel Command (AMC) who wanted to contract out the replacement of the existing Army-wide integrated system didn't understand the existing legacy system. They thought it was obsolete - no, it wasn't, that was part of their ignorance, because the HQ AMC people who had originally sponsored the system had long since left and nobody then at HQ AMC really knew anything about the existing system..

The contractor, CSC, had no clue whatsoever about the level of difficulty of government procurement, requisitioning, supply management, finance, provisioning, maintenance, etc. They thought because their consulting staff knew how SAP worked, they could implement an off-the-shelf solution "with a little tweaking for public law and Army & DoD policy". The supply stuff is the easy part; probably 90% of the existing system - the largest, most integrated and complex supply system in the world - was "public law and policy guidance". And both law and policy changed rapidly - one such major change occurred after the contract was let, and there was a 5-year argument over whether it had to be included in the new SAP system, or if they should field the new system first, and then ask for more money for the out-of-scope "change" to be added on afterward (they didn't understand that the needs of the troops in the field meant that _all_ changes were in-scope when it came to operating the worldwide production Army logistics system).

CSC picked up about 200 of the original 300 Army system developers and functional experts on the Wholesale software side; something similar was done on the Depot side of the Army logistics house. They thought they were going to be able to fire them all after 3 years. These people were hired primarily to maintain the old systems while the new SAP system was developed but, because all non-critical changes were frozen, there was quite a bit of spare time to help with development of the new system. Again, CSC thought they knew it all, and refused to let the ex-Army civil servants help with the development of the new system (even though they received SAP training specifically to do so), nor even to work on data cleansing and migration from the old system to import into the new one. The Army system developers and functional experts had over 30 years experience doing that - moving from the Vietnam-war era systems custom-built at each Army supply command and Depot to the "new" Army-wide standard Wholesale and Depot systems, and then doing major upgrades as functionality was added or public law and Army/DoD policy changes required major changes. Along the way, Unix and other mini-computer systems, as well as PC's, were added to the original mainframe system with it's custom-written DBMS specifically designed for the hierarchical nature of supply data.

GAO has some valid criticisms of the whole mess but, typical of GAO, still doesn't understand major computer systems after 40-50 years of auditing them, and misses many things done wrong, and completely misunderstands the legitimate reasons the process was going to take that long whether well or badly managed.

My perspective on all this is as a programmer, analyst, and team leader for 3 years helping to develop, implement, and run one of the Vietnam-era systems for one of the Army supply commands, then the same for the new AMC-developed Army-wide standard system for 30+ years, then finally working for CSC for 10 years maintaining the Wholesale legacy system and, occasionally, working on the new SAP (LMP) system in minor ways. An "insider" perspective, for sure.

Pretty much everything the Pentagon does is over budget, behind schedule, and budget-wise, generally a spawn of wishful thinking. The "cheap" Littoral Combat Ships were sold to Congress as sub-$250 million craft. They're currently just under $700 million apiece. The "cheap" F-35 was promised to be no more than $60 million a copy or so. They're now just under $200 million a copy, flyaway (more expensive than the F-22 they were supposed to compliment). The new Ford class carriers... an evolutionary development of the current Nimitz class.... will now cost 2 1/2 times as much as the last Nimitz that was launched just a few years back.

Why should DOD software be any different than DOD hardware when it comes to wishful thinking from the brass?

There's engineering solutions and military solutions.Engineering solutions have known goals (don't laugh, I'm writing about an ideal situation here).Military solutions cover contingencies which may shift over time. They are also highly vunerable to political whim, up to and including outright bribery.

Pretty much everything the Pentagon does is over budget, behind schedule, and budget-wise, generally a spawn of wishful thinking

And by "wishful thinking" you mean the wish that a whole bunch of pork will land in a barrel somewhere, right?

Wish? More like planned that way. The Pentagon knows how to play the game: lowball your estimate for a weapon system you're selling as critical to national security, get the process flowing to as many Congressional districts as possible (one factor that raises costs, in fact) in order to gather maximum support, and then when production actually starts, you know that Congress won't have the courage to cancel the program.

I'm very hawkish, but over the years, I've also become very, very cynical about how we bu

Maybe we'd be better off going back to designing and building our own ships and aircraft (the Navy especially was into doing this... they even had their own aircraft factory, and they found that it kept costs down in the 20's and 30's as it kept 3rd party contractors honest).

You know, I've had this thought myself. We've reached the point that we're hiring companies to produce weapons for us where we retain full rights to the technology and design, and are thus the ONLY customer.

If we're going to be funding the design and be the only users, might as well do it entirely in-house. Contract in support when necessary, but have our own company doing it.

In a lot of ways we have stacked the deck against the government doing successful projects.

The public gets upset when government workers are highly paid or get nice perks (see the recent GSA Las Vegas conference fiasco), so it is not easy for them to attract top talent.

The public wants more oversight of government projects than is typical for industry. This results in non-standard accounting practices that require custom management software. The accounting requirements can also significantly distort enginee

reminds me of SpaceX Dragon as compared to LM Orion (ok, bad comparison in many ways), where Dragon was developed along with a launch vehicle and flown twice for about $1 billion (or of that magnitude). Orion is costing billions and billions... not sure when it will ever be flown. What is this difference? An effective cost control is when the money comes out of your own pocket. Unlike the other which is not.

In my brief stint in the Military/Industrial complex, I noticed that there were always uniformed personal on the government side of the table, and retired military people on the "civilian" side of the table. When officers retire, they leave and go to the private sector, where they end up managing projects for the military.

I expect that the contractors were staffed with lots of "project planners" and "requirements specialists" who went straight from the service to work on these projects. And you can be sure that the ex-military are extremely unlikely to buck the system and stand up to uniformed types. And those in uniform know that they can climb on the retirement gravy train as long as they don't make life too hard for the contractors who they expect to work for when they get out.

It's a recipe for disaster. Nobody is going to make waves, because they are all too busy looking out for their common interest. It's another example of the endemic corruption that is steadily eroding the fundamentals of US society.

Of course this is small change compared to what goes on in the financial sector...

Of course this is small change compared to what goes on in the financial sector...

I'd have to disagree. The financial giveaways only every few years when some socialization of risk needs to go down. This sort of thing happens every year to IMHO the tune of hundreds of billions per year.

Anecdotal, but in my experience no military 'officer', whether Commissioned or not(enlisted), who makes it to retirement has no problem finding a job within view of their old office if they want it. Well, there are 'problems', but they're more in the line of helping to ensure that said retiring military people have the best shot at said jobs, whether contract, GS, etc...

Generally, the contractor will start looking and like acting the customer, especially if you only have one customer.

Also, technical projects are hard. Requirements can change by a lot. And in some segments of the military government officials are only in their positions 2-3 years, so they don't get the level of expertise they need.

You described a situation where you have people who understand the customer identifying requirements and planning. Makes a lot of sense since an outsider would take many years to understand customer needs. I think the real part is the second part of your statement "ex-military are extremely unlikely to buck the system and stand up to uniformed types" - the problem has nothing to do with uniform, prior service are LESS likely to be impressed with that, what you really have is the age-old problem of not w

But the whole point in outsourcing is that you pay a fixed amount to third parties to complete a specific job, and they take over the responsibility for making a profit (or, at worst, breaking even).

That's true if the job remains the same. But it seldom does. A tried-and-true method is to give a lowball amount for the basic contract, but charge out the ass for change requests. (And there will always be change requests.) This is true whether the contractee is a private company or the federal government.

So wait a minute...if you're always forced to go with the lowest bidder this can happen? I would have thought the lowest bidder would have also been the most reliable and skilled. Who would have known?

Common misconception. You are *not* required to go with the lowest bidder. You just need a good reason to pick someone more expensive. "I think they're bullshitting and can't really do this" is a valid reason.

"I think they're bullshitting and can't really do this" is a valid reason.

Depends on your contracting office. In one of my old units we had a choice between two radio companies. One company's shit was cheaper, but broke all the time. To buy the more expensive ones we had to do a study(NOT a multimillion dollar one, more like a week of a guy's time collating stats) showing that the cheap company's radios didn't really meet standards and broke so often that the additional required spares(due to reliability) actually made them cost more.

Apparently it is the sacred duty of governments to waste money, rather give it away to random corporations than ever run the risk of making a profit or hint at competing in any market. So it looks like this government department is doing a fine job, making its corporate chums filthy rich off of taxpayer money in the process. Redistribution of money was what taxes are all about, wasn't it?

Or at least, that seems a pretty accurate description of this country's government's actions over the years. Any economis

The Navy's ERP system has already been "deployed" at certain commands such as NAVAIR and SPAWAR, so it isn't like they have spent this money and not produced a product. The system has issues, some of which are outright bugs (e.g. losing employee time sheets), some are poor design choices (e.g. the purchase request part of the system had issues with $0 line items on orders, previously we had to put in $0.01 to make it work), and some are dumb policy choices by the DoN or the local command (e.g. we are not allowed to use the built in leave request system, we still pass around e-mails or signed PDF documents).

Overall it is an improvement over the systems it replaced - for me that was a series of in house systems that essentially emulated the original WANG environment and re-used COBOL code. It does make it more difficult to fix things and I still believe the system has a hard time working with the DoD's "accounting" system i.e. different colors of money, lines of accounting, money that expires, etc.

ERP has a user interface that sucks so bad, it has to be a deliberate joke. Imagine plugging in a week's worth of time-keeping with Visicalc on an Apple II, when your funding number for each piece is just as long as a credit card number.

From TFA: "The department is racing to meet a statutory September 2017 deadline for passing a full financial audit."

Ya gotta love it. Any publically traded company has its accounts audited annually. The government is so out of control that it looks unlikely to meet a deadline of a successful audit five years in the future.

The government ought to be required to follow the same standards required of companies. No one has any idea what the financial status of the US government really is, least of all the government itself...

It's the Department of "Defense". I would be surprised if it wasn't over budget. (I'd also be surprised if there wasn't some pork being doled out to contractors in key congressional districts from these programs.)

I love examples like this, in a perverse sort of way, because it give me yet another example to give to my students. Government contracts are essentially always done with the "waterfall" model, because the government insists on complete specs before funding. Any competent software engineer educated in the past 20 years knows that waterfall-projects fail if they are above a certain size. Hence, there is an endless supply of examples like this, and will be as long as the government software contracts are mana

The perhaps you are best placed to understand the point that Im trying to make, its not as bad as the article is painting the projects out to be.

8 Billion over budget at 110% ~ 8 Billion budge for 6 projects, each project has about 1.5B1.5 B over 12 years ~ 100M per year. Probably includes hardware and software, these guys started in 1998, dont think 100M gets you much in terms of ruggerdized equipment and Im sure software needed to be rewritten each time new software. These companies probably wanted a prof

good thinking, but problem is you're just taking the latest happy number lies from these huge defense contractors and doing what they intended you to do with them. There have been very few completed large projects for US Federal government for more than 20 years as far as I can tell. They're all disasters and none working very well. If there are success stories I'd like to hear about them but haven't.

Mostly true for large projects at all government levels. People blame governments but I blame the software i

Limited hardware capability (which meant you had to use every CPU cycle and bit (an on/off state, a computer bit) of RAM optimally) meant that those coding had to be very skilled at telling the computer what to do. Nowadays, bloated and unoptimized code can do the same things faster on modern hardware, but it'd be many times faster yet, have fewer bugs, and be more secure if all programmers were as skilled and creative using modern hardware as those who are currently known as graybeards, who know the import