Often we as programmers see large organisations wasting huge sums of money on bloated and inefficient solutions to problems. This pains me greatly because I like organisations to benefit from best of breed solutions. However, my abilities as a programmer are limited when it comes to influencing the key decision makers and often my perspective on the matter is constrained to my own little technical world.

So, my question is this. After encountering an egregious waste of money on some software and/or hardware that really got your goat, what did you do about it to get it fixed or were you doomed to bite the bullet and mutter forever under your breath? I'm interested in hearing your overall experiences and especially what lessons you learned about how to tackle this sort of thing in the future. Let's not name names, the experience of how to tackle the problem is more important than the actual offending product.

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

9

+1 both for a good question and for using the word egregious.
–
Jon HopkinsNov 12 '10 at 12:51

I've seen too many examples to name a favourite, but I've noticed a few general trends in my main field, web-development:

Vanity Websites. These are websites that serve no useful purpose to anyone outside the small organisation that commissions them and are built around an obsessive compulsion with logos, photos of themselves and self-indulgent waffle. The worst part is these are usually public-sector funded and commissioned by people who have no clue about the web. (For instance, once had a NHS hospital trust who wanted to develop a mini-version of Facebook for their own staff intranet).

Paid for is Best. The mindset that insists that
paid-for software must intrinsically
be better than open-source. After
all, it's paid for, right? I've seen
so many clients insist on making
stupid choices simply because they
work in a culture that automatically
discounts anything open-source as a
matter of policy.

Design by committee. This is where a huge group of people have a "brainstorm" and then try to incorporate every crack-pot idea there is into the design, inevitably resulting in a ill-thought through mess that compromises on everything in favour of trying to please everyone (and by everyone they mean the committee making the decisions, not the people having to use the application).

Consultants. This is where you pay a middle-man (who knows neither business practices nor software development) to get in the way and cream-off money by protracting the development process with confusing techno-babble and business-speak.

+1 for Vanity Websites. Going into a law firm as a development manager my biggest achievement was actually putting a price on the development of these up front which killed them stone dead (strangely no-one was willing to sign off £100k).
–
Jon HopkinsNov 12 '10 at 10:07

7

Re: (3) "A camel is a horse designed by a committee"
–
JBRWilkinsonNov 12 '10 at 14:00

Hiring consultants (freelance) just to add more production capacity, while they should invest in their own employees instead, by hiring consultants to bring new knowledge and coaching their existing people.

Hiring project managers that manage other project managers that manage other project managers that finally (think) they manage the development team. While they should let the team self manage and focus on business instead. I've seen software projects where they had more project managers than developers. Imagine the meetings.

Sometimes companies need extra production capacity on a temporary basis to overcome a short term demand without requiring skills transfer. This is a key function of the freelancer. If this continues for longer than a few months without skill transfer then your point definitely stands.
–
Gary RoweNov 12 '10 at 9:32

6

In 10 years of consultancy, I never seen that working properly. The Mythical Man-Month.
–
user2567Nov 12 '10 at 9:40

4

@Gary Rowe, the law is "Assigning more programmers to a project running behind schedule will make it even later". But hiring consultants to start a new project because you can't find any permanent staff is 100% valid. I wanted to clarify that. So my statement is about "just add more capacity" to an existing team (of permanent staff).
–
user2567Nov 12 '10 at 9:52

3

My current project is me as only dev and 2 project managers. Yes, the meetings are far from the best I've ever been in.
–
Matt LaceyNov 12 '10 at 17:21

I think its taught in Business 101 to not give employees raises. A secondary case is to limit salaries of star performers because they need to fit inside of some certain salary range.

Eventually employees will realise their pay-scale is not in-line with their industry (or output). The people that have the resume and skills will eventually leave, and take with them all their knowledge and probably a few of their friends. The people remaining (who are the bottom performers) will have to pick up the slack and then spend more time hiring a new person (at market rate). So the company just traded a star employee for JR level one, and just lost all the "savings" of keeping the salaries low.

As this continues, the development team will struggle to stay on par, and will probably get worse and worse until something drastic is done.

+1 for pointing out that talent should be rewarded. Owners of businesses make a lot of money when they sell out, but it is often the talent employed that makes the business worth buying. Owners - pay your talent well. Everyone wins.
–
Gary RoweNov 13 '10 at 1:24

2

Not paying properly for skills = the skills walk out the door. They do this every evening anyhow, just one day they do it for the last time. And the managers wonder why.
–
quickly_nowNov 13 '10 at 4:50

2

I was at a company that cut a $40/head team-building expenses to save on the bottom line. I left shortly after that. That was probably the most expense $40 the company ever saved, cause I'm sure I wasn't the only one.
–
CaseyNov 13 '10 at 9:55

1

Sadly too many people think that paying a pittance is good enough. If they know you're making X, they'll offer X+1 instead of Y, where Y is the average, and then wonder why you leave in under a year there.
–
Wayne MAug 31 '11 at 17:57

This answer is somewhat different than most: not firing an employee soon enough, or stated differently, being overly tolerant of an employee's mistakes habits. These were things I've observed and couldn't do much about as a consultant.

The dev that poorly drove the design decisions of a project that led to its eventual rewrite (it was a complete mess).

The dev that sent sensitive unencrypted data to Google Charts because they thought it would be cool to show a pie chart (was a pie chart a requirement? Nope!).

The dev that consulted with a company in the past and accepted a position with them directly. He did an about face and turned into a prima donna that sought the Technical Lead position and went as far as talking to the lead's manager stating they thought it would be good for him to take over as the lead. Talk about audacity! A lot of devs no longer like the guy and he burnt a lot of bridges within his first 2 weeks as an employee. To top it all off he's a very green dev that only graduated 2 years ago but thinks he's awesome.

A few mistakes are understandable, but when there's a consensus between many devs about someone's attitude or skill level companies ought to get rid of them sooner rather than later.

Several times I witnessed management bringing in consultants for the sole purpose of spending money. Most of the time this happened at the end of the year when they were under-budget frantically trying to spend the money. Usually these consultants would be paid hundreds of dollars an hour and they would spend weeks on a PowerPoint presentation that would never be used.

+1 for the "must spend the budget or it'll be taken away" management anti-pattern. What can be done about that?
–
Gary RoweNov 12 '10 at 15:25

2

"Frantically spending money because they are under-budget"---a friend and classmate of mine, who was working shifts in the physics computer lab, told me they had to spend the rest of their budget or it would get cut the next year. So they bought $5000 worth of new printers, paper, and a scanner, I believe.
–
Mark CNov 12 '10 at 15:55

14

@Mark C - See, that's the way to do it. If you're under budget, and absolutely must spend money now, splurge on gear for your team. Maybe get some new chairs, or dual 32" monitors for everyone, or maybe just a powerful new integration server. If your equipment doesn't fall under the same budget, you'd be surprised what most companies let you get away with as a "team building exercise".
–
InaimathiNov 12 '10 at 17:44

Many companies have one goal - to increase shareholder wealth.
What they produce is irrelevant.
How they produce it is irrelevant.
How much waste they produce is irrelevant.
The cost to society and the planet is irrelevant.

So - go work for or start a company that does something of benefit for society / the planet.

Paying large software companies not only for their product, but for their "support."

I was working at a Government agency for a team that was deep in bed with Oracle. Over the course of many years, they had been paid bajillions of dollars for their software. Coming from a startup background this made no sense to me - "why not use MySQL or Postgres?" I was told that it was mainly because of the support that Oracle provides, if something goes wrong they help you find the solution quickly.

The support was an absolute joke. There was a problem where one web app kept crashing the whole system. It appeared to be a result of a slow database query with a combination of horribly written code (which was written by a team of consultants, which should be a whole other answer). A "task force" (groan) was assembled to pinpoint the issue and fix it. Included on the task force was an Oracle support member. Every day at EOB there would be a conference call where the task force members would update the rest of the team with the findings. It was a long enough call that nobody wanted to be on b/c it started at 5, and the Oracle person just made it worse. Why? Well, saying "person" isn't even correct. It was a number of people. It seemed like every two or three conference calls, the Oracle rep would be somebody new, who explained their predecessor was now on another project or had gone on vacation. The new people were never briefed by anybody at Oracle, so every time somebody new came in we had to waste ten minutes of the conference call explaining the problem again. Their contribution would then be asking for J2EE log files, which not only can any monkey read, but were also useless because the horribly written code was doing things like throwing IndexOutOfBounds exceptions when the programmer found errors in parsing XML.

I know that this is an old question & I'll be lucky if 3 people read this answer, but it's a fun story to tell, so what the hell.

I came into a project (embedded systems, safety-critical firmware, very high stakes) and I was appalled by what I found. People using C (especially pointers) incorrectly, no static analysis, no code reviews, no testing other than "integrate it together, run it, beat on it, see what breaks."

I wrote a very long email my first week there (as a consultant). It was dicey because I was basically saying it was mis-managed, the developers were in over their heads, no process was being followed, etc. It should have gone to the corporate VP, but instead I sent it to the development manager who hired me. He was not entirely defensive about it, in fact he acknowledged many of the shortcomings & told me I wasn't the first to point them out (no kidding, right?)

To answer the crux of the original question: I offered to spend AT MOST 1 man-week getting Gimpel's Lint (PC-Lint / Flexelint) static analysis tool configured & running on their platform, and to run a full report of everything that was found. I told them I was absolutely sure that we'd find several lurking "timebombs" as a result.

They calculated my hourly rate, multiplied it by 40, and determined it was "too expensive to do that." Long story short, I left there within 60 days. About 3 years later, I learned of a product recall, the cost approached 9 figures ($100M), not to mention damage to the company's reputation.

I won't mention the company, the product, or the industry, but I still keep in touch with one of the engineers there, and when he explained to me what caused the recall, my eyes rolled - it was a problem that would have been caught by even a basic static analysis tool (accessing an array out of bounds). In fairness, I cannot say with certainty that the problem was in the code when I was there, but I'm sure if they'd spent the money on some kind of static analysis tool, that bug would not have escaped.

So they saved $295 by not buying PC-Lint (OK, they also saved a week of paying me, at most) - but I'm nowhere good enough to charge $100M for a week.

That's what I call a pretty damn big waste of money.

Reminds me of a joke that many of you may have already heard:

Ever heard the story of the giant ship engine that failed? The ship’s owners tried one expert after another, but none of them could figure but how to fix the engine. Then they brought in an old man who had been fixing ships since he was a youngster. He carried a large bag of tools with him, and when he arrived, he immediately went to work. He inspected the engine very carefully, top to bottom.

Two of the ship’s owners were there, watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away. The engine was fixed! A week later, the owners received a bill from the old man for $10,000.

"What?!" the owners exclaimed. "He hardly did anything!"

So they wrote the old man a note saying, "Please send us an itemized bill."

Bloated development teams and terrible productivity in software companies.

This is a consequence of the common pattern in the business world: the importance of a manager is measured by the number of subordinates, therefore a manager's number one concern is not productivity but quite the opposite: poorer productivity is the best justification to hire more people.

In a company that sold software...giving salespeople full commission on all custom mods sold, so that selling something that already existed and we could just profit on was not nearly as profitable for them as selling one-offs. This was combined with moving the sales staff halfway across the country from the technical staff.

This also meant that we in Development couldn't possibly meet the sales deadlines, making customers unhappy, and had a great deal of difficulty in getting any core work done that would make the product better for everybody. The increased pressure caused code quality to decrease and hurt morale, particularly when we heard stories about the sales office (which I never confirmed).

Lots of us resented Sales, but in fact it wasn't their fault. They were going out and selling as much as they could, doing what they were rewarded for in accordance with the limits that had been placed on them. It was bad management that caused all these problems.

+1. Commissioned sales is a very bad idea in general. (Think about this: How much of the housing bubble would never have happened if it weren't in both the realtors' and the bank loan agents' self-interest to sell people houses at prices they can't afford?)
–
Mason WheelerNov 12 '10 at 18:03

1

@David: That's just the point. Commissioned sales creates an inherent conflict of interest, especially in a product that's sold on debt and not for cash up front. The people who were making that decision were the loan officers, the very ones benefiting from commissions on bad sales. "It is difficult to get a man to understand something when his salary depends upon his not understanding it." -- Upton Sinclair
–
Mason WheelerNov 12 '10 at 18:35

1

Yes, but no one at any level did, because it was making short-term money for everyone. Now, if we had a law that made it a crime to receive a commission payment for any transaction before it had been paid in full, the entire problem would go away almost instantly. Suddenly it would be in the loan agents' and realtors' best interests to give people loans they could afford to pay off, and pay off quickly. Absurdities like 30-year mortgages would disappear overnight, and everyone would be happy except the parasites who cause problems like this in the first place.
–
Mason WheelerNov 12 '10 at 18:44

1

@Xepoch: I am thinking of the effect; I'm just looking to cause different effects than the status quo. Long-term financing is not a good thing. A worker is generally expected to enter the workforce around age 20, give or take a few years, and leave around age 65. If he wants something as fundamental as a home to call his own, he's supposed to be in bondage to a bank for two-thirds of his productive life?!? I don't know how anyone ever got the idea that that was supposed to be a good thing, but I call it a crime against humanity.
–
Mason WheelerNov 12 '10 at 22:07

I've seen a couple of horrible outsourcing projects that succeeded in significantly increasing costs while either failing to increase or actually reducing efficiency.

In the worst case the new outsource team were put in place and up skilled but the existing on-shore team remained in place because the outsource team weren't trusted to actually do any of the critical work.

At this point the logical thing to do would obviously have been to accept failure and shut down the outsource team but because management weren't willing to publicly admit that it hadn't worked both teams were left in place (at a significant increase in cost with no increase in efficiency or usable capacity) until the whole thing could be buried.

In another instance development was outsourced and the original team laid off. Two years later they realised that it hadn't worked and paid to bring the whole lot in-house again only to find that in addition to the very significant costs of another handover, the impact of lost knowledge, recruitment fees, contract terminations and so on, the outsource organisation had lost a significant chunk of the source code.

(Note: I'm not saying outsourcing can't work, just that too many times people are seduced by potential savings and don't consider the realities of their new world, the change to process and working practices and so on which leads to majorly screwed projects)

+1 for outsourcing badly. I've seen one outsourcing team pushed on us by a major investor and everything they did had to be rewritten, another who just didn't do what they were asked (when contractors are asked to produce an MSBuild-based build script, MSBuild wrapping NAnt isn't good enough), and one team who did a great job but they've started on v2 and have been working for us so long they're effectively just really expensive employees
–
JohnLNov 12 '10 at 13:10

1

+1 for the "can't admit failure" anti-pattern (a very sweeping pattern that crosses management and enters international politics but let's not go there...)
–
Gary RoweNov 12 '10 at 17:38

I've seen is the chronic "beating the dead horse" of legacy code. Or more to the point, from the viewpoint of the trenches, countless hours spent in maintenance mode when the whole team knows we should be in replacement mode.

What we've done.... is still on going. Trying to invoke positive change from within

Performance Testing

Simply, not doing it. Again, still working on the positive change from within.

I've been working with a few state institutions and they are amazing at wasting money on IT. From buying bloated middleware to solve extremely simple problems to paying thousands and thousands of dollars to a vendor to have them create a CSV. Without in-house people with sufficient experience it seems they either get fleeced on the upfront cost or on the maintenance.

In non-software companies (banks, insurance) with in-house IT the money comes from various business groups. The business groups directly gets sales pitch from vendors and will push it on to the IT. They are paying for the software/hardware and your salary so your protests will go no where.

Paying for bloated applications and middleware that costs mid-high five figures and doesn't even fit within the existing system architecture

Using expensive software like HP QualityCenter, BMC Remedy, HP LoadRunner etc where better and cheaper options are available

With multi city teams a lot of travel costs, sometimes for only a few hours of meeting

Paying for windows 7 license that somes with new machines and then paying again to downgrade to windows XP as the new SOE (designed in 2010) is still XP

I work in the performance testing profession and I witness (literally) millions of dollars a year being flushed down the drain by organizations for four reasons

Hiring an outsourcer based on price alone, not qualifying skills and not regularly auditing the skills of the performance testers. Hiring an amateur performance tester is a lot like hiring an amateur plumber or an amateur electrician, it will take them a lot longer to work through basic tasks, a lot of checks and balances in process are lost, and when you do find out just how bad they were its horribly expensive to fix (in production). As a moderator for half a dozen forums in this field I observe regularly people showing up who lack fundamental skills in testing, communication, project management, development, systems analysis, etc... and they have simply been thrown at a tool. To the person that noted LoadRunner as a waste of money earlier, if you throw a fool at a tool there is only one result you should expect. The irony is that open source tools require an even more mature skill set to be successful with them.

Not collecting performance requirements. This impacts the whole organization because you will have a different perspective on performance in Architecture, Platform engineering, Application Engineering, Functional QA and performance QA, none of which may actually match the business stakeholders (and frequently doesn't). This is a process problem for in many organizations the performance test team is asked both to collect the performance requirements and to test against them. For proper checks and balances you should do one and not the other. Related to 1 above with immature staff you will have people who cannot even recognize a proper performance requirement, do not have a measurement point to validate against with a load profile, and yet they are still building "scripts to run." This is a collosal waste of time and effort and does little to improve quality. Performance needs a common perspective across the organization and is not something that can just be tacked on at the end if it wasn't engineered in to begin with.

Performance Test Environment Management. I cannot tell you how many organizations are delayed to to test environments not being ready to run at the time that the test organization is ready to proceed. Just in one client I can see this as a multi-million dollar problem in terms of hours lost while waiting

Project managers who have no understanding of what performance testing is, what tasks are involved or the level of effort in place, but who are dictating how long the activities should take place. This leads to variances in the project schedule which are entirely related to how items were scheduled (and cost overruns as a result). This is directly related to 1 above as well for immature testers are not able to accurately project either the number and types of tasks nor how long the tasks should take. It is an axiom that if you allow someone who does not understand what you do and why you do it to dictate how you work and how long you will take, then this path will lead to failure. It happens all too often in performance testing.

Sort of have to disagree: after using a proprietary and expensive Version Control System and then moving on... by comparison, CVS and SVN are horribly horribly sucky. The system I used was expensive, and by heck we used it hard. And it stood up for year after year, simple to understand, use. No experience of Git or Mercurial (which seem to be todays big things), but some of the other free stuff is just hideous. Once you have used quality, going to something lesser is HARD.
–
quickly_nowNov 13 '10 at 10:55

1

@quickly_now - Which VCS was that? I've used a lot over the years & haven't found anything free or paid that I'd use in preference to Hg
–
mcottleSep 1 '11 at 1:12

An organization I worked in was all about weekly status reports - rolled up at 3 different levels. The dev leads and test leads for each of the 4-6 projects in flight report their progress in a lengthy email, which then get rolled up by the next manager up, which in turn gets arbitrarily summarized by the next one.

The following business day, all project leads gather in a 1 hour meeting to go over the report.

Effectively one day each week is spent on reporting that week's progress. Keep in mind that this is all separate from the daily standups and weekly demo / retrospective meetings.

I work for a public body. There's really no way to adequately explain the level of waste that can go on when the workplace is so heavily legislated and unionised that sacking someone is practically almost impossible.

Managers play pass the parcel with bad staff, and hope to remove them all at once under cover of restructuring. Some bad staff are promoted, just to move them out of an area that needs improvement. Any good staff end up struggling constantly just to make up for the bad staff's work. Staff you wouldn't keep for 3 months forge 40 year careers. The amount of money they waste over such careers is astronomical.

I worked in the private sector previously, and saw a lot of waste, but public sector waste is a whole different sport, let alone ballgame.

It was suggested in a comment that establishing sinecures for underperforming staff would help. It would help in that it would limit the damage they could do, but wouldn't impact the root causes of the problem. I think the best thing would be the adoption of some private sector hiring and management procedures, and changes in legislation to make it easier for public bodies to let staff who underperform go. Unions should also change their policies in consultation with the government - their role of protecting their members is important, but they should recognise that sometimes their members are truly out of their depth, and should be moved on

It would help in that it would limit the damage they could do, but wouldn't impact the root causes of the problem. I think the best thing would be the adoption of some private sector hiring and management procedures, and changes in legislation to make it easier for public bodies to let staff who underperform go. Unions should also change their policies in consultation with the government - their role of protecting their members is important, but they should recognise that sometimes their members are truly out of their depth, and should be moved on.
–
Dan ONov 30 '10 at 9:52

One project I worked on with a large financial institution. There were huge amounts of conference calls daily, and I estimated that they burned about $100k per day just on conference calls. The project lasted about 2 years. They had tons of legacy systems and when the daylight savings changes were made a couple years ago, they paid Microsoft about half a million dollars to come up with a DST patch for NT 3.51.

We were having a smallish amount of work and barely making bills and payroll in a small shop in which I worked. The solution: hire an efficiency consultant and a personal secretary for the boss so he could go perform more "meat and potatoes" work.

Solve a budget shortfall by increasing spending...fail.

On the plus side - the efficiency expert supplied a dry erase board where we tracked our billable hours and paid hours...guess who had the least amount of billable hours.

Let's see, we once spent well over half a million dollars doing the work to win a million dollar contract. So much for the profit on that one. Some of us on the project proposal development team tried to point this out but it had become a thing of pride for our little company to win over the Fortune 500 companies we were competing with. We did win and lost money hand over fist onteh contract for that and other reasons but we had bragging rights.

As a government contractor once, I was forced to work unpaid overtime because the contract allowed it and the contractor got paid for my overtime. Not only that I was caught up on my work and spent 4 hours every Sunday surfing the Internet with no work to do. Needless to say I moved on very quickly after they started that nonsense.

Buying Clarity as our project management system, a commercial app that is so bad, 100% of the people who use it have begged to go back to our old home grown system (the one guy who liked and chose it has moved on to some other company), people even volunteered to work on their own time to add the reporting they wanted to our old system. But we have invested the money so we are stuck with it. In other words, refusing to ditch something that doesn't work just because it was expensive.

Sheer waste. An IT spend that had to be cut by many millions. So the way to do this was to fly the IT people in from all over the world. Put them up in a flash hotel for a week. Then in the building where the meetings were held, lay a new floor. Marble of course. And overnight, between the meetings each day, the building was redecorated. Thats every evening for a week.