The 25 costliest tech screw-ups of all time

All organizations depend on IT to keep operations up and running. That means tech mistakes – even seemingly minor ones – can have a huge impact on the organization and its bottom line.

Those errors can include everything from security and coding mistakes to faulty judgments and predictions about technology products. And the costs can include money spent on clean-up efforts, legal fines, damage to reputations, and a host of other tangible and intangible expenses.

While the actual financial damage isn’t always possible to measure, these are 25 of the biggest, most famous and costliest tech screw-ups of all time:

1. Outdated software causes massive data breach

In the spring of 2011, Sony suffered several massive data breaches that were estimated to have affected more than 100 million PlayStation Network and Sony Online Entertainment users. Data stolen included personal information as well as credit card numbers. Experts estimated the breach would cost Sony $1.5 billion in clean-up costs — not to mention the damage to the company’s brand after such a large breach. The IT mistake behind the breach: running upatched, outdated Apache web server software.

2. Disgruntled IT employee cripples network, gets himself arrested

A long legal fight began in 2008, when Terry Childs, network administrator for the city of San Francisco, blocked access to parts of the city’s network and refused to turn over passwords to government officials. What was his motivation? That depended on who you asked. Some claimed he was a disgruntled employee intentionally sabotaging his employer, while others felt he was just doing his job by refusing to give passwords to people not authorized to have them. Either way, the city was widely criticized for creating a system where only one person had critical passwords. The incident had significant costs for both sides: In May 2011, Childs was sentenced to four years in prison and ordered to pay $1.5 million in restitution — which is how much officials claimed the standoff cost the city.

3. Malware steals 100 million credit card numbers

In what has been estimated to be one of the costliest data breaches of all time, Heartland Payment Systems, a payment processor based in Princeton, NJ, suffered a major cyberattack in 2008 that led to an estimated 100 million credit and debit cards being compromised. The source of the breach: malicious software installed on the company’s network. Total estimated costs: $140 million.

4. AOL accidentally publishes 20 million web inquiries

In what was intended to be a collection of data for research purposes, in 2006, AOL publicly released more than 20 million search terms entered by 650,000 of its subscribers. Though numeric IDs were substituted for the users’ names, many of the searches contained personally identifiable information, as well as banking and financial data. The information was mistakenly posted to a web page accessible by the general public. After learning about the mistake, the data was taken down, but not before people made copies. It’s not clear if any criminal action was taken based on the breach, but damage to AOL’s reputation and customer trust was certainly extensive.

5. Simple output error takes down $327 million space mission

In September 1999, NASA lost communication with the Mars Climate Orbiter, a space probe that was built to study Mars’ climate and atmosphere. The spacecraft approached the planet at the wrong altitude, causing it to enter the upper atmosphere and disintegrate. A few months later, the cause of the failure was revealed: Ground-based computer software controlling the probe produced output in English pound-seconds instead of metric Newton-seconds, which were supposed to be used.

6. Stolen laptop leads to $25 million breach

In 2006, the names, birth dates and Social Security numbers of 17.5 million military veterans and personnel were stolen in a data breach at the Department of Veterans Affairs. The cause of the breach: A VA employee took home a laptop containing the data without permission. His house was then burglarized, and the computer was stolen. At the time, experts estimated the breach would total at least $25 million, including money for running call centers, sending out mailings, and paying for credit-monitoring service for victims. In 2009, the VA settled a lawsuit and agreed to pay $20 million to victims.

7. Breach at marketing firm has wide reach

Often, the costliest IT security incidents not only affect the business that was attacked, but also customers of that business. That was the case in a 2011 breach when marketing firm Epsilon had 60 million names and email addresses stolen by hackers. Epsilon’s customers included major retailers and banks such as Best Buy, JPMorgan Chase and Walgreens. Some experts estimated the breach could cost Epsilon and its customers between $225 million and $4 billion, depending on what happened with the stolen data.

8. Microsoft releases the doomed Windows Me

Though Vista got a lot of bad press a few years ago, it’s Windows Me, Microsoft’s 2000 release, that’s consistently named one of the worst tech products of all time. Users reported a whole host of problems with the OS, including a faulty System Restore feature that would sometimes re-install malware that had been previously wiped.

9. VeriSign fails to report huge breach

Sometimes during a data breach, the most glaring error is the one that enables the attack to happen in the first place. Other times, the real issue is how the affected organization responds. The latter was the case in an attack against VeriSign, the company that operates two of the Internet’s 13 root nameservers. Though the organization was breached repeatedly throughout 2010, no announcement was made until 2011. Even after the incident was made public, the organization admitted that it did not know as much as it should about the attack. As the scope of the breach is still not known, it’s not clear what the exact costs were.

10. Software bug leaves thousands without phone service

In 1990, a software bug caused an estimated 60,000 people to be without long distance phone service for nine hours. Problems were caused by a bug in a new release of the software that controls AT&T’s #4ESS long distance switches. Apparently, the glitch caused the switches to crash after receiving a message from a neighboring machine recovering from a crash, leading to 114 switches crashing and rebooting every six seconds. Engineers eventually fixed the problem by re-installing the previous version of the software.

11. RSA breach may have left government contractors vulnerable

In yet another example of a breach with far-reaching consequences, the impact of an April 2012 breach of RSA’s SecurID system is still not fully known today. What is known is that two separate hacker groups worked in collaboration with a foreign government to launch a series of spear phishing attacks against RSA employees to penetrate the company’s network. The company reported it spent $66 million on remediation, but claimed no breaches were launched against its customers — which include several government contractors. Many observers don’t buy it, as subsequent attacks on Lockheed-Martin, L3 and others are believed to have been partially enabled by the RSA breach.

12. Google lists every website as malicious

To protect Internet users, search engines such as Google maintain a registry of sites that may contain viruses or otherwise harm someone’s computer. That’s obviously beneficial, but it did lead to one incident in 2009 when Google briefly warned users that all websites were malicious and blocked access to them. Apparently, while updating the registry of flagged sites a Google employee mistakenly entered a line containing just “/”, which the system interpreted to mean as the entire Internet. Fortunately for everyone, the issue was fixed less than an hour later.

13. Vendor passes on the iPod

This screw-up wasn’t the result of something a company did, but rather something it chose not to do. Before Apple released the incredibly popular iPod mp3 player, the gadget’s inventor Tony Fadell shopped his idea around to other companies, including Real Networks. That company turned down Fadell, and he brought the idea to Steve Jobs. The rest is history.

14. Transcription error takes down space probe

In 1962, a different NASA mission was ruined by a technical error. The Mariner I space probe, launched to fly past Venus, was diverted from its intended path on launch, and mission control had to destroy the rocket over the Atlantic Ocean. The cause of the problem: A hand-written formula was improperly transcribed into computer code, causing the computer to incorrectly calculate the rocket’s trajectory.

15. Y2K causes expensive scare

It’s actually debatable whether the Y2K scare should be considered a tech screw-up, or a legitimate response to a screw-up made years earlier. Some say the $300 billion spent fixing the Year 2000 problem helped avoid the massive catastrophes many experts predicted, while others say it was a lot of hype over nothing.

16. Background check firm sells data to cybercriminals

In 2005, information about 163,000 people was compromised after background check service provider ChoicePoint mistakenly sold that information to a group of criminals posing as a legitimate customer. As a result of the breach, ChoicePoint agreed to pay $10 million in civil penalties and $5 million for consumer redress, as well as limit the sale of information products containing sensitive consumer data, including Social Security and driver’s license numbers — which meant a lot of lost revenue for the company.

17. Yahoo! passes on Facebook

In another case of a failed business opportunity in the tech arena, back in 2006, search company Yahoo! considered buying a relatively new social networking site called Facebook, before deciding that $1 billion was too expensive. Of course, the site’s popularity and value have grown rapidly in the past few years, with the most recent valuation coming in at $104 billion.

18. McDonald’s bites off more than it can chew

Sometimes IT projects fail because they’re too ambitious. That was the case in 2001 when McDonald’s planned a massive ERP system implementation to connect its thousands of locations across the globe — including some in countries with a less-than-robust technology infrastructure in place. The company eventually decided to shelve the project, but not before spending $170 million on consultants and planning.

19. System crash spreads power failure

In 2003, a tech error was partially blamed for a massive blackout that left more than 50 million people in the Northeast U.S. and parts of Canada without power, forced 12 airports to close, and caused billions of dollars in losses in both countries. The problem was initially caused by falling trees downing power lines, but the issue was exacerbated immensely by a computer glitch at Ohio-based power company FirstEnergy. After the initial outage, IT staff learned the glitch had caused the company’s alarm system to go offline, and employees thought they fixed the issue by rebooting the system. However, they failed to verify that everything had come back online. It hadn’t, and problems quickly piled up, leading to the massive incident.

20. Sensitive data left on resold copier hard drive

One key component of any IT security plan should be properly dispose of used equipment by wiping all sensitive data off those devices. However many organizations fail to do that with copiers, scanners and other devices that use hard drives that might save sensitive data. One example was found in 2010 after a team of investigative journalists bought a used copier that had previously belong to Affinity Health Plan. The copier’s hard drive contained a wealth of sensitive data about more than 400,000 people, including Social Security numbers and medical information. Luckily, the machine wasn’t purchased by a team of cybercriminals.

21. Botched upgrade makes fraudsters happy

In 2006, the IRS planned to install a new fraud detection system in the hopes that it would allow the agency to better catch tax cheats. In anticipation, the IRS pulled the plug on the old system. The problem: The new one didn’t work yet. When the system was supposed to go live, users generated more than 500 problem tickets — basically anything that could go wrong with the system did. Experts estimate the lack of an effective fraud detection system cost the government about $318 million in lost revenue because fraudulent returns weren’t properly identified.

22. Management dooms FBI’s IT modernization project

Another case of an IT project being abandoned before completion, but not before costing a lot of money: In 2005, the FBI killed off its Virtual Case File project, which was part of an effort to modernize the agency’s IT infrastructure. The project went through five CIOs and nine program managers, costing roughly $170 million dollars. Reasons cited for the project’s failure included rotating management resulting in frequent specification changes, micromanagement of software developers, and the inclusion of too many people with no technical expertise in the planning process.

23. Sony releases exploding batteries

A manufacturing error in lithium-ion batteries manufactured by Sony for companies like Dell, Acer and Apple caused the batteries to combust if they were hit or dropped hard enough. The issue was fairly rare, but as news reports of exploding batteries circulated, recalls were demanded and Sony was forced to replace 4.3 million battery packs.

24. Alleged Cold War sabotage went too far

This tech screw-up was the result of an alleged hacking attack doing more damage than originally planned. In 1982, the CIA allegedly had a logic bomb inserted in the programming code into the control system for the Trans-Siberian Pipeline, a system for exporting natural gas from Russia. Though the alleged attack was supposed to create various disruptions in pump speeds and valve settings, the end result was a huge explosion that could be seen from space.

25. Company’s network breached for nearly a decade

As important as it is to keep hackers off the network, some companies make the mistake of failing to detect attacks that do occur. That was apparently what happened at telecommunications firm Nortel Networks, which was the victim of an attack that lasted from 2000 to 2009. Throughout that period, hackers stole technical papers, research-and-development reports, business plans and other documents. The breach began when passwords were stolen by hackers and not changed until years later, after which hackers had installed sophisticated spyware to keep the data theft going. The incident didn’t really end until the company filed for bankruptcy and began selling off its business divisions in 2009.

Top Trending Resources

About IT Manager Daily

IT Manager Daily, part of the Catalyst Media Network, provides the latest IT and business technology news for IT professionals in the trenches of small-to-medium-sized businesses. Rather than simply regurgitating the day’s headlines, IT Manager Daily delivers actionable insights, helping IT execs understand what technology trends mean to their business.