Similar to the question I read on Server Fault, what is the biggest mistake you've ever made in an IT related position. Some examples from friends:

I needed to do some work on a production site so I decided to copy over the live database to the beta site. Pretty standard, but when I went to the beta site it was still pulling out-of-date info. OOPS! I had copied the beta database over to the live site! Thank god for backups.

And for me, I created a form for an event that was to be held during a specific time range. Participants would fill out the form for a chance to win, and we would send the event organizers a CSV from the database. I went into the database, and found ONLY 1 ENTRY, MINE. Upon investigating, it appears as though I forgot an auto increment key, and because of the server setup there was no way to recover the lost data.

I am aware this question is similar to ones on Stack Overflow but the ones I found seemed to receive generic answers instead of actual stories :)

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

Haha I think everyone with some SQL under their belt has done this at some time, me included
–
billy.bobOct 25 '10 at 11:55

3

Which is why updates should be done on dev first! No hot fixes directly to prod. And be very grateful for audit tables if you have them when you do this.
–
HLGEMOct 25 '10 at 15:05

13

I did that once to reset someone's password. After I set everyone's password to the same thing, I simply told my boss that tech support would be getting a lot of phone calls and that we should tell the caller we had to change their password for security reasons.
–
Barry BrownOct 25 '10 at 16:35

See the error? It's the semicolon at the end of the else if. I couldn't figure out why my removed comments weren't removing. Dug into the database, the AJAX request I was using, POST variables, had error_log's and alertseverywhere. Ended up re-writing the method and it worked. Then I did a diff with the original version and noticed the semicolon.

Typo's are the hardest to track down on a language that isn't compiled or pre-checked by something. Even errors like this would be difficult on a compiled language. Change a == to = and suddenly you have assignment inside an if.

@Rogue Coder: the difference between single and double quotes is vanishingly small. It's a misuse of time to spend brain cycles worrying about it. See the test at the bottom of phpbench.com
–
Joeri SebrechtsOct 25 '10 at 9:54

2

@back2dos: Errr... right. So you recommend dropping all those mentioned languages in favor of haXe? Something that first appeared in 2005? Go right ahead with that.
–
Josh KOct 25 '10 at 15:06

10

Thank you for the argument in favor of the One True Brace Style. +1
–
eswaldOct 25 '10 at 17:33

When I first started I too had this thought. Then I learnt there is far more maintenance of other peoples than anything. That's why I try now to write the best code (with comments) that I can.
–
fortheworldFeb 11 '11 at 4:50

2

There's still plenty of new code to write. A lot of open source projects require coding from scratch that implements the same functionality as a closed system.
–
PP.Feb 11 '11 at 8:31

Accidentally deleted the database that stored all our customer info, order history and invoices going back to the start of the company (several years worth).

To be fair, my employer has to share some of the blame though. They had the only copy of that database stored on the Mac SE (yes this was a long time ago) that they gave me (a brand new employee, first job out of college) as my workstation and had never even considered making a backup.

So, thinking it was a copy of the DB I dragged it to the trash can. Because of the size of the file it deleted it immediately. We eventually got it back after paying an ungodly amount to a data restoration service, but for about 5 days we couldn't fulfill or bill any orders, and had no way to access information about any customer. It pretty much brought the (3 person company) to a standstill.

There was enough blame to go around. I was just a kid back then and probably more naive than I would be today about assuming that a real company would do anything so silly as putting the only copy of the DB on my desktop. Experience has shown me that it is foolish to overestimate my employer's preparedness, even at a big company.
–
JohnFxJan 14 '11 at 23:34

If you got into programming for the money, you should get a new career. Not that the money isn't there, it is, but you need the obsessive love for programming that true developers have, or you will burn out very quickly
–
johncOct 26 '10 at 0:02

1

You made a good point. I really did not get into programming for the money, but I really thought that it would be a lot more easy to live from it. Programming is my passion and I would do that even if I have to pay for.
–
Marcelo de AguiarOct 26 '10 at 16:42

4

@johnc same would go for most professions. I honestly don't understand people who go to university to "learn programming". I started when I was 14 with K&R. I went to university to "learn electronics" - but guess what, I ended up as a programmer anyway. The very best musicians taught themselves. The very best programmers taught themselves. That's life.
–
PP.Feb 11 '11 at 8:30

Great example of when defensive programming would have helped prevent major issues. If you're doing something where the repercussions could be grave, like perhaps dropping tables/databases, default the user response to "No" so that they must ACTIVELY agree to the action.
–
HugoOct 25 '10 at 3:22

5

@Hugo: +1. I had a professor at uni (ex Unix/database sysadmin) who used to say "whenever you are doing anything serious, always take your hands off the keyboard and sit on them for about ten seconds and think about what's about to happen."
–
Bobby TablesOct 25 '10 at 5:00

1

@Hugo: I go farther--"You are about to DESTROY a complete database! If you really want to DROP DATABASE xxx, type DELETE:"
–
Loren PechtelJan 14 '11 at 5:31

3

@Hugo @Loren Pechtel - "You are about to DESTROY a complete database! A web search of your login has turned up multiple occurrences of 'lol', and therefore this action is being denied. Please see your administrator or learn to communicate."
–
detlyJan 14 '11 at 6:18

Not sure it was my "biggest mistake", but definitely a memorable one. In my first week or two at a new job, I was assigned to do a small "feature enhancement" that involved modifying the sort order of some items on one of the most popular pages on the site. I wasn't too familiar with the code base, but quickly found the relevant Comparator and added calls to a couple harmless-looking methods inside the compareTo method without thinking too much about it. The code tested locally and in QA w/o issues and went live.

I had a 1 on 1 with my boss that same day and with a big smile on his face, he congratulated me on getting my first "feature" live so soon after joining. We both looked on as he visited the corresponding page to see the feature in action. The page took 47 seconds to load. My heart sank. He refreshed the page: 53 seconds. His smile vanished. I went back to my desk and spent a long evening debugging and pushing critical patches to the live site.

It turns out one of the harmless looking methods I added was a remote service call that resulted in at least one DB hit. In each compareTo call. So on a page where 2,000+ items were being sorted, I was making ~6000 (nlogn) DB hits. Ouch.

The customer wants a mailout to their entire userbase that will basically send a personalised email about an event and when they click on a link in the email will automatically log them in to a form with half their data filled in.

I write the code, test the code, it works fine, the links work, it's pretty smooth. Having checked and double checked everything, I switch over to the live list and away we go.

The live list is significantly bigger than my sample data and the mail server falls over. I have to stop my console app, then realise I'll have to restart it from where it stopped. Fortunately this feature is not too hard to add and I have logged what users the mailout has already gone to, so having got the mailserver up and running again, and reconfigured the code to be able to restart from the point it failed previously and to pause to let the mailserver catch up, away we go.

Unfortunately I had somehow managed to not get all my code updating right. I don't remember the details of what I did, but the query to get the user data was getting one set of data, the query to generate the hash was getting another, so if a user clicked on the link they would get to a form that contained someone else's personal data from their account details. And this in a niche industry where there were lots of competetive small businesses on the list.

It didn't take long for the phone to start ringing at the customers' office...

That was the only mistake so far that has lead to me offering my resignation.

Actually your biggest mistake was in not developiong against a database of the appropriate size. Never use a small test datbase to develop for a large production database. You can't write performant code unless you have the right sized dabase.
–
HLGEMOct 25 '10 at 15:08

Allowed my manager to brow beat me into leaving a job I loved. I should have let him mess things up, get fired and then I should have stepped up and clean up the pieces.
Instead I quit and have regretted it ever since.

Managers rarely get fired over such things. To get to manager level you have to be very good at covering your rear; you probably became the scapegoat the minute you left the company. Maybe you could have handled it differently, but don't beat yourself up over it.
–
Mark RansomOct 26 '10 at 4:41

Seven years ago my boss & company owner, whom I'd not yet met I was so new to the job, was in another state doing a demo of our research web site to a couple of potential clients. Memberships fees ran in the low to mid five figures, so these demos were a big deal to our fledgling company.

In the middle of his demo, as I was doing some database work, I made a change to the live database and updated every survey question in every survey to the same text, something like "This is a test question" because of a flaw in the WHERE. My co-worker and I scrambled for the backup and cringed, hoping he wasn't demoing that section of the site right then but waiting for the email in all caps that gratefully never appeared.

I was running Xenu's link sleuth on our intranet to try and clear up some of the many broken links that have built up over the years (most, unsurprisingly enough were links from the wiki to the shared network drive).

After about 30 minutes I started noticing some of the new items had some odd pictures, it looked like people had used some of the grainy old stock photos that were on the system but I ignored that thinking that maybe there just wasn't anything newer that had what they wanted.

Another 10 minutes later I noticed the featured items were changing randomly for some reason. At this point it dawned on me what was happening. The intranet uses windows authentication and some of the functions (such as selecting news images and featured items) were coded to response to HTTP GET requests. The link checker had been using my authentication and had crawled to pages on the admin side, loyally doing its job and following everylink it found including ones like

So far my only stomach-twisting, heart-rate-increasing, suddenly-get-hot-and-itchy mistake was running an UPDATE query without a WHERE clause. Luckily there was a backup from the last 15 minutes and the data didn't change very often which meant I was able to fully restore the data within 10 minutes without anybody knowing.

Nothing too bad but I've had some close calls. They are and remain close calls because I've heard enough horror stories in this industry that everything I do that could screw things up gets a good going over.

@Char, I have a personal preference for maintaining all DB interactions within transactions. Those who don't, they carry risk or have business cases which don't care. The specification assumption and MANDATEs are that we will begin and maintain code projects with transactional DB management.
–
XepochJan 13 '11 at 23:31

For a robot I work on, we keep a log file of every run. We have the robot and a simulator, both which generate these log folders. The simulator's log files are useless, but the robots log files are very useful and are stored forever.

Because the robot itself has limited hard drive space, they're transferred to another computer and stored there. This computer happens to be the main operating computer for communicating with the robot.

Well, I had just started working on the robot a couple months earlier and didn't know this. I thought the logs on the operating computer were the useless ones generated by the simulator and deleted them. We lost all of the old logs because there wasn't a proper backup.

The biggest mistake I ever made was not having my work in source control off of my development machine. My hard drive crashed and I lost weeks of work. It's a lesson hard-learned and something I'll never let happen again.

Built a version of our software on my computer instead of the build computer as it was faster (about 4 hours faster, which meant that it could go into QA that day instead of the next) and the publisher wanted it asap.

I had a special define for debugging on my computer which caused a bug that wasn't detected in QA. They only detected it at the end of 4 weeks of validation testing, but it was bad enough to fail validation.

Drop a production database by accident instead of a development version. I didn't see what server I was working. Fortunately, the last backup was produced 4 hour ago and the user didn't made a lot of changes on the system, so no big data loss.

I told my Director that just because he is OLD and has more years of experience than I have does not mean that I will respect him. The only thing that I would respect is ABILITY. I didn't have to face the consequences of such a pathetic statement may be because I was a junior Programmer than. Looking back this seems to be my bisggest mistake. Where was my common sense\humility ?? :-(

I agree partly with you, just because someone has more experience (in years) doesn't mean that (s)he's a better programmer. There are a lot of programmers out there, which can write 'xx years of experience' on their resume but have no clue what they are doing.
–
BobbyOct 25 '10 at 12:28

He was fixing a bug that occurred when you send an SMS to a lot of cellphones. Usually these messages won't actually be sent to save on costs but due to a misconfiguration in our database they did get sent.

On a manager's request, I copied /etc/sudoers from one machine to another (even though it wouldn't have fixed the problem at hand, but that's beside the point). Unfortunately, I used sudo to move the copied file into place, without noticing that its owner and permissions were completely wrong. At that point, nobody had a root shell open, nobody could sudo, and nobody could log in as root because its shell was set to /bin/false. And the machine was in a remote data warehouse...

I wanted to create metadata files for each file in a directory (i.e. for every file somedir/foo.bin, I'd want to create a file somedir/foo.bin.meta). For some stupid reason I decided to create the files by opening and closing a file stream via Python:

for fn in os.listdir(path):
open(os.path.join(path, fn), 'w').close()

For some even stupider reason I thought it was clever to test this script against a directory on my harddrive with several gigs of personal data on it. Only after running it I realized that I had forgot to actually modify the filename prior to passing it to open and that I had just truncated every single file in that directory (ouch).

Luckily that was on a personal computer and didn't cause any damage to anything important, but I still learned my lesson.

I was working for a company who handles one of Sweden's biggest telecom companies customer support. On our server we had some software that regulated the incoming call queue, how many calls we could take and so forth. If it wasn't working we weren't getting any calls and customers weren't getting any support.

Aaaaanyway, I did a (minor) change to the software. I could have waited for a service window to change it but I thought "Hey, it's going to take one minute to restart the service, why bother. Fortune favors the bold". So I restarted it, unfortunately the thread was locked so I couldnt replace it. Starting to panick I decide to quickly reboot the machine (I was remoting in). Unfortunately in my panic I click "install updates and shutdown" instead :P

Needless to say there wasn't much customer support in Sweden to be had for the next half hour before we managed to restart it.

A colleague was worse then me though, he we debugging the automated voice response system one night and rerouted the number to his mobile. Only thing he forgot to switch it back, had the day off the next day and left his mobile at work. We we're wondering why it was ringing constantly that day :)

Definitely TSQL mishap - too many TSQL statements in one editor window. Was a bit tired, and people were interrupting me left and right, and I issued a command that updated 47,000 records address, city, state, and zip - oops. Had it fixed in about 25 minutes, though.