Today I got yelled at for developing an application on a production server. Quote, "developing on a production server is not acceptable - ever!"

Here is the situation.

I set up a development instance: http://example.com:3000

The production instance is: http://example.com

I complete all my development work on http://example.com:3000 and when the client is pleased with the changes, I move them over to http://example.com.

The application I am working with is an old Ruby on Rails application, and I have to say that initially, I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

Again, am I an idiot? Probably so, but I've been doing web development for a couple of years now, and I have never encountered a situation like this. Who is right and why?

To me, the real problem here is that you have a production system where you have no idea what's running or how it is configured. What would you do if your production machine crashed, losing all its data? If you can't set up a proper development environment now, what will you do when that's your only option?
–
Greg HewgillJan 11 '13 at 4:15

25

Greg is right, if you don't enough about the application to reproduce the environment on a development machine, then not only should you not develop and test on the production machine, you shouldn't even be allowed access to deploy to that machine. You were clearly in the wrong, but I would have yelled at your boss for giving you access in the first place before you fully understood what you were doing.
–
maple_shaft♦Jan 11 '13 at 10:26

2

If you can't setup local development environment, you should not develop at all - ever.
–
JasJan 16 '13 at 13:42

Yes, that is a good point. The more I think about it you guys are totally correct.
–
luk3thomasJan 11 '13 at 4:30

6

There's a third issue: security. What if someone portscans the production server and discovers your (development) application -- and as a result compromises the production application? Again, while this isn't a likely scenario that's pretty much what everyone says after a server or system has been compromised.
–
MarcoJan 15 '13 at 20:58

@Marco Actually that's very likely if the server is publicly accessible. Development servers typically have very bad security (because conveniences like debuggers and stack traces are liabilities when it comes to security). If an attacker can escalate privileges by exploiting the dev server, he/she can completely own the production app.
–
mehaaseFeb 24 '13 at 20:34

As others have stated, coding on the PROD environment exposes your users to your bugs. Even if you've started a different instance, you've still got shared hardware resources and can still access production files and databases. And as some of the comments point out, if your Dev instance gets hacked (for example, because you forget to wipe it and someone then discovers a massive security exploit in Rails), you've now got a publicly accessible machine with your app acting as a gateway in. Which would be... unfortunate.

Different businesses have different responses to this, but it can be generally broken down like this:

Did a screw-up occur?

How long would it take to revert a change (I primarily work in C++, so rolling back a binary can take significantly longer than in Ruby, especially when you've "lost" the old binary and have to recompile)

What the effect of the change (rough guide: screwing up stored data is so much worse than not storing or displaying data, which in turn is worse than not showing the page at all)

If you screwed up then walked out the door, would anyone know what you'd done?

Was there another deployment option that would have prevented/minimized/detected the screw-up before impact?

This gives you the final calculation:

How much would this completely preventable screw-up have cost the business?

This is now how much less your entire management structure is worth to the guy making budget decisions. Hence shouty.

If you're working on the company's internal "About Us" page and typo your own name to be L. "God-like" Thomas, embarrassing nickname problem; if you're working on the business-critical purchasing app, and it ends up accidentally plain-text debugging out credit card details to the homepage... lawsuit problem. Between those extremes lie everything from mischarging, crippling productivity, and all the other things that can drive away customers.

The reason not to allow it even for the "About Us" page is because coding directly in production is addictive. You start by only doing it for the minors, but over time, it's so much quicker not to have to get the DEV env up to scratch.

Beyond that, the size of the business can have a big effect. In a two-man team, when something goes phut, you lean over your shoulder and go "Oi, jackass, put it back". In a 300-person company, you have to start worrying about whether it was incompetence or malice, managers can be held responsible for stuff they had no control over, etc.

At the end of the day, if you follow procedure and screw up, they check what's wrong with the procedure. If you skirt procedure and screw-up, it's now your responsibility alone, even if the blame gets spread out a bit. Whether you want to roll the dice on that is up to you.

I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

I DO support the statements to AVOID development on a production server. You may only be justified to do under the GUN, if it is a typo correction in config file and insisted by your manager.

WHY NOT? - Because, it is very risky and pron to become a habit later on that would catch you up badly. Because, fatal production errors/failures may cost you to be fired from your work.

Let me re-iterate it again, even-though if you requested to do typo correction on production server, first do it on Staging. or in another words test your changes, test it, and again test it before placing in production.

This is something that happens often in places where culture of "do it quick and dirty" seems to be a norm.

Edit: Developing on production server, as a separate environment is also NOT acceptable. Any issues caused in your work may simply bring down the production server, and affect the production application performance. As example, i do remember a case when there was a security hole, when my previous co-worker tried to use production server WinServer 2003 for development purposes.

This is really a protocol issue. Generally, this is not something that you would want to be doing. You want to leave production machines alone. They may contain sensitive data, and you don't want to compromise the performance or stability of production sites with non-production ready code.

That said, there are times where this is commonly done. If you are in a position where you are pumping out low traffic brouchureware or simple CMS sites, this will likely be less of an issue. But if you are working on anything with sensitive data or "business critical" processes, you really shouldn't be risking having development code on the same machine.

Ok, thanks. Can development code make a machine unstable? I am working on an old rails app. It seems to me (naive person) that development code for http://example.com:3000 would not affect http://example.com.
–
luk3thomasJan 11 '13 at 3:16

2

@luk3thomas: well, sure, it shouldn't. What if there's a bug in the webserver or Rails framework, though, that crashes the server? What if, as Jacob Terry suggested in his answer, an infinite loop or memory leak in your dev code sucks up all the resources on the production server and compromises the performance of the live site?
–
Carson63000Jan 11 '13 at 20:41

1

Sometimes it's a requirement. Such as shops that license expensive software and don't have the budget for a second copy just for dev work.
–
Brian KnoblauchJan 16 '13 at 12:52

Another important reason not to develop directly in production is that a development instance will usually produce and show verbose errors and stack traces. You never want to expose that to the user.

Yes, you can log them instead of showing them to the client, but that makes debugging that much less amusing than it already is.

Added Addressing your side-problem of having trouble with your development instance: I've had great success deploying a VirtualBox-based VM which duplicates our production environment (exclusive of hardware, of course) with an Ubuntu Server.

I'm quite astonished no one mentioned the most important reason yet, why it's absolutely forbidden to develop on production servers:

Don't mess with production data, which can happen oh so easily!

A tiny error in one place leads to gigantic troubles in other calculations and then, on the next day, all the data is garbage and the customer is pissed. This is much, much worse than a bug in the UI or a little crash here and there.

For most applications, the value lies in the data and not in the routines.

Another problem with developing on prod is that, sometimes, these things get missed in source control and a future release might undo your quick-fix change.

If you are in a publicly traded company in the US, you should not even have access to prod for regulatory reasons. In general, in no company should a developer have prod access (for the resons stated in all the answers as well as possible legal reasons), so your manager is also in the wrong for allowing you the rights to that server.

Rules that use "always" or "never" are usually ill defined. There will be edge cases where breaking a best practice will be justified. Much better advice will be "Don't touch the production servers unless you have very good reasons"

In my career I have found only two reasons to change code on production servers:

Bugs or behavior that happen only there and are not reproducible on the development environment. These are rare but can be very annoying and hard to find.

Directly fixing critical bug that you just cannot afford to wait to pass trough the normal deployment process if it takes more than a few minutes. After this has been cleared with management. If you are lucky you should only have few of those for your entire career.

Both are best left to senior developers that know the systems intimately.

If you have bugs that only occur on the production environment, your development environment is not close enough to the production environment. You should AT THE VERY LEAST have a staging environment with the exact same configuration as the production environment, down to the exact OS settings, processing hardware and installed software.
–
Nate KerkhofsNov 28 '13 at 10:45