Today I got yelled at for developing an application on a production server. Quote, "developing on a production server is not acceptable—ever!"

Here is the situation.

I set up a development instance: http://example.com:3000

The production instance is: http://example.com

I complete all my development work on http://example.com:3000 and when the client is pleased with the changes, I move them over to http://example.com.

The application I am working with is an old Ruby on Rails application, and I have to say that initially, I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

Again, am I an idiot? Probably so, but I've been doing Web development for a couple of years now, and I have never encountered a situation like this. Who is right and why?

Follow the rules

This is really a protocol issue. Generally, this is not something that you would want to be doing. You want to leave production machines alone. They may contain sensitive data, and you don't want to compromise the performance or stability of production sites with non-production ready code.

That said, there are times where this is commonly done. If you are in a position where you are pumping out low traffic brouchureware or simple CMS sites, this will likely be less of an issue. But if you are working on anything with sensitive data or "business critical" processes, you really shouldn't be risking having development code on the same machine.

Clean things up for the client

Another important reason not to develop directly in production is that a development instance will usually produce and show verbose errors and stack traces. You never want to expose that to the user.

Yes, you can log them instead of showing them to the client, but that makes debugging that much less amusing than it already is.

Addressing your side-problem of having trouble with your development instance: I've had great success deploying a VirtualBox-based VM which duplicates our production environment (exclusive of hardware, of course) with an Ubuntu Server.

The only thing a production server should be doing is live production work. The most that should ever happen in terms of development would be to do something like deploy symbols and attach a debugger to a running process if you are having issues that can't be replicated in the test environment, but are happening in the production environment.

In the old days most places only had one piece of equipment of big equipment. Development, test and production usually ran off of a single system. Today with hardware and virtualization so cheap why would anyone want to do this. If your business tries to meet iso or sox compliance, you can forget access to develop in production. Now sometimes "fixes" might be tried out in production (after being rvwd and tested) for various reasons data volumes, usage patterns, etc., but broad development access to production seems dangerous and unnecessary.

Disclaimer: I am a sysadmin, so the natural enemy of developers everywhere.

That said, the short answer to the question in the title is "Yes, it is bad to develop on a production server."

The longer answer involves a lot more profanity, some questions as to the legal relationship between your parents, and invitations to perform activities which are technically still illegal in some states. And the revocation of your privileges to said production server (which you shouldn't even have in the first place).

My IT group was nice enough to setup a DB instance that clones production every morning. So instead of developing in production, I develop against that instance, where the data is just a day old. Works well enough for me.

I second sum100 with vms, this should normally never be an issue. However I am allergic to strict rules. Let's say developing on a production server is inadvisable but in a pinch?

After all it depends on a couple of factors.

What happens if the production server goes down? Will a million airline passengers be stranded (don't do it) or will the website of a small online shop be down for 20 minutes reboot time? (Much less bad)

What you are developing and how dangerous it is is another point, an insulated JVM with a fixed upper memory limit will normally not bring down your server. On the other hand if you are developing unfenced C++ UDFs in the production database server I would strongly advise against it.

But given the normal state of inhouse development I would say get the hell off the production server.

As soon as I read "ruby on rails app", I thought, well that explains that. Then I was again proven correct when it was followed by, "I could not get it running locally". Yes, yes, broad paint brush, there are bad developers using every platform... That is true, but its also true that many of the same crowd that wreaked havoc with PHP jumped on the RoR wagon. My experience with the RoR community( short-lived, because I abandoned that shitty platform soon after 1.0 came out ) was one of hostile egos and dangerous people such as the question poster.

In any tight shop, you'd have never even had the chance to touch the production server. Shit, you wouldn't be allowed near anything higher than DEV.

For those developers like the one quoted who "could never get [a local development environment] running", I suggest approaching another tekkie on the team for assistance on getting yourself running locally as developing locally is simply easier and more productive for all the reasons mentioned by other commenters.

If asked, most IT engineers &/or senior developers will happily spend time getting a junior developer online as it can only reduce the chances that they get paged to fix a server crash, right as their hot date is removing that last and most critical piece of clothing...

Disclaimer: I am a sysadmin, so the natural enemy of developers everywhere.

That said, the short answer to the question in the title is "Yes, it is bad to develop on a production server."

I am QA, also the natural enemy of developers, and I approve this message.

I'm a developer who's been around the block more than a few times, so the natural enemy of young developers, and I approve the approving of the approved message.

And I'm the Product Manager who will get raked over the coals (by clients and many internal stakeholders) if that development brings my service down and so I approve of the approving of the approving of the approved message.

I work in a very old, specialist, highly complex, integrated environment where the client is highly risk averse.

I still sit on an XP box and only some of us just moved to IE8 from 6. Agile is being tried in some areas and hopefully from my description you can imagine how it's being received.

Given that, I just want to say how mortifyingly horrible I find the tone of Iscott3's four steps. "Possibly push it..."? Many of our applications have to go through four separate environments before we even consider rolling it live. 8 months from first conception to live for a relatively small change is the norm.

Disclaimer: I am a sysadmin, so the natural enemy of developers everywhere.

That said, the short answer to the question in the title is "Yes, it is bad to develop on a production server."

I am QA, also the natural enemy of developers, and I approve this message.

I'm a developer who's been around the block more than a few times, so the natural enemy of young developers, and I approve the approving of the approved message.

Yeah? Well, I'm a developer who's had to deal with too many idiots who got into a sysadmin, QA or senior development position and I your approving of the approval of said approved message.

That said, kennedye is spot on. DEV, TEST, PROD isn't the default at most companies for no reason. It's far too easy for development code to do terrible things. Things that you want nowhere near your live systems.

If you can't get a working environment set up according to your company's requirements then talk to someone. Get someone, possibly another developer on the project (or doing something in the same ballpark) to assist you and show you what you're doing wrong.

If you're in a situation where the company wants a new dev environment for each new project then discuss with someone ranking about the possibility of setting up a default dev image or virtual machine that can be cloned for new projects. That can save you enormous amounts of time in the long run if you're starting new projects frequently.

>> but I've been doing Web development for a couple of years now, and I have never encountered a situation like this

Depending on the details you're leaving out about just what this Production Server is doing and how critical it is for optimal uptime, you're definitely living on the edge. Just because you have 'a couple of years' of luck doesn't mean squat, if you do something either directly causing a crash or inadvertently creating some kind of buggy behavior, hope your skills at hiding logs and covering up your history are much, much better than your web development.

I work in a small organization with an IT department of three, including myself. I officially the database administrator, but we no longer have a web developer.

I do develop locally on my machine. I use the test server for testing, and a there's a beta site on the same server as production. All of this was created when we had a web development team of three. Now we have none.

Over the last few weeks I've pushed some emergency updates out quickly. I did quick testing locally before deploying. There's a framework to email me any unhandled exceptions that pop up, so I know when things go badly.

Of the three of us in IT, I'm the only one who can write code. I'm the only one who knows enough about the servers to deploy, test, or configure. So to say developers shouldn't have access to production might be a bit too absolute.

This is all assuming your company has less than 12 employees total (not just IT staff). If you've never worked in such an environment, you're really missing out.

I wouldn't recommend recompiling binaries or that sort of thing, but often there are issues that are related to production data being vastly different than expectations, and most places don't push down production data to development servers. The payoff of doing so often enough to be useful usually isn't worth the trouble....

So there you are trying to figure out what is going on and you know it's related to difference in production and development data. In most environments there are ways to add some code in the uncompiled UI sections to get that data you need to solve your problem out... usually to HTML comments or hidden sections so you only people looking at the source at those few moments can see that information.

-- so my answer --

Production investigation, in this special situation where it's a data issue, is perfectly acceptable. If you policies don't allow for this special case, you should strongly consider changing your policies.

I also have no problem with minor UI band-aids on production servers, so long as they fix something a client is complaining about that may take more than a few hours to get released to production. Protocol is never worth losing a sale.

This is not the same as development, which would be a much riskier undertaking. I have not given any justification here for production development.

My IT group was nice enough to setup a DB instance that clones production every morning. So instead of developing in production, I develop against that instance, where the data is just a day old. Works well enough for me.

So you have real data on a development machine? That's going to put you in breach of privacy laws in various jurisdictions.

If it is a legal requirement that dev boxes do not handle real data, sometimes you're going to have to debug (ie. develop) in production.

Sweet Jesus. Not every app is a banking app. Some small companies would just like to be able to afford a simple website without paying the tens of thousands charged by the sort of people who advocate five boxes for every actual live box. They also don't want to pay a grand and wait five months to get a web page changed.

We don't build cars to the same tolerances as space shuttles. And so cars explode more often. But most of us can afford a car.

My IT group was nice enough to setup a DB instance that clones production every morning. So instead of developing in production, I develop against that instance, where the data is just a day old. Works well enough for me.

So you have real data on a development machine? That's going to put you in breach of privacy laws in various jurisdictions.

If it is a legal requirement that dev boxes do not handle real data, sometimes you're going to have to debug (ie. develop) in production.

If SirOmega's IT group is capable enough to automate the daily refresh of a DB instance, they could also easily set up some sprocs to anonymize (strip, encrypt or replace with dummy data) any personally identifiable information (PII) prior to going live in DEV, thus alleviating the privacy problems you mention should that be a requirement of their jurisdiction. We did this for our test environment to create a production-sized database for performance testing and it works great and has passed all security audits.

With a shared server service level, you need a separate user account for development.With a virtual private service level, you need a separate virtual private server for development.With a dedicated server service level, you need a separate dedicated server for development.

But generally its best and fastest if you do development on your local machine (laptop, desktop, "workstation"). If you are resource constrained, you can do testing on your development machine. Upgrade your production server during off-hours, so you don't bear the wrath of users and bosses, WHEN (not IF) something goes wrong.

mr_dee is a smart man. I was fortunate enough to convince our CEO years ago to never store private data on our servers and find a partner to manage that information. The space shuttle / car analogy is perfect. These are all reasons I prefaced my comment as I did.

If you can't even replicate the production environment such that it works, then you have even bigger problems than developing in production. You have a timebomb waiting to explode.

This...

I've been privileged to have been involved with some pretty large site launches, and have seen the good and the bad. Some nasty things can go down in the hands of even experienced devs--and cause the resulting headaches and drama.

Personally, one good rule of thumb to go by is by asking how hard it would be to roll back, and then how costly it would be to have your online presence down for any amount of time.

As others far more experienced than myself have posted--by all means do this as a last resort only. Better to find out the deal killers locally than live. Sure, there may be issues that crop up only in wild that can't be forseen locally, but it's far better to get things straight (and QC'd) before taking the big leap.

I work in a small organization with an IT department of three, including myself. I officially the database administrator, but we no longer have a web developer.

I do develop locally on my machine. I use the test server for testing, and a there's a beta site on the same server as production. All of this was created when we had a web development team of three. Now we have none.

Over the last few weeks I've pushed some emergency updates out quickly. I did quick testing locally before deploying. There's a framework to email me any unhandled exceptions that pop up, so I know when things go badly.

Of the three of us in IT, I'm the only one who can write code. I'm the only one who knows enough about the servers to deploy, test, or configure. So to say developers shouldn't have access to production might be a bit too absolute.

I would argue (semantically) that you have removed your DEV hat and are wearing your OPS hat when working in production in this manner, as your comments indicate compliance with normal SDLC protocol.

That being said, I don't believe that the intent of the other forum members' comments is that developers should never have access to production, just that they not develop and/or do stupid things in production (like turning on verbose logging and then leaving for the evening without telling anyone).

It is quite often the case that developers have read access to prod and in some cases write access to prod if working in an escalation support role in a large shop, or in the normal course of business in a small shop such as you describe.

I've been reading ars for years and never had an account. I signed up for one just now, purely because I wanted to comment on this. That involved setting up another, single use, email address, for registration. I wanted to paint that picture, just so it's clear how strongly I preach this.

Are you an idiot? I don't know you well enough to judge. Are your actions idiotic? Absolutely. I've been doing devops style work for a long time and I can promise you that the number one cause of production incidents is people and a major subset of causes is by doing _anything_ on production machines, by hand. The only time I'm comfortable with people accessing production systems is for break/fix operations where an screw up isn't going to matter as much because you're already broken.

The fact that you're working with an application that you can't mock locally screams that your operational understanding of the components needed to get your app working is limited, at best. You have absolutely no business on a production machine because you've proven that you can't fix it, if you break it.

You do your dev work on an alternate port and show it to the client for approval? I hope what you meant to say is "approval of UI and new functionality but it goes through a strict QA/testing process before we open the firewall to allow access from * to the alternate port." Since I'm sure you didn't mean that, you're telling us that you've opened a world accessible port, with an in development version of a Ruby app (ruby can do many dangerous things, if you're not careful) and you periodically advertise this port to the customer, so they can rubber stamp things. How do you secure it so you're not caught with your pants down and a half developed app with a giant gaping hole for me to stroll through and pwn your system(s)?

At least you're doing periodic checkins of your in development code to your revision control system, so it'll be easy to restore when your machine goes sideways. Right? Right?

I know I'm coming off like a jerk, so I'll stop ranting but these are serious problems and are things that you need to stop doing ASAP. They're also an indication that there are major problems in your development environment as a whole that you should start addressing yesterday.

You have customers. I assume they pay you money for things. How much money are they going to be willing to pay when they hear "some dev guy was monkeying around on a prod box and exposed all your private info via an SQL injection hole"?

Find an ops person, have them mock the environment elsewhere (AWS is incredibly cheap these days) and play there. Do proper QA and testing _before_ doing a customer demo. Have a build and deployment pipeline (Jenkins is free an easy) that releases versions to prod. Stay out of prod!

That being said, I don't believe that the intent of the other forum members' comments is that developers should never have access to production, just that they not develop and/or do stupid things in production (like turning on verbose logging and then leaving for the evening without telling anyone).

It is quite often the case that developers have read access to prod and in some cases write access to prod if working in an escalation support role in a large shop, or in the normal course of business in a small shop such as you describe.

If you don't think that was their intent, then let me be clear that it is exactly my intent. Developers have no business in production. In fact, I don't think Operations folks should have access to production, except in emergency situations and no one should have direct access to login as root, without going through at least one out of band hurdle (I like the physical key to a fire safe with a sealed envelope containing the root password and mandatory rotation once unsealed). That being said, this is a hard model to actually achieve but I think it should always be the target.

If you don't think that was their intent, then let me be clear that it is exactly my intent. Developers have no business in production. In fact, I don't think Operations folks should have access to production, except in emergency situations and no one should have direct access to login as root, without going through at least one out of band hurdle

It really depends on what uptime you would like to achieve and the $€£¥ you can spend on it. Sure, you can have a team for each steps of development, testing a production. Then multiple redundant frontend and backend system, with scheduled offline backups...

But try to think why should one not also have a development site on a shared web server with a production website.

The simple answer is that whatever policy the company you are working with sets is ultimately the correct way to do it. Even if it's stupid.

There are many unknowns about the environment in question, but some inference can be made from the fact that it's a Ruby on Rails application. For example, this clearly doesn't involve compiling binaries - it involves web-facing scripts implemented through a mature full-stack framework. That narrows the scope of potential negative server impacts considerably.

In setting policy, it is a balance of available resources and the nature of the application - both in terms of the software being developed and how the server is configured and operated in daily use. Superficially, a policy whereby coders must configure a local environment to duplicate a reasonable facsimile of the production environment for themselves as a precursor to working on code is a potentially problematic approach. Before live deployment, it is basic diligence to ensure an application will operate as expected within the production environment. Moving from a local-machine development environment (XAMPP/windows in my case) directly to production is a *way* worse idea than creating a development/debug instance on a production server.

If the resources are available, obviously you run two identical "production" servers and do final QA in total isolation and safety - secure in the knowledge that the environments are *almost* certainly analogous. However, in many cases that is prohibitively expensive ... and in many it genuinely is unnecessary overkill. Even if a script loops-out (probably the most drastic likely-case in the current scenario), unless a developer is sitting there running their broken code over and over and over, the real impact on a server used for production should not be remotely critical. There are plenty of scenarios where the balance of equities favors simply making a debug instance.

Usually, a strict "deployment-only" policy for production servers is accompanied by a policy of providing a development server on which to prepare for deployment. A company that doesn't explicitly provide a development platform should probably put more effort into organizing a core suite of tools and configuration files to get their team up and running with a common framework rather than yelling at their coders on the back-end for solutions reached as they were left feeling for a way in the dark.

My IT group was nice enough to setup a DB instance that clones production every morning. So instead of developing in production, I develop against that instance, where the data is just a day old. Works well enough for me.

So you have real data on a development machine? That's going to put you in breach of privacy laws in various jurisdictions.

If it is a legal requirement that dev boxes do not handle real data, sometimes you're going to have to debug (ie. develop) in production.

If SirOmega's IT group is capable enough to automate the daily refresh of a DB instance, they could also easily set up some sprocs to anonymize (strip, encrypt or replace with dummy data) any personally identifiable information (PII) prior to going live in DEV, thus alleviating the privacy problems you mention should that be a requirement of their jurisdiction. We did this for our test environment to create a production-sized database for performance testing and it works great and has passed all security audits.

You've missed the point. Your two environments are now not the same. The data is DIFFERENT. Personal data such as ... ooooh ... names have been changed. Maybe a bug is caused by a name like Peter O'Toole ...

Does your anonymisation routine cater for the infinite variety of human names and any and all other edge cases?

If you can't even replicate the production environment such that it works, then you have even bigger problems than developing in production. You have a timebomb waiting to explode.

Especially with Ruby-on-Rails where there's a lot of peeing in the same pool by applications on the same machine (unless you're carefully using something like RVM). At some point you're going to need to add or update a gem for a new feature of your development code, or make some kind of configuration change that's system wide, and at that point the same weird combination which happens to keep your production server running is going to change and it's going to break production. Then you're going to really regret not figuring out what made the production server work when you're trying to rebuild it and have no template to base it on. As Peter said, the key thing is being able to replicate the environment, it's the most important thing you can do.

No, it is NEVER acceptable. If your employer says it is, find a new job ASAP.

The first time you mess up prod... and you WILL at some point, it's not the stupid protocol you dutifully followed that will get blamed, it'll be YOU. Don't allow yourself to be in such a situation.

Another thought: you said you couldn't get a local development environment set up. Frankly, this seriously calls your skill into question in my mind, which makes me want to keep you away from the prod boxes under Any circumstances.

Either of these reasons is enough to say never develop on a production server, the former is a general reason why not, the latter specific to you... but they both lead inexorably to the same answer, which happens to be the only non joke answer anyone but a rank amateur should EVER give to this question.

Disclaimer: Since 12 years ago, I've worked in everything, from dev-duction environments, up to SOx compliant, multi-tier deployment environments.

To borrow a meme from cycling:

There are three types of developers in your situation: Those who are going to hose production. Those who are hosing production. Those who have hosed production.

Trust me, you don't want to be known as the guy who hosed production. At least spend the effort to get a dev environment set up on a separate machine.

In fairness, there are two more critical categories you need to add to the mix:

-Those who are going to hose production by uploading something that worked on a separate dev machine without testing in the production environment.-Those who have hosed production by uploading something that worked on a separate dev machine without testing in the production environment.

If it is a question of either/or, in the absence of a viable production-equivalent machine on which to test, I suspect the risk of killing production with a bad upload is far more likely than a Ruby on Rails app doing something terrible in it's own sandbox on the machine.

Another thought: you said you couldn't get a local development environment set up. Frankly, this seriously calls your skill into question in my mind, which makes me want to keep you away from the prod boxes under Any circumstances.

He (assuming "luk" is a man) said it's an old app. Maybe it requires a mess of obselete packages tied together with a brand of string that no one either sells or supports. Exactly how much time and money do you think he should spend trying to get a development local copy working? Exactly how much time did you spend thinking about such business issues before calling him unworthy?

Try being a little more business-oriented and realistic. Nothing is perfect, and the pursuit of perfection is prohibitively expensive. Anyone touting "NEVER" will soon show themself to be a risk to the business, either through their insistence on the unattainable, or through their inabaility to play nice with others.