Today I got yelled at for developing an application on a production server. Quote, "developing on a production server is not acceptable—ever!"

Here is the situation.

I set up a development instance: http://example.com:3000

The production instance is: http://example.com

I complete all my development work on http://example.com:3000 and when the client is pleased with the changes, I move them over to http://example.com.

The application I am working with is an old Ruby on Rails application, and I have to say that initially, I did try to set up an development environment locally, but I could never get it running. After trying for a while, I gave up and decided to develop on the production server.

Again, am I an idiot? Probably so, but I've been doing Web development for a couple of years now, and I have never encountered a situation like this. Who is right and why?

Follow the rules

This is really a protocol issue. Generally, this is not something that you would want to be doing. You want to leave production machines alone. They may contain sensitive data, and you don't want to compromise the performance or stability of production sites with non-production ready code.

That said, there are times where this is commonly done. If you are in a position where you are pumping out low traffic brouchureware or simple CMS sites, this will likely be less of an issue. But if you are working on anything with sensitive data or "business critical" processes, you really shouldn't be risking having development code on the same machine.

Clean things up for the client

Another important reason not to develop directly in production is that a development instance will usually produce and show verbose errors and stack traces. You never want to expose that to the user.

Yes, you can log them instead of showing them to the client, but that makes debugging that much less amusing than it already is.

Addressing your side-problem of having trouble with your development instance: I've had great success deploying a VirtualBox-based VM which duplicates our production environment (exclusive of hardware, of course) with an Ubuntu Server.

120 Reader Comments

Welcome to other people. Discussions wander. Or are you some weird kind of conversation facist? And really, making a flip-flop joke means I have personal issues?

Yes, they do... and they tend to more when someone realizes their arguments are ridiculous. I'll note you were the one that took the conversation in another direction. You can draw the line from A to B I suspect, right Mr. Physicist?

mr_dee wrote:

Hi. Theoretical physicist here. There is no such thing as "facts" or "truth". Just data and theories. Sorry about that.

And while we all agree that "dev-in-prod is bad", no-one of even passing intelligence would seriously classify that as "a fact".

So, you're thesis, Mr. Physicist, is that there is nothing that is objectively true? There are no facts and no truth? All things are just data and theory? Hmm... so, people don't have two legs, generally? The sky is not typically blue? The Earth does not actually revolve around the Sun? 2+2 does not, in fact, equal 4? These are all just data and theories? Interesting. I believe you're just started a scientific revolution! All of mathematics will have to be rewritten for a start! Amazing! Have you informed all your colleagues yet of this fantastic discovery of yours?

You're either wrong, a liar (read: not a physicist) or the worst physicist in the history of science. You're choice.

mr_dee wrote:

Both are renowned for being unreliable indicators. What was your point?

I shouldn't think I'd need to explain it to you, Mr. Physicist, but fine: shared knowledge, experience and collective agreement on best practices is how the human race makes progress in most cases. Sure, there's the occasional "you're all wrong, I'm right, now do it this way" that causes a paradigm shift, but most of the time it's "hey, here's how to do X, now let's iterate on it to learn to do it best"... and at some point the method to do X, M1 let's say, becomes M2, then M3, each time a little better, and at each point in time, Mx is deemed the "best" way. That's reality.

But, you're a theoretical physicist, so of course you know this.

mr_dee wrote:

Actually, in your first post you opine "[d]on't allow yourself to be in such a situation" in reference to your own hypothetical example - "it is NEVER acceptable. If your employer says it is, find a new job ASAP" - and this doesn't apply to Luk. You never actually state that HE should leave HIS ACTUAL job. What you call "picking at words" I call "accuracy", "reading comprehension", and "having a clue".

And I, and the rest of the world, call it being willfully ignorant, Mr. Physicist.

If we're having a conversation about guns, and you have a loaded gun to your head, and I say "you should NEVER shoot yourself in the head, that's NEVER acceptable", should you go ahead and pull the trigger because I was talking theoretically and you didn't realize it applies to the situation we're in, the conversation we're having?

Well, at least I won't have to talk to you anymore I suppose.

Did you really, for a second, have any doubt I was referring to him as well? Did you really, for a second, not understand that the hypothetical applies to reality as well? If so you're simply proving my point further. You either have no ability to comprehend or you're being willfully ignorant, Mr. Physicist.

mr_dee wrote:

Yes, quite obviously. So what the hell was the first 90% of your post and your previous posts about?

Come on, you know what it was about. You had a problem with the "harshness" of my post. You had a problem with the "absoluteness" of my post. That's what we've been arguing about. That's what you've been wrong about all along, Mr. Physicist.

mr_dee wrote:

You need take something down from a website urgently for legal reasons. Perhaps you need to disable a service because it infringes someone else's copyright or license. Someone else has royally screwed up and now it's down to you to make sure the business doesn't get sued into oblivion. Do you a) make the business go through dev, testing, QA, and god knows what else, dragging it out to several weeks if you can, making sure all t's are dotted and i's crossed, thereby ensuring an expensive lawsuit, negative press, devalued stock, etc, just because "NEVER" or b) just pull the service and patch around the missing tool, taking about twenty minutes to do so, before anyone outside the business becomes aware of the error?

You know perfectly well that's not what we're talking about here, Mr. Physicist. Emergency fixes are just that: emergencies. They need to be dealt with immediately and we all know that. They need to be handled differently. I never said otherwise.

We're talking about DEVELOPING on a PRODUCTION server, THAT is what this whole thing is about. That's a very different animal.

Ugh, I'm so tired of repeating myself for your benefit. It's wasted effort, I see that now.

mr_dee wrote:

Your company is about to launch an illegal lottery (yes, again someone else has screwed up, and you've stepped into the breach). Do you a) take a month or two to follow every single best practice you can think of, or b) save people from jail time and fines and leave them with a workable last-minute alernative that saves them a lot of face and the brand, and only just manage that by the skin of your teeth because you cut every corner you could?

I chose b) in both scenarios and earned myself fame and fortune and the love of many a fair-haired maiden. So your years of experience ... just cost two businesses. I know, I know, it was their fault for being broken in the first place.

See above. I already answered this. Except I'll add that it's my years of experience that likely helped avoid such a situation in the first place because the type of process you seem to loath so much is what in most cases stops things like this from happening. That's why it's a best practice, Mr. Physicist.

mr_dee wrote:

Would you like some salt with that hat?

No need. You haven't even remotely made it necessary.

mr_dee wrote:

Right. NEVER dev in production, except maybe tomorrow. Or yesterday. Except unequivocally, absolutely never. Except when it changes. Hilarious. Leave the thinking to the thinkers, okay?

Wow, you really are just an obnoxious...

...you know what, Mr. Physicist? Never mind. I won't even bother insulting you. I'm done. And, you get the final word! I won't reply to you again. Have at it. Claim intellectual victory, take the final shot, go right ahead. Hell, you've in fact EARNED it: I fell for it, haha, joke's on me, I got sucked in by your inane ramblings. My bad, lesson learned. I've fed the troll far more than I ever should have and I'm done. Have a nice life.

When I saw port 3000, I knew what was it going to sound like. It is not bad if you are only doing static websites. With more complicated stuff, you are surely setting yourself up for an epic FAIL in the long run.

Brace yourself, the Rant is coming.

<rant>I speak from experiences working in-house with a group of over 20 developers where we thankfully have source control, local and test environment to handle before actual live deployment to multiple servers (and no, we don't just FTP the code to live servers). Every deployment requires committing a new revision and the servers pull the latest updates from trunk version (not automated yet, but it's rather easy to set up with Jenkins). So things are rather smooth until the bugs appear. A serious bug was still very difficult to track and re-produce in the local environment due to our hybrid infrastructure. Think cache and message queues sitting in front of relational databases.

Our system engineers already did a lot to minimise the inconsistencies among local, test and live environments to make software people' jobs easier so that we could focus on what we do best: code. Let's say that if we know that the bug is definitely in the code, imagine tracing it by going through the commit logs versus running like a headless chicken trying to figure out who did what and when with teams having their own release schedules. Let's not even mention the times when you are so sure that the code works correctly in your environment but breaks when going live, how many factors could have been at play? It is everyone's part to reduce as many variables out of that equation as possible. The less you have, the easier it is to fix bugs, which would take up time cumulatively faster than you think.

</rant>

My point is, before anyone starts pushing code, they should have their own playground first, with or without the sysops' help. If you can just grab a VM image from them and fire it up in your workstation, that's fine. The damage you could do out of pure ignorance could be tremendous (I myself was responsible for a not-so-short downtime during my 1st month). So to anyone who thinks coding in production is fine, take my advice and at least put out a test environment as close as humanly possible to the live one and please test at least once before going live, even if you are a one-man shop doing one-page website (actually, one-page app could be pretty complicated beasts, now that I think about it).

> If you can't even replicate the production environment such that it works, then you have even bigger> problems than developing in production. You have a timebomb waiting to explode.

What if replicating the production environment costs real money (monthly fees, setup fees)?

Our product communicates with several dozen external systems from other companies. Some of them do requests to us via preconfigured URLs. If we want a separate URLs for staging or development we have to pay for more accounts, manage more VPN tunnels etc.

I soon gave up doing all locally. Maintenance was too complicated: Callbacks going to the wrong system, VPNs removed by the other companies because they were not used for some time...

Now we build locally and deploy to the same linux box where production runs but use several instances of a Java servlet container.

I'll note you were the one that took the conversation in another direction.

When did I do that? I've been quite consistent in my remark that "NEVER" is a ridiculous construct to live by. I have of course used metapohr, example, jokes, other words ....

fzammetti wrote:

So, you're [sic] thesis, is that there is nothing that is objectively true? There are no facts and no truth? All things are just data and theory? Hmm... so, people don't have two legs, generally? The sky is not typically blue? The Earth does not actually revolve around the Sun? 2+2 does not, in fact, equal 4?

Oh dear.

(a) "The sky is blue". The sky isn't blue, it appears "blue" to the human eye, although "blue" is a subjective concept and people disagree as to its perceptual quality (or quale). And the ancient greeks thought the sky was bronze.

We do of course have Quantum Electrodynamics, a very successful theory supported by lots of data. At no point does it concern itself with crass statements like "the sky is blue". Unless you think Brian Cox is where it's at science-wise.

(b) "People generally have two legs". In other words, people have two legs except when they don't. That's what you consider a truth? Something of use to science? We call that "trivial" and "redundant".

(c) "SSO + SSO = SSSSO". It's a construct and only true by definition. Axiomatic truth is not semantic truth. Just because you can count things doesn't mean numbers exist. Plato and all that. It was pretty much a given that you'd seek recourse in maths. Got any other abstracts you'ld like to reify?

fzammetti wrote:

These are all just data and theories?

Er, no. None of those were either experimental data, or coherent theories.

fzammetti wrote:

Interesting. I believe you're just started a scientific revolution!

No. That was several other people. Read some books. Or a web page. Since Mr Experienced clearly hasn't yet learned to google:

All of mathematics will have to be rewritten for a start! Amazing! Have you informed all your colleagues yet of this fantastic discovery of yours?

Yeah. Seriously. Read some internet before you really embarrass yourself.

fzammetti wrote:

I shouldn't think I'd need to explain it to you, Mr. Physicist, but fine: shared knowledge, experience and collective agreement on best practices is how the human race makes progress in most cases.

Don't shift the goalposts. You wrote "we have these things known as 'facts'... they come from our collective experience and shared knowledge". I believe your intention there was to show that truths are defined collectively. And I agree with that remark, and it's also why I, and others, reject truths and why I wrote what I wrote. Truths are shifting and mercurial and aren't useful arbiters of anything. They are semi-religious and often used as an excuse to kill.

But I digress, you're now changing your remark from the genesis of facts to the causes of progress: "shared knowledge, experience and agreement on best practices drives progress". The first has some basis, the second is pretty dodgy (experience can retard progress, for example senior members of a group often reject new knowledge, especially if it rubbishes their life's work), and the third is completely wrong. Progress is always the result of a change to accepted practices.

But I digress again. The point here is that you're pretty miserable at arguing a point. You were supposed to be explaining how facts are dervied from collective experience. In attempting to explain that to me, you wandered off somewhere else ... and never did get back to your original point. Yes I like to poke through the sideshows - they're interesting - but I generally return to the ring. You, you've driven your clown car right out of the fairground ...

fzammetti wrote:

Did you really, for a second, have any doubt I was referring to him as well? Did you really, for a second, not understand that the hypothetical applies to reality as well? If so you're simply proving my point further. You either have no ability to comprehend or you're being willfully ignorant, Mr. Physicist.

You were describing a counterfactual, a hypothetical that specifically did not apply to him or his reality (he was called out for using dev-in-prod, you were discussing a contrary example where dev-in-prod was the rule). It is your lack of comprehension (of your own damn thoughts and words) that is the issue here.

fzammetti wrote:

mr_dee wrote:

Yes, quite obviously. So what the hell was the first 90% of your post and your previous posts about?

Come on, you know what it was about. You had a problem with the "harshness" of my post. You had a problem with the "absoluteness" of my post. That's what we've been arguing about.

Yes. I know what it was about. I remarked that "NEVER" isn't really a useful way to talk, and you spent several posts dancing around your own straw man. Eg: "if you can PROVE that doing dev on a prod box is actually good, contrary to all current industry knowledge, I'll be the first to bow down and admit I was wrong". I have never tried to argue this. I have said that it is sometimes expedient. But not good. Can you tell the difference? The killing of men is not good. It is sometimes expedient.

fzammetti wrote:

That's what you've been wrong about all along."

Nope. I'm still right that absolutes belong in the 19th century.

fzammetti wrote:

You know perfectly well that's not what we're talking about here. Emergency fixes are just that: emergencies.

Emergency fixes that require development. And that's still development. Thanks for playing.

fzammetti wrote:

They need to be dealt with immediately and we all know that. They need to be handled differently. I never said otherwise.

Flippity-floppity.

And do "we all know that"? The junior devs are now running around telling themselves "NEVER touch production, NEVER touch production", and that's precisely why your attitude is so toxic, and very much my point.

fzammetti wrote:

Except I'll add that it's my years of experience that likely helped avoid such a situation in the first place because the type of process you seem to loath so much is what in most cases stops things like this from happening.

Giving legal advice to business owners is best practice for a developer? Especially when said develoepr is not yet aware that business owner is making with the sillies? And the Oscar for Most Arrogrant Asshat goes to ...

fzammetti wrote:

Wow, you really are just an obnoxious...

Well, to quote a Very Experienced Person, "Kid gloves is fine for 6-year olds, not for professionals in any given field". How's that petard working out for you? Do you promise to not lead the youngsters astray with NEVER any more? Good.

One of my colleagues was "developing" in a production instance once. She SSH'd to production using puTTY. Since it was just a black screen with white letters, she soon forgot she was on production and started cleaning up some "unnecessary files" taking up lot of space. Needless to say, this created a shitstorm and took days to recover.

Moral of the story is this. You don't develop in production because sometimes you simply just forget that you're logged into production box and do something stupid.

I just want so say that I agree with fzammetti and disagree with the other guy. And this post is hilarious:

b4lr0g wrote:

One of colleagues were "developing" in a production instance once. She SSH'd to production using puTTY. Since it was just a black screen with white letters, she soon forgot she was on production and started cleaning up some "unnecessary files" taking up lot of space. Needless to say, this created a shitstorm that took days to recover.

Moral of the story is this. You don't develop in production because sometimes you simply just forget that you're logged into production box and do something stupid.

One of colleagues were "developing" in a production instance once. She SSH'd to production using puTTY. Since it was just a black screen with white letters, she soon forgot she was on production and started cleaning up some "unnecessary files" taking up lot of space. Needless to say, this created a shitstorm that took days to recover.

Some of the developers I know have different profiles on PuTTY so when they log into one instance, the screen has (say) a different background colour to another instance.

I think a lot of commenters mis something important. First do a risk assessment! The wiki server you set up for your development team is also a production server but do you really need to do all the testing and planning when you change something there of course not. Maybe it is not really that big of a problem. On the other hand if this application is a very hi impact hi risk application then you have a bigger problem and the owners first priority should be to get a grip on the app now. Imagine what happens if the hardware fails then it is end of the story maybe even for the business.

As easy as it is to set up virtual server's and virtual machines these days, no one has an excuse not to have developer environments. I can run VirtualBox on a laptop with 4GB of RAM. I don't know if you need more for Windows or Mac, but it works great on my Ubuntu systems.

Answer this 1 simple question though before you develop on production:

"Can I absolutely prove, with 100% certainty, that my actions on the production server will not cause any issue now or in the future? ... Ever?"

You see, answering that is impossible! Even if you have done nothing bad on the server, there may be just the suspicion that you did ... There may be doubt. And if there is doubt, you may be explaining to management or to a damages court why it wasn't you! That's something you can simply avoid and something way worse than the pain of getting a test environment sorted.

If your management are put in a position where they have to explain to a damages court about your actions, you could bet that either way, they would view you as a liability more than an asset...

So go right ahead, but there may be consequences somewhere lurking in your future!

Short answer: your boss is correct. Consider yourself lucky you just got yelled at and didn't have your badge seized on the spot. Take heart: they must see something in your abilities they really like.

First issue: you need to figure out why you can't set up a development environment. If you don't have the resources at your workplace to accomplish this mission, set up some VMs on your home system and work on this problem there. Go through the steps setting up RoR and familiarize yourself with that process. You'll learn something useful doing this; it may not be directly related to your job right now, but it will be someday.

Second issue: your professional outlook. You should accept as holy writ the fact that your experience level (if it is only a few years in the profession) is scarcely better than the lowliest apprentice. If you really want to do this as a professional then you have signed on to a lifetime of investigation and learning. If your brain isn't consumed by the question "Why can't I set up a development environment that looks just like production" then maybe this life isn't for you.

Third issue: you'll always be working as part of a team, you need to understand the perspectives of each part of that team, and, like a football team, when someone on that team doesn't do their job and causes the team to fail they get cut. The people administering the production system want it to be boring: always up with a zero-length trouble ticket queue. The auditors want it working within the parameters of the business. The developers want the state of production to be predictable and completely understood, so they can add new features to it. The people using the system, and relying on the system, want it to be like an electric utility: always there, always doing what it should be.

One of colleagues were "developing" in a production instance once. She SSH'd to production using puTTY. Since it was just a black screen with white letters, she soon forgot she was on production and started cleaning up some "unnecessary files" taking up lot of space. Needless to say, this created a shitstorm that took days to recover.

Some of the developers I know have different profiles on PuTTY so when they log into one instance, the screen has (say) a different background colour to another instance.

I thought that was quite a good idea.

I think it's an essential one. It's the same reason that root accounts, if they even have a gui, should have a clear warning on the wallpaper.

That being said, I don't believe that the intent of the other forum members' comments is that developers should never have access to production, just that they not develop and/or do stupid things in production (like turning on verbose logging and then leaving for the evening without telling anyone).

It is quite often the case that developers have read access to prod and in some cases write access to prod if working in an escalation support role in a large shop, or in the normal course of business in a small shop such as you describe.

If you don't think that was their intent, then let me be clear that it is exactly my intent. Developers have no business in production. In fact, I don't think Operations folks should have access to production, except in emergency situations and no one should have direct access to login as root, without going through at least one out of band hurdle (I like the physical key to a fire safe with a sealed envelope containing the root password and mandatory rotation once unsealed). That being said, this is a hard model to actually achieve but I think it should always be the target.

Not every shop has the luxury of having dedicated personnel for separating duties at every layer of the task. In small shops, you often have to wear many hats. That's just business reality.

While wearing your DEV hat, log into your DEV account and use the DEV environment. (this may be an old 286 or a dedicated VM, but it will be a standalone machine that can crash and burn in isolationWhile wearing your QA hat, log into your QA account. Depending on where you store your tools you may or may not need to move the app being tested. The important thing is that you are doing QA before allowing access to anything you don't wish to damageWhile wearing your production hat you log into your production account. Before allowing changes you verify that QA has verified that the changes will not cause problems & create a rollback procedure to undo any changes.

All of the above can be a single person on a single computer as long as they maintain a strict Chinese Wall between roles and have procedures for isolating each environment from the others. In particular when running all on a single machine, the DEV & QA environments must be incapable of EVER crashing the physical hardware or VMS external to themselves. If it is possible to crash the hardware, then restarting the PROD code must be quick and easy or you will suffer the consequences of unscheduled downtime.

If ( Kevin Mitnick == your lover ) develop in productionelse { clone production to another server develop there }

You're putting yourself as serious risk of being fired,and your company as risk of a serious hack attack.Any cross-site scripting attack, the most likely, will leave a hacker with your total SOURCE for your site, and the tools to back-door it to death.

Disclaimer: Since 12 years ago, I've worked in everything, from dev-duction environments, up to SOx compliant, multi-tier deployment environments.

To borrow a meme from cycling:

There are three types of developers in your situation: Those who are going to hose production. Those who are hosing production. Those who have hosed production.

Trust me, you don't want to be known as the guy who hosed production. At least spend the effort to get a dev environment set up on a separate machine.

In fairness, there are two more critical categories you need to add to the mix:

-Those who are going to hose production by uploading something that worked on a separate dev machine without testing in the production environment.-Those who have hosed production by uploading something that worked on a separate dev machine without testing in the production environment.

If it is a question of either/or, in the absence of a viable production-equivalent machine on which to test, I suspect the risk of killing production with a bad upload is far more likely than a Ruby on Rails app doing something terrible in it's own sandbox on the machine.

Your two examples, while common, can be recovered from backup and the cause investigated offline. That is precisly why you have a seperate DEV/QA/Prod instances. The OP's three examples lead to real time troubleshooting on a live production box.

If you can't even replicate the production environment such that it works, then you have even bigger problems than developing in production. You have a timebomb waiting to explode.

Especially with Ruby-on-Rails where there's a lot of peeing in the same pool by applications on the same machine (unless you're carefully using something like RVM). At some point you're going to need to add or update a gem for a new feature of your development code, or make some kind of configuration change that's system wide, and at that point the same weird combination which happens to keep your production server running is going to change and it's going to break production. Then you're going to really regret not figuring out what made the production server work when you're trying to rebuild it and have no template to base it on. As Peter said, the key thing is being able to replicate the environment, it's the most important thing you can do.

On the one hand, this makes a good point. On the other it totally re-specs the job in order to make it. My guess is that the person holding the checkbook behind this assignment sees whatever app changes they hired done as the most important thing the person they hired to do them could accomplish.

Unless the coder also has a contract for the server management/configuration side of things, isn't this just suggesting they do a bunch of work sorting out this client's configuration quirks for free? Typically this is something folks get paid money to do. If the only cash on the table out of this is for a simple code update to the app, that is a boatload of overhead merely to claim success at duplicating a configuration which the developer will likely never willingly employ in any other context. Seems largely worthless knowledge absent cash incentive and/or promise of relevant future server work (neither of which were mentioned in this case).

This is the biggest issue I see here. Thanks for summing it up so nicely. Putting my MBA hat on the question at hand is "what kind of business are you in?". If you are a coder, than you produce code, burn to cd and mail to the client. Done. If your service is to provide a live environment to the clients customers then you are not just a development shop, but also implentation, tech support etc.

If the service you are providing is a "hosted application" you had better to be able to recreate that environment, at will, on demand when needed. If you can't do that, as Peter Bright already mentioned, you are a disaster waiting to happen.

I have moved projects from 'all work on one box' to dev/qa/prod environments. Client had a small intranet app with handful of departmental users. When it went corporate wide, with 1000+ users, nightly oracle dw ETL processes and more it needed to be placed into a real development environment. Our service to the client changed from simle coding to application delivery and so we changed our delivery model to match.

Its good that most people are 100% against working in production, which is as it should be, but there are different situations that dictate different behavior.

For the most part, a professional development environment should have development/staging/production. After that, it really depends on other factors. Development is for, well, development, but staging is just as critical because it tests the deployment process. In the same way that the most innocuous thing in code can bring down production, the most innocuous thing in deploying code can do the same. Having a solid roll-back plan is a given, you just need it, because things will go wrong. That, really, is all that is generic for all development, everything else depends on the what and why of the development target.

If you're doing work that needs to comply with PCI or HIPAA, you have other requirements in terms of data storage and accessibility in development/staging. Your testing/QA is also going to need to be more rigorous. If you're doing work on an application that is mission critical, you have to deal with release schedules and scheduled downtime (and potentially SLA type of requirements for downtime) which may lead to a more robust staging setup. If you're working on an online store you have other downtime issues to deal with in terms of scheduling.

Beyond that, I didn't see anyone address the issue of lossiness. Many mission-critical applications with integrate with other real-time systems that are not going to be down while you are updating. How you deal with this to ensure no loss of data factors into the development environments as well (for example if you are in an application cluster in production are you in one in staging and/or development? it SHOULD be the same, but it's really not).

On a final note, I see some IT/Ops people saying that developers should never even have access to production. All developers shouldn't have access to production, but the development team/department should, even if it's restricted to one or more senior developers. There are monitoring, logging and auditing tasks that are necessary enough to warrant that in most cases, but more importantly its likely that the senior developers are going to be somewhere on the chain of phone calls if/when things go pear shaped.

If you're doing work that needs to comply with PCI or HIPAA, you have other requirements in terms of data storage and accessibility in development/staging.

my job recently announced that we're not allowed to keep any customer databases on our PCs or the current test servers, but they haven't gotten around to making a secure test server yet so all testing is currently being done on the customers' production servers. this is really stupid but they're probably not going to give us the resources we need until we break something with an untested patch and the customer escalates all the way up.

On the blame issue... if something is perceived as going wrong, you will be held responsible whether or not you follow protocol. And you will have the same risk to your job. That's the nature of the world.

Anyone else feel like this conversation is getting as bad as the CSS purists who say to never use tables except for tabular data?

It wasn't meant to be that way mind you. I set up a CentOS box, setup Apache, setup MySQL and MongoDB. I installed git and cloned the repo. I started perusing the code, changed a few bits and pieces for the interface. All good. Then I started on the nitty gritty and started changing some DB functions.

That was when I discovered that the git repo contained the database settings for the production databases rather than pointing to localhost.

If it's for a business server then DON'T. There are liability issues for the business that you can expose them to and developing on a production server would look really really bad in lawsuit. It's NOT all about you.

FWIW, I tend to go with the Black's Law Dictionary definition of "should:"

Ought to, but not necessarily will.

So while I agree that one should never develop on a Production server, my definition recognizes that there may be some instances where it can't be helped - hot patching a critical system, a legal issue, etc. - but, as a general rule, you don't do it.

Can people live with this interpretation, or shall we keep on arguing?

If that server is for 5 people and the change is small, go ahead, until you cause it to crash and be down for a week over something stupid.

In a larger organization, best practices do not let developers have accounts on production servers. This is critical to ensure configuration management methods are used. Deployments can be repeated, entire servers can be re-deployed without some magical developer-only knowledge.

Most of the time, making any changes on a production server with human hands is a bad idea, that includes developing new code - or even modifying a tiny script. That tiny script change made late on a Friday, just before leaving for a week of vacation out of the country will probably be the first time any error takes down the server. Murphy always works on Fridays.

Peter makes a very good point. If you can't create a separate reliable development environment then you don't know your system. How can you therefore state that you understand the implications of what you are doing in production or on the production server?

If you're doing work that needs to comply with PCI or HIPAA, you have other requirements in terms of data storage and accessibility in development/staging.

my job recently announced that we're not allowed to keep any customer databases on our PCs or the current test servers, but they haven't gotten around to making a secure test server yet so all testing is currently being done on the customers' production servers. this is really stupid but they're probably not going to give us the resources we need until we break something with an untested patch and the customer escalates all the way up.

Not having the resources for appropriate change control is another issue entirely, and it occurs far more often than it should. It's such a shame that so many managers need to learn from experience that the price of production failure often exceeds the price of effective change infrastructure.

1) Why didn't the OP ask for help? If there is no one who knows how to set up the system, FIND ANOTHER JOB before you get a call at 2AM to setup a new server after the prod server crashes! If there is someone to ask, ask them, not asking question of others is a very bad trait to have, almost as bad as developing on the prod server.

2) There must also be no backup server. Typically any production system should either have either a fail-over server or it will be run in parallel on 2 or more servers such that the failure of any single server will not take the system down. In the worst case you will have a production image on your staging or dev system so you can reboot it into prod if needed. I always liked to have a hot stand-by server which I can deploy updates to and then swap it to production so that I can have the old system with the old version of prod handy in case there as issues with the new prod and I have to fail back.

3) Obviously there is no development box or development image.

4) Obviously again, there is no documentation, source code control, or setup scripts. The funny thing here is this is insane for a small company, particularly one that uses consultants and probably has a high turnover. For a big company this lack is just an eventual cost overrun, for a small company it can mean going out of business. (Yea, I know all the dweebs that hate doc are going to respond "Well I never and it always works for me." Yes, but what about the guys that get called in after you?)

5) No one noticed that there was development going on on the production server. Is it even being monitored? Would the first one to notice a server crash be their customers?

6) Ok, so dev on a prod server is bad, but having a developer that doesn't even understand how to configure the dev environment developing on the prod server is just plain insane. (Not picking on the OP here, he is just tying to get his job done. the best he can. There are several major, major production systems I am aware of [on legacy platforms] where people are still forced to do development on production servers.)

7) Dare I say Cloud, Amazon EC2 anyone?

8) In todays world of promiscuous virtualization it is just plain silly not to have a prod image that people can use for dev on their laptops. My laptop has 17 VMs on it now and it had even more when I was still working :-). Download VirtualBox for free if you can't afford VMWare.

Having said all this, there have been times where I have done manual changes in a production server while it was running. Usually these were DB changes and usually there were scripted and I had tested the scripts in dev and staging, but not always. There are times where the risk of the issue I was resolving was worse than the risk of going through the process. The key point is balancing the risks; how much money are you saving versus how much will you lose if you screw up and understanding what are the odds that you will screw up. This requires a deep understanding of both the system and the risks of your work. Its normally a decision that must be made in concert with the other decision makers of the company being put at risk by the work.

If you're doing work that needs to comply with PCI or HIPAA, you have other requirements in terms of data storage and accessibility in development/staging.

my job recently announced that we're not allowed to keep any customer databases on our PCs or the current test servers, but they haven't gotten around to making a secure test server yet so all testing is currently being done on the customers' production servers. this is really stupid but they're probably not going to give us the resources we need until we break something with an untested patch and the customer escalates all the way up.

You should be working with test data, not real data. Test data is created from real data by copying all your real data while scrubbing all of the customer data from it,i.e. use a random number/character generator that replaces the original data with a string/number of the same size. Then add a bunch of test data that includes:

1) Tests cases of all of the specific bugs you have encountered before.2) Boundary condition data such as the largest and smallest numbers strings, etc. 3) Random garbage and error conditions that should be caught (testing should also test that things break right too.)4) Test cases added for the current development you are doing.

MySQL and other modern small databases are small and lightweight enough that you can easily run them on you laptop, I do it all the time and have had as many as 5 instances going at the same time on my MacBook Pro.

Using the same data base server for dev and prod falls into the same category as developing on the prod server, DO NOT DO IT, even if you use a different database. If you do not even have access to the production data to clean in, then use a random data generator to generate it all. I have never failed to find new bugs in a system filled with a large set of random data.

With MySQL I have created scripts that the developers used to setup and load a DB from scratch, your DBA or (more likely) a dev that knows SQL should be able to do this. A VM may be even easier.

In a previous job I kept all of the DB code in a source code control repository. When a developer checked out their source code to build a new development environment they also got a new dev DB created on the Oracle dev server and loaded with the test data. I have seen this done with MySQL as well as long as there is a DB already installed.

I say sure! Just not on the _real_ production server. Either replicate all settings to a VM, or - as I did once out of pure laziness to be honest - replicate the whole server into a VM (I went from a debian stable server into a virtualbox vm). Then you don't have to worry about breaking anything, you still have everything as it is on the real thing, plus, if you have to change something, you can without worrying about breaking something on the live thing.

Back when I used to play a web developer in the late '90's, we used to develop live on the production server all the time--mostly because no one knew any better. Until the day when there was a miscommunication between me and a new programmer hire as to what a function he wrote for me was supposed to do in case of error.

I called the function and it quickly set all 150,000+ users in the live db to resubscribed today. Once I figured out WTF had happened, I booted up the Website Under Maintenance banner and went out and had a Mountain Dew in the sunshine while contemplating which form of suicide to use. Luckily, I realized that we had a backup of the database that was less that 24 hours old, so I just rolled it back, told customer service that they would be getting calls from about 30 irate users who had really renewed in the last 24 hours and then spent the rest of the weekend setting up a development server. (We had a machine dedicated for it but no one had had the time to set it up in the 9 months since we bought the thing.) And on Monday, I gathered the troups and explained that no one was to develop live on the production server ever again.

The point being, just because you haven't killed the production server yet doesn't mean the consequences can't be completely disastrous if and when you finally do, and it can all be down to one simple miscommunication between you and a colleague, which means you'll likely never see it coming.

Speaking from personal experience, the developers where I work have no problems putting in emergency code changes without testing them at all (or at a bare minimum, say for 1 hour in load test environment). Heck, for some applications they only have a production environment.

To this day I'm amazed we maintain as much stability as we do. We frequently tell them not to do this, but they go over us and get management approval (as no one seems to care what my team thinks - at least we don't get any blame usually).

Developing on a production server is NEVER acceptable. Yes, that's absolute. NEVER. Ever. There is not situation or exception that makes it acceptable. It doesn't matter if it's a two person company.

Developing on a production server probably means you are not using source control. Developing without source control in a business setting is NEVER acceptable. Yes, that's absolute. NEVER. Ever. There is not situation or exception that makes it acceptable. It doesn't matter if it's a two person company.

Anyone that condones or facilitates these practices deserves to have their ass chewed hard, at the very least. Fired is a more appropriate response. In some environments, jail might also be an appropriate consequence. The OP would have had his ass fired on the spot at my workplace.

Developing in production is not agility. Documenting IS agility. If you have a documented process to stand up a replica of a production environment, then you can do it quickly and reliably. That's agility. If you have to spend an all nighter figuring out why it doesn't work, that's stupidity.

Heck, I prefer multiple development environments as well, at least when there's more than just a few people on a project. That was you have a stable dev environment with working versions of everything as well as a "playground" where developers can test out new versions during integration without affecting other developers working on other features who need a stable environment for their work. That's in addition to a separate QA environment and, of course, production.