Forums

Blue green deployment?

Still just prototyping on trial plan, but I want to start figuring out how I can do blue green deployment in production - basically having two production systems so you can push changes to one, do some more tests, switch over but be able to revert to the other if needed while minimizing risks and downtime. Would like to do this on PA if possible rather than go from dev/test to right to prod. If there is a way to do this or sort of do it that you've done, would like to hear about it. Thanks!

You could do this on PythonAnywhere by having two web apps on the "Web" tab.

Let's say that your production site is viewed by your users on www.mydomain.com. At any one time, you'd have two web apps on the "Web" tab:

www.mydomain.com

alternate.mydomain.com

They'd have different codebases, but be connected to the same database on the backend. (Keeping everything consistent so that different versions of the code could run with different versions of the database, but this is a general problem with this kind of deployment, not PythonAnywhere.)

When you had some changes that you wanted to try, you'd push them to the source code directory associated with alternate.mydomain.com. You could access that site, and check that it was OK. If it was, you'd then:

Rename www.mydomain.com to somethingelse.mydomain.com

Rename alternate.mydomain.com to www.mydomain.com

Rename somethingelse.mydomain.com to alternate.mydomain.com

You would see some warnings on the "Web" tab about CNAMEs, but they should be ignorable.

Thanks Giles, this is good to know. I'm so practically far away from these concerns, still, but starting to think about them already. Ideally I think this deployment is done via a load balancer that can shift everything to the new system when ready or back if needed pretty much instantaneously. I suppose beta features for a limited subset of users could be done in some similar way. You have a canary in a coal mine warning system early on. I heard the Chromebooks work sort of like this. There are actually 2 production OSes on your machine. The updates patches go to the hidden one so the user experiences no downtime like with Mac, Windows, Linux, etc. Google can revert you to the other one if needed.

I've the same question as @ikongf. The approach @Giles describes is a big bang cut-over. Any ideas how you could implement some sort of routing rules to move a particular x% of your traffic to the other server?

Hmm, that would be interesting. If you want to do proper A/B testing of that kind of thing, then you'd need two completely different running websites, as of course they'd have different codebases.

The only way I can think of doing that on PythonAnywhere would be to have two different sites (eg. www1.yourdomain.com and www2.yourdomain.com) and then host something on the domain people normally came to (eg. www.yourdomain.com) to redirect people to one or the other based on some kind of algorithm.

Not ideal, as of course the URL in the location bar would change and there would be a risk of someone bookmarking/linking to one of the testing subdomains.

the prod ideally has blue/green that is configurable with no domain name changes. This is good for both a/b tests and for zero downtime when you reload your app.

One would need the first three first. Ideally this would all be available on PA, maybe in different tiers. At hobby level, you don't need it. At real company level, you do. It'd be ideal to keep PA as you go from hobby to startup. I like PA and don't ever want to switch PAAS or learn to do this level of sysdmin manually on IaaS.

Right now to do dev test prod, it seems easiest to dev on local host then do the two domain name plan (one test one prod). Startup plan actually should have the blue/green on the third domain. Then the first two could be dev and test. Or something like that.

Thanks for the suggestions! We're gradually rolling out changes to the way our web request routing works; historically the code that's running and the domain name have been inextricably linked, but we're changing that over time. (The request routing is so core to our system that we definitely don't want to risk a big-bang change breaking every website we host.)

So far we've been thinking of this in terms of routing multiple domains to the same website code (so that we can handle websites that use multitenancy hostnames like user1.adomain.com and user2.adomain.com). But once that infrastructure is in place, routing one domain to different websites based on some kind of trigger definitely sounds doable.

No promises as to timelines, of course, but I've made a note of this as a useful feature, with at least two people expressing an interest -- and if anyone else out there would like to see something like this, please do post here so that we can tally how much interest there is.

Wondering if anyone has some new thinking on this topic. What I find in testing is my app breaks a lot so I make changes and reload. When I go production, I'm concerned about reloading my prod app (downtime and also server errors not caught for some reason in testing).

Nothing new from our side. That said -- a reload should involve minimal downtime if your website doesn't try to do too much processing before it starts handling requests. A reload on our own site, which is a pretty complex Django app, takes seconds.

Thanks a lot for the update. Ok that actually doesn't seem too bad, really. I'm still prototyping, but I'm thinking what I'll end up doing is having one test account and one prod account. The method you mentioned above could be interesting, but I'll cross that bridge later...