Avoiding ecommerce deployment disaster: 10 areas to watch

Replatforming and deploying major updates are some of the most stressful moments for an ecommerce team.

These moments are vital for staying ahead of the competition, for introducing innovative new features or responding to user testing, but they’re also the point at which things can go most wrong.

Too often when you or your agency throw the hypothetical switch you end up with a site that’s got serious bugs or, even worse, no site at all.

What can you do to ensure that the deployment of your new platform, or of important revisions to your existing one, run seamlessly and effectively?

1. Always have a roll back strategy

A roll back strategy is the way to prepare for the unexpected, giving you a second chance to undo the deployment and go back to where you started.

Having a clear strategy to undo your changes is vital, however, before you put it into action, remember that sometimes fixing the problem can be quicker than rolling it back.

Accepting an amount of downtime whilst you fix an issue rather than instigating a time consuming rollback and going through the whole process again can sometimes be the better option.

Also, some deployments, especially of major changes, may include changes to the database. This is something that isn’t included in a version control system rollback.

2. Check payment processes

One of the most common area issues can emerge in a newly deployed site is in payment processes. Payment integrations can be difficult to properly test in the build process so make sure you do manual tests again and again before you go live.

There is also software available to test payments in a mimicked production environment and if you’re introducing new payment processes, then this sort of technology is vital.

3. Have at least four development environments

Having separate desktop (developer’s computer), development server, staging server and production server environments gives you the freedom to develop whilst maintaining control of final code and allows you to refine deployment scripts as code is pushed through the environments.

It means you can test at each stage without disrupting the development cycle.

It’s also important to make sure that all of your environments, including developers’ local setups, are running the same versions of software and hardware so you can be sure that the code is operating consistently at each step of the process.

This avoids the all too common 'but it works on my machine!' excuses.

4 Think about deployment timing

If it’s a big deployment that involves planned downtime (or the risk of unplanned downtime), do it out of hours if possible.

If it’s an international site operating in multiple time zones then there may not be a ‘down’ time, in which case choose the least worst timeslot based on historical trading patterns.

It might seem counterintuitive to take a site down in the middle of the day UK time but, if a lot of your business is coming from the Far East, this might make the most sense.

5. Never deploy on a Friday

It can be tempting to get an update out before the weekend but, quite apart from the fact that Friday can be a busy trading day, it can also set you up for hard-to-solve issues over the weekend when people can be difficult to reach.

Problems can sometimes take hours or even days to emerge, potentially meaning that your site could go down out of hours like early on a Sunday morning.

Deploying earlier in the week means that issues are less likely to happen at inconvenient times.

6. Consider database processes

If you have setup scripts in your deployment, be aware of database processes that could take a long time to undertake.

For example, if you’re adding a column to a database with a million rows it may take less than a second for each row to update, but that time adds up.

Test deployments with production levels of data before you deploy so you have a realistic idea of how long the deployment is going to take, enabling you to plan realistically for potential downtime.

7. Communicate

When it comes to any deployment there should be an open line of communication between you, your agency, your management, your marketing and your developers.

Retailers should know about the potential impact of deployments, how long they could take and what might go wrong.

Also, agree fixed deployment slots in advance to stop people pushing for inappropriate, rushed deployments. This also minimises the risk that the marketing department could send out a 50% off voucher at the same time as you’re deploying your changes.

8. Minimise the size of deployments

It’s not always possible, but doing less, more often is usually the best approach.

Breaking up one big release into lots of smaller deployments means you can more quickly identify where issues are coming from if they emerge. It will also break up downtime into smaller pieces, lessening the overall impact on consumer experience.

9. Document

Documenting is something everyone should do, but it’s easy to forget or ignore.

By using a ticketing and bug tracking system to document your code, you can check commits against ticket numbers in order to ensure that all issues are resolved before deployment.

This approach not only enables a rigorous checking process, it also feeds into your rollback strategy making sure you know exactly what each commit was for should key team members leave/go ill/fall under a bus.

10. Have all the code that needs to be deployed ready

It sounds obvious but, without the right development processes and policies in place, it’s all too easy for a developer to forget to send all of the code to the server prior to deployment.

Adherence to proper development protocol, along with rigorous checks and testing along the way, will means that complete code should be ready to switch from staging to production each and every time.

You can never completely de-risk a deployment. However, by taking some simple logistical steps you can prepare for the worst whilst also doing your best to ensure the worst never happens!

Comments (7)

There's an ongoing struggle for businesses with little or no IT staff to adjust to the rapidly adjusting demands of shoppers. With a new platform like Mozu, the SaaS updates and ease of use on the back end enables a marketing manager to adjust their commerce platform on the fly!

For any extended deployments / outages: serve a holding 'maintenance' page to handle all page requests, from another box if necessary. *Importantly* though, serve this page with an HTTP 503 status. This should keep search engines happy, and avoid 'Sorry, we're down for maintenance...' getting into page descriptions in search engine results. Been there. Ooops.

Ensure key staff *still* have all the necesary log-in and access credentials at the critical deployment time. Ongoing network maintenance and updates can wreak havoc on a good implementation plan. Been there. Ooops again.

almost 3 years ago

Olly Secluna

With a risky deployment it can be worth extending point 1 to actually testing both your deployment and rollback with 'dry run'. This can be time consuming of course.

This is a great point that we omitted, as part of the automated deployment process there should be a maintenance flag (and page), although it's not always required. In our work, this is only usually required when a database schema change is made.

On your second point, this can be achieved by allowing only users from specific IPs to access the underlying site, while hiding it from the wider world with the maintenance page.

Thanks for your comment. By having 4 environments and an automated deployment process, this should be 'tested' during the promotion to each subsequent environment. Your point on rollbacks is well made as people often forget to test for failure.

Roll-back processes are great advice - we help alot of sites through release processes; and for a number of reasons rolling-back can be quite tricky in some environments.

And I'd add to the list - that it's good to monitor User Experience before and after release, in terms of speed of the core User Journeys.

Because sometimes 'everything still works but some Journeys are now noticeably slower' can be very damaging to conversions if left for several weeks.

<self promotion on>

We do real-time Before/After reports, during the vital first few hours and day(s) after each release, as part of our Site Release Management add to our performance monitoring services.

<self promotion off>

almost 3 years ago

Liam McDowell, Commercial Director at MagenTys

I see there is mention of 'testing as you go along' but what we see is that most companies have no regression testing, let alone automated regression testing.

Regression test packs should be built upfront alongside development so when code is released, it will either Pas or Fail. These regression test packs are automated and can also cover compatibility testing across web and mobile so also taking most of the tedious testing work away from humans so they can focus on the good stuff.

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you
ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.