SDL Web/Tridion – How could the hot fix process be smoother?

All software needs updating at some point due to unforeseen customer configurations, security updates or fixing bugs that slipped past QA. SDL’s software is no different.

Edit: Jan Horsman pointed out below that hot fixes may also be enhancements that do not warrant a service pack and so may not be a bad thing!

There are many ways to deliver updates to installed software. The challenge with enterprise software is that it often can’t just update itself automatically, especially in production when the product is powering a website that could potentially be the main revenue stream or customer portal.

SDL has traditionally delivered hot fixes as zip files with a text file of instructions and while this has worked OK in the past it doesn’t really scale and it makes it very difficult to keep track of what is already installed. This challenge has got much bigger now that there are potentially many more servers involved due to the more dynamic nature of the websites now being built on the SDL Web platform with DD4T and DXA and the micro-services architecture.

However, I have realised that I am just saying there is a problem and not really offering a solution. So here is my attempt to suggest some possible improvements.

Idea #1 – Have an install Script

Even if all it does is unzip the zip file to somewhere. Having to remote desktop to 10 servers and unzip a zip file on top of a running service (remembering to back up what was already there) is not practical and is prone to errors.

This can cause issues because it’s difficult to know what hot fixes have been installed on a system after the fact and also, now that there are so many micro services, deploying a zip file can cost a significant amount of time and testing with potential downtime in an on-premise install.

A Powershell/shell script to do this would be a great start! A GUI would probably be overkill.

Idea #2 – Record the hot fixes in a database table

There has been an attempt to keep track of hot fixes with the Hot fix manager custom page: http://tridion.stackexchange.com/questions/11450/tridion-2013-sp1-what-hot-fixes-are-installed but it

We now have a connection between CM and CD via Topology Manager and the Discovery Service. Could we somehow record which hot fixes are installed on which CD environment in the Discovery Database or Topology Manager’s DB? Keeping this outside the CM database would help with keeping this knowledge in upgrades.

Idea #3 – Rolling updates

When I discussed the hot fix process with Renze at the SDL Web 8.5 Developer Bootcamp, his suggestion was to move to the cloud with docker containers. I didn’t agree with this then but I have since been investigating docker more and rolling updates could solve some of the problems we’re seeing withthe more scaled out environments.

How this would work is up for debate, but Kubernetes and Docker Swarm allow for this sort of rolling update with 2 containers running different versions at the same time to occur. If SDL could help with how this could be set up that would help a lot!

Idea #4 – Use a Package Manager

1

>npm updatesdl-web-8.5-discovery-service:CD.1.2.1234

That would be a nice command right? What about this one?

1

>yum info sdl-web-8.5-discovery-service

These package managers have been around for ages and are well understood. Seems like a good thing to leverage to offer some sort of reasonable update mechanism.

Idea #5 – Use proper versioning!

A few times, hotfixes needed to be replaced and SDL provided a new hot fix with the same version number as the faulty one. This is really annoying to explain to IT why they need to install the same hotfix they already installed last week.

Idea #6 – Be open

Many software companies are scared of hot fixes because they imply problems and bugs. But we need to know about them so we can advise our customers proactively.

– Hot fixes are not always to repair bugs. Sometimes a hot fix updates functionality, like updating an integration with another product (i.e. hotfix XPM_8.1.1.3571). So hot fixes are not necessarily bad.
– #1 Install script: Absolutely agree, maybe the hot fixes should be shipped with a script. A GUI would make things worse, it would bloat the hot fix, and a script is easier to run against the 10 servers from your example.
– #2 Record hot fixes in database: I am not so sure about this one. The database lives separately from the software; if you would restore an old database, restore a backup of your CM server, or deploy an old version of your web app, then the list of installed hot fixes would be out of sync. This also applies is the data is kept outside of the CM database.
– #3 Rolling updates with Docker: sounds promising
– #4 Package Manager: The CIL (content interaction libraries we use on the web application site) is in Maven and NuGet. Also the CIL library is backwards compatible, and the SDL cloud versions (i.e. 8.3) are released as well. This means you can us the 8.5 or 8.3 CIL (client library) on the older 8.1.1 CIS (micro services.) So no need to install hotfixes on the web application when running with the Web 8 micro services.
– #4 Package Manager: It would be nice if the CIS (micro service) gets repackaged instead of shipping a hot fix. When using automated deployments it might be easier to re-deploy the micro services than to install a hot fix.
– #5 and #6: agree 🙂

Good point Jan, I have included a note about hot fixes adding functionality.

Also, interesting about the database. I guess a better approach would be examining the actual checksums of the JARs or this could be superseded by having sensible versions! Either way, I think what versions are installed needs to be more visible.