Tag Archives:dark launch

Sometimes when it’s time to deploy a new feature, there’s no way to test it properly without actually putting it into production. Maybe you’ve got to use the production environment and networking to be sure everything is OK with the new version, or maybe you won’t know if a new feature works until it’s in use.

One way to handle that is with a dark launch. You can use dark launches to provide new features to only certain users or to test new features without changing how the main production system works for most users. There are lots of ways to handle a dark launch, depending on how you want to make the new features selectively available. I’ve described one way in Revealing new features slowly with dark launches.

Now I can run a process with that tag as an argument. The process removes the nodes with that tag from the load balancer so no users access the dark launch node. Then, the process deploys the new version of the application to only the nodes with that tag. Now one of my nodes is running the new version of the application, and people with the direct URL can access it and try it out. I could set my load balancer to send a little traffic to the new version to see how it works, or I could use it strictly for my own internal testing.

When it’s time to go live on every node with the new feature, I run a similar process that updates each node and enables all nodes on the load balancer.

That’s just one way to handle a dark launch in IBM UrbanCode Deploy. Let us know if you’ve got other strategies or different requirements for your applications.

A dark launch for a software product or service is one way to reveal new features incrementally, with minimal disruption to a production system. You can use dark launches to provide new features to only certain users or to test new features without changing how the main production system works.

Some of this is covered by a rolling deployment, because in that case you update a multi-node system one node at a time, each time giving another block of users access to the new version. However, dark launches go a little farther by leaving the new features enabled on some nodes for a longer period of time — long enough for A/B testing, usability testing, or metrics-based analysis.

There are many different ways to implement dark launches in IBM® UrbanCode™ Deploy; which strategy you choose depends on how your application is set up and how you want the dark launch nodes to be available. Sometimes, all nodes have the dark launch features enabled, but feature flags or other settings prevent most users from accessing them. In other cases, only one node or a few nodes have the new features enabled, and those nodes are either restricted to specific users or hidden from direct customer use altogether until the new feature goes live to all nodes and customers.

In this article, I’ll go through a scenario that handles dark launches in a way similar to how I covered rolling deployments. In this scenario, the load balancer sends no traffic to the updated node until it’s time to go live. In the meantime, the dark launch node is available for testing and trial use by people who know the direct URL.

Here’s the basic topology I’m using in this case. The application is running on four nodes in four different geographical areas. Before the dark launch, all nodes are running the same version of the application and the load balancer sends traffic to all geographical areas equally:

The basic steps for the dark launch are as follows:

The application process removes one node (the dark launch node) from the load balancer so that no end users can access it. In the topology below, note that the load balancer is no longer sending traffic to the North America node.

The process updates the dark launch node to a new version of the application and runs tests as appropriate. The node is now updated, but it is available only to users with the direct URL because the load balancer is still not sending traffic to that node, as shown in the following topology:

3. When it’s time to provide the new version to all users on all nodes, the process enables the dark launch node on the load balancer. In the topology below, the load balancer is again sending traffic to the dark launch node.

4. The process updates the other nodes to the new version.

Using resource tags to control the deployment

Using resource tags and the For Each Tag loop simplifies the automation for this scenario. For example, in a scenario similar to one of the scenarios that I used in demonstrating rolling deployments, I’ve tagged four nodes with the resource tags “Asia,” “Europe,” “North America,” and “Africa.” I can use the resource tags to identify one node to use in a dark launch. As in my blue-green deployment scenario, the load balancer component is mapped to each node so I can access operational component processes to add and remove each node from the load balancer. For more information on how I’m doing that with the load balancer, see Load balancer setup. I’m also using the same JPetStore sample application that I’ve used in other articles and in the tutorial Deploying a simple web application.

The application process for the dark launch is similar to the application process for the rolling deployment. This process has the following steps:

The process updates the database. In this scenario, I’m assuming that the database runs on a separate node and is backwards-compatible; therefore, it is always updated to the newest version, even in a dark launch.

The For Each Tag loop runs the following steps over all selected resource tags. In most cases, I dark launch the application to a single resource tag.

The “Set node to dark” step is an operational component process on the load balancer component. This process removes the node from the load balancer so no traffic is sent to it.

The application process installs the JPetStore application component and web component on the selected resource tags as usual.

I’ve also set up a process for the public or “light launch” to make the nodes available to users. This process is similar; the only difference is that at the end of the deployment, the nodes are added to the load balancer instead of being removed from it:

To do the dark launch in a single process, I could include a manual approval to pause the process while I evaluate the dark launch node. Then, the same process could update the rest of the nodes and update the load balancer after the approval. This time, the separate processes made the most sense to me, but of course there are many ways that you could structure these processes.

Running the dark launch and the public launch

As I explained earlier, when I run the dark launch application process, I can select the resource tag or tags on which to run the deployment. In this case, I’ll select just one tag to use one node (in this case, the North America node) as the dark lunch node:

When the process is done, the environment is noncompliant. That doesn’t mean that there is anything wrong with the environment; it means that I deployed a new version of the application, and only one node got that new version.

Now I can access the new version by going directly to the URL of the dark launch node. The other nodes run normally on the load balancer with the old version.

When it’s time to make the new version public on all nodes, I can use the similar light launch process. I request this process with some important settings:

I select the new version, which is already deployed to the dark launch node.

I select all of the resource tags, including the dark launch node.

I select the Only Changed Versions check box. This way, the node that already has the new version is not changed.

Now the environment is compliant, all nodes have the new version, and all nodes are enabled on the load balancer:

Also, all that’s necessary to make this public launch into a rolling deployment is to limit the For Each Tag loop to one tag at a time.

Load balancer setup with HAProxy

The way you set up your load balancer for dark launches depends on how your organization handles its IT and the specific way you’re implementing the dark launch, but just in case it helps anyone, I’m going to cover how I’ve implemented HAProxy for this scenario. If you’ve got another load balancer like F5, you might be able to find a plugin that will handle all this for you.

The load balancer requirement for dark launches is to be able to enable or disable nodes as part of a process. A straightforward way of doing that is to comment out or uncomment each node in the HAProxy configuration. For example, here’s the relevant part of my haproxy.cfg file for this scenario:

To enable or disable nodes, all I have to do is comment out or uncomment a line and then restart HAProxy.

Like in the article Setting Up Blue-Green Deployments in IBM UrbanCode Deploy 6.2.4, I’ve added the load balancer as a component to each node in the environment. This way, I can call component processes on the load balancer in the context of the node that is being updated. Then, the component process automatically has properties that give it information about the node, including the host name.

Of course, the load balancer is not actually running on the application nodes; it’s running on a dedicated node somewhere else. I can use the information in the component process to call a generic process that runs on the load balancer node. This generic process actually edits the configuration file and restarts the load balancer.

The component process is very simple; all it has to do is to call the generic process and pass a few values from the context of the component process. Here’s the process that disables nodes. It has a single step that calls the generic process. To that generic process, it passes the host name of the node and the resource path to the load balancer agent resource.

The process to enable a node is the same; it calls a generic process that enables nodes and passes the same resource path and host name.

The generic process that disables a node first gets a lock. That lock prevents any other processes from accessing the configuration file. Then it runs a shell script step that comments out the line that matches the passed host name. Here’s the shell script I’m using for the purpose:

With these two generic processes and matching component processes, you can enable and disable nodes as part of an application process, so you can dark launch nodes and make them public again programmatically.

That’s just one way to handle a dark launch. There are many other strategies, depending on how you wanted to make the new version partially available. For example, you could deploy the new version to all nodes, but show the new feature only on nodes with a certain resource property, like a feature flag. That way you can enable and disable the feature without any changes to the load balancer.