Dynamic A/B Testing with NGINX Plus

The key‑value store feature was introduced in NGINX Plus R13 for HTTP traffic and extended to TCP/UDP (stream) traffic in NGINX Plus R14. This feature provides an API for dynamically maintaining values that can be used as part of the NGINX Plus configuration, without requiring a reload of the configuration. There are many possible use cases for this feature and I have no doubt that our customers will find a variety of ways to take advantage it.

This blog post describes one use case, dynamically altering how the Split Clients module is used to do A/B testing.

The Key-Value Store

The NGINX Plus API can be used to maintain a set of key‑value pairs which NGINX Plus can access at runtime. For example, let’s look at the use case where you want to keep a blacklist of client IP addresses that are not allowed to access your site (or particular URLs).

The key is the client IP address, which is captured in the $remote_addr variable. The value is a variable named $blacklist_status that is set to 1 to indicate that the client IP address is blacklisted and 0 that it’s not blacklisted.

To configure this, we follow these steps:

Set up a shared‑memory zone to store the key‑value pairs (the keyval_zone directive)

Give the zone a name

Specify the maximum amount of memory to allocate for it

Optionally, specify a state file to store the entries so they persist across NGINX Plus restarts

For the state file, we have previously created the /etc/nginx/state_files directory and made it writable by the unprivileged user that runs the NGINX worker processes (as defined by the user directive elsewhere in the configuration). Here we include the state parameter to the keyval_zone directive to create the file blacklist.json for storing the key‑value pairs:

Split Clients for A/B Testing

The Split Clients module allows you to split incoming traffic between upstream groups based on a request characteristic of your choice. You define the split as the percentage of incoming traffic to forward to the different upstream groups. A common use case is testing the new version of an application by sending a small proportion of traffic to it and the remainder to the current version. In our example, we’re sending 5% of the traffic to the upstream group for the new version, appversion2, and the remainder (95%) to the current version, appversion1.

We’re splitting the traffic based on the client IP address in the request, so we set the split_clients directive’s first parameter to the NGINX variable $remote_addr. With the second parameter we set the variable $upstream to the name of the upstream group.

Using the Key-Value Store with Split Clients

Prior to NGINX Plus R13, if you wanted to change the percentages for the split, you had to edit the configuration file and reload the configuration. Using the key‑value store, you simply change the percentage value stored in the key‑value pair and the split changes accordingly, without the need for a reload.

Building on the use case in the previous section, let’s say we have decided that we want NGINX Plus to support the following options for how much traffic gets sent to appversion2: 0%, 5%, 10%, 25%, 50%, and 100%. We also want to base the split on the Host header (captured in the NGINX variable $host). The following NGINX Plus configuration implements this functionality.

As mentioned for the initial use case, in an actual deployment it makes sense to base the split on a request characteristic like the client IP address, $remote_addr. In a simple test using a tool like curl, however, all the requests come from a single IP address, so there is no split to observe.

For the test, we instead base the split on a value that is more random: $request_id. To make it easy to transition the configuration from test to production, we create a new variable in the server block, $client_ip, setting it to $request_id for testing and to $remote_addr for production. Then we set up the split_clients configuration.

The variable for each split percentage (split0 for 0%, split5 for 5%, and so on) is set in a separate split_clients directive:

Finally, we have the rest of the configuration for the upstream groups and the virtual server. Note that we have also configured the NGINX Plus API which is used for the key‑value store and the live activity monitoring dashboard. This is the new status dashboard in NGINX Plus R14:

Using this configuration, we can now control how the traffic is split between the appversion1 and appversion2 upstream groups by sending an API request to NGINX Plus and setting the $split_level value for a hostname. For example, the following two requests can be sent to NGINX Plus so that 5% of the traffic for www.example.com is sent to the appversion2 upstream group and 25% of the traffic for www2.example.com is sent to the appversion2 upstream group:

Conclusion

This is just one example of what you can do with the key‑value store. You can use a similar approach for request‑rate limiting, bandwidth limiting, or connection limiting. Dynamic IP Blacklisting with NGINX Plus and fail2ban discusses a detailed IP address blacklisting use case, and there are many other possibilities.

If you don’t already have NGINX Plus, start your free 30‑day trial and give it a try.

Have a Cookie? :)

Our site uses cookies to provide functionality and performance as well as for social media and advertising purposes. Social media and advertising cookies of third parties are used to offer you social media functionalities and personalized ads for NGINX content and offers. To get more information about these cookies and how we process personal data, check our Privacy Policy. Do you accept the use of cookies and the processing of personal data involved?

Your Cookie Settings

Site functionality and performance

These cookies are required for NGINX site functionality and are therefore always enabled. These include cookies that allow you to be remembered as you explore the NGINX site, help make the shopping cart and checkout process possible as well as assist in security issues and conforming to regulations. To use the NGINX website, you have to consent to these cookies and the processing of personal data according to the NGINX website terms of use and privacy policy.

Social media and advertising

Social media cookies offer the possibility to connect you to your social networks and share content from our website through social media. Advertising cookies (of third parties) collect information to help better tailor NGINX advertising to your interests, both within and beyond NGINX websites. De-selecting these cookies may result in seeing advertising that is not as relevant to you or you not being able to link effectively with Facebook, Twitter, or other social networks and/or not allowing you to share content on social media.