Pages

In this post I'll demonstrate one of the many ways to setup a multitenant and highly available Redis cluster using Amazon Web Services, OpenVZ containers, Open vSwitch with GRE tunneling, HAProxy and keepalived on CentOS 6.5.

This is what the architecture looks like:

I'll use one VPC, with two Subnets, 3 EC2 instances for the Redis and Sentinel containers, and two more EC2 instances for the HAProxy and keepalived proxy layer.

Each OpenVZ container will be part of an isolated network, the entry point to which is the HAProxy node. This is achieved by using Open vSwitch with a mesh of GRE tunnels.

First create the VPC subnets and the EC2 instances, then on the Redis instances install the OVZ Kernel:

With this the 3 Redis/Sentinel nodes are complete. Next let's move on to EC2 Node 4 and 5.

First, create the same GRE tunnels to the other 3 EC2 nodes as shown previously, the main difference here is that the bridge interface will have an IP that's part of the GRE mesh subnet to allow the other hosts to reach the containers:

Keepliaved will be using unicast messages between Node 4 and Node 5. When the state of the node changes it will trigger a script defined on line 29, which will re-assign the private IP from one of the EC2 instances to the other, and keepliaved will raise the IP and send an unsolicited ARP broadcast. For this to work we need to have the aws cli installed (lines 3-5), and auth working (lines 71).

If Node 4 and 5 have a public IP you should be able to connect to the current Redis master, as decided by the Sentinels by using the redis-cli command. To test a failover just shut down redis-server on the current master container and watch the sentinels promote a new master and HAProxy detecting the change (lines 107-113).