Categories

Setting up Inter Region AWS VPC Peering and Latency Tests

Most of our infrastructure and client facing services are in us-east-1 and we have lots of users connecting from different parts of the world including India. Of course there was a significant latency involved when users connect to US from other part of the world. And we wanted to test that, if a user from India connects to Mumbai region(faster handshake) and then that region uses VPC peering to us-east-1 to talk to other services. Here’s how it went:

First thing we looked out for was AWS provided VPC peering, it does exist, but for limited number of regions (4, 3 US, 1 UK, at the time of writing this) and it did not include Mumbai. So we had to setup our own IPSec VPN tunnels.

The tool we chose for IPSec tunnel was OpenSwan. Found it easy to setup. We tried two different setups:

EC2 <=== tunnel ===> EC2

EC2 <=== tunnel ===> AWS Managed VPN

VPC Peering: EC2 <=> EC2

Following steps can be taken for setting up EC2, on both the regions.

Spin up an EC2 instance. Have an EIP, disable source-destination checks.

Open ports 500, 4500 UDP and Custom Protocol 50. Allow from the other side/EC2.

Modify route tables to forward traffic for other VPC to this EC2 instance. (And this EC2 will tunnel it to the other VPC)

Change kernel conf: add following in /etc/sysctl.conf and to bring it in effect: sudo service network restart

1

2

3

net.ipv4.ip_forward=1

net.ipv4.conf.all.accept_redirects=0

net.ipv4.conf.all.send_redirects=0

5. install OpenSwan: sudo yum install openswan

6.sudo vi /etc/ipsec.conf and uncomment last line to include files from ipsec.d folder.

7. Create conf files:

/etc/ipsec.d/us-mum.conf

1

2

3

4

5

6

7

8

9

10

11

conn us-mum

type=tunnel

authby=secret

left=%defaultroute

leftid=<your pub IP>

leftnexthop=%defaultroute

leftsubnet=<your VPC CIDR>

right=<opposite side pub IP>

rightsubnet=<opposite side VPC CIDR>

pfs=yes

auto=start

/etc/ipsec.d/us-mum.secrets

1

<your pub IP><other side pub IP>:PSK"changemeplease"

You would want to do similar setup in the other region's VPC EC2. Obviously, the PSK will the same and new conf files will be created ie:/etc/ipsec.d/us-mum.secrets and us-mum.conf with values changed appropriately.

Establishing tunnel:

1

2

3

sudo service ipsec start

sudo ipsec verify

sudo service ipsec status

If you see any problems with verify output, you may want to rectify it. For example if you have not set send_redirects` or not set it properly, you can do:

1

2

3

4

echo0>/proc/sys/net/ipv4/conf/all/send_redirects

echo0>/proc/sys/net/ipv4/conf/default/send_redirects

echo0>/proc/sys/net/ipv4/conf/eth0/send_redirects

echo0>/proc/sys/net/ipv4/conf/lo/send_redirects

VPC Peering: EC2 <=> AWS Managed VPN

In this case, one instance will be taken care by AWS and one will be EC2 as setup above.

To setup AWS side VPN:

Create Virtual Private Gateway on one VPC (let’s assume Mumbai) and attach it to VPC.

Latency Tests:

Here’s how we performed latency tests. We had a webserver running in US region. We had one instance in both US and Mumbai region, both had nginx, proxying requests to US region.

Here are the results:

Connecting Directly to US from Bangalore:

Initial connection: Bangalore to US

Socket reuse: connection from Bangalore to US

Connecting to Mumbai from Bangalore , request will be tunneled to US:

Initial connection: Bangalore to Mumbaisocket reuse: from bangalore to mumbai, tunneled to US

As you can see, for the first request, handshake is much faster (approx 10x) to Mumbai as it is near to client. But when you have the socket established, it’s clear that if you take Mumbai route (VPN) to US instead of going directly US, it is approx 1.5x slower as we encounter penalty for VPN encryption and decryption operations.

So that’s how that one went. Let me know if any doubts or you’re stuck anywhere. Also I would love to know how did it work for you and what improvements you saw with your setup.