Elsewhere

I spent the better part of today and yesterday trying to fix something that was broken because I was typing the wrong IPv4 address! Instead of connecting to (fake) address 36.51.55.1 I was typing 36.55.51.1. And since it wasn’t working I kept thinking it was a configuration error. Grr!

Here’s what I am trying to do:

I have a Linux server with a /64 IPv6 block assigned to it. The server has IPv6 address 2dcc:7c4e:3651:55::1/64, IPv4 address 36.51.55.1/24. (The server is a virtual machine on my laptop so these aren’t real public IPs).

I want a Windows client to connect to this server via OpenVPN, get/ assign itself a public IPv6 address from the /64 block, and have IPv6 connectivity to the Internet. Easy peasy?

I could use services like SixSx or Hurricane Electric to get a tunnel, but I am curious whether it will work over OpenVPN. The idea is to do it in a test lab now and replicate in the real world later (if I feel like it). It’s been a while since I used OpenVPN so this is a good chance to play with it too.

Note

I want a single client to connect to the server so I don’t need a fancy server setup. What I am trying to achieve is a peer-to-peer setup (aka point-to-point setup). Below I refer to the client as OpenVPN client, the server as OpenVPN server, but as far as OpenVPN is concerned it’s a peer-to-peer setup.

Bridging & Routing

OpenVPN has two ways of connecting machines. One is called routing, wherein the remote machine is assigned an IP address from a different network range and so all remote machines are as though they are on a separate network. This means the OpenVPN server machine acts as a router between this network and the network the server itself is connected to. Hence the name “routing”.

The other method is called bridging, and here the OpenVPN server connects the remote machines to the same network it is on. Whereas in routing the OpenVPN server behaves like a router, in bridging the OpenVPN server behaves like a switch. All remote machines have IP addresses from the same network as the OpenVPN server. They are all on the same LAN essentially.

The advantage of bridging is that it works at the layer 2 level (because the remote machine is plugged in to the real network) and so any protocol in the upper levels can be used over it. In contrast routing works at layer 3 (because the remote machine is plugged into a separate network) and so only layer 3 protocols supported by OpenVPN can be used. Previously it was only IPv4 but now IPv6 support too exists. I could use either method in my case, but I am going ahead with bridging because 1) I want the remote machines to behave as though they are connected to the main network, and 2) I just have a /64 block and I don’t want to muck about with subnetting to create separate networks (plus blocks smaller than /64 could have other issues). Maybe I will try routing later, but for now I am going to stick with bridging.

OpenVPN: Quick setup on the Server side

A good intro to setting up OpenVPN can be found in the INSTALL file of OpenVPN. It’s not a comprehensive document, so it’s not for everyone, but I found it useful to get started with OpenVPN. The steps I am following are from the “Example running a point-to-point VPN” section with some inputs from the manpage. A point-to-point setup is what I want to achieve.

OpenVPN has various switches (as can be seen in the manpage). You can invoke OpenVPN with these switches, or put them in a config file and invoke the config file. For a quick and dirty testing it’s best to run OpenVPN with the switches. Here’s how I launched the server:

1

openvpn--dev tap

In practice, you shouldn’t do this as it launches the tunnel with NO encryption. You even get a warning from OpenVPN:

1

*******WARNING*******:all encryption andauthentication features disabled--all data will be tunnelled ascleartext

A better approach is to generate a shared key first and copy it to the client(s) too. A shared key can be generated with the --secret switch:

1

openvpn--genkey--secret secret.key

And OpenVPN can be launched with this shared key thus:

1

openvpn--dev tap--secret secret.key

Note

Remember to copy the shared key to the client. If a client connects without the shared key it will fail:

Notice I am using the --dev switch to specify that I want to use a TAP device.

TAP devices

TAP and TUN are virtual devices. TAP is a layer 2 virtual device, TUN is a layer 3 virtual device.

Each OpenVPN instance on the server creates a TAP device (if it’s configured to use TAP) – names could be tap0, tap1, and so on. It’s important to note that each OpenVPN instance on the server uses a single TAP device – you don’t create TAP devices for each client. Similarly OpenVPN on the client side creates a TAP device when it’s launched. After creating the TAP device, OpenVPN client contacts OpenVPN server and both setup a communication between their respective TAP devices. This “communication” is basically an encrypted UDP connection (by default; though you could have non-encrypted UDP or TCP connections too) from the client to the server. The client OS and applications think its TAP device is connected to the server TAP device and vice versa, while the OpenVPN programs on both ends do the actual moving around of the packets/ frames.

In my case, since I am doing a peer-to-peer connection, the TAP device created on the server will be used by a single client. If I want to add another client, I’ll have to launch a new OpenVPN instance – listening on a separate port, with a separate TAP device, preferably with a separate secret.

If multiple clients try to use the same TAP device on the server, both clients won’t work and the server generates errors like these:

May1409:41:58cen-svr01 openvpn[2796]:Authenticate/Decrypt packet error:bad packet ID(may beareplay):[#150 / time = (1400045966) Wed May 14 09:39:26 2014 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

This is because packets from both clients are appearing on the same TAP device and confusing the OpenVPN server.

Creating a bridge

When OpenVPN on the server creates a TAP device it is not automatically connected to the main network. That is something additional we have to do. On Linux this is done using the bridge-utils package. Since I am using CentOS, the following will install it:

1

yum install bridge-utils

A bridgeis the (virtual) network switch I was talking about above. It’s a virtual device, to which you connect the machine’s network adapter as well as the TAP adapters you create above. Keep in mind a bridge is a layer 2 device. It doesn’t know about IP addresses or networks. All it knows are MAC addresses and that it has (virtual) network adapters plugged in to it.

The bridge can have any name, but let’s be boring and call it br0. I create a bridge thus:

1

brctl addbr br0

Next I add the physical network card of my machine to the bridge. Think of it as me connecting the network card to the bridge.

1

brctl addif br0 eth0

Here’s a catch though. Your physical network adapter is already connected to a switch – the physical switch of your network! – so when you connect it to our virtual switch it is implicitly disconnected from the physical switch. There’s no warning given regarding this, just that you lose connectivity. As far as the machine is concerned your physical network adapter is now connected to the virtual switch (bridge), and this virtual switch is connected to the physical switch.

What this means is that all the network settings you had so far configured on the physical adapter must now be put on the bridge. Bridges are not meant to have IP addresses (remember they are layer 2 devices) but this is something we have to do here as that’s the only way to connect the bridge to the physical network.

To make this change explicit let’s zero IP the physical interface. It’s not really needed (as the interface automatically loses its settings) but I think it’s worth doing so there’s no confusion.

1

ifconfig eth00.0.0.0

Next I assign the bridge interface the IP settings that were previously on the physical adapter:

1

ifconfig br036.51.55.1netmask255.255.255.0up

Since I am on CentOS and I’d like this to be automatically done each time I start the server I must make changes to the /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-br0 files. I just copied over the ifcfg-eth0 file to ifcfg-br0 and renamed some of the parameters in the file (such as the interface name, MAC address, etc).

Here’s my /etc/sysconfig/network-scripts/ifcfg-eth0 file. It’s mostly the defaults except for the last two lines:

1

2

3

4

5

6

7

8

DEVICE=eth0

HWADDR=00:50:56:24:7F:3F

TYPE=Ethernet

UUID=88e62576-9c19-4489-a0a9-62fd2e7b8078

ONBOOT=yes

NM_CONTROLLED=no

BRIDGE=br0

PROMISC=yes

This Fedora page is a good source of information on setting up a bridge in CentOS/ RHEL/ Fedora. The PROMISC=yes line is added to set the physical adapter in promiscuous mode. This is required by OpenVPN. (By default network adapters only accept frames corresponding to their MAC address. Promiscuous mode tells the adapter to accept frames for any MAC address. Basically you are telling the adapter to be promiscuous!)

Next, here’s my /etc/sysconfig/network-scripts/ifcfg-br0 file:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

DEVICE=br0

TYPE=Bridge

DELAY=0

UUID=2c498c46-722b-4486-bbab-403b811a6860

ONBOOT=yes

NM_CONTROLLED=no

IPADDR=36.51.55.1

NETMASK=255.255.255.0

GATEWAY=36.51.55.254

DEFROUTE=yes

BOOTPROTO=static

USERCTL=no

PEERDNS=no

NETWORKING_IPV6=yes

IPV6INIT=yes

IPV6ADDR=2dcc:7c4e:3651:55::1

# setting it to no for now

IPV6_AUTOCONF=no

# either specify zone id here ...

#IPV6_DEFAULTGW=fe80::254%br0

IPV6_DEFAULTGW=fe80::254

# ... or specify the default device here

IPV6_DEFAULTDEV=br0

The first three lines are what matter here. They give the device a name, set it as type Bridge, and tell the kernel it needn’t delay in passing frames through the bridge (by default there’s a 30s delay). The other lines are from the original eth0 configuration file, modified for br0.

Once these two files are in place, even after a reboot the bridge interface will be automatically created, assigned an IP address connecting it to the physical switch, and the physical adapter will be put in promiscuous mode and connected to this bridge.

Creating TAP devices

Back to TAP devices. Once OpenVPN creates a TAP device it must be added to the bridge previously created. This is what connects the TAP device to the main network.

Typically a shell script is used that adds the dynamically created TAP device to the bridge. For instance OpenVPN’s Ethernet Bridging page has a BASH script. Similarly the OpenVPN INSTALL document gives another BASH script (which I like more and for some reason made more sense to me).

CentOS installs an OpenVPN wrapper script in /etc/init.d/openvpn which lets you start and stop OpenVPN as a service. This wrapper script automatically starts an OpenVPN process for each config file ending with the .conf extension in the /etc/openvpn directory and if a /etc/openvpn/xxx.sh script exists for the xxx.conf file it executes this file before launching the OpenVPN process. Thus CentOS already has a place to put in the script to add TAP interfaces to the bridge.

(CentOS also looks for two files /etc/openvpn/openvpn-start and /etc/openvpn/openvpn-shutdown which are run when the wrapper script starts and shutdown – basically they are run before any of the configuration files or scripts are run. I don’t think I’ll use these two files for now but just mentioning here as a future reference to myself).

The shell script can behave in two ways – depending on how TAP devices are created. TAP devices can be created in two ways:

They can be created on the fly by OpenVPN – in which case you’ll need to invoke the script via an up switch; the script will be passed a variable $dev referring to the TAP device name so it can be set in promiscuous mode and added to the bridge.

To create TAP devices on the fly use the --dev tap switch – notice it doesn’t specify a specific TAP device such as tap0. OpenVPN will automatically create a tap0 (or whatever number is free) device.

They can be pre-created as persistent TAP devices – these will stay until the machine is rebooted, and are preferred for the reason that you don’t need --up and --down switches to create them, and the devices stay on even if the connection resets. In this case the script can create TAP devices and configure and add them to the bridge – independent of OpenVPN.

To use a specific TAP device use the --dev tap0 switch (where tap0 is the name of the device you created).

I will be using persistent TAP devices. These can be created using the --mktun switch:

1

openvpn--mktun--dev tap0

Next I set them in promiscuous mode, bring up the device, zero the IP, and set it in promiscuous mode:

1

ifconfig tap00.0.0.0promisc up

Lastly I add these to the bridge:

1

brctl addif br0 tap0

Assuming my configuration file is called /etc/openvpn/ipv6.conf I’ll create a file called /etc/openvpn/ipv6.sh and add the following:

1

2

3

4

#!/bin/bash

openvpn--mktun--dev tap0

ifconfig tap00.0.0.0promisc up

brctl addif br0 tap0

Don’t forget to make the file executable:

1

chmod+x/etc/openvpn/ipv6.sh

Now whenever the OpenVPN service on CentOS starts, it will run the script above to create the TAP device, configure and add it to the bridge, then launch OpenVPN with a configuration file that will actually use this device. This configuration file will refer by name to the TAP device created in this script.

Firewall

The OpenVPN server listens on UDP port 1194 by default. So I added a rule to allow this port to iptables.

In CentOS I made the following change to /etc/sysconfig/iptables:

1

-AINPUT-mstate--state NEW-pudp--dport1194-jACCEPT

Then I reloaded the iptables service so it loads the new rule:

1

service iptables reload

Forwarding

Next I enabled IPv4 & IPv6 forwarding. I added the following lines to /etc/sysctl.conf:

1

2

net.ipv4.ip_forward=1

net.ipv6.conf.all.forwarding=1

And got sysctl to read the file and load these settings:

1

sysctl-p

IPv4 forwarding is not needed in my case, but I enabled it anyways. Also note that I am enabling IPv6 forwarding for all interfaces. That’s not necessary, the forwarding can be enabled for specific interfaces only.

In addition to enabling forwarding at the kernel level, iptables too must be configured to allow forwarding. The IPv6 firewall is called ip6tables and its configuration file is /etc/sysconfig/ip6tables. The syntax is similar to /etc/sysconfig/iptables and in CentOS forwarding is disabled by default:

1

2

:FORWARD ACCEPT[0:0]

-AFORWARD-jREJECT--reject-with icmp6-adm-prohibited

I commented out the second line and reloaded the service.

1

2

:FORWARD ACCEPT[0:0]

#-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited

1

service ip6tables reload

Lastly, I added the following line to /etc/system-config/network:

1

IPV6FORWARDING=yes

This tells the initscripts bringing up the interfaces that IPv6 forwarding is to be enabled on all interfaces. Again, this can be specified on specific interfaces – in which you you’d put it in the /etc/system-config/network-scripts/ifcfg-xxx file for that interface – or enabled globally as above but disabled in the file of specific interfaces. For the curious, the initscripts-ipv6 homepage describes more options.

To summarize

To summarize what I’ve done so far:

Create a bridge, add the physical adapter to the bridge, configure the bridge.

Create persistent TAP devices – best to create a script like I did above so we can set it to run automatically later on.

Create a secret key, copy the secret key to the client(s).

Modify firewall, enable forwarding.

Launch OpenVPN, telling it to use the secret key and TAP. No need to specify the TAP device name, OpenVPN will figure it out.

1

openvpn--dev tap--secret secret.key

Mind you, this is a quick and dirty setup. Using shared keys won’t work scale, but they will do fine for me now. And rather than launch OpenVPN directly I must put it into a conf file with some logging and additional options … but that can be done later.

Troubleshooting

A very useful switch for troubleshooting purposes is --verb x where x is a number from 0 to 11. 0 means no output except fatal errors. 1 (the default) to 4 mean normal usage errors. 5 to be more verbose, and also output R and W characters to the console for each packet read and write, uppercase is used for TCP/UDP packets and lowercase is used for TUN/TAP packets. And 6-11 for debug range output.

I usually stick with --verb 3 but when troubleshooting I start with --verb 5.

Also, while UDP is a good protocol to use for VPN, during troubleshooting it’s worth enabling TCP to test connectivity. In my case, for instance, the client seemed to be connecting successfully – there were no error messages on either end at least – but I couldn’t test connectivity as I can’t simply telnet to port 1194 of the server to see if it’s reachable (since it was uses UDP). So I enabled TCP temporarily, got OpenVPN on the server to listen on TCP port 1194, then did a telnet to that port from the client … and realized I had been typing the wrong IPv4 address so far!

Here’s how you launch OpenVPN server with TCP just in case:

1

openvpn--dev tap--secret secret.key--verb3--proto tcp-server

Be sure to open the firewall and use the corresponding switch --proto tcp-client on the client end too.

OpenVPN: Quick setup on the Client side

The server is running with the following command (remember to add --verb x if you want more verbosity):

1

openvpn--dev tap--secret secret.key

I launch the client thus (again, remember to add --verb x if you want more verbosity):

The --dev tap switch tells OpenVPN to use a TAP device. On Windows TAP devices are persistent by default so this will be a persistent TAP device.

Port is 1194, the default, so I don’t need to specify explicitly. Similarly protocol is UDP, the default.

The shared key is specified using the --secret switch.

And that’s it! Now my client’s connected to the server. It doesn’t have an IPv6 address yet because the server doesn’t have DHCPv6 or Router Advertisements turned on. I could turn these ON on the server, or assign the client a static IPv6 address and gateway. The address it gets will be a global IPv6 address from the /64 block so you should be able to connect to it from anywhere on the IPv6 Internet! (Of course, firewall rules on the client could still be blocking packets so be sure to check that if something doesn’t work).

Wrapping up

Here’s the final setup on the server and client. This is where I put the settings above into config files.

On the server side I created a file called /etc/openvpn/ipv6.conf:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

# network settings

proto udp

port1194

dev tap0

# crypto

secret/etc/openvpn/ipv6.key

# restart control

# see manpage https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage for details

persist-key

persist-tun

ping-timer-rem

ping-restart60

ping10

# compression

comp-lzo

# UID

user nobody

group nobody

# verbosity

verb3

I renamed the previously generated key to /etc/openvpn/ipv6.key.

Created /etc/openvpn/ipv6.sh:

1

2

3

4

#!/bin/bash

openvpn--mktun--dev tap0

ifconfig tap00.0.0.0promisc up

brctl addif br0 tap0

Also created /etc/openvpn/openvpn-shutdown to remove the TAP devices when the OpenVPN service stops:

1

2

3

#!/bin/bash

brctl delif br0 tap0

openvpn--rmtun--dev tap0

Apart from this I created /etc/sysconfig/network-scripts/ifcfg-br0 and /etc/sysconfig/network-scripts/ifcfg-eth0 as detailed earlier. And made changes to the firewall rules.

On the client side I created a file C:\Program Files\OpenVPN\config\ipv6.ovpn:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

remote36.51.55.1

# network settings

proto udp

port1194

dev tap

# crypto

secret"C:\\Program Files\\OpenVPN\\config\\ipv6.key"

# restart control

# see manpage https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage for details

persist-key

persist-tun

ping-timer-rem

ping-restart60

ping10

# compression

comp-lzo

# verbosity

verb3

script-security2

up'C:\\Progra~1\\OpenVPN\\config\\ipv6-up.cmd'

down'C:\\Progra~1\\OpenVPN\\config\\ipv6-down.cmd'

I copied the shared key to C:\Program Files\OpenVPN\config\ipv6.key. Note that it is important to escape the path name in the ovpn file on Windows.

I created two command files to be run when the client connects and disconnects. These are specified by the --up and --down switches.

Here’s ipv6-up.cmd:

1

2

3

c:\windows\system32\netsh.exeinterfaceipv6 set address"Local Area Connection"2dcc:7c4e:3651:55::3

I noticed that it was trying to run the ipv6-up.cmd file without escaping the space between “Program Files”. Must be an OpenVPN bug, so to work around that I used the short name path.

To connect to the server I launch OpenVPN thus:

1

C:\Program Files\OpenVPN\config>..\bin\openvpn.exe--config ipv6.ovpn

OpenVPN for Windows also installs a GUI system tray icon (called OpenVPN GUI). If your config files end with the ovpn extension and are in the C:\Program Files\OpenVPN\config\ folder you can right click this tray icon and connect. Very handy!

Note

This post was updated to include the wrapping up section above. And some errors in my understanding of TAP devices were corrected.

This post was further updated to add the netsh files for the client and also to clarify that this is a peer-to-peer setup. It will only work with one client at a time. Multiple clients will fail unless you create separate OpenVPN instances (on different ports and TAP adapters) to service them.

The problem was this: I have specified the default gateway as fe80::254 but that’s a link-local address. How does the system know which link to use this with? I’d either have to specify a scope with the gateway address, or clarify which network interface to use. That’s what the error message is trying to say.

So the solution is one of the below.

Either specify the scope id:

1

IPV6_DEFAULTGW=fe80::254%eth0

Or specify the device to use:

1

2

IPV6_DEFAULTGW=fe80::254

IPV6_DEFAULTDEV=eth0

Another thing, if your network router is sending Router Advertisements you can skip specifying the gateway altogether and ask the interface to auto configure.

1

IPV6_AUTOCONF=no

This is how you’d configure your network on a VPS. The VPS provider will have allocated to /64 block to you. Unlike IPv4 where the provider will take an IPv4 from the same network as you to use as the router for your network, with IPv6 they don’t need to do that as link-local addresses can be used. So they’ll either provide you with a link-local address, or you can turn on auto configure and pick it up yourself (provided it’s being advertised).

Lastly, I needn’t have put this configuration in the interface specific files. I could have put it in /etc/sysconfig/network too.

Nothing new here, just stuff I can refer to later when I forget. I’ve usually worked with Debian but am not working with CentOS for a change so some things are new to me.

If you install CentOS in a machine with less than 1024 MB RAM you don’t get the graphical installation option. So best to install with 1024 MB RAM and then downgrade RAM if you want to. I have a CentOS VM with less than 512 MB RAM. I don’t necessarily need a graphical installation but I prefer it as it’s friendly. I get to choose the hostname for instance, select the packages I want to install, and so on. Without a graphical install you are given a minimal install only with no options to choose.

Set the hostname in /etc/sysconfig/networking.

Network interfaces are managed by NetworkManager, which is useful if you have a GUI or like to use the nmcli command. I prefer old fashioned ifconfig so I manage my network devices via /etc/sysconfig/network-scripts/ifcfg-* files.

Set NM_CONTROLLED=no in this file to tell NetworkManager not to bother with this interface.

Set ONBOOT=yes to up this interface on boot up.

Use IPADDR=a.b.c.d, NETMASK=x.y.z.f, GATEWAY=p.q.r.s to set the IPv4 address, netmask, and gateway. Use BOOTPROTO=dhcp if you’d prefer DHCP instead.

Using IPV6INIT=no and IPV6_AUTOCONF=no is supposed to turn off IPv6 for that interface but it doesn’t seem to. A better way to turn off IPv6 (and/ or control its parameters) is via sysctl.

To disable IPv6 on an interface the following sysctl setting helps: net.ipv6.conf.eth0.disable_ipv6 = 1. Add entries like these to /etc/sysctl.conf.

The following sysctl setting tells the OS to act as a router (i.e. forward packets): net.ipv4.ip_forward = 1

Use /etc/resolv.conf to specify name servers, domain suffix (via the domain keyword), and list of search suffixes (via the search keyword with a space separated list of domains). Domain suffix is the domain name to be added to hosts without a domain name. Search suffixes are additional domain names that can be added to hosts when searching. Use only one of these. If both are specified the last one wins.

Its best to put the Intranet DNS servers first in /etc/resolv.conf followed by Internet DNS servers. In my setup the Intranet servers don’t have access to the Internet and so any Internet name queries to them timeout. When this happens the DNS resolver will automatically try the next server in the list … up to when it reaches the Internet servers and get an answer. To speed the process up use option timeout:1 to set a timeout of 1 second. In contrast if I put the Internet servers first the don’t timeout – they try to resolve the non-existent domain and reply that it does not exist – so the Intranet servers aren’t queried.