Monthly Archives: April 2015

Direct Server Return requires raw access to network traffic. There is no portable way to do that, so Pen supports it so far only on Linux and FreeBSD.

On Linux, the probe into the network stack is created pretty much like any other socket:

fd = socket(AF_PACKET, SOCK_RAW, IPPROTO_RAW);

It works literally like a probe, the packets are duplicated out and received both by the kernel and by the program creating the socket, Pen in this case. Care must be taken to prevent the kernel from interferring, for example by responding to TCP traffic.

Here, d is not a socket but a “netmap descriptor” and reading and writing is done to rings of buffers, matching what is available in the network card. The regular network stack is cut off from the traffic. The details are hidden behind a pcap-like api:

nm_nextpkt(d, &h);
nm_inject(d, b, len);

Now, wouldn’t it be fun to compare these two? Of course it would!

Two VMs are prepared with Apache and the address 192.168.100.1 on a loopback interface.

One VM runs ApacheBench like this:

ab -c 20 -n 10000 http://192.168.100.1/1000k

One Linux and one FreeBSD VM are prepared with the latest Pen from Git. On Linux the command line looks like:

Interesting observation made while testing direct server return through Pen in VirtualBox.

Using e1000 drivers will send 1514 byte frames. Pen receives the frames, scribbles a bit on the headers and puts them back on the network. Iperf performance through Pen is pretty decent, 200 Mbps. Direct iperf without Pen in between is 400 Mbps. That makes sense, total throughput is limited by the host running all three virtual servers.

With virtio drivers, iperf performance without Pen is 1 Gbps. But performance with Pen drops to < 100 kbps! What's up with that? It turns out that the virtio nic drivers support TCP Segmentation Offload and will send oversized frames, expecting the physical nic to do the frame segmentation. With an entirely virtual network, no segmentation takes place. As a result, Pen receives oversized frames which it can't forward. The frames are dropped and TCP gets grumpy.
To get rid of this behaviour, TSO must be turned off on the iperf client:
ethtool -K eth1 tso off

However, this makes direct iperf performance between two vm:s drop from 1 Gbps to 365 Mbps. Also, iperf performance through Pen drops from 200 Mbps to 60 Mbps. So virtio nics are actually slower for this purpose than software emulated e1000.

Direct server return means that traffic from the client to the server goes through Pen, but the traffic from the server to the client does not. There are a number of advantages to doing it this way, especially when it comes to performance – the load balancer never has to touch the return traffic. There are disadvantages as well, for example the load balancer cannot do SSL termination and the servers need special configuration.

Very, very early development, but it already kind of works. Ignore the mediocre performance, the code is running with debug level “blazing madness”.

150408 Updated ocsp stapling to be compatible with server name indication.
Added pen-ocsp.sh script.
Released 0.27.4.

150407 SSL code broken out into ssl.[ch]. SSL context creation broken
out from ssl_init to ssl_create_context.
Server Name Indication support. New command to enable:
ssl_sni_path PATH
where PATH is the name of a directory containing domain.key,
domain.crt and domain.ca files for each domain.

150406 OCSP stapling. New command ssl_ocsp_response filename
specifies the location of the ocsp response to be stapled.
The response must be pre-fetched. The idea was borrowed
from Rob Stradling.

150403 New command ssl_client_renegotiation_interval specifies the
minimum number of seconds the client must wait between
renegotiation requests. Default 3600.

Server Name Indication is a TLS extension which allows multiple virtual hosts, and multiple digital certificates, to coexist on the same IP address and port. It is among the features destined for Pen 0.27.4.

To enable SNI in Pen, all that needs to be done is this line in the configuration file:

ssl_sni_path /etc/pen/sni

Then add the private key, your certificate and the CA’s certificate to that directory, using this naming scheme:

Pen 0.27.4 introduces support for OCSP stapling. What the heck is this, you may ask.

Https websites use certificates to prove that they are who they claim they are. The certificates are signed by trusted entities, called certificate authorities. Trusted by whom? The person using the browser most likely have no idea what the CA:s are or who works there. But the browser has a built-in list of CA:s that it trusts. It is at least a little bit of trust, so let’s say that a little trust is better than no trust.

The certificates are good for a year or more, after which time they need to be renewed. But what if the certificate needs to be revoked before its time is up?

Kludge upon kludge upon kludge

One such scheme is OCSP, “online certificate status protocol”, where the browser connects to the CA and asks whether the certificate is still valid. Unlike the certificate itself, which has a life span of years, the OCSP response has a much shorter life of hours.

One may think that if the certificate had a shorter life to begin with, none of this would be necessary. But let’s ignore that for now. Surely this fixes the problem with trusting certificates that shouldn’t be trusted?

Well, not quite. There are a bunch of problems with OCSP. For one, it requires that the browser is free to connect to the OCSP responder (the thingy on the CA:s web site that says the certificate is good). If the guys administrating the network haven’t opened the firewall to every potential OCSP responder (remember that list of “trusted” certificate authorities in the browser), OCSP doesn’t work.

By the way, care to guess what a browser does by default if the certificate validity can’t be verified? It doesn’t care. It just carries on as if the certificate were good.

There are more problems with OCSP: it creates a lot of traffic to the CA:s, traffic that they have to pay for, making operating a CA more expensive. And the queries disclose to the CA what web sites the user visits. And the queries are typically unencrypted, so anyone with a wiretap knows too. Let’s make a list.

– The user visiting https://shadybusiness.com/ knows (we hope)
– Shady Business knows
– Shady Business’ CA knows
– Every governmental agency on the way from Shady Business to the CA knows

This is obviously totally broken. It is less secure than if Shady Business had used a self-signed certificate. It is less secure than if Shady Business had used plain http.

Pull out the Swingline

What if the web server, instead of relying on the browser connecting to the CA to check the certificate status, sent the OCSP response itself? The response would still need to be pulled from the CA, but by the certificate owner, and only every few hours. This solves the traffic problem for the CA, it solves the privacy issue, and it saves the networking guys a bit of firewall configuration.

So how does one implement this? Let’s say, in an SSL-terminating load balancer?

A clue

Rob Stradling proposed this approach for Nginx:

You’ve got me thinking…

In the Apache httpd implementation, mod_ssl contains code which regularly
downloads and caches OCSP Responses in shared memory so that they’re ready to
be Stapled.

It occurs to me that the same result could be achieved without having to put
the downloading/caching code inside the webserver. A cronjob could run a
shell script that…
– runs the OpenSSL command-line tool to download the appropriate OCSP
Responses, saving them to the hard disk.
– signals the webserver to reload its config file.

With this cronjob-based approach, the webserver would only require a small
amount of new code:
– a new config directive for specifying the filename of an OCSP Response (or
perhaps the code could just look for .ors: if present, use
it for Stapling; if absent, Stapling is disabled).
– code to read the OCSP Response file.
– a call to SSL_CTX_set_tlsext_status_cb().
– some sanity checks on the OCSP Response (e.g. is it still time-valid?)
– a call to SSL_set_tlsext_status_ocsp_resp() to actually staple the OCSP
Response.

This cronjob-based approach should perform better than Apache/mod_ssl’s
approach, because it would avoid the use of a mutexed shared-memory cache.

This is actually the typical situation when you try to do anything with OpenSSL. The manpages, when they exist, document details but not the big picture, and when they don’t exist they don’t document anything.

Implementation

After an actually very small amount of trial and error, here is a simplified version of the callback that does the stapling:

Your private key is in the file your.domain.key. The file your.domain.csr contains your certificate signing request, which needs to be sent to your certification authority. The details of that procedure is different depending on the CA, but should result in you having your new certificate in your possession. Save the certificate as your.domain.crt.

The final piece of information you need is the CA’s certificate, which the CA will provide. Save the certificate as intermediate.crt.

Assuming you managed to cobble together all these files in the directory /etc/pen, the certificate installation is now finished.

Protocol Support

This is easy. Nobody supports SSL 2.0 anymore. SSL 3.0 is only for IE6 on Windows XP, a dwindling user base. TLS 1.0 is still acceptable, but this is not an exercise in acceptability (or compatibility). Throw out everything but TLS 1.2 by putting the following in /etc/pen/https.cfg: