Server Administration

So just a moment ago I was wondering why my Let’s Encrypt certificate on a particular server wasn’t renewing. I’m using a simple script called acme-tiny for Let’s Encrypt as it’s just about the lightest implementation for it. However it failed with this error message:

I checked one of the sites I use Let’s Encrypt free SSL certificates on SSL Labs’ SSL Test recently and found to my dismay that they only got a “B” grade while I was pretty sure I got an “A” grade before. Looking at the details it showed the intermediate certificate chain to be incomplete and the dreaded “extra download” warning.

When I set up Let’s Encrypt I used the simple acme-tiny tool and naturally also configured Apache to to use the generated certificate properly, including sending the intermediate certificate. Some searching suggested that it might be because the relevant directive was deprecated in Apache 2.4.8. But the affected sites (incl. this blog) run on a VPS with Ubuntu 14.04 which includes Apache 2.4.7, so that change was not relevant to me (it will be when you upgrade Apache to a later version, like when upgrading to Ubuntu 16.04!).

So after some experimenting I finally figured it out: when the Let’s Encrypt beta went live all the certificates were signed by the “Let’s Encrypt Authority X1”. However sometime in March they changed this and now all certificates are signed by “Let’s Encrypt Authority X3”. My server was sending the wrong intermediate certificate (X1 instead of X3). It’s one of those problems that once you find out what is wrong, is very easy to fix, just understanding what was wrong took some effort.

Links for Let’s Encrypt intermediate certificates can be found here. Acme-Tiny has updated their readme as well.

HTTP is the most important protocol on the web, but hasn’t had a major update in a long time (HTTP 1.1 was introduced in 1997). In May 2015 however the HTTP 2.0 (HTTP/2) standard was published. The new standard promises increased speed.

As of right now http/2 will only work on HTTPS sites. Although the HTTPS requirement isn’t in the standard (although it nearly did), the most popular browser implementations (Chrome, Firefox, etc) don’t support http/2 with anything but HTTPS.

Now to enable http/2 on your Ubuntu server, just run this command (as root, prefix with sudo when necessary):

add-apt-repository ppa:nginx/development

Then run apt-get update and apt-get install nginx as usual. This will install the latest mainline version of nginx (http/2 support is included as of version 1.9.5). Despite the name of the PPA repository this is actually the recommended version: “We recommend that in general you deploy the NGINX mainline branch at all times”.

Now there is a slight problem: if you use this method on the current (as of writing) version of Ubuntu LTS (14.04) then an important part of http/2 will not work: ALPN. This TLS extension is used to enable http/2 during the connection phase. Without it another roundtrip to the server is needed for browser and server to agree on using http/2.

To enable ALPN you need a version of nginx that is built with OpenSSL 1.0.2. This version of OpenSSL is included from Ubuntu version 15.10 onwards (including 16.04 LTS). When installing on Ubuntu 14.04 the nginx PPA repository assumes you have the default version of OpenSSL installed (v1.0.1), hence no ALPN support.

That leaves three options: upgrade to 15.10 (not an LTS release, so not preferable on servers), build the latest version of nginx and OpenSSL yourself (which creates an upgrade burden) or live with having slightly crappy http/2 support until Ubuntu 16.04 LTS is available.

I have Cacti installed so that I have pretty graphs (CPU/load/memory/disk/network) for all my servers (VPS) in one place. I just enabled full IPv6 on one of them and Cacti stopped working. The problem wasn’t Cacti though, it was the server it was now trying to contact through IPv6.

As I only need SNMP for Cacti, the snmpd.conf file on all my servers is very simple, just:

rocommunity public

However for IPv6 that isn’t enough. On Ubuntu 14.04 (what the server is running) snmpd will only work on IPv4 this way. I quickly found out you can use an agentAddress directive in the snmpd.conf file to enable the daemon to listen on IPv6, but more changes are required if you want to actually return data through IPv6. As it turns out the “rocommunity” directive is apparently also IPv4 only.

The final solution was this:

agentAddress udp:161,udp6:161
rocommunity public
rocommunity6 public

Warning: above configuration will cause snmpd to listen on all network interfaces. Be sure to lock down access to UDP port 161 in your firewall to only authorized hosts. And make sure your firewall supports IPv6.

Let’s Encrypt is now in public beta phase. It allows you to create free unlimited trusted SSL certificates that work in all browsers (all relevant browsers anyway). There are now no excuses for adding SSL encryption to your website(s).

That isn’t to say that Let’s Encrypt isn’t without flaws: the certificates you can get are only valid for a maximum of 90 days. While you can of course renew them, if you don’t have some kind of automated way of renewing your certificates it can quickly become a pain to keep them up-to-date.

The other issue (at the moment) is that Let’s Encrypt own monolithic tool for requesting certificates requires to be run as root on your server, which no doubt some people will have issues with. Fortunately the protocol (ACME) used is public and alternative clients are available. I personally used letsencrypt-nosudo to issue my first certificate (this tool allows you to register an account with an email address with Let’s Encrypt, which might be useful) and intent to issue further certificates (and renewals) with the very simple acme-tiny.

If you haven’t set up SSL on your server before, these resources might also be useful:

I intend to use Let’s Encrypt for all my less important domains (which includes this site) until its reliability is proven and CheapSSLSecurity (Comodo PositiveSSL 3 years for $4.99/year) for the rest in the mean time. No need to pay any more.