Saturday, December 10th, 2016

I’ve been having a few issues with my certificate renewals with Certbot (the artist formerly known as Let’s Encrypt). If I did a dry-run for renewing my certificates…

/etc/certbot-auto renew --dry-run

… I kept getting this message:

Encountered vhost ambiguity but unable to ask for user guidance in non-interactive mode. Currently Certbot needs each vhost to be in its own conf file, and may need vhosts to be explicitly labelled with ServerName or ServerAlias directories. Falling back to default vhost *:443…

It turns out that Certbot doesn’t like HTTP and HTTPS configurations being lumped into one .conf file. Instead it expects to see all the port 80 stuff in a domain.com.conf file, and the port 443 stuff in a domain.com-ssl.conf file.

I installed the Let’s Encertbot client with this incantation (which, like everything else here, will need root-level access so if none of these work, retry using sudo in front of the commands):

wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Seems like a good idea to put that certbot-auto thingy into a directory like /etc:

mv certbot-auto /etc

Rather than have Certbot generate conf files for me, I’m just going to have it generate the certificates. Here’s how I’d generate a certificate for yourdomain.com:

/etc/certbot-auto --apache certonly -d yourdomain.com

The first time you do this, it’ll need to fetch a bunch of dependencies and it’ll ask you for an email address for future reference (should anything ever go screwy). For subsequent domains, the process will be much quicker.

The result of this will be a bunch of generated certificates that live here:

/etc/letsencrypt/live/yourdomain.com/cert.pem

/etc/letsencrypt/live/yourdomain.com/chain.pem

/etc/letsencrypt/live/yourdomain.com/privkey.pem

/etc/letsencrypt/live/yourdomain.com/fullchain.pem

Now you’ll need to configure your Apache gubbins. Head on over to…

cd /etc/apache2/sites-available

If you only have one domain on your server, you can just edit default.ssl.conf. I prefer to have separate conf files for each domain.

Make sure you update the /path/to/yourdomain.com part—you probably want a directory somewhere in /var/www or wherever your website’s files are sitting.

To exit the infernal text editor, hit ctrl and o, press enter in response to the prompt, and then hit ctrl and x.

If the yourdomain.com.conf didn’t previously exist, you’ll need to enable the configuration by running:

a2ensite yourdomain.com

Time to restart Apache. Fingers crossed…

service apache2 restart

If that worked, you should be able to go to https://yourdomain.com and see a lovely shiny padlock in the address bar.

Assuming that worked, everything is awesome! …for 90 days. After that, your certificates will expire and you’ll be left with a broken website.

Not to worry. You can update your certificates at any time. Test for yourself by doing a dry run:

/etc/certbot-auto renew --dry-run

You should see a message saying:

Processing /etc/letsencrypt/renewal/yourdomain.com.conf

And then, after a while:

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
Congratulations, all renewals succeeded.

You could set yourself a calendar reminder to do the renewal (without the --dry-run bit) every few months. Or you could tell your server’s computer to do it by using a cron job. It’s not nearly as rude as it sounds.

You can fire up and edit your list of cron tasks with this command:

crontab -e

This tells the machine to run the renewal task at quarter past six every evening and log any results:

You know, I probably should have said this at the start of this post, but I should clarify that any advice I’ve given here should be taken with a huge pinch of salt—I have little to no idea what I’m doing. I’m not responsible for any flame-bursting-into that may occur. It’s probably a good idea to back everything up before even starting to do this.

To really up your paranoia (and let’s face it, that’s what security is all about; justified paranoia), you can throw this in too:

Header unset Server
Header unset X-Powered-By

That means that your server will no longer broadcast its intimate details. Of course, I’ve completely reversed that benefit by revealing to you in this blog post that my site is running on Apache on Ubuntu.

I’ll tell you something else too: it’s powered by PHP. There’s some editing I did there too. But before I get to that, let’s just finish up that .conf file…

Hit ctrl and o, then press enter. That writes out the file you’ve edited. Now you can leave nano: press ctrl and x.

You’ll need to restart Apache for those changes to take effect. Type:

service apache2 restart

Or, if permission is denied:

sudo service apache2 restart

Now, about that PHP thing. Head over to a different directory:

cd /etc/php5/fpm

Time to edit the php.ini file. Type:

nano php.ini

Or, if you need more permissions:

sudo nano php.ini

It’s a long file, but you’re really only interested in one line. A shortcut to finding that line is to hit ctrl and w (for “where is?”), type expose, and hit enter. That will take you to the right paragraph. If you see a line that says:

expose_php = On

Change it to:

expose_php= Off

Save the file (ctrl and o, enter) then exit nano (ctrl and x).

Restart Apache:

service apache2 restart

Again, you might need to preface that with sudo.

Alright, head on back to securityheaders.io and see how your site is doing now. You should be seeing a much better score.

There’s one more thing I should be doing that’s preventing me from getting a perfect score. That’s Public Key Pinning. It sounds a bit too scary for a mere mortal like me to attempt. Or rather, the consequences of getting it wrong (which I probably would), sound too scary.

Tuesday, March 10th, 2015

After listening to Scott rave on about how much of a perceived-performance benefit he got from inlining critical CSS on first load, I thought I’d give it a shot over at The Session. On the chance that this might be useful for others, I figured I’d document what I did.

The idea here is that you can give a massive boost to the perceived performance of the first page load on a site by putting the most important CSS in the head of the page. Then you cache the full stylesheet. For subsequent visits you only ever use the external stylesheet. So if you’re squeamish at the thought of munging your CSS into your HTML (and that’s a perfectly reasonable reaction), don’t worry—this is a temporary workaround just for initial visits.

My particular technology stack here is using Grunt, Apache, and PHP with Twig templates. But I’m sure you can adapt this for other technology stacks: what’s important here isn’t the technology, it’s the thinking behind it. And anyway, the end user never sees any of those technologies: the end user gets HTML, CSS, and JavaScript. As long as that’s what you’re outputting, the specifics of the technology stack really don’t matter.

Generating the critical CSS

Okay. First question: how do you figure out which CSS is critical and which CSS can be deferred?

To help answer that, and automate the task of generating the critical CSS, Filament Group have made a Grunt task called grunt-criticalcss. I added that to my project and updated my Gruntfile accordingly:

I’m giving it the name of my locally-hosted version of the site and some parameters to judge which CSS to prioritise. Those parameters are viewport width and height. Now, that’s not a perfect way of judging which CSS matters most, but it’ll do.

The end result is that I’ve got two CSS files: the full stylesheet (called something like main.css) and a stylesheet that only contains the critical styles (called critical.css).

Cache-busting CSS

Okay, this is a bit of a tangent but trust me, it’s going to be relevant…

Most of the time it’s a very good thing that browsers cache external CSS files. But if you’ve made a change to that CSS file, then that feature becomes a bug: you need some way of telling the browser that the CSS file has been updated. The simplest way to do this is to change the name of the file so that the browser sees it as a whole new asset to be cached.

You could use query strings to do this cache-busting but that has some issues. I use a little bit of Apache rewriting to get a similar effect. I point browsers to CSS files like this:

<link rel="stylesheet" href="/css/main.20150310.css">

Now, there isn’t actually a file named main.20150310.css, it’s just called main.css. To tell the server where the actual file is, I use this rewrite rule:

That tells the server to ignore those numbers in JavaScript and CSS file names, but the browser will still interpret it as a new file whenever I update that number. You can do that in a .htaccess file or directly in the Apache configuration.

Right. With that little detour out of the way, let’s get back to the issue of inlining critical CSS.

Differentiating repeat visits

That number that I’m putting into the filenames of my CSS is something I update in my Twig template, like this (although this is really something that a Grunt task could do, I guess):

{% set cssupdate = '20150310' %}

Then I can use it like this:

<link rel="stylesheet" href="/css/main.{{ cssupdate }}.css">

I can also use JavaScript to store that number in a cookie called csscached so I’ll know if the user has a cached version of this revision of the stylesheet:

The absence or presence of that cookie is going to be what determines whether the user gets inlined critical CSS (a first-time visitor, or a visitor with an out-of-date cached stylesheet) or whether the user gets a good ol’ fashioned external stylesheet (a repeat visitor with an up-to-date version of the stylesheet in their cache).

Here are the steps I’m going through:

First of all, set the Twig cssupdate variable to the last revision of the CSS:

{% set cssupdate = '20150310' %}

Next, check to see if there’s a cookie called csscached that matches the value of the latest revision. If there is, great! This is a repeat visitor with an up-to-date cache. Give ‘em the external stylesheet:

You can see the production code from The Session in this gist. I’ve tweaked the loadCSS script slightly to match my preferred JavaScript style but otherwise, it’s doing exactly what I’ve outlined here.

Friday, September 12th, 2014

Now, here’s the thing with any of these walkthroughs; they’re all very specific to host/server combo in question. My particular combination is:

Hosting on a Digital Ocean virtual machine running Ubuntu 14.04 (typing lsb_release -a on the server will tell you which version of Ubuntu you’re running).

Running Apache 2.4.7 (typing apache2 -v will tell you which version of Apache is running).

If you’re using another flavour of Linux, or running Nginx, this walkthrough will probably not help you. You’ll also need admin access to the server. If you’re on shared hosting, you may well be screwed from the get-go.

To start with, you need to get your magic certificates from a Certificate Authority. So the first question is: who is a good Certificate Authority?

Choose a Certificate Authority

There is no such thing as a good Certificate Authority.

It’s a racket.

The best you can hope for is to deal with a Certificate Authority that doesn’t charge too much and doesn’t make the process as painful as applying a power drill to your teeth.

My pain was mitigated by my DNS provider, DNSimple. They have a one-clickish process for applying for a certificate. They charge $20 a year for a single hostname, which is not too bad. If that’s still too rich for your blood, you can look into navigating the interface at StartSSL. Good luck with that.

Until last week, DNSimple were using RapidSSL to issue those single-hostname certificates. RapidSSL still issues SHA-1 certificates by default, which is not good. But after a little prompting, DNSimple switched to Comodo for their single-hostname certificates, which means you can get yummy SHA256 encryption.

Make a private key and CSR

DNSimple can even take care of generating a private key and a Certificate Signing Request (the two chunks of encrypted gibberish you need to send to the Certificate Authority to get your certificate). Otherwise you’ll need to generate those for yourself. To do that, ssh into your server and run this, swapping out yourdomain_com for, you guessed it, your domain:

..and answer the probing questions that your nosy server will ask of you. You’ll then have two files:

/etc/ssl/private/yourdomain_com.key
/etc/ssl/certs/yourdomain_com.csr

You can then copy and paste the contents of those files when you’re applying for a certificate from your CA of choice. I can’t walk you through that process because, like I said, I managed to skip over it completely by having DNSimple do all the work.

I still had to put the private key on my server so I created the file:

nano /etc/ssl/private/yourdomain_com.key

And then I pasted the private key in there, hit ctrl o and then hit enter to write out the file, followed by ctrl x to close it.

(If some of these command line things don’t work for you, try prefixing them with sudo, try again, and enter your password when prompted.)

See how this is getting more and more specific to my particular combination? Now I’m talking about installing a Comodo certificate (acquired through DNSimple) on Apache 2.4 on Ubuntu 14. If you’re using a Certificate Authority other than Comodo, the next step won’t mean much to you.

Open a text editor on your computer and create a new file by copying and pasting the contents of these files in this order:

COMODORSADomainValidationSecureServerCA.crt followed by

COMODORSAAddTrustCA.crt follwed by

AddTrustExternalCARoot.crt

On your server, create a new file by typing:

nano /etc/ssl/certs/yourdomain_com.ca-bundle

Paste in the combined contents of the text file from your computer. Hit ctrl o (followed by enter) to write out the file and ctrl x to close it.

That leaves one file, which is your actual certificate, yourdomain_com.crt

On your server, type:

nano /etc/ssl/certs/yourdomain_com.crt

Copy and paste the contents from the certificate file (yourdomain_com.crt) in there. Hit ctrl o to write out the file and ctrl x to close it.

Set up https on Apache

Remember, this is for Apache 2.4. Things will be subtly different on previous versions. For instance, on Apache 2.4, all the files in /etc/apache2/sites-available must end with .conf; that wasn’t the case with previous versions.

First things first. You need to enable the ssl module for Apache. On your server, type:

a2enmod ssl

(Again, if that doesn’t work straight away, try prefixing it with sudo.)

Now restart Apache:

service apache2 restart

Your Apache server is now capable of serving over https, but you still need to give it more details.

Go into the sites-available directory by typing:

cd /etc/apache2/sites-available

You can see all the files in this folder by typing ls. I’m going to assume that you’ve got the server serving up just one site and that the details of that site are in the 000-default.conf file which serves up your site over port 80. You’ll want to copy any details you have set up in 000-default.conf over to default-ssl, but with the difference that the port number is 443. The top of your default-ssl file should look something like this:

Now you’re pointing at your hard-earned certificate and your private key. You still need to point to that bundle file you created earlier.

Comment out this line by putting # at the start of it:

#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt

Right underneath it type:

SSLCertificateChainFile /etc/ssl/certs/yourdomain_com.ca-bundle

Now page down to near the end of the file by hitting ctrl v. Right before the closing /VirtualHost tag, add these lines:

SSLProtocol All -SSLv2 -SSLv3
SSLHonorCipherOrder On
SSLCompression off
SSLCipherSuite 'EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA'

That line beginning with SSLCipherSuite needs to be all one line so watch out for line breaks if you’re copying and pasting. It’s quite an impressive bit of unintelligible gibberish, isn’t it?

Write out the default-ssl.conf file by hitting ctrl o, followed by enter, and exit this hellish text editor by hitting ctrl x.

Type this (you might need to sudo it):

a2ensite default-ssl.conf

Do the Apache restart dance by typing:

service apache2 reload

You might see a message like “Could not reliably determine the server’s fully qualified domain name…” Don’t worry about it. If, on the other hand, you see an error message that Apache can’t restart, worry about it.

Open your website in a browser; http://yourdomain.com — you should see no difference whatsoever. But now type https://yourdomain.com (but, y’know, swapping out yourdomain.com for your domain). Again, everything should look exactly the same but with one crucial difference: a shiny green lock in the corner of the URL bar indicating that the site is secure.

Redirect http to https

Once you’re happy with the way the https version of your site is working, you can make it the default.

Before doing this, it’s worth making sure that you haven’t hardcoded any images, scripts, or other external files with http:// URLs. If you’re pointing to third-party resources, most of them should also be available over https. Don’t forget that you can use protocol-relative URLS: //fontdeck.com/etc instead of http://fontdeck.com/etc or https://fontdeck.com/etc.

To switch http requests for your site over to https, you’ll need to edit the file that has your port 80 details. That’s probably000-default.conf in /etc/apache2/sites-available.

Again, the details might be slightly different for you: your DocumentRoot might by /var/www/html. Either way, add this line right after that DocumentRoot line:

Redirect / https://yourdomain.com/

(Swapping out yourdomain.com for your domain.)

Do the keyboard shortcut mambo: ctrl o, enter, ctrl x.

Do the Apache restart shuffle:

service apache2 restart

Now try visiting http://yourdomain.com in a browser. You should be automatically redirected to https://yourdomain.com …I really hope it works for you.

Enable HSTS

If you’ve made it this far and everything is working, well done! You have patience and fortitude in equal measure.

Go to ssllabs.com/ssltest and paste in your site’s domain name to see how well your site is doing in the security stakes. You’ll be given a grade, which will bring back all sorts of horrible memories of school tests.

If you’ve been given an A grade, there’s a way to level up to A+. You can enable HTTP Strict Transport Security. No, I don’t understand what it means either.

Go back to the command line on your server. You need to enable the headers module for Apache. Again, you may need to sudo this one:

a2enmod headers

Restart Apache:

service apache2 restart

Now dive back into that default-ssl.conf file:

nano /etc/apache2/sites-available/default-ssl.conf

Hit ctrl v to page down to right before that closing /VirtualHost tag. Add this line right after the gobbledygook you added previously:

Troubleshooting

I hope that this walkthrough will be of some use to you. But the chances are that unless you are also using Comodo as your Certificate Authority and you’re running Apache 2.4 on Ubuntu 14, there will be some sort of difference between your setup and mine that could render all of this null and void.

If you run into problems, I probably can’t help you. I’m a complete n00b at this stuff, and if it hadn’t been for Tim’s hand-holding, I doubt I ever would’ve managed.

If you do make the switch from http to https, please, please, please document each step along the way, and then publish it. There are plenty of articles and blog posts telling us why we need to switch on TLS, but not nearly enough articles and blog posts telling us how.

Saying “You should enable TLS—it’s easy!” can be damaging on two counts:

The first part is redundant: we all know that we should be doing this.

The second part just makes us feel bad. It’s like telling someone “You should play guitar—it’s easy!” Yeah, it’s easy if you already know how to play guitar.

If you’re a l33t server wizard and you care about this https stuff—which you almost certainly do—I urge you to divert the time and energy you might consider putting into advocacy and instead put that time and effort into helping n00bs like me. If you can gather willing web developers in the same physical space—like Tim did at Indie Web Camp—I think you can achieve maximum knowledge dispersement. (Perhaps Google/Mozilla/Opera/Microsoft dev rels could sponsor TLS days in various cities and towns?)

The issue with https is not that web developers don’t care or understand the importance of security. We care! We understand!

Thursday, December 27th, 2007

Christmas is a time for giving, a time for over-indulgence, a time for lounging around and for me, a time for doing those somewhat time-consuming tasks that I’d otherwise never get around to doing… like upgrading my operating system.

I used the downtime here in Arizona to install Leopard on my Macbook. I knew from reading other people’s reports that it might take some time to get my local web server back up and running. Sure enough, I had to jump through some hoops.

I threw caution to the wind and chose the “upgrade” option. Normally I’d choose “Archive and Install” but it sounds like this caused some problems for Roger .

The upgrade went smoothly. Before too long, I had a brand spanking new OS that was similar to the old OS but ever so slightly uglier and slower.

My first big disappointment was discovering that my copy of Photoshop 7 didn’t work at all. Yes, I know that’s a really old version but I don’t do too much image editing on my laptop so it’s always been good enough. I guess I should have done some reading up on compatibility before installing Leopard. Fortunately, I was able to upgrade from Photoshop 7 to Photoshop CS3—I was worried that I might have had to buy a new copy.

But, as I said, the bulk of my time was spent getting my local LAMP constellation back up and running. I did most of my editing in BBEdit—if you install the BBEdit command line tools, you can use the word bbedit in Terminal to edit documents. If you use Textmate, mate is the command you want.

Leopard ships with Apache 2 which manages virtual hosts differently to the previous version. Instead of keeping all the virtual host information in /etc/httpd/httpd.conf (or /etc/httpd/users/jeremy.conf), the new version of Apache stores it in /private/etc/apache2/extra/httpd-vhosts.conf. I fired up Terminal and typed:

bbedit /private/etc/apache2/extra/httpd-vhosts.conf

That file shows a VirtualHost example. After unlocking the file, I commented out the example and added my own info:

The good news is that Leopard doesn’t mess with the hosts file (located at /private/etc/hosts). That’s where I had listed the same host names I had chosen in the previous file:

127.0.0.1 localhost
127.0.0.1 adactio.dev

But for any of that to get applied, I needed to edit the httpd.conf file:

bbedit /private/etc/apache2/httpd.conf

I uncommented this line:

# Include /private/etc/apache2/extra/httpd-vhosts.conf

While I was in there, I also removed the octothorp from the start of this line:

# LoadModule php5_module libexec/apache2/libphp5.so

That gets PHP up and running. Leopard ships with PHP5 which is A Good Thing.

Going into Systems Preferences, then Sharing and then ticking the Web Sharing checkbox, I started up my web server and was able to successfully navigate to http://adactio.dev/. There I was greeted with an error message informing me that my local site wasn’t able to connect to MySQL.

Do not fear: MySQL is still there. But I needed to do two things:

Tell PHP where to look for the connection socket and

Get MySQL to start automatically on login.

For the first step, I needed a php.ini file to edit. I created this by copying the supplied php.ini.default file:

cd /private/etc
cp php.ini.default php.ini
bbedit php.ini

I found this line:

mysql.default_socket =

…and changed it to:

mysql.default_socket = /private/tmp/mysql.sock

I had previously installed MySQL by following these instructions but now the handy little preference pane for starting and stopping MySQL was no longer working. It was going to be a real PITA if I had to manually start up MySQL every time I restarted my computer so I looked for a way of getting it to start up automatically.