Random Tips & Hints

Ok, 7 seconds doesn’t seem like much… but if you’re rebooting a live server then every second of downtime counts – so why not?

This is a simple change that reduces the timer on the initial FreeBSD menu from 10 seconds to 3 seconds. (I don’t recommend reducing it below 3 seconds, as it may make it more difficult for you to use the menu should you ever need to!)

Ok, so you need to edit the /boot/loader.conf file (or create it if it doesn’t exist) and add the following line:

autoboot_delay=”3″

and that’s all you need to do. Next reboot, the timer will start at 3 instead of 10 and you have a reboot that takes 7 seconds less than before.

On Friday 22nd January 2016, I formally gave 30 day notice to Nominet for the closure of my registrar account. This post details the reasons for this.

Nominet contacted me stating that they were going to link my registrar account with that of my employer due to their “connected persons” section of their AUP. I should note that I had never seen this section before, so it must have been added since I became a registrar.

I contacted them detailing that my personal account was for the express purposes of keeping my private and business life separate… including that I go as far as to have provider independant IP address space, my own Autonomous System, BGP routers and servers – all separate from my employer. I stated the fact that I had been (save for a short break) a Nominet member for longer than I had been an employee of my current employer too.

In response I received:

I have spoken with our legal team with regards to your information, however, they have advised that we need to link the services in regards to the acceptable use policies (Anti-avoidance and Connected Persons link below).

Obviously my employer would be rather upset if anything I could do in my personal capacity could affect them in any way (it’s a fair expectation of an employee), so I was left with no alternative but to serve notice on my registrar account with the minimum possible term (30 days)

Nominet’s AUP (whenever it was added/updated) contains the following description of connected persons:

A person is ‘connected’ to another person if:

they are the same person, have the same Nominet account, or have connected memberships under the voting rights policy;

they make any declaration that they are connected or when challenged by us they fail to make a declaration in the reasonably required legal form that they are not connected;

they have social, family, ownership or business links (directly or indirectly) which mean that they either:

do not appear to operate truly independently of one another, or

it could reasonably be assumed that they will not operate truly independently of one another; or

the object or effect of their activities is such that we reasonably think that not linking them would compromise one or more of the AUP Principles.

For the purposes of (2) we may decide that two parties are not connected even if one declares that they are, if the other disputes this and our investigations support the denial (so, for example, you cannot just declare that you are linked to another person and then use all their limit up to disrupt their business).

For the purposes of (3) there is a strong presumption that:

group companies (or other businesses) are connected to other businesses in the group;

a company and its employees/officers/partners are connected to one another;

members of the same family group are connect to each other (including in-laws, co-habitees, civil partners, adopted children and others who may not have a relationship by blood but who are part of the family group); and

a business owned, run or ultimately controlled by one member of a family is connected to a business owned, run or ultimately controlled by another member of the same family.

The final decision rests with us. When deciding whether to link people under (2) to (4) above, we will take into account the AUP principles.

I’m unhappy with so many things in the above snippet that I don’t even know where to start…

In short, Nominet have lost my annual membership fee (i’m sure they won’t be concerned about this given that they make more profit than their non-profit status allows anyway), but also one of their last registrars who had implemented their entire feature set.

I have recommended to any people with domains on my account to migrate to another registrar ASAP. For those without a preferred registrar, I have suggested PortFast as they appear to have a similar feature set to that I had offerred.

This follows a disturbing trend within Nominet which I wont go into in this post. Feel free to use your favourite search engine to lookup recent events at Nominet if you want to read further.

In an age where ICANN are dishing out new TLDs constantly, you would expect Nominet to be trying to strengthen the .uk brand but instead recently it seems to be trying to harm it.

In light of this, I have been considering migrating my main domain name away from the .uk namespace to one of the more level-headed registries.

As we move into an more secure environment, simply have HTTPS isn’t sufficient. Many cases of forged SSL certificates for man-in-the-middle attacks have appeared recently.

These can be obtained through deception or hacking attemps on SSL CAs.

One method to help combat this is HTTP Public Key Pinning (HPKP) – this is where the webserver can communicate to the web browser an allowed certificate path, with a sufficiently long expiry time to be of use to return visitors. It’s not perfect or ideal, but it’s better than nothing.

This post does NOT detail the most secure method of HPKP but a compromise in terms of usability vs security. It will ensure that an unauthorised certificate would need to have been generated by one of your chosen SSL CAs only. The more secure method of having dual online and offline keys is out of scope for this blog post and a more advanced topic.

The way this works is to publish a header in the HTTPS response detailing the allowed hashes of public keys of webserver keys or keysigner keys that are permitted to be in the certificate chain. This is in addition to the normal process of validation of the full SSL certificate chain.

BEWARE: misconfiguration of this header can lead people to be unable to view your site for MONTHS so be careful!

First let’s look how to obtain the SHA-256 hash needed for a certificate. We use OpenSSL to do this based on the .crt file… Let’s assume the certificate is in the cert.crt file…

Now we provide the information to the webserver… i’ll detail Apache 2.4 here, but other servers will have similar methods of adding header outputs. We add the following into the VirtualHost block for the SSL site:

HPKP requires a minimum of 2 hashes to be present, including one hash that is NOT present in the certificate chain. This is to encourage you to ensure you have a backup plan.

In my examples, I use the hash of my SSL public key and the hashes of my chosen primary and secondary SSL certificate authority’s intermediate certificates.

In my case, I use StartSSL’s class 2 intermediate and RapidSSL’s SHA-2 under SHA-2 root intermediate. To generate their hashes, use the above OpenSSL method on their CA certificates.

This means that I can use my existing key/csr with any SSL CA, or I can generate a new key with my primary or secondary CA. It also satisfies the requirement for one of the hashes to not exist (my current certificate is not present via my secondary CA)

The max-age above is set to 30 seconds for testing purposes… once you’re completely happy with your choice, this should be raised to a much longer figure. I use a 30 day period (2592000 seconds)

A good method of testing is to use Qualys SSLlabs tester at https://www.ssllabs.com/ssltest/ – this will show you the pinned list, including any currently in the chain in a different colour.

The new FreeBSD 10 installer makes a lot of things easier (such as installing ZFS or encrypted filesystems), but there’s a few quick hints I have:

If installing using a UFS fileystem type (the default), I prefer to re-partition so that the swap space is at the start of the disk instead of the end.

This makes it easiest to expand the root filesystem later (especially if you’re installing onto a virtual machine)

If installing using ZFS and using multiple disks the installer will spread your swap partition across all disks – however these will NOT be protected by ZFS.

If you have a disk failure, any swap space on that disk will disappear and your machine will potentially crash and reboot.

I prefer to set the swap space to “0G” which causes the installer not to create a swap partition, and then I configure a swap file instead (i’ll post another blog post about creating swap files in FreeBSD 10+) which is protected by the ZFS subsystem.

This doesn’t apply if you’re installing onto a single drive (or a hardware raid array)

If you want any kind of performance from encrypted ZFS, make sure your CPU supports the ‘AESNI’ (or AES New Instructions) flag – it really makes a huge difference to the speed achieved.

There are two ways to increase the capacity of a ZFS pool… either add more disks to the pool (e.g. 3 more disks in RAIDZ1), or replace all the existing disks with larger ones… this is the method discussed here today. These instructions assume that you have followed my installation guide for ZFS. If you have varied from that guide at all, you may need to vary the instructions below. I am not responsible for any data loss by following any of these instructions!

NOTE: you can only increase the size of a mirror or raidz1/2/3 pool using this method.

I am replacing the 4 x 3TB disks in my storage array for 4 x 4TB disks. This is a time consuming process and is risky if using mirror/RAIDZ1 (as you have to degrade the array !) – if you do not have full backups of the contents, do so at your own risk. (if you’re using RAIDZ2 then you’re just at reduced resilience and a little safer)

First, we want to make sure that the autoexpand option is enabled, this can be run at any time with the following command:

zpool set autoexpand=on zroot

Next, check the status of your ZFS pool to make sure it is healthy… here’s the command and the output from my array:

As you can see, my array consists of 4 members (ada0p2 through ada3p2) and is currently healthy. We’re good to proceed!

First we shutdown the machine, and replace one of the disks… I prefer to start with the last disk and work backwards so i’m going to replaceada3… Once replaced, start the machine up again.

Now we can confirm that the disk is missing (and confirm which one) as follows… (command and output listed):

zpool status

pool: zroot
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using ‘zpool online’ or replace the device with
‘zpool replace’.
scan: scrub repaired 0 in 6h38m with 0 errors on Thu Nov 8 16:06:21 2012
config:

The above creates a GPT partition table, adds a small boot loader parition and the remainder of the disk for ZFS. It then installs the boot loader into the small partition.

We are now ready to re-add the disk into the ZFS pool. This will trigger an auto-resilver of the disks (a rebuild of the disk)…

zpool replace zroot ada3p2 /dev/ada3p2

This command takes a little while to process, so be patient. The resilver stage can take a long time (it depends how much data you have on the pool, how many disks are in it and how fast you can read from them!)

You can check on the status of the rebuild with the following command:

zpool status zroot

Here’s an example output so you know what to look for:

pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Nov 14 18:34:57 2012
16.8G scanned out of 6.59T at 116M/s, 16h30m to go
4.19G resilvered, 0.25% done
config:

Once the disk has been fully reconstructed, the array will be healthy again (like at the start), and you can move onto the next disk. Repeat until all disks have been replaced and resilvered.

You will only see the new space once all the disks have finished resilvering.

I will note again that your array is vulnerable if a mirror or raidz1 configuration while doing this. If a 2nd disk fails during the resilver of any of the disks and you’re doing a mirror or raidz1 pool, you will LOSE your data.

Assuming you have created your ZFS FreeBSD system using the instructions on my site, here is how to do full system backups to an extra attached disk.

You can adjust these instructions if you need to store the backup remotely – but they are out of scope of this post.

First, in case you haven’t already… here is how to format/dev/da1 as a dedicated ZFS backup drive. You can configure the backup drive however you want (it doesn’t even need to be ZFS-based) but you will also have to adjust these instructions accordingly to restore too.