Some time ago I found a good, reliable way of using and installing FreeBSD and described it in my Modern FreeBSD Install[1] [2] HOWTO. Now, more then a year later I come back with my experiences about that setup and a proposal of newer and probably better way of doing it.

1. Introduction

Same as year ago, I assume that You would want to create fresh installation of FreeBSD using one or more hard disks, but also with (laptops) and without GELI based full disk encryption.

This guide was written when FreeBSD 9.0 and 8.3 were available and definitely works for 9.0, but I did not try all this on the older 8.3, if You find some issues on 8.3, let me know I will try to address them in this guide.

Earlier, I was not that confident about booting from the ZFS pool, but there is some very neat feature that made me think ZFS boot is now mandatory. If You just smiled, You know that I am thinking about Boot Environments feature from Illumos/Solaris systems.

In case You are not familiar with the Boot Environments feature, check the Managing Boot Environments with Solaris 11 Express PDF white paper [3]. Illumos/Solaris has the beadm(1M)[4] utility and while Philipp Wuensche wrote the manageBE script as replacement [5], it uses older style used at times when OpenSolaris (and SUN) were still having a great time.
I last couple of days writing an up-to-date replacement for FreeBSD compatible beadm utility, and with some tweaks from today I just made it available at SourceForge [6] if You wish to test it. Currently its about 200 lines long, so it should be pretty simple to take a look at it. I tried to make it as compatible as possible with the 'upstream' version, along with some small improvements, it currently supports basic functions like list, create, destroy and activate.

There are several subtle differences between mine implementation and Philipp's one, he defines and then relies upon ZFS property called freebsd:boot-environment=1 for each boot environment, I do not set any other additional ZFS properties. There is already org.freebsd:swap property used for SWAP on FreeBSD, so we may use org.freebsd:be in the future, but is just a thought, right now its not used. My version also supports activating boot environments received with zfs recv command from other systems (it just updates appreciate /boot/zfs/zpool.cache file).

My implementation is also style compatible with current Illumos/Solaris beadm(1M) which is like the example below.

The boot environments are located in the same please as in Illumos/Solaris, at pool/ROOT/environment place.

2. Now You're Thinking with Portals

The main purpose of the Boot Environments concept is to make all risky tasks harmless, to provide an easy way back from possible troubles. Think about upgrading the system to newer version, an update of 30+ installed packages to latest versions, testing software or various solutions before taking the final decision, and much more. All these tasks are now harmless thanks to the Boot Environments, but this is just the tip of the iceberg.

You can now move desired boot environment to other machine, physical or virtual and check how it will behave there, check hardware support on the other hardware for example or make a painless hardware upgrade. You may also clone Your desired boot environment and ... start it as a Jail for some more experiments or move Your old physical server install into FreeBSD Jail because its not that heavily used anymore but it still have to be available.

Other good example may be just created server on Your laptop inside VirtualBox virtual machine. After you finish the creation process and tests, You may move this boot environment to the real server and put it into production. Or even move it into VMware ESX/vSphere virtual machine and use it there.

As You see the possibilities with Boot Environments are unlimited.

3. The Install Process

I created 3 possible schemes which should cover most demands, choose one and continue to the next step.

3.1. Server with Two Disks

I assume that this server has 2 disks and we will create ZFS mirror across them, so if any of them will be gone the system will still work as usual. I also assume that these disks are ada0 and ada1. If You have SCSI/SAS drives there, they may be named da0 and da1 accordingly. The procedures below will wipe all data on these disks, You have been warned.

The procedure is quite different for Laptop because we will use the full disk encryption mechanism provided by GELI and then setup the ZFS pool. Its not currently possible to boot off from the ZFS pool on top of encrypted GELI provider, so we will use setup similar to the Server with ... one but with additional local pool for /home and /root partitions. It will be password based and You will be asked to type-in that password at every boot. The install process is generally the same with new instructions added for the GELI encrypted local pool, I put them with different color to make the difference more visible.

7. Create snapshot called configured or production
After You configured Your fresh FreeBSD system, added needed packages and services, create snapshot called configured or production so if You mess something, You can always go back in time to bring working configuration back. mess something.

# zfs snapshot -r sys/ROOT/default@configured

5. Enable Boot Environments

Here are some simple instructions on how to download and enable the beadm command line utility for easy Boot Environments administration.

Now we have a working ZFS only FreeBSD system, I will put some example here about what You now can do with this type of installation and of course the Boot Environments feature.

6.1. Create New Boot Environment Before Upgrade

1. Create new environment from the current one.# beadm create upgrade
Created successfully

2. Activate it.# beadm activate upgrade
Activated successfully

3. Reboot into it.# shutdown -r now

4. Mess with it.

You are now free to do anything You like fo or the upgrade process, but even if You break everything, You still have a working default working environment.

6.2. Perform Upgrade within a Jail

This concept is about creating new boot environment from the desired one, lets call it jailed, then start that new environment inside a FreeBSD Jail and perform upgrade there. After You have finished all tasks related to this upgrade and You are satisfied with the achieved results, shutdown that Jail, set the boot environment into that just upgraded Jail called jailed and reboot into just upgraded system without any risks.

Lets assume, that You need to upgrade or do some major modification to some of Your servers, You will then create new boot environment from the default one, move it to other 'free' machine, perform these tasks there and after everything is done, move the modified boot environment to the production without any risks. You may as well transport that environment into You laptop/workstation and upgrade it in a Jail like in step 6.2 of this guide.

1. Create new environment on the production server.# beadm create upgrade
Created successfully.

You can now add your users, services and packages as usual on any FreeBSD system, have fun ;)

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

3.1. Make new Jail dataset mountable.# zfs set canmount=noauto sys/ROOT/jailed

3.2. Mount new Jail dataset.# zfs mount sys/ROOT/jailed

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

hi here i describe the procedure to encrypt everything including the freebsd system that i use on my laptop and use a usb key with the bootcode, kernel and keys, you can detach the usb media after system boots (btw you'll have to enter two passphrases).

now enter the 2 passphrases to decrypt system (freebsd os) and local (/home and /root) as suggested above make a virgin state snapshot; also i would suggest using mtree to check that the kernel on the usb key wasn't tampered to snoop your pass phrases, i'll add the script later on if i'm able to edit the post. best.

Its pointless to create both sys and local pools as You encrypt both of them, just create the encrypted sys pool.

With USB thumbs that size below, its even aplyable to laptops:

... but the main question is: How it beadm working with it? (as this is this tutorial all about)

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

hi vermaden, what prompted me to share this is not beadm but the fact that your howto for zfs is the best around the interwebs and believe me i looked at so many. so all in all my addition was about encrypting the whole system and using a usb bootkey. on a side note i'm using the config above for a nas, didn't check beadm as of yet, so do you imply that it won't work with the system pool encrypted?. now back to your question i'm having 2 pools on the os disk because i feel it's easier to backup, recover the system should any upgrade go bad. the local part has an important essential subset backup from the raidz, in other words i'm just using the free space left on the OS disk for extra backups. I was hesitant to use a USB disk OS and maybe i'm wrong. other than that i'm using a small ssd for the zil. Well i think that's all about my zfs experience and thank you for sharing this as I said it's probably the best online .

hi vermaden, what prompted me to share this is not beadm but the fact that your howto for zfs is the best around the interwebs and believe me i looked at so many.

It does not cover 4k drives (gnop devices), so its not the best, but thanks

Quote:

Originally Posted by silex

so all in all my addition was about encrypting the whole system and using a usb bootkey. on a side note i'm using the config above for a nas, didn't check beadm as of yet, so do you imply that it won't work with the system pool encrypted?

Encryption is not the problem.

The MAIN problem, is that FreeBSD Bootloader is not able to boot FreeBSD from ZFS which is on encrypted GELI drive, so we have to do it other way.

One of the things that beadm does is it changes bootfs property of ZFS pool and vfs.root.mountfrom line in the /boot/loader.conf, so beadm will have to be modified to do that on the separate / or /boot or separate pool.

Its not impossible, its just pain in the ass

Quote:

Originally Posted by silex

now back to your question i'm having 2 pools on the os disk because i feel it's easier to backup, recover the system should any upgrade go bad. the local part has an important essential subset backup from the raidz, in other words i'm just using the free space left on the OS disk for extra backups.

In some advanced configuration, sure, 2 drives in ZFS mirror for sys and other drives in some fancy stripes and mirrors, or raidz configurations for local pool.

Quote:

Originally Posted by silex

I was hesitant to use a USB disk OS and maybe i'm wrong.

IMHO nothing wrong with that.

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

It does not cover 4k drives (gnop devices), so its not the best, but thanks

Right! and there's lots of madness lying there. So here's what I did, and please correct me if I'm wrong, on my nas pool, my logic was to tackle the Advanced Format bs by partitioning the disk so that it stays in the boundaries of 4096 bytes at a time for whatever operation, then well Geli was set to feed ZFS 4K chunks and ZFS would just align (ashift 12), I didn't bother using gnop since I'm already at 3 software layers for disk access without counting ZFS. Is there any issue apparent in here?

The MAIN problem, is that FreeBSD Bootloader is not able to boot FreeBSD from ZFS which is on encrypted GELI drive, so we have to do it other way.

Got it. Personally I found this ZFS setup way too complex already, I'm may be accustomed to KISS stuff from OpenBSD but I really needed ZFS for this server, it took me about 10days to get everything sorted out with not much trial and errors so for now i think i'd better pass on Beadm although I can see how useful it is.

Right! and there's lots of madness lying there. So here's what I did, and please correct me if I'm wrong, on my nas pool, my logic was to tackle the Advanced Format bs by partitioning the disk so that it stays in the boundaries of 4096 bytes at a time for whatever operation, then well Geli was set to feed ZFS 4K chunks and ZFS would just align (ashift 12), I didn't bother using gnop since I'm already at 3 software layers for disk access without counting ZFS. Is there any issue apparent in here?

These instructions seem to be OK, but I haven't tried them at home yet ;p

Quote:

Originally Posted by silex

Got it. Personally I found this ZFS setup way too complex already, I'm may be accustomed to KISS stuff from OpenBSD but I really needed ZFS for this server, it took me about 10days to get everything sorted out with not much trial and errors so for now i think i'd better pass on Beadm although I can see how useful it is.

If You take some time to understand what really ZFS snapshot and ZFS clone is, beadm is no-brainer then

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

Vermaden, something extra we forgot to mention on ZFS and disk alignment: I've seen a noticeable improvement when the ZIL on the SSD is properly aligned, in that case i've used Gnop to 4K-align a mounted memory drive then instructed ZFS to mirror log on the SSD with the properly aligned memory drive, I then deleted the MD and the Gnop device yet ZFS keeps a 12 ashift on the log disk and it's what we want. [strike]I'll add the instructions later.[/strike]

I have read some about 4K (ashift=12) align on L2ARC and ZIL devices, post these instructions here as many people will find them useful.

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

@vermaden if you could start a FreeBSD on armish cross compiling how-to that covers kernel, world and ports for SoCs like Raspberry Pi, Beaglebone, Pandabox etc.

I do not own any of these devices and I do not plan to get one. My Mini-ITX storage box based on mobile Intel Core 2 Duo is more then enough for me.

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd

__________________religions, worst damnation of mankind"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus TorvaldsLinux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.vermaden's:linksresourcesdeviantartspreadbsd