yakLab Part 1c: Bootstrapping Bifrost with Ansible

In this scene I’ll explore some of the bootstrapping I’ve been working on for a
while that will result in a clean, shiny new Bifrost deployment, populated with
inventory, executed from your laptop to a virtual machine.

Also to note this blog post makes the whole endeavour seem a lot
more complicated than it really is. I’m mostly giving you all the background,
overview, and lab topology so that things make a lot more sense so that
you’ll be successful in your own local deployment (as not all networks are
created equally :)).

Assumptions

You’re running KVM / libvirt on your virtual host

You’ll be deploying a CentOS 7 virtual machine

Your baremetal nodes can be controlled via IPMI

Your network is setup the same as mine :)

Less Automation Quickstart

This is pretty much what the documentation tells you to do. It works, and it
doesn’t take very long to setup. The issue I have is that it then leaves you
to your own devices to add your inventory. I’d like to automate that part of
the bootstrapping process as well.

Also, it installs Ansible via pip and the version installed by default is
Ansible 2.1.6.0 (which is starting to get fairly dated). While use of the
env-setup.sh is convenient to get all the dependencies bootstrapped, it is a
pain to use with Ansible to automate the deployment, as you start having to
much around with virtual environments.

Instead of using env-setup.sh we’ll install some dependencies then let
Bifrost install any other dependencies it needs after the fact.

A More Automated Bootstrap

Both @dougbtv and I have been working on some
bootstrapping Ansible playbooks that will also bring in your inventory, and
generally make it a little easier to import a set of hardware initially.

Most of the work is just basically what you need to do after you run the
quickstart, but then if you destroy your Bifrost install, you have to do all
that data entry again. So we automated your automation.

The Topology

In my lab, I have multiple low power nodes that I used for deployment testing.
They are wired up with IPMI so that I can remotely power them on and off, and
then they are provisioned from a virtual machine running Bifrost.

Here is a diagram of the physical topology of the lab:

Image 1-1: yakLab Physical Topology

Bifrost lives on the virthost using a bridged interface to service the IPMI
and provisioning interface. The IPMI interfaces (green) are statically assigned
IP addresses on the 192.168.25.0/24 subnet. Bifrost then advertises DHCP for
the provisioning interfaces in the same subnet, between 192.168.25.64 and
192.168.25.95 (yellow).

The other network interface (purple) is then provided by my main router via
DHCP using the 192.168.26.0/24 subnet (which provides internet access via the
uplink).

The Logical Topology

So that you understand where things are on the network, here is the logical
topology.

Post Install Manual Tweak…

For now, I need to add another interface with Network Manager to the virtual
host on boot. I haven’t figured out why when I do it with the kickstart file
that the tagged VLAN interface doesn’t quite work. It’s pretty easy to run
after the fact, and you’ll only need to do it once.

NOTE

Bifrost will run at static IP 192.168.25.251 on the provisioning network.
IPMI of yak machines 1 through 5 will live at static IP addresses
192.168.25.1 through 192.168.25.5.

Let’s add the external network to our virtual machine by logging into it, and
running nmcli.

ssh admin@192.168.25.251
sudo nmcli con add type vlan id 26 dev eth0

Bootstrapping The Virtual Machine

There are always a few things I like to do to a machine once it has been
instantiated, and instead of running all that manually, I create a little
Ansible bootstrap playbook.

The primary reason I bootstrap the nodes, is so that the admin user (as
created by kickstart and virt-install) can sudo to root without a password.
It makes using Ansible a little easier in the next steps.

Then you’ll want to create a new inventory file. There is an example in
inventory/example.inventory. You can cat the following file contents into
the local.inventory file. Don’t worry, everything except the example is
ignored, so you won’t clutter up the git repo.

Feel free to clone this bootstrap and add anything you want. If there are other
cool bootstrap things you think all machines should have, feel free to open a
pull request. Of course, the bootstrapping is biased towards my preferred
machine deployment, and your bootstrapping needs may differ.

Clone The Dr. Octagon Bifrost Repository

On your control machine, clone the Dr. Octagon Bifrost repository, which
contains the Ansible playbooks we’ll run in a few minutes.

Building Our Inventory

One of the most time consuming parts of deploying Bifrost is really just
gathering up the information about your hardware, setting the IP addresses for
your IPMI interfaces, documenting the MAC addresses for the provisioning
network interfaces, and putting all that data into JSON (or in our case, YAML
because we’re going to generate the JSON inventory with our Ansible playbook).

In order to make it easier to generate the inventory and keep a bunch of your
data in a single file, we’ve created a template that will generate the
inventory for you. However, you’ll still need to enter the aforementioned data
into a YAML file.

We’ve provided an example inventory for you in the inventory/example/
directory. You can start with this by copying it, or building out your own from
scratch.

I copied the example inventory directory to staging with cp -r
inventory/example/ inventory/staging/ and then put in the contents of my
hardware into the inventory/staging/group_vars/bifrost.yaml. Here is my local
example for the yakLab.

Using the random | to_uuid filter does result in a known bug currently. I
probably just need to drop the random filter, since if you run this again,
you’ll get conflicts from Bifrost when it tries to enroll the hardware, since
the UUID will have changed in the template.

Stepping through this file, we’ve got a few things to note. First, the Bifrost
version we’re going to clone. In this case I’m using master, but you could
also use a branch name, or version tag.

The pxe_boot_server_ip is our static provisioning IP address we set up when
we instantiated our virtual machine. It’ll be the IP address that Bifrost will
advertise DHCP on with dnsmasq.

The pm_addr is the IP address of the IPMI interface you would have previously
configured. In my case I went through and statically assigned an IP address for
the nodes. You could of course have created another network and assigned them
via DHCP (if the machines you’re using provide it). In that case you’d probably
want to make sure you could communicate with them via a DNS name, but I haven’t
tried that (or set that up locally).

The mac value is the MAC address that Bifrost will look for when the nodes
boot up, then over the provisioning interface, request an IP address via DHCP
during the PXE boot process. The idea is that once the node has successfully
been provisioned, Bifrost will no longer server the required files to boot via
PXE on subsequent boots for that MAC address.

The pm_user and pm_password are the values for logging into the IPMI
management interface on your baremetal nodes.

We’re almost there!

Deploying Bifrost

OK, we’ve done all the hard work now. If things are setup properly on your
network, then you’ll be able to deploy Bifrost to your virtual machine, and
service the baremetal nodes over your bridged interface.

Moment of truth, time to run the Bifrost deployment.

From your droctagon-bifrost directory, run the following command.

ansible-playbook -i inventory/staging/ site.yml

Now be patient and let things progress. Hopefully it all finishes correctly,
and that you’ve automated your deployment. At this point, you’re pretty much
done, other than the actual deployment. So far, we’ve only brought the playbook
as far as the enrollment process, so deploying the nodes is left as a separate,
non-automated step.

Primarily, when I see a failure, it’s because…

I don’t have the network setup quite correctly (can’t download packages from
the external network),

Issue with Bifrost itself from upstream (not all that common actually)

I’ve deployed it twice with the known random filter bug I introduced

Next, let’s see how we can deploy our hardware.

Preparing To Deploy Our Baremetal Nodes

Now the fun part. Deploying an operating system and powering on the nodes
remotely from the comfort of our virtual console. As part of our bootstrapping
of Bifrost, we’ve resulted in a built CentOS image, and are now ready to deploy
it to our baremetal nodes.

First, we need to SSH into our virtual host and run the commands there.

ssh admin@bifrost.yakdata.61will.space

(I’m setup with local DNS resolution for the 192.168.26.0/24 network that we
added as a tagged VLAN interface, but you could just as easily login to the
192.168.25.251 static IP as well.)

OK, now that we’re logged into the Bifrost server, we need to do a couple of
preparatory steps. We’ll source the env-vars and export the
BIFROST_INVENTORY_SOURCE environment variable.

In this case, you can see that I’ve lazily shown you a post-deployment output
(you can tell because the Provisioning State is set to active meaning that
our nodes are fully provisioned and active). If you’ve just run the commands,
you’ll see that the Provisioning State should be set to available. If you
see anything else, likely something has gone wrong. (It’s possible enroll is
also a valid state to be in as well, so you could still proceed and see if it
works, but I’ve not tried deploying from that state.)

You may also see the Power State is a value of None. Sometimes the state
just isn’t necessarily readily available, depending how good your nodes IPMI
implementation is. I find that things will work fine even if initially the
state isn’t known. Your kilometerage may vary.

Deploying Our Baremetal Nodes

OK, we’ve done all the preparatory work, and we can finally try deploying our
nodes.

First, change into the bifrost/playbooks/ directory on the Bifrost virtual
machine.

cd /opt/bifrost/playbooks/

And then we can run the following command to deploy our nodes dynamically (if
you get an immediate error, make sure you’ve sourced the env-vars file and
exported the BIFROST_INVENTORY_SOURCE to your path.)

I then watch the output of ironic node-list to see if the nodes are
deploying.

watch -n10 "ironic node-list"

We should see things in the wait call-back state, but those should eventually
change into active. Nodes that sit in wait call-back for a very long time
are likely to eventually time out and fail.

The main reason for this in my experience is that the nodes are not connecting
to the Bifrost server over the provisioning network. It could either be failing
to get an IP address from dnsmasq during PXE boot, or the MAC address being
used on the provisioning network isn’t the correct network interface that was
provided.

You can check some of this via journalctl -fu dnsmasq.service and via
journalctl -fu ironic*.

Conclusion

Hopefully this post gets you on your way to deploying Bifrost and managing the
deployments of your baremetal hardware. I actually had quite a bit of a rocky
road getting this far, but after trying several times, I finally understood how
best to deploy Bifrost, and to automate the bootstrapping of the virtual
machines. Now I can move onto more exciting things, and know that I can at
least get my provisioning system going without too much trouble if I need to
recreate it.