Tag Archives: chef

A while back, there was a discussion on Munki-Dev floating the idea of local-only manifests. After some long discussion, the final Pull Request was created and merged.

The idea behind local-only manifests is simple: if you specify a LocalOnlyManifest key in the preferences, Munki will look for that manifest name in /Library/Managed Installs/manifests. If it finds it, it’ll look for any managed_installs and managed_uninstalls specified inside, and concatenate those with whatever it gets from the Munki server. It’s an extra place to specify managed installs and uninstalls that is unique to the client.

Essentially, what it does is move the unique-client logic from the server to the client. As you scale upwards in client numbers, having huge numbers of unique server-side manifests induces significant overhead – potentially 10,000+ unique manifests in your Munki server’s manifests directory gets unwieldy. With the uniqueness moved client-side, the server only has to provide the common manifests.

There’s a lot of neat things you can do with this idea, so let’s explore some of them!

Hang Out With The Locals

While the basic idea of the local-only manifest is simple, the implementation has some fun details you can take advantage of.

Local-only manifests do not have any catalogs of their own. Instead, they inherit from whatever catalog is provided by the manifest given from the ClientIdentifier key. Thus, if your main manifest uses the catalog “release”, any items specified in the local-only manifest must also be in the “release” catalog (or they will simply be treated like adding any item to a manifest when it is not in a catalog – which is to say that you will receive warnings).

Local-only manifests also don’t have their own conditional items. This is where interaction with third-party tools really begins to shine, but we’ll explore that later.

Because this is a unique manifest, you get the benefits that “real” manifests get. You can specify items to be installed here that are not provided as optional items in the server-side manifest (as long as they’re in the catalog). You can still get the server’s provided list of optional installs, and use the local-only manifest to determine what items become managed installs or removals.

This doesn’t absolve the Munki admin of taking care, though. It’s still possible for an item to be specified as a managed install in one manifest and a managed uninstall in another manifest – and therefore trigger a collision. Local-only manifests are just as vulnerable to that as server-side manifests, and it’s easy for a client to contravene the server-side manifest and result in undefined (or undesireable) behavior.

One Manifest To Rule Them All

Because of the slightly limited nature of local-only manifests, it’s important to think of them as addenda to server-side manifests. The way to mentally separate these functions is to also separate “mine” vs. “yours” – the things I, the Munki admin, want your machine to have vs. the things you, the client, want your machine to have (or not have).

The easiest way to accomplish this is to completely remove managed_installs and managed_uninstalls from your server-side manifest. The server-side manifest thus becomes the self-service list and gatekeeper to all optional software. The Munki admins determine what software is available because they control the optional installs list as well as the catalogs, but the clients now have essentially free customizability without needing any ability to modify the servers.

Because the unique aspects of clients are now done client-side and not server-side, this allows an external management mechanism, like Chef or Puppet, to control what Munki manages on a client, without needing the ability to make changes to the repo. If your repo is in source control (and it should be!), this means that the only commits to the repo’s manifests are done by the Munki admins, and will only involve changes that generally affect the whole fleet.

Whence Does This Mystical Manifest Come From?

The local-only manifest moves the work from maintaining the manifest relationships on the server to maintaining them on the client. This is really only beneficial if you already have a mechanism in place to manage these files – such as a config management tool (Chef, Puppet, etc.).

Facebook CPE handles this with our cpe_munki cookbook for Chef. In addition to managing the installation and configuration of Munki, we also create a local-only manifest on disk and tell clients to use it. Manifests are just plists, and plists are just structured-data representations of dictionaries/hashes.

Nearly every programming language offers a mechanism for interacting with dictionaries/hashes in relatively easy ways, and Ruby (in both Chef and Puppet) allows for simple abstractions here.

Abstracting Local Manifests Into Simple Variables

I’m going to use pseudo-Ruby via Chef as the base for this, but the same principles will apply to any scripting language or tool.

The Process in pseudocode:

The point of the pseudocode above is to show how simple it is to abstract out what amounts to a complex process – deciding what software is installed or removed on a machine – and reduce it to simply two arrays.

To add something to be installed on your client, you add to the local managed installs variable. Same for removals and its equivalent variable.

What you now have here is a mechanism by which you can use any kind of condition or trigger as a result of your config management engine to determine what gets installed on individual clients.

Use Some Conditioning, It Makes It All Smooth

Veteran Munki admins are very familiar with conditional items. Conditions can be used to place items in appropriate slots – managed installs/uninstalls, optionals, etc. They’re an extremely powerful aspect of manifests, and allows for amazing and complex logic and customization. You can also provide your own conditions using admin-provided conditionals, which essentially allow you to script any logic you want for this purpose.

Conditions in Munki are critical to success, but NSPredicates can be difficult and unintuitive. Admin-provided conditionals are a convenient way to get around complex NSPredicate logic by scripting what you want, but they require multiple steps:

You have to write the scripting logic,

You have to deploy the conditional scripts to the clients

You still have to write the predicates into the manifest.

They’re powerful but require some work to utilize.

In the context of a local-only manifest, though, all of the logic for determining what goes in is determined entirely your management system. So there’s technically no client-side evaluation of predicates happening, because that logic is handled by the management engine whenever it runs. This unifies your logic into a single codebase which makes maintaining it easy, with less moving parts overall.

Some Code Examples

In Chef, the local-only managed_installs is expressed as a node attribute, which is essentially a persistent variable throughout an entire Chef run. This array represents an array of strings – a list of all the item names from Munki that will be added to managed installs.

If you look inside that file, you’ll see a plist with your managed installs and removals:

When managedsoftwareupdate runs, it will concatenate the server-side manifest with the local-manifest, as described above. The sample plist above will ensure that six items are always going to be installed by Munki on my machine, and that “Tableau8” will always attempt to uninstall if needed.

With a setup like this, anyone who can submit code to the Chef repo can easily configure their machine for whatever settings they want, and thus users have individual control over their own machines without needing the ability to access any of the server manifests.

Even If You Don’t Have Config Management

You can still benefit from local-only manifests without needing config management. Manifests, including local ones, are just plists, and there are lots of ways to manipulate plists already available.

Note the issue mentioned above, though, which is that it’s trivial for someone to add an item name that doesn’t exist in the catalog. Should that happen, the Munki client would generate warnings to your reporting engine. The benefits of using an external config management is the ability to lint or filter out non-existent items and thus prevent such warnings.

Summary

Ultimately, the benefits here are obvious. Clients have the ability to configure themselves without needing any access to the Munki repo. In addition, your users and customers don’t even need to have an understanding of manifests or how they work in order to get results. The entire interaction they’ll have with Munki will be understanding that items added to managed_installs get installed, and items added to managed_uninstalls get removed.

Stay tuned for a follow-up blog post about how this fits into Facebook’s overall managed Munki strategy, and how source control plays an important role in this process.

The issue is that the version of OpenSSL on OS X for the last several years is still 0.9.8zg. That version simply doesn't have an implementation of the pbkdf2_hmac() that Ruby wants to use. However, Python does, thanks to hashlib.

To recreate the same process in Python that the Chef documentation recommends for generating a 10.8+ password hash, use the following steps:

We use 25,000 iterations as a nice arbitrary number, but you should use anything above 10,000 to be at least minimally safe. Of course this is a local user account on a service machine in my context, so I’m not entirely worried about its security as much.

Once we have all the variables, we can use the actual pbkdf2_hmac() function. In the example above, we’re using the SHA-512 digest, with a derived key length of 128 as the Chef documentation suggests. Once again, the result of that command is binary data, and Chef requires a hexadecimal representation, so we turn to our trusty friend binascii.hexlify again.

The goal here is to be able to deploy a new computer with Chef and Munki preinstalled via DeployStudio (which runs over HTTPS), and then bootstrap Munki using SSL client certificates – meaning every part of the network deployment process is over a secure channel.

On First Boot:

LaunchDaemon that waits for the existence of /etc/ssl/munki2.sacredsf.org.csr triggers:

It will keep running the [x509::munki2_client] recipe while the CSR exists.

The CSR will only be deleted when the CSR is signed on the Chef server.

When the recipe succeeds and the CSR is removed and the .crt file is created, run the [munkiSSL::munki] recipe to copy the certificates into /Library/Managed Installs/certs/.

Touch the Munki bootstrap file.

With the certificates in place, Munki ManagedInstalls profile installed, and the bootstrap trigger file present, Munki can now successfully bootstrap.

The Detailed Process:

Preparing the Deployment:

For my deployments, I like using Greg Neagle’s CreateOSXInstallPkg (henceforth referred to by acronym “COSXIP”) for generating OS X installer packages. Rather than crafting a specific image to be restored using DeployStudio, a package can be used to both install a new OS as well as upgrade-in-place over an existing OS.

One of the perks of using COSXIP is being able to load up additional packages that are installed at the same time as the OS, in the OS X Installer environment.

As mentioned above, we’re going to use a number of specific packages. Here’s what the COSXIP plist looks like:

The Packages:

AddChefToHosts and AddMunkiToHosts are payload-free packages that just add the IP addresses for my Chef and Munki2 server to /etc/hosts, since this is just in testing and those services don’t yet exist in DNS. The scripts look like this:

#!/bin/sh
echo "10.0.0.1 chef.sacredsf.org" >> "$3/private/etc/hosts"

chef-12.1.0-1.Dist is a specially repackaged-version of the Chef client. You can find the recipe for this in my AutoPkg repo.

The reason I did this is because the Chef-client’s postinstall script assumes that the target volume is a live booted OS X install – which is not true of the OS X install environment. The OS X install environment doesn’t have all OS X features, and the Chef client postinstall script will fail to do certain things like run basename and uname, and the symlinks will not work properly as they are executed in the OS install environment. My AutoPkg recipe addresses these issues and repackages the Chef client in a manner that is more compatible with the OS X install environment.

The validation.pem file is the private key of the organization. See this blog post for details.

ClearRegistration, CreateAdmin, Profile-SetupAssistant are packages that bypass the OS X first-time boot setup process, by skipping the device registration, creating a local Admin account, and then skipping the iCloud Setup Assistant on first login. This allows me to boot straight to the Login Window and then login straight to the Desktop with no interruption.

Outset is the distribution package of Joseph Chilcote’s Outset, a fantastic tool for easily running scripts and packages at boot time and login time (which is easier than writing a new launch agent or launch daemon every time).

The script sets the hostname to the serial number (which I’m just using in my test environment so I can boot multiple VMs without having all of them be named “Mac.local”), and then runs the Chef client to trigger the generation of the CSR.

XCodeCLITools installs the Xcode Command Line tools from the Developer site. This isn’t strictly necessary, but if you run Chef-client manually it will prompt you to install them, so preinstalling saves me some testing time.

ChefCSRTrigger installs the Launch Daemon that watches the path /etc/ssl/munki2.sacredsf.org.csr. So long as that path exists, this Launch Daemon will continue to trigger. The CSR is generated by the first run of the Outset Chef script, and this will keep making a request until the CA signs the CSR. The Launch Daemon looks like this:

It runs this script:

Once the CSR is found, the script will attempt to run the same recipe again. If the recipe succeeds, the CSR will disappear and instead, /etc/ssl/munki2.sacredsf.org.crt will appear. If this file exists after the Chef-client run, the script will proceed to try and run the [munkiSSL::munki] recipe until it has successfully copied over the cert into /Library/Managed Installs/certs/clientcert.pem (which should theoretically only take one run). Then, it will create the Munki bootstrap file.

With all of these packages, you can build your OS X installer to use in DeployStudio:sudo ./createOSXinstallPkg --plist=InstallYosemite-ChefMunki.plist

Deployment:

When a computer is NetBooted into DeployStudio, and the OS is installed (along with all the packages above), that’s when the fun stuff happens.

On first boot, Outset will execute the run_chef.sh script (installed by the Outset-ChefClient package). This script will wait for network access, and then use scutil to set the HostName, LocalHostName, and ComputerName to the serial number. Then, it will run execute the first Chef client run with the [x509::munki2_client] recipe, which generates a private key and submits a CSR to the Chef server.

The creation of the CSR file at /etc/ssl/munki2.sacredsf.org.csr triggers the execution of the org.sacredsf.chef.csrtrigger LaunchDaemon (installed by the ChefCSRTrigger package), which will continually run the chef_munki_cert.sh script while that CSR file is present.

On the Chef server/workstation, the CSR needs to be signed (this is assuming the ChefCA is set up according to previous blog posts):chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

When the CSR is signed, the LaunchDaemon that is spinning in circles around the CSR file will finally have a successful chef-client run. The successful run will delete the csr file and create the signed certificate file at /etc/ssl/munki2.sacredsf.org.crt.

Once this file exists, the script will then trigger the [munkiSSL::munki] recipe, which copies the certificates and private keys from /etc/ssl/ into /Library/Managed Installs/certs/ with the appropriate names.

Finally, the Munki bootstrap file is created at /Users/Shared/.com.googlecode.munki.checkandinstallatstartup.

The appearance of the Bootstrap file will cause Munki to execute immediately (as we’re still at the Login Window at this point). Munki will read the preferences from the ManagedInstalls profile settings, which tells it to use the certificates in /Library/Managed Installs/certs/ to check the server https://munki2.sacredsf.org/repo for updates.

If the certificates are valid, Munki will proceed with a normal bootstrap run, except through a secure SSL connection that uses its client certificates to communicate with the Munki server, which has been configured to require these certificates (see the previous blog posts).

Conclusion

It’s now possible to securely bootstrap a new OS X machine using Chef to set up SSL client certificates to use with Munki. The best part is that it doesn’t require hands-on attention on the OS X client. The downside is that, at this point, it does require hands-on attention on the Chef server, where the CA is.

There are some possible easy fixes for that, though. The easiest solution would be to run a cronjob on the Chef server that automatically signs all CSRs every X amount of time, which would eliminate any need for manual intervention on the Chef CA. That’s not a desirable method, though, because that’s essentially letting any client who runs the right recipe get a free SSL certificate to the Munki repo. There’s no verification that the client is one we want to allow.

Another possibility to use a more industrial-strength internal CA not managed by Chef, which can have its own policies and methods for signing certificates. This is more common in enterprise environments where they tend to have their own root CAs and Intermediary CAs for internal-only services. More commercial offerings of this sort of thing probably have better methods for determining which CSRs get signed and which don’t.

The chef-ssl client can also be used to generate CSRs for third-party external CAs, but you probably wouldn’t want to sign individual clients with an external CA.

At least we can bootstrap a large batch of machines at once. With 30 machines running, they’ll all submit CSRs and sit there waiting until they get signed. In one command on the CA server, you can sign all 30 CSRs and they’ll automatically proceed to the next step, which is to get the certs and then bootstrap Munki. So we’re at mostly unattended install. But hey, as a proof of concept, it works!

This run command sets the open port to 443, for SSL connections, and uses the munki2-data data container to access the repo. In addition, I’ve set the hostname here manually as well, using the -h option.

If you are just testing this out and don’t have your Chef server entered into your DNS, you can fix that inside the container using the --add-host option like so:docker run -d --name munki2 --volumes-from munki2-data -p 443:443 -h munki2.domain.com --add-host chef.domain.com:10.0.0.1 nmcspadden/munki-chef

The first step after running the container is to check in with Chef:docker exec munki2 /usr/bin/chef-client --force-logger --runlist "recipe[x509::munki2_server]"

Here, we’re using the [x509::munki2_server] recipe to generate a private key and send a CSR to the Chef CA.

On the Chef server or workstation, you’ll need to sign the CSR:chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

Back on the Docker host, run the x509::munki2_server recipe again to receive the signed certificate:docker exec munki2 /usr/bin/chef-client --force-logger --runlist "recipe[x509::munki2_server]"

Now that the certificates are present, it’s time to add in the new Nginx config file to tell the webserver to use client certificates:cat munki-repo-ssl.conf | docker exec -i munki2 sh -c 'cat &gt; /etc/nginx/sites-enabled/munki-repo.conf'

The Nginx configuration looks like this:

The three important file paths that must be correct are ssl_certificate, ssl_certificate_key, and ssl_client_certificate. If any of these paths are wrong or can’t be found, Nginx will not start and your Docker container will immediately halt.

For reference, the ssl_protocols and ssl_ciphers are configured for perfect forward secrecy.

Otherwise, the configuration for Nginx for the Munki repo remains the same as the non-SSL version – we’re serving the file path /munki_repo as https://munki2/repo/.

Now restart the container to reload the Nginx configuration:

docker stop munki2 && docker start munki2

We have a working Munki server on port 443, now it needs to be populated.

Configure the Clients to use Munki with SSL certificates:

If you are testing this out, you probably don’t have munki2 or Chef in your DNS entry. You’ll need to add them to your /etc/hosts file on the clients first:10.0.0.1 munki2 munki2.domain.com10.0.0.2 chef chef.domain.com

Detailed instructions on configuring Munki with SSL certificates can be found on the official wiki, but I’m going to recreate the steps here.

On the client, run the [x509::munki2_client] recipe to generate a CSR:sudo chef-client --runlist "recipe[x509::munki2_client]"

On the Chef Server or workstation, use chef-ssl to sign the CSR:chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

On the OS X client, run the recipe again to receive the signed certificate:sudo chef-client --runlist "recipe[x509::munki2_client]"

With the client set up with its certificate, now it’s time to configure Munki. Run the [munkiSSL::munki] recipe:sudo chef-client --runlist "recipe[munkiSSL::munki]"
This recipe copies the client certificates from /etc/ssl/ into /Library/Managed Installs/certs/ where Munki can use them.

Note: Despite my best efforts, I could not get this to work with the default Chef self-signed certificate. This only functioned when I switched to a real SSL certificate. See my previous blog post for a guide.

Set Up the Server:

Most of these instructions will require root or sudo. Also, it’s helpful to use the Ruby provided by Chef, so check which ruby to make sure you’re using the one in /opt/chefdk/embedded/bin/ruby
Let’s go through the process of setting up the x509 cookbook to create a CA for us:

Now that the recipe is created, we need to upload the cookbooks to the server:knife cookbook upload -aThat command uploads all cookbooks. If you have more than the x509 and vt-gpg cookbooks installed, you can be more precise:knife cookbook upload x509 vt-gpg

Finally, create the actual CA. Substitute the ca-path with a valid one of your own:chef-ssl makeca --dn '/CN=ChefCA' --ca-path /home/nmcspadden/chefCA
Enter a passphrase when prompted.

Test It:

Now, it’s time to test it on a Chef client. I tested this on a Mac OS 10.10.2 Yosemite VM, which I set up in the original article about OS X client configuration:sudo chef-client --force-logger --runlist "recipe[x509::chef-ca]"

Back on the Chef server / workstation, you can check the CSR in the node’s data. If you don’t know the name of the node, run this command to get a list:knife node list
In my case, my node’s name is TestVM.local.

Sign It:

Now we can sign the CSR on the server:chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA
Enter the passphrase you used earlier to generate the CA.

This command will search for all outstanding CSRs and ask you to sign them (which is horribly inefficient and requires lots of manual effort for production use, but that’s a future topic for discussion).

Once you have signed all outstanding CSRs (in this example, only one) by entering “yes” when prompted, the run completes:Saved OKAll CSRs processed.

You can use openssl to verify your certificate:openssl x509 -in /etc/ssl/chef.sacredsf.org.cert -issuer | head -1

issuer= /CN=ChefCA

Conclusions

Chef now provides a simpler way of generating client certificates than creating your own CA using openssl. However, there’s still some major pitfalls here that get in the way of a strong SSL client certificate solution.

Another issue is that because the CSRs require manual approval, it currently takes twochef-client runs to get an SSL cert. The next goal is to get this step down to being more automatic and streamlined.

By default, Chef Server uses a self-signed certificate. This is fine for small testing purposes, but this becomes a significant problem in production, especially when using other Ruby tools or modules that require SSL verification.

Thus, I highly recommend configuring the Chef server to run with a real, trusted SSL certificate. For this example, I got a free one from StartSSL. I chose StartSSL because it’s free, and the root CAs are already in the default OS X and iOS trust stores. As in the previous article, this is with Chef Server 12 running on CentOS 6.6 in VMWare.

In this post, all of my examples will be for “chef.sacredsf.org”. Substitute in your own example.

NOTE: you do not need to have this server exist in DNS. As long as the server knows its own hostname, you can do this all on one VM.

Once you go through the StartSSL certificate wizard, you’ll be given four things (not all filenames will be accurate, I’ve renamed them for clarity):

The actual SSL certificate: chef_sacredsf_org_startssl.crt

The encrypted private key: chef_ssl_encrypted.key

The intermediate certificate: sub.class1.server.ca.pem

The root CA certificate: ca.pem

With all of these downloaded, here are the steps for setting up Chef to use these SSL certificates:

(Note that I put this into a separate “Complete” folder to keep track of which one to copy)

Copy Complete/chef.sacredsf.org.pem and chef.sacredsf.org.nopassphrase.key to your Chef server (via scp, or whatever mechanism works for you – since I did this in VMWare, I just used the drag-n-drop capability).

Finally, reconfigure the server with the new SSL certs:sudo chef-server-ctl reconfigure

If you have a problem here, or nginx takes too long to start up (or fails to start up), it’s almost certainly because of a problem with the certificates. Use cat to check the contents of the certs to make sure there are no typos, there is space between each BEGIN CERTIFICATE and END CERTIFICATE line (there should be 3 total), and that you copied the correct files.

Chef is a configuration management tool, similar in concept to Puppet. As I test out various configuration management tools and methods, I wanted to try my hands at Chef to see what the process would be like.

I find that Chef’s documentation is a bit ambiguous, so I’ve recreated it here to show the steps I used to get something working.

The server / workstation setup was all done in a CentOS 6.6 VM.

The client configuration is done on an OS X Yosemite 10.10.2 VM.

Server Setup:

In these instructions, “ssh” refers to the short name for my organization “Schools of the Sacred Heart SF”.

This “ssh.pem” file is the private key for my organization, with short name “ssh”. If you create an organization called “test” you can name the private key “test.pem” and use that wherever you see “ssh.pem” in these instructions.

Now, set up the Chef Workstation where you can get things done. This can be on the same computer, and in my case is in the same VM – the Server and Workstation are the same virtual machine. Follow instructions for Installing Chef Workstation:

Copy the ssh.pem created earlier to /etc/chef/validation.pem on the client (name is important).

You need to copy the trusted certs from your chef-repo (~/chef-repo/.chef/trusted_certs/) to the client as well, into /etc/chef/trusted_certs/. If you don’t have these trusted certs on the chef server, use this command to generate them: knife ssl fetch

Write a file client.rb to /etc/chef/client.rb:

Run sudo chef-client to trigger the initial chef run.

You now have a working Chef client install with a working Chef server!