In our CoreOS clusters, Puppet + Foreman integration provides us with the following benefits:

End-to-end automation for node provisioning (i.e. installation and configuration), including IPMI, means you can treat nodes as cattle, not pets

Assign IP addresses centrally for all use cases (in EPFL-STI clusters we use separate subnets for the following: IPMI for lifecycle management, RFC1918 IPv4 for privileged services, and routable IPv6 for tenants)

Poor man's monitoring: was the node at least alive in the last 30 minutes?

How it works

Puppet-in-Docker

To run Puppet on CoreOS, we rely on two Docker images:

one for the Puppetmaster, which is actually bundled with Foremaninside thecluster.foremanproject,

and one for the Puppet agent (see puppet-agent/Dockerfile in this project).

As is customary for a number of CoreOS services, the Docker containerfor the Puppet agent is run as a systemd service, and both thecontainer and the service are named identically (puppet.service).

Two-Stage Bootstrap

In the standard deployment scenario,cluster.foreman andcluster.coreos.installcooperate to run the Dockerized Puppet agent a first time after CoreOSis installed on the node being provisioned, and before it reboots. Atthis stage, Puppet is responsible for installing itself into theon-disk system image so that it operates normally after reboot.

In order for the Puppet code to to distinguish the bootstrap andsteady-state stages, Puppet is passed an environment variableFACTER_lifecycle_stage=bootstrap, which translates to$::lifecycle_stage == "bootstrap" in the code. Puppet arranges torun itself with FACTER_lifecycle_stage=production after reboot.

This two-stage setup is made necessary by the fact that beforerebooting, the provisioning host needs to transition from the"building" to "built" states in Foreman; this is so that even if theBIOS is still configured to boot through PXE, the pxelinuxconfiguration on Foreman's TFTP server will instruct the provisionedhost to boot from the local disk.

Not all Puppet classes are aware of $::lifecycle_stage; for thosethat aren't, manifests/init.pp simply excludes them at bootstraptime. See the comments at the top of the individual manifests/*.ppfiles for more details.