Cobbler or just straight kickstart for VMware ESXi?

I’m working on automating some installs that are going to happen during the infrastructure upgrade, and I need to decide what I want to use for automation.

I have used Kickstart before, and it’s essentially a single file that contains instructions for the RedHat installer (although Debian is in on that action, too). The idea with Kickstart is that your “normal” installation (whether that be through DVD, USB key, PXE, or whatever) points to the kickstart file, and the installation proceeds according to those instructions.

Cobbler goes the extra steps and becomes the installation server, PXE/DHCP boot provider, etc etc, in addition to working with kickstart files. In fact, it can even do crazy kickstart templating. It certainly seems full featured, and I’ve heard people recommend it before.

One of the coolest things I’ve seen it be able to do is automate new virtual machines. As I understand it, you basically hit a button and a VM is created, powered on, and installed according to the kickstart templates. That’s slick.

Unfortunately, the best support for Koan (the Cobbler client) is on Qemu/KVM. The site mentions support for VMware Server, but that’s anathema. There doesn’t appear to be support for ESXi (certainly not 4.1, which was just released last month), but I was hoping for something more recent than a question on VMware Communities from 2007.

So I come to you. If you’ve got an ESXi infrastructure, do you automate rollouts? Am I just doing this wrong? I’m leaning toward manually spinning up machines and using Cobbler / Kickstart to perform the installs (maybe with customized boot media, in the case of just kickstart). What do you suggest?

Post navigation

Actually ESXi 4.1 does have Kick Start support. Kendrick Coleman (@KendrickColeman) just last week posted an ESXi 4.1 KS script to his blog page. It is currently work in progress but should be an excellent starting point.

I’ll be building out a couple new 4.1i boxes this week and plan to give it go.

We’ve built an automated bare-metal-to-functional-server installer at $JOB. The network installer environment consists of a TFTP/PXE/DHCP/DNS/NFS/HTTP server with various forms of deployment scripts/configs (preseed files for Debian/Ubuntu, Jumpstart for Solaris) which allows a host with blank disks to end up installed with a base set of packages in about 5 minutes or so, with no interaction after selecting your OS.

VM’s or physical doen’t really make any difference here; if it can see the right network and is capable of PXE booting (ESXi VM’s are), you’re good to go.

The next steps are to automate some personalisation during the install (hostname / IP address), and automatic registration with the Puppet infrastructure to allow for automated provisioning to the point of the new host doing real work.

http://rejectreality.net Greg

@Jeff – I’d be reluctant to use something that directly edits files in Tech Support mode if you’re keen on keeping your environment supported.

As an alternative, look into VMware’s vCLI, shipped either standalone or as part of the vSphere Management Assistant (vMA) to either implement or emulate almost everything in that script. By scripting a chain of vCLI commands, you can achieve the same goal without accessing tech support mode.

We use this to deploy both physical and virtual servers. Cobbler does not support VSphere, but we just have to create a VM manually, report the MAC address in a new Cobbler “system” and that’s it.

Having to create the VM system manually in Cobbler is just a minor annoyance for me given the huge benefits Cobbler brings: we can deploy complex / ready to use servers in less than 20 minutes.

Regards
@Farzy

http://www.standalone-sysadmin.com Matt Simmons

@Jeff

That looks really promising! Thanks for the link. I’ll read through it and see how it works out for me.

@chewy
Yuck. I hate black boxes. Good luck figuring out the incantation to make it work :-)

@Greg
Nice! I know that whatever solution I put in place will have to be an all-in-wonder like yours, since when I’m going to be installing these, there won’t be network services to speak of. I’m not finalized on which solutions I’ll be rolling onto the box. I suppose this discussion is part of that process

http://www.pragmatic-source.com Farzad FARID

I forgot to say that we’re about to integrate kickstart files & scripts for the installation of ESX 4.0 servers themselves in Cobbler.

But for the moment we have some difficulties to get the kickstart installation to run standalone as some “esxcfg-*” commands do not seem to be executed correctly during the installation..

http://www.standalone-sysadmin.com Matt Simmons

@Lee – Interesting, I hadn’t heard of SystemImager. That does sound cool. Thanks for the link

@Farzad – That sounds pretty much exactly like what I’m trying to do, sans the Oracle stuff. Thanks for the comment. At least I know I’m not headed in completely the wrong direction :-)

http://badcheese.com Steve Webb

I use cobbler with LSXi at $JOB and we just get a mac address from ESXi and go from there. There’s nothing special that we use with ESXi to get cobbler to work friendly with it. Just boot from the network card, if your VLANs are set up properly, cobbler will hear the DHCP broadcast and your machine should connect and do its thing.

http://www.pragmatic-source.com Farzad FARID

Oh I forgot another interesting information :) We use Opscode’s product, <b<Chef, for configuration management. I fully inegrated Cobbler and Chef by making Cobbler deploy the Chef Client and register the new server in Chef.

For this I have written a Cobbler extension. This extension also maps Cobbler “classes” into Chef “roles”.

And it works like a charm: as soon as the newly installed server reboots, it connects to the Chef server and updates its configuration according to the roles it belongs to.

I haven’t written a documentation or article on the subject yet but if anybody’s interested fell free to contact me and I’ll help you.

Stan Chan

Just trying to figure out what the requirements are for the ESXi integration. Can someone give me a general workflow for the use case? I have forked Cobbler/Koan for some of my clients projects to integrate some additional requirements and I would like to add additional support with ESXi/OVF images if it is a trivial effort. I’m hoping to get it merged upstream eventually to get that support in.

Stan Chan

On a side note, at a glance, libvirt is capable of accessing ESX/ESXi drivers using the vSphere API.

http://www.cmdln.org/ Nick Anderson

You are using puppet now right? You could create a small template and then use puppet to do the rest. This has the additional benefit of getting more configuration documented in your configuration management system.

I still prefer to do initial installs via a kickstart or debian preseed as opposed to cloning a template. But I think it’s 6 of 1 and half dozen of another if your only looking at the one rollout. Long term though being able to network install virtual or physical followed by configuration management is best.

http://andrew.j.cosgriff.name/ Cos

FWIW, I’ve been doing PXE boot with Ubuntu preseeding to get started, then using Puppet to do the rest for VMs on our new ESXi servers.

http://www.big-bubbles.org.uk/ Ade

Like others above, we’re using Cobbler and Puppet to provision new VMs with no major problems; just make sure your VM is on the same network as the servers initially, or generate an ISO in Cobbler. Oh, and don’t configure the VM with a VMXNET3 NIC, as the driver isn’t in default RHEL so it won’t net boot. You can use VMware’s OSP repo as a source to automate VMware Tools package installs if necessary (although our VMware admins don’t like this as they prefer to manage the Tools themselves).

The trickier part is scripting the actual VM creation, which I haven’t looked at yet. Could do with some sort of framework wrapped around vCLI.

natxo asenjo

We’re doing pxe with FAI and cfengine to get everything working (debian shop here). For testing purposes I tried pxe + kickstart + cfengine for vmware (3.5 still) and it works a treat.

http://blogs.law.harvard.edu/hoanga Al Hoang

Cobbler is great if you are going the Redhat route as importing a new version of Centos/Fedora/Redhat is quite simple once you get Cobbler installed and bootstrapped. If you are trying to be more agnostic on (Linux) OS, I would look more closely at SystemImager

Spending time to get puppet installed and inserted into your provisioning setup will definitely save time configuring the servers as you see fit. If you google around for Cobbler and puppet there are some other people that went this route and it should not be that difficult to follow in their footsteps.

http://blogs.law.harvard.edu/hoanga Al Hoang

One more note… I think it would be prudent to spend time simplifying the process where you spin up a new ESXi guest and have it do a PXE boot / installation on guest bringup. Some other people have already mentioned the same approach in the comments. That part will be ESXi specific and can probably be encapsulated in a (not so) simple script that is tailored to your infrastructure needs (usual parameters are memory, disk, and CPUs).

Once that part is completed, you can use more ‘standard’ bootstrap / installation tools like SystemImager, fai, JumpStart or Cobbler to provision a bare metal guest then hand it off to a configuration management system like Puppet, Chef, Cfengine to handle the rest.

http://gadget.geek.nz Canllaith

I tend to use kickstart, and then have all of the post-install customisation done with Puppet or Cfengine. I haven’t looked into Cobbler – but it looks pretty interesting. I might have to see if I can integrate it into the way I do things.

http://benincosa.com Vallard

We use xCAT and it handles ESXi 4.1 kickstart files (templating) very nicely. It’s pretty full featured so takes over your DHCP, TFTP, DNS environment. The thing I love about it is that it also handles remote hardware control so you can install or reinstall nodes very quickly. xCAT is open source though it was started at IBM.

I’d love to see a blog or article on using chef or puppet for configuring it after its installed with kickstart.

Ian

I use cobbler for PXE booting VMs extensively. We have a build vlan where our VM’s PXE boot,and a base OS is installed. After the PXE process is complete, kickstart makes a web call to a little webservice i wrote, that changes the VLAN assignment of the newly created VM, to its “production” vlan.

IMHO image based solutions like SystemImager or VM templates do not lend themselves to software updates very well. If you constantly are having to run updates on your base image that is an extra step in making sure you have the latest patch set.

Kickstart allows you to perform a yum update as the last step of the pxe process, ensuring that you are getting the latest patch set for that time. Other machines can then be updated using chef/puppet, with a call to yum update.

In short, image systems add another element of upkeep to your infrastructure.

About the Author

Hi! I'm Matt Simmons, author of the Standalone Sysadmin blog. I've been a small infrastructure administrator since aound 2001, nearly always on my own, hence the "Standalone Sysadmin" title.
This year, I started working at Northeastern University's College of Computer and Information Science, so I'm getting a taste of a larger environment,but it's still small in the grand scheme of things, but I'll be passing along this new experience as I go through it!
I started this blog in May of 2008 and have thoroughly enjoyed every interaction I've had because of it. Thank you for visiting!