Automating ESXi 5 Kickstart Tips & Tricks

There are some minor changes with kickstarting ESXi 5.0 but the majority of your existing ESXi 4.1 kickstart configurations can be re-used with a few modifications. One of my goals during the vSphere 5.0 beta was to automate as much of the configurations of an ESXi host as possible within the kickstart process. I also converted as many of the legacy esxcfg-* commands as I could over to the enhanced esxcli namespaces as the esxcfg-* commands will eventually be deprecated in favor of esxcli. Hopefully the tips & tricks and the example kickstart configuration file will be useful in aiding the transition to ESXi 5.0.

As usual, before diving in and creating an ESXi 5.0 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide. If you would like to get a sense for what a ESXi 5.0 kickstart could look like, please jump to the bottom of this post to get a complete working example displaying the various types of configurations.

Tip #1

If you want to have your ESXi pxelinux configuration boot up to a kickstart configuration file, you have two methods of specifying this with ESXi 5.0.

The first is using the defualt installer method which specifies a new boot.cfg configuration file to contain all the boot parameters.

Here is an example of what the pxelinux default configuration file would look like:

Here is an example of what the boot.cfg looks like:

As you can see the default boot.cfg that is included in the ESXi 5.0 installer in the same directory as all the modules needed to boot up ESXi.

The second method is specifying the kickstart configuration file in the actual pxeboot file versus in a seperate boot.cfg file. This is similar to the old method of doing things, but you will have to also include all the entries that are in the boot.cfg if you decide to go down this route.

Here is an example of specifying ks.cfg in pxelinux default file:

Notice the "pxebooting ks=" stanza used to specify the ks.cfg configuration file and list of modules in boot.cfg. If you need to append additional parameters such as "IPAPPEND 1", you will need to add "+++" (three plus characters) at the very end of the module list separated with a new line for your additional entries. If you forget the "+++" symbols, you will not be able to successfully boot up the installer and an error will be thrown.

The default method of relying on the boot.cfg is the recommended approach. You can still append custom kernel parameters such as "IPAPPEND 1" which will still be in the pxelinux file, but your kickstart entry will now be part of the boot.cfg configuration file. Below is an example of the above configuration but leveraging the additional boot.cfg configuration file.

Here is an example of what the pxelinux file should look like:

As you can see, we still need to specify the "+++" to include additional parameters, but the pxelinux file is much cleaner now.

Here is an example of what the boot.cfg should look like:

As you can see, all we needed to do is append the following line "kernelopt=ks=http://172.30.0.108/esxi5/ks.cfg" which specifies the source to the kickstart configuration file.

Tip #2

This is not really a new tip, but definitely make use of the "dryrun" mode in the ks.cfg and reviewing the esxi_install.log and hostd.log for any errors. The logs will only persist upon the first reboot, if you specify a secondary reboot for whatever reason, these logs will be lost. If you would like to automate the copying of the installation logs to help debug/troubleshoot, you can copy them to a local VMFS. Here is a quick snippet that can be used (This assumes your local VMFS will have the name with the following format $(hostname -s)-local-storage-1):

VMware has once again and hopefully for the last time, changed what was known as Busybox Console, Unsupported Mode, Tech Support Mode to ESXi Shell. In doing so, the old vim-cmd to enable both local (TSM) and remote (SSH) to ESXi Shell needs to be updated.

To enable local ESXi Shell (previously known as Tech Support Mode) you will need to run the following commands to enable and start ESXi Shell.

vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

To enable remote ESXi Shell (previously known as TSM SSH) you will need to run the following commands to enable and start remote ESXi Shell for SSH support.

vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh

Note: During the beta, there was actually an intermediate name for this which was known as ESX Shell but through our feedback, we had recommended ESXi Shell so that users do not confuse old classic ESX with ESXi and this may or may not change from the actual vim-cmd's

Tip #4

The --levelXX that would specify when a %firstboot script would execute is no longer supported and deprecated in ESXi 5.0. By default all %firstboot script will automatically execute after all default ESXi start up scripts have finished running. The location of the %firstboot scripts are in /etc/rc.local.d/001.fireboot_001

Tip #5

You shouldbe able leverage the new changes in esxcli to configure majority of your ESXi 5.0 hosts, the are various examples below in the example kickstart. There are still a few things that are currently not included in esxcli such as configuring NTP, host licensing, ESXi Shell, SSH, and a few others that rely on vim-cmd or some minor hacks. VMware recommends you start to get familiar with esxcli, as the old esxcfg-* will eventually be deprecated and removed in future vSphere releases and be completely replaced with esxcli.

Tip #6

You now have the ability to configure multiple syslog hosts with ESXi 5.0 but also the ability to control individual loggers such as for vmkernel, hostd, vpxa, fdm, etc. Here is an example of changing the default syslog rotation from 10 to 20 and specifying two syslog hosts:

In ESXi 4.1, to enable the SSH security banner, you had to make some minor hacks. With ESXi 5, there is not an official sshd_config and you can configure the security banner by editing /etc/issue. You can also edit the motd under /etc/motd.

If you are going to SSH out of an ESXi host at any point (%post, %firstboot) ensure that you enable sshClient via the ESXi firewall else you will get a connection denied. To enable the ESXi firewall use the following command:

Sounds like you may have some syntax errors in your boot.cfg. Did you create the boot.cfg on a Linux system or Windows, you may have some hidden Windows carriage return which is causing bad parse as mentioned by the error message

I also have a beta version of vSphere 5, build 381646. I tried putting the modules line in the pxelinux.cfg/default file as well, but the pxelinux.o fails to parse the configuration and present the menu. I’m stuck and really need to have this functionality. Any assistance would be greatly appreciated.

Will, I’m guessing you use DHCP to get an IP to the vmk0, we don’t have DHCP running on our server vlan nor do we have a kickstart server so I was using kernel options to pass info to the ks script. In ESXi 4.1 for example, I modified the isolinux.cfg so I could hit Tab and be presented with the following line:

variables should be obvious (ENV is to specify prod, dev, or lab which kicked off different KS.cfg commands for different environments.) With the kernel variables going to the boot.cfg file, and no longer visible from the initial boot command line, how would i modify the “APPEND -c boot.cfg” line to get these variables into my KS script?

Anyone knows how to pass multiple kernel options in the boot.cfg file? If I try to use multiple kernelopt= lines, then it will take into account only the last definition. I’d imagine a separator has to be used… but I haven’t been able to find any examples anywhere.

where BOOT.CFG looks something like:title=Loading ESXi installerkernel=http://SRVNAME/ESXi/5.0/tboot.b00kernelopt=ks=http://SRVNAME/cblr/svc/op/ks/system/esxsrv01 BOOTIF=ff:ff:ff:ff:ff:ffmodules=…where of course BOOTIF contains the server’s mac address and modules= contains a list of URLs for modules to be loaded.

If someone is facing a problem when booting from pxe and the screen is blank, check your pxelinux.0 version..

PXELINUX 3.10 2005-08-24 – gives only blank black screen after booting from pxe.PXELINUX 3.30 2006-09-18 – shows the graphics, but fails to load all the installation files.. guess there is somekinda limitation on amounth of files that can be downloadedPXELINUX 3.50 2007-06-09 and newer – work just perfectly

I ran into the same problem. Thanks for sharing the info (PXELINUX versions). I initially stared with whaever pxelinux version that comes with centos5.8. Ran into the same issue as yours. Then I downloaded the recommended version – pxelinux 3.86 and it worked fine!

What am I missing? I cannot get the %pre section of the above script to obtain the IP information from an ESX 4.1 host. Is it because there are very limited commands available at the time when this portion of the script is executed? All of the parameters to the network command wind up being blank!

The syntax of these commands look more like ESXi 5 commands. Is that the case?

@William, I can understand the idea for doing this in an install….but the piece I am missing is with an Offline Bundle using AutoDeploy, how do I call a script to run when ESXi is starting up(not installing)…thx

Not sure I follow your question, Auto Deploy is stateless, so if you’re able to get the suggestion working, then you can technically embed anything that’ll run upon startup. Again, this probably would not be officially supported by VMware, but you probably could get it to work.

Yes….that is my question, how do I actually embed those commands to run at startup. That is the piece I am missing. What file would those commands go in and how do I call that script to run as part of the startup with AutoDeploy? I assume those commands could just be the esxcli commands. I am not doing this in production, just testing in the dev environment.

@William..Yes….that is my question, how do I actually embed those commands to run at startup. That is the piece I am missing. What file would those commands go in and how do I call that script to run as part of the startup with AutoDeploy? I assume those commands could just be the esxcli commands. I am not doing this in production, just testing in the dev environment.

I am trying to find a way to prompt for the hostname at the start of the install. Is this possible? We’ll be provisioning about 70 servers and then shipping them all over the world and I want the provisioning team to just type in the hostname so that I can then use that to do more host-specific config later on like using the hp utility to configure the iLO and embedding the server name in the datastore name.

How can I review the logs from the kickstart script to see if anything failed? All of my %firstboot commands worked except for the wget command to copy the authorized_keys. The wget command works fine if I manually run it later but does not during the kickstart.

I am trying to do non-interactive installation of 5.1.It is downloading the kernel from tftp server but it is going to interactive install and it is not using the kickstart file.I checked the boot options by pressing shift+O while loading the installer and it has the ks file path.I have tried all possible things I can but I did not succeed.

I modified this excellent script to do a fresh install and not upgrade ESXi, which works like magic. Is there anyway one can add the server to vcenter from within the script to join an existing cluster?

In other words are there any command line that one can run from the ESX prompt to join a cluster on vcenter?

Hi, I am new to ESXi, i was able to use KS.cfg file by ur reference.
i want to achieve following automation.
1) boot machine using USB and install esxi using KS.cfg.
2) USB has windows iso. copy the iso from usb to datastore.
3) create vms and map iso to vm.

Trackbacks

[…] configurations between ESXi 5.1 and ESXi 5.0, majority of the tips and tricks noted in the ESXi 5.0 kickstart guide are still relevant for ESXi 5.1. Below are a few new tips and tricks (some old) as well as a […]

[…] Note: You will also be asked to fill out a few properties such as the license and password for your ESXi host as well as the network information. If you are interested in the kickstart that is being used for the ESXi deployments, you can take a look at /opt/razor/lib/project_razor/model/esxi/5/kickstart.erb. If you would like to adjust the kickstart file, be sure to take a look here. […]

Primary Sidebar

Search this website

Author

William Lam is a Staff Solutions Architect working in the VMware Cloud on AWS team within the Cloud Platform Business Unit (CPBU) at VMware. He focuses on Automation, Integration and Operation of the VMware Software Defined Datacenter (SDDC).