While I was upgrading our vSphere to 5.5 I was put to the task of going the extra step to replace the self generated SSL certificates on our ESXi hosts with certificates generated by our internal CA. This was going to help us pass various security audit requirements so I wanted to put it into the upgrade plan. I found various posts about it and this VMware KB2015499. I followed the KB article and when I tried to restart the services the host would not connect back to vCenter. After a little more research I found that the KB was missing a very important step that converted the certificate issued by our CA into a x509 certificate. Here are the steps that I used to finally get the certificates to work. Some of these steps are taken directly from the VMware KB.

You must have OpenSSL installed on your computer to complete these steps.

Launch a command prompt and navigate into the OpenSSL directory. Mine is C:\OpenSSL\bin>

Execute the command:
C:\OpenSSL\bin>openssl genrsa 2048 > rui.key

Answer all of the prompts and at the Common Name prompt make sure you enter the fqdn of the host you are configuring the certificate request for. (hostname.company.com) Note: You can change the 2048 to whatever size certificate you want, I wanted 2048 bit.

Now that you have your certificate request you will need to log into your Microsoft CA server and get the certificate.
1. Log in to the Microsoft CA certificate authority web interface. By default, it is http://<servername>/CertSrv/
2. Click Request a certificate.
3. Click advanced certificate request.
4. Click Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
5. Open the certificate request in a plain text editor. This is the rui.csr file.
6. Copy from —–BEGIN CERTIFICATE REQUEST—– to —–END CERTIFICATE REQUEST—– into the Saved Request box.
7. Click Web Server when selecting the Certificate Template.
8. Click Submit to submit the request.
9. Click Base 64 encoded on the Certificate issued screen.
10. Click Download Certificate.

Now for the important part that the KB is missing.

Copy the certificate file that you downloaded, to your openssl directory that you have been working in. The file is usually certnew.cer and run the following command.
C:\OpenSSL\bin>openssl x509 -in certnew.cer -out rui.crt

Now the final step is to copy the certificate and the key to the ESXi host. I prefer using WinSCP but you can use any method you like to get the files to /etc/vmware/ssl.
1. Log in to vCenter Server
2. Put the host into Maintenance Mode.
3. Enable SSH on the host from the host configuration tab.
4. Log in to the host with WinSCP and navigate to the /etc/vmware/ssl.
5. Rename the existing rui.crt and rui.key to .old
6. Copy the newly created rui.crt and rui.key to the directory.
7. Log onto the host with Putty and restart the management agents. (services.sh restart)
8. Reconnect the host in vCenter. You will need to use the root password and accept the new certificate.
9. Exit the host from Maintenance Mode.