Metrics and Thresholds of RESXTOP

Overprovisioning of vCPUs, excessive usage of vSMP or a limit(check %MLMTD) has been set. See Jason’s explanation for vSMP VMs

%CSTP

3

Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM. This should lead to increased scheduling opportunities.

%SYS

20

The percentage of time spent by system services on behalf of the world. Most likely caused by high IO VM. Check other metrics and VM for possible root cause

%MLMTD

0

The percentage of time the vCPU was ready to run but deliberately wasn’t scheduled because that would violate the “CPU limit” settings. If larger than 0 the world is being throttled due to the limit on CPU.

If less than 80 VM experiences poor NUMA locality. If a VM has a memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM and “remotely” uses memory via “interconnect”. Check “GST_ND(X)” to find out which NUMA nodes are used.

Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason.

DISK

RESETS/s

1

The number of commands reset per second.

DISK

CONS/s

20

SCSI Reservation Conflicts per second. If many SCSI Reservation Conflicts occur performance could be degraded due to the lock on the VMFS.

Solucion error conexion SSL para RESXTOP

Automating CA Self-Signed Certificates for ESXi 5.1 for use with resxtop

Last week I wrote an article about resxtop failing to connect to an ESXi 5.1 host due to SSL Certificate validation which has been implemented in resxtop and I provided a few workarounds that you can use until a fix is released for resxtop. As promised at the end of that article, I will show you how you can automate the creation proper certificates for environments using CA self-signed SSL Certificates so you can continue using resxtop with ESXi 5.1 until a fix is released.

Note: VMware definitely recommends as a best practice to replace the self-signed SSL Certificates that are automatically generated during an ESXi installation with proper CA signed SSL Certificates for additional security. For more details on SSL Certificate replacement, please refer to the following VMware KB.

Disclaimer: These script are provided for informational/educational purposes only. It should be thoroughly tested before attempting to use in a production environment.

Here is a high level overview before using the script:

The script will generates a one time CA cert & key which will then be used to to sign the SSL Certificates that are generated for each ESXi 5.1 hosts all using the openssl utility.

You will then need to upload the SSL Certificates (rui.crt & rui.key) generated for each of the respective ESXi 5.1 hosts under /etc/vmware/ssl directory

You will then need restart the rhttpproxy service on each of the ESXi 5.1 hosts for the SSL Certificates to take effect

OPTIONAL: If you are using vCenter Server to manage your ESXi hosts, there is an additional required step of re-connecting the ESXi hosts as they will automatically be disconnected from your vCenter Server. This is due to new SSL Certificates on the hosts and the thumbprint that vCenter Server knows about is no longer valid

Set the two required environmental variables HTTPS_CA_DIR and HTTPS_CA_FILE to the CA Cert that was generated in step 1 which will allow us to connect to our ESXi hosts that have had their SSL Certificates replaced

Note: Steps 2-4 are currently manual as it depends on when you want the SSL Certificates to go live on the ESXi hosts which will also require a service restart for the changes to go into effect.

You can download the script called generateSSLSelfSignCert.sh which is a shell script that can be executed on any UNIX/Linux system. In the example below, I running the script using the vMA appliance.

The script accepts a single command-line argument which is a file that contain the hostnames (FQDN) of all the ESXi 5.1 hosts you wish to replace SSL Certificates for. In the example below, I created a file called hostlist which contains the following:

Echo pod23-esx-01a.pml.local > hostlist

Echo pod23-esx-02a.pml.local >> hostlist

pod23-esx-01a.pml.localpod23-esx-02a.pml.local

Make sure you set the execute permission on the script before running. Here is a screenshot of the script executing based on the input file that I have provided:

The first section which is highlighted in RED, is creating a temporarily directory called ssl-certs in the current working directory and generates a one time CA Cert & Key which will then be used to self-sign the SSL Certificates for each ESXi 5.1 hosts. Please note that you will need the cacert.pem file which is generated from this step to use resxtop to connect to an ESXi 5.1 host. If you delete or lose this file, you will need to re-generate all new SSL Certificates.

Note: You also have the ability to run the script for additional hosts afterwards and if the cacerts.pem already exists, then it will not re-regenerate a new one and just use the existing one to sign new SSL Certificates.

The next section that is highlighted in BLUE is creation of the SSL Certificates (rui.crt & rui.key) for each of the ESXi hosts which is stored in their own directory based on the hostname.

Once all the SSL Certificates have been generated, you will now need to copy the rui.crt and rui.key files to each of the respective ESXi hosts under the /etc/vmware/ssl directory as noted in the GREEN highlighted section and then restart therhttpproxy service for the SSL Certificate to go live on the ESXi host.

You can either use SCP which relies on SSH being enabled on the ESXi host or you can leverage the vifs utility which is part of the vCLI to upload your SSL Certificates. Take a look at this article for more details. In this example, I am using SCP and here is the command that I am using:

Once you have uploaded the SSL Certificates to each of the ESXi hosts, we then need to restart the rhttpproxy service. To do so, you will need to login into the ESXi Shell and running /etc/init.d/rhttpproxy restart or restart the management service via the DCUI. In this example, I will be SSHing into the host and restarting the service using the following command:

Here is a screenshot of restarting the rhttpproxy for both ESXi hosts:

Note: If the ESXi hosts are being managed by vCenter Server, after the restart of the rhttpproxy or management service, the hosts will disconnect from vCenter Server. This is due to a change in the SSL Thumbprint that vCenter Server had recorded about the ESXi host when added to vCenter. You will need to re-connect each host back into vCenter Server.

The final step which is highlighted in ORANGE is to set the two required environmental variables HTTPS_CA_DIR and HTTPS_CA_FILE to specify the location of the CA Cert that was generated earlier before using resxtop. To set the two variables, run the following command:

Here is a screenshot of exporting the two environmental variables and running resxtop to connect to one of the ESXi 5.1 hosts:

Note: The exact path will be automatically displayed for you in the script output, you just need to copy and run the command. Once you have finished, you can then connect to any of your ESXi 5.1 hosts using resxtop and you no longer will get an error connecting to your ESXi 5.1 host. Please note that it is important you keep the cacert.pem safe, do not lost it or accidentally delete it, else you will need to re-generate new SSL Certificates.