Under Networking and Security, Installation Status showing Not Ready on the Clusters and Hosts, clicking on Resolve does not fix the issue.

ESXi host VXLAN VIB installation fails.

ESXi host fails to access VIBs from the vCenter Server. In the VC system events table, you see:

Event Message:'Installation of deployment unit failed, please check if ovf/vib urls are accessible, in correct format and all the properties in ovf environment have been configured in service attributes. Please check logs for details.', Module:'Security Fabric'

In the /var/log/vmkernel.log file of the ESXi hosts, you see entries similar to:

Purpose

Overview

vSphere ESX Agent Manager (EAM) is an application that sits between NSX and vCenter. NSX Manager requests EAM prepares a given NSX Cluster by creating an Agency with Scope that covers that Cluster. EAM automates the process of deploying and managing vSphere ESX agents.

The services that ESX Agent Manager provides include out-of-the-box integration of agents with vSphere features such as DRS, AddHost, High Availability, DRM, and maintenance mode. All of these features can be difficult to integrate with manually. ESX Agent Manager also allows users to monitor the health of ESX agents, and blocks users from performing certain operations on ESX agents that might affect the virtual machines that use them. For example, ESX Agent Manager can prevent an ESX agent virtual machine from being powered off or moved from an ESX host that contains other virtual machines that use that agent.

Agency: In EAM terms, defines what enhancements should be done on all hosts/clusters in the scope.

Enhancements: In EAM terms, comprise installing an agent VM and/or VIB on each ESXi host in the agency scope.

Agent: In EAM terms, is an enhancement on a given ESXi host defined by an Agency. There might be more than one Agency that cover the same host. In this case each one of them correspond to a different Agent on that particular host.

To summarize, an Agency is comprised of agents, one per ESXi host from the scope.

Host Preparation

There are 3 components for NSX ESXi host preparation:

vCenter Server

NSX Manager

EAM (ESX Agent Manager)

The vCenter Server manages the Compute Infrastructure and is tightly connected to the NSX Manager. There is a 1 to 1 relationship between the NSX Manager and the vCenter Server.

The NSX Manager is responsible for the ESXi host preparation. It installs on the ESXi hypervisor the various vSphere Installation Bundles (VIBs) to enable VXLAN, Distributed Routing, Distributed Firewall(DFW) and a user world agent(UWA) used to communicate at the control plane level.

VIBs are packages of files that get installed in the Kernel space of the ESXi hypervisor.

Software components that provide Control Plane communication from the ESXi hypervisor to the NSX Manager and the Controller Cluster nodes are:

RabbitMQ Message bus provides communication between the RMQ client (vsfwd process on the ESXi hypervisor) and the RMQ Server process hosted on the NSX Manager.

User World Agent (UWA) process (netcpa on the ESXi hypervisor) establishes TCP over SSL communication channels to the Controller Cluster nodes. Controllers use this channel to populate the local MAC,ARP and VTEP tables to determine where workloads are connected in the deployed logical networks.

Host Preparation Workflow

Connect NSX Manager to vCenter Sever/VCVA.

Deploy NSX Controllers (If using Unicast,Hybrid mode or LDR).

Host Prep.

NSX Manager > EAM > vCenter Server > ESXi host

Agency is created and EAM calls vCenter Server to install VIBs

EAM initiates VIBs scanning on each agency create/update/delete and on ESXi host restart (for the sake of installing vibs on stateless hosts). The actual scanning is done by vSphere Update Manager. EAM calls VUM providing NSX VIBs. VUM determines the delta between the provided VIB metadata and the actual VIBs installed on the ESXi host. The result is used by EAM to request VIB install from VUM.

EAM Agency/Agents Health Status

An agency and its agents each maintains a status field which can be either Red, Yellow, or Green.

RED State: The RED health status is used to indicate that the solution must somehow intervene for EAM to proceed.

YELLOW State: The YELLOW health status indicates that EAM is actively working on reaching a given goal state. The goal state can be one of Enabled, Disabled, or Undeployed . E.g. when a solution is first registered its status is YELLOW until EAM has deployed the solution's agents to all the specified compute resources. A solution does not need to intervene when EAM reports its EAM health status as YELLOW.

GREEN State: The GREEN health status is used to indicate that a solution and all its agents have reached the desired goal state.

Resolution

IMPORTANT: Please note that this knowledge base article is no
longer being updated. For the most up-to-date information, see the
latest version of the NSX Troubleshooting Guide.

Validate that each troubleshooting step is true for your environment. Each step provides instructions or a link to an article to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Please do not skip any step.

Ensure that the EAM is able to access the VIBs from NSX Manager. Review the eam.log file and look for VIB inaccessible messages. Also check the vCenter Server registration in NSX Manager. For more information, see Configuring the SSO Lookup Service fails (2102041).

Ensure that the ESXi hosts are able to access VIBs from the vCenter Server. You see this error: esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known in the esxupdate.log file. This can occur if the ESXi host is unable to access the vCenter Server's Fully Qualified Domain Name (FQDN). For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).

Verify that the cluster contains the correct agencies. If an EAM agency is missing on an already prepared cluster, new hosts can fail to prepare. To work around this issue, create a dummy cluster and prepare it. This forces NSX/EAM to update the configuration of all existing clusters, creating a new EAM Agency for the problematic cluster.

Note: Starting with VMware NSX for vSphere 6.1.3, an alarm is raised when no agency is detected.

Ensure that the ESXi hosts are rebooted after an upgrade. Failure to manually reboot the ESXi host marks them as red in the Installation Status.Note: In a DRS enabled cluster, an ESXi host reboot can be initiated by clicking the Resolve option on the cluster.

In VMware vSphere 6.0 Update 1, host scan operation by EAM fails but NSX host preparation status shows as green incorrectly as the returned unhandled error 99. In releases with a fix for this issue, EAM correctly reports that the host scan failed and raises a vibNotInstalled state. The following example message from the /var/log/eam.log file on the affected ESXi host shows the error code 99.

If you see the error esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Errno -2] Name= or service not known) in the esxupdate.log file, this may happen if the ESXi host cannot reach the vCenter Server Fully Qualified Domain Name (FQDN). To resolve this issue, set the correct the vCenter Server Managed IP address. For more information, see Verifying the VMware vCenter Server Managed IP Address (1008030).

Sometimes after restarting NSX Manager, EAM or vCenter Server, you notice the cluster status reported Not Ready in Host Preparation tab, in Installation section of NSX Manager. This is a false-positive status that is a result of restarting one of the components. To get the state updated, click the Resolve All button.

Ensure that the EAM service is aware when the vCenter Server certificate has been replaced. When the EAM service is not aware of the new vCenter Server certificate , it is not able to properly log in and displays the error similar to:

VMware vSphere 6.0 supports VIB downloads over port 443 (instead of port 80). This port is opened and closed dynamically. The intermediate devices between the ESXi hosts and vCenter Server must allow traffic using this port.

Additional Information

vSphere Installation Bundles (VIBs)

When you prepare an ESXi host for VMware NSX for vSphere, vSphere Installation Bundles (VIBs) are automatically pushed by the NSX Manager through VMware vSphere ESX Agency Manager (EAM). On the ESXi hosts, you see these VIBs installed: