Restoring vCenter access after being blocked by a Deny All rule in NSX DFW

The issue

I recently had an issue to solve with an NSX(-v) instance in which the default firewall rule had been set to reject without an exception defined for the vCenter Server.
As a result, the communication between the vCenter Server appliance and NSX Manager was interrupted, which blocked vCenter Server completely from accessing the network.
I decided to document the procedure, which was necessary to get it working again, just in case I would have to solve it again, but I also thought that sharing it would do no harm.

Basically the following procedure is based on VMware KB 2079620:vCenter Server access is blocked after creating a Deny All rule in DFW (2079620), but as I faced additional issues during troubleshooting, I document the whole procedure which I had to do.

Unblock vCenter

The first thing I did was get vCenter Server to communicate again. vCenter and NSX Manager were completely unavailable, which is why the first thing I had to do was to remove the NSX vibs from the underlying host.

First shut down vCenter Server appliance, and other remaining VMs on the ESXi host.

Then put host in maintenance mode, which is required to remove the NSX vibs.

Connect via ssh to the host (e.g. using PuTTY). Note that if ssh is not enabled on the host, it has to be enabled first.

Check for the NSX vibs installed on the host using the command esxcli software vib list | grep nsx

Side note:

Depending on the situation you might be facing, or the point you step in the troubleshooting, the situation can be slightly different. Sometimes somebody else has already begun troubleshooting, and vCenter Server can be disconnected already from the vDS.

In the case that the portgroup is a non-ephemeral one, no new ports can be allocated as the vCenter Server is down. In that particular case a new standard vSwitch has to be created, vCenter needs to be connected to the VSS for the boot process, until it can be reconnected to the vDS again.

If you encounter the same issue, simply follow the procedure from the above linked article.

—————–

Remove the blocking firewall rule in NSX Manager via an API call

Now the vibs are removed and vCenter Server can be accessed again. However, as in this case there is no exclusion defined in the distributed firewall of the vCenter Server, the default firewall rule has to be reset to default.
In order to do so, you will have to work with REST API. If you have already a favourite REST API client, you are free to use what you want. If not, you can download some for free, as for instance Postman, which also exists as extension for Chrome. (The standalone version of Postman can be downloaded here: https://www.getpostman.com).

Note: If you want to know more about Postman, feel free to check Romain’s blogpost here.

Form a request using the method GET to test connectivity to the NSX Manager. Using the URL below you will be able to download the actual firewall configuration, which might help in identifying the blocking rule.

Now your blocking firewall rule should be gone. You can test it by connecting to the vCenter Server Appliance.

Go to Menu –> Networking and Security

Note that the connection to NSX Manager has been re-established if the NSX Manager can be found.

Now you can reconfigure the firewall rules as you wish.

Additional Note:

To avoid this situation in future, it is recommended to add the vCenter Server in the exclusion list of the NSX distributed firewall settings.
This can be done by going to Firewall Settings à Exclusion List à User Excluded VM(s).

Click ADD to add the vCenter to the exclusion list.

Select your vCenter VM, then click the à arrow to ad dit to the selected objects. Then hit OK.
Now the vCenter will not be blocked anymore by the distributed firewall rules of the NSX Manager.