and other brilliant error messages

EqualLogic, iSCSI and the Windows Server 2008 R2 firewall

I recently migrated a backup server from Windows Server 2003 to Windows 2008 R2 in order to install Backup Exec 2012 at the same time. Once I had configured everything I noticed in the iSCSI Control Panel that only one path would ever connect to the array, and I was getting regular iSCSI timeouts and failures in the System Event Log, which I hadn’t seen while running Windows 2003:

The errors were event 129:

The description for Event ID 129 from source iScsiPrt cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

event 39:

Initiator sent a task management command to reset the target. The target name is given in the dump data.

and event 9:

Target did not respond in time for a SCSI request. The CDB is given in the dump data.

Crucially these were spaced (all three together) at intervals of four minutes.

Solution

I spoke to the EqualLogic support team and, after a little while spent focusing on NIC drivers, one of the senior technicians fortunately realised that this four minute time interval coincides with the approximate frequency with which the array pings the initiators on the host and may send reconnect requests for additional path setup and load balancing. He recommended that I disable the Windows Firewall and sure enough the problem vanished. So it’s quite easy to inadvertently break iSCSI storage MPIO by making firewall settings changes to your system later on, and it’s easy to forget that these two things are related.

The problem for me was that this backup server has a NIC on the DMZ for faster backups (bypassing the hardware firewall). The pre- and post-backup job scripts enable and disable this NIC as required, but it does nonetheless need to be firewalled restrictively. In Windows 2003 the Windows Firewall can be enabled on a per NIC basis, however not in Windows 2008. Instead the firewall is configured instead in Network and Sharing Center on a per security zone basis (Domain Networks, Private Networks, Public Networks). The problem here is that the iSCSI NICs automatically end up in the Public zone, which is the most likely to be restricted. In my case, I had selected the option Block all connections including programs on the list of allowed programs. Even though the EqualLogic HIT Kit had specified an exemption rule, this was being denied.

Excluding iSCSI adapters

Relaxing the firewall in my scenario was not desirable, so I spent a while searching for a way to force the iSCSI NICs into the Private Networks zone. I couldn’t find one, though I did spot a method to exclude the NICs from the Network And Sharing Center altogether. In fact this same issue had been bothering people running VMware Workstation (because the VMware virtual NICs would get firewalled as Public Network connections), and fortunately someone had found a fix:http://www.petri.co.il/exclude-vmware-virtual-adapters-vista-2008-network-awareness-windows-firewall.htm

The solution posted there uses a PowerShell script which automatically targets VMware adapters, but we can use the same registry modification. So, on your server use Regedit to navigate to HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}. There is a child branch here for each NIC. Find your dedicated iSCSI NICs and for each one, create a new DWORD value called *NdisDeviceType (including the asterisk) and give it a value of 1. Now disable and re-enable each modified NIC. You will see that they disappear from Network and Sharing Center, and are now unaffected by the Windows Firewall.

By setting *NdisDeviceType to a value of 1 the NIC is designated as an endpoint device and is not considered to be connecting to an external network, which is probably quite appropriate for a dedicated iSCSI storage connection. I wonder whether this is the sort of thing that ought to be automated by the HIT kit in future in fact.

Preference Order

Another thing that’s easily overlooked on servers with iSCSI storage (because it’s so well hidden) is that if you have been changing NIC configs (changing drivers, adding hardware, P2V converting, etc.) then it’s quite likely that you may have affected the preference order in which network services use physical adapters. You don’t generally want the iSCSI ones to become the higher priority ones, and I have experienced strange issues with Exchange Server in the past owing to this, as well as licence issues with copy-protected software that relies on generating a unique hardware-dependent machine ID. To set the order, open Network and Sharing Center, then click on Change Adapter Settings on the left hand side. Now hold Alt, then Advanced -> Advanced Seettings. Now you can configure the LAN NICs with higher priority: