Overview

Virtual SANs (VSANs) provide a method of isolating devices that are physically connected to the same storage network, but are logically considered to be part of different SAN fabrics that do not need to be aware of one another. Each VSAN can contain up to 239 switches and has an independent address space that allows identical Fibre Channel IDs (FC IDs) to be used simultaneously in different VSANs.

VSANs provide the following capabilities:

•Isolate devices physically connected to the same fabric.

•Reduce the size of a Fibre Channel distributed database.

•Enable more scalable and secure fabrics.

Best Practices for VSAN Implementation

This section provides the best practices for implementing VSANs.

•Avoid using VSAN 1 (the default VSAN) for production network traffic. Create at least one VSAN to carry your network traffic.

•Isolate devices in VSANs whenever practical.

•Leave fabric timers and FSPF timers at their default settings, unless changes are required because of interoperability with an existing fabric or long-haul links are being deployed.

•Use Inter-VSAN routing (IVR) only when necessary to selectively connect devices across VSANs. If IVR is used without NAT, ensure that domain IDs are statically configured and unique across all VSANs.

•Place FCIP gateways in their own native VSAN to isolate disturbances when problems in the IP cloud (such as flapping links) occur.

•Use VSAN-based roles to control and limit management access to your switches.

Best Practices for Domain ID Assignment

This section provides best practices for implementing domain ID assignments.

•Use static domains in most environments. To use static domains, choose Fabricxx > All VSANs > Domain Manager and select static from the Config Type drop-down menu in Fabric Manager or use the fcdomain domain n static vsan xCLIcommand. You must then issue a disruptive restart so that the configured domain ID matches the running domain ID. Select the Configuration tab and select disruptive from the Restart drop-down menu in Fabric Manager, and then click Apply Changes. Or you can use the fcdomain restart disruptive CLI command.

Note You cannot issue a disruptive restart for VSANs that are in any of the interop modes. Use a nondisruptive restart as needed.

•Disable the Domain Manager to disable the principal switch selection process. This is possible if all domains are statically assigned. Disabling the principal switch selection can reduce disruption when switches are rebooted or added to the fabric. This must be done on each switch that should not participate in principal switch selection. A disruptive restart of the fabric is required to apply this change. To disable the Domain Manager, choose Fabricxx > All VSANs > Domain Manager and uncheck the Enable check box in Fabric Manager or use the no fcdomain vsan x CLI command.

•Keep domain ID allowed lists the same on all switches in a fabric for consistency. If the principal switch changes, the allowed domain lists will remain the same.

•Assign domain IDs between decimal 97 and 127 if the domain may be used for standards-based interop mode.

•Do not perform frequent changes to the Domain Manager on production fabrics. Experienced administrators familiar with switch operations should be responsible for Domain Manager changes. Plan your domain configuration carefully so that you avoid the need to make disruptive changes at a later time.

•Save Domain Manager changes. When you change the configuration, be sure to save the running configuration by choosing Switches > Copy Configuration in Fabric Manager or by using the copy running-config startup-config CLI command. The next time you reboot the switch, the saved configuration is used. If you do not save the configuration, the previously saved startup configuration is used.

•Enable reconfigure fabric (RCF) rejection on every ISL port if high availability is mandatory. Choose Switches > Interfaces > FC Physical in Fabric Manager, and select the Domain Manager tab in the Information pane and then check the RcfReject check box on all ISL ports to enable rcf-rejects. Or use the interface CLI command on a TE or E port and then use the fcdomain rcf-reject vsan CLI command in interface configuration mode to enable the RCF reject option. RCF reject prevents other switches from sending an RCF and potentially causing a disruption in your production traffic.

Best Practices for FSPF

This section provides best practices for implementing FSPF.

•Use the default FSPF link cost, which can be configured on a per-VSAN basis for the same physical link, to provide preferred and alternate paths. If you must alter the FSPF link cost, use caution to avoid asymmetric Fibre Channel routing.

•Use the default FSPF load-balancing configuration unless you are required to load balance based on your unique fabric; for example, if you have FICON VSANs.

•Use the default FSPF timer configuration. If FSPF timers are misconfigured, then the switches will not reach the "two-way" state and FSPF will not operate properly.

License Requirements

VSANs, domain IDs, and FSPF are bundled with Cisco SAN-OS and require no additional licensing.

Initial Troubleshooting Checklist

Most VSAN problems can be avoided by following the best practices for VSAN implementation. However, if needed, you can use the Fabric Analysis tool in Fabric Manager to verify different categories of problems such as VSANs, zoning, FCdomain, admin issues, or switch-specific or fabric-specific issues.

Note When suspending or deleting VSANs, make sure that you suspend and unsuspend one VSAN at a time, and that you wait a minimum of 60 seconds after you issue the vsan suspend command before you issue any other config command. Failure to do so may result in some Fibre Channel interfaces or member ports in a PortChannel becoming suspended or error-disabled.

Troubleshooting a SAN problem involves gathering information about the configuration and connectivity of individual devices as well as the status of the entire SAN fabric. In the case of VSANs, begin your troubleshooting activity as follows:

Checklist

Check off

Verify the FSPF parameters for switches in the VSAN.

Verify the domain parameters for switches in the VSAN.

Verify the physical connectivity for any problem ports or VSANs.

Verify that you have both devices in the name server.

Verify that you have both end devices in the same VSAN.

Verify that you have both end devices in the same zone.

Verify that the zone is part of the active zone set.

Common Troubleshooting Tools in Fabric Manager

Use the following Fabric Manager procedures to verify the VSAN, FC domain, FSPF, and zone s:

•Choose Fabricxx > VSANxx to view the VSAN configuration in the Information pane.

•Choose Fabricxx > VSANxx and select the Host or Storage tab in the Information pane to view the VSAN members.

Verifying VSAN Membership Using Fabric Manager

To verify VSAN membership for host and storage devices using Fabric Manager, follow these steps:

Step 1 Choose Fabricxx > VSANxx and select the Host or Storage tab in the Information pane. Verify that both devices are in the same VSAN.

Step 2 If the host and storage are in different VSANs, verify which port is not in the correct VSAN and then follow these steps to change the port VSAN:

a. Highlight the host or storage in the Information pane. You see the link to that end device highlighted in blue in the map pane.

b. Right-click on the highlighted link and select Interface Attributes from the pop-up menu.

c. Set the PortVSAN field to the VSAN that holds the other end device and click Apply Changes.

Step 3 Right-click any ISL between the switches and select Interface Attributes. Select the Trunk Config tab and verify that the allowed VSAN list includes the VSAN found in Step 1.

Step 4 If the trunk is not configured for the VSAN, set the Allowed VSANs field to include the VSAN that the host and storage devices are on and click Apply Changes.

Verifying VSAN Membership Using the CLI

To verify VSAN membership for host and storage devices using the CLI, follow these steps:

Step 1 Use the show vsan membership command to see all the ports connected to your host and storage, and verify that both devices are in the same VSAN. Use this command on the switches that connect to your host or storage devices.

switch# show vsan membership

vsan 1 interfaces:

fc2/7 fc2/8 fc2/9 fc2/10 fc2/11 fc2/12 fc2/13 fc2/14

fc2/15 fc2/16 fc7/1 fc7/2 fc7/3 fc7/4 fc7/5 fc7/6

fc7/7 fc7/8 fc7/9 fc7/10 fc7/11 fc7/12 fc7/13 fc7/14

fc7/15 fc7/16 fc7/17 fc7/18 fc7/19 fc7/20 fc7/21 fc7/22

fc7/25 fc7/26 fc7/27 fc7/28 fc7/29 fc7/30 fc7/31 fc7/32

vsan 2 interfaces:

fc2/6 fc7/23 fc7/24

vsan 3 interfaces:

fc2/1 fc2/2 fc2/5

vsan 4 interfaces:

fc2/3 fc2/4

Step 2 If the host and storage are in different VSANs, use the vsan databasevsan vsan-id interface command to move the interface connected to the host and storage devices into the same VSAN.

Step 3 Use the show interface command to verify that the trunks connecting the end switches are configured to transport the VSAN found in Step 1.

switch# show interface fc2/14

fc2/14 is trunking

Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e

Port mode is TE

Speed is 2 Gbps

vsan is 2

Beacon is turned off

Trunk vsans (allowed active) (1-3,5)

Trunk vsans (operational) (1-3,5)

Trunk vsans (up) (2-3,5)

Trunk vsans (isolated) (1)

Trunk vsans (initializing) ()

475 frames input, 8982 bytes, 0 discards

0 runts, 0 jabber, 0 too long, 0 too short

0 input errors, 0 CRC, 3 invalid transmission words

0 address id, 0 delimiter

0 EOF abort, 0 fragmented, 0 unknown class

514 frames output, 7509 bytes, 16777216 discards

Received 30 OLS, 21 LRR, 18 NOS, 53 loop inits

Transmitted 68 OLS, 25 LRR, 28 NOS, 32 loop inits

Step 4 If the trunk is not configured for the VSAN, use the interface command and then the switchport trunk allowed vsan command in interface mode to add the VSAN to the allowed VSAN list for the interface that connects the host and storage devices.

Resolving an Isolated E Port Using the CLI

To resolve VSAN isolation on an E port using the CLI, follow these steps:

Step 1 Use the show interface command to verify that the port is isolated because of a VSAN mismatch.

switch# show interface fc2/4

fc2/4 is down fc2/4 is down (isolation due to port vsan mismatch)

Hardware is Fibre Channel, WWN is 20:44:00:05:30:00:63:5e

vsan is 4

Beacon is turned off

30 frames input, 682 bytes, 0 discards

0 runts, 0 jabber, 0 too long, 0 too short

0 input errors, 0 CRC, 0 invalid transmission words

0 address id, 0 delimiter

0 EOF abort, 0 fragmented, 0 unknown class

30 frames output, 583 bytes, 0 discards

Received 2 OLS, 2 LRR, 2 NOS, 5 loop inits

Transmitted 5 OLS, 3 LRR, 2 NOS, 4 loop inits

Step 2 Use the show vsan membership command to verify that the ports are in separate VSANs.

switch# show vsan membership

vsan 3 interfaces:

fc2/1 fc2/2 fc2/3 fc2/4 fc2/6 fc2/7 fc2/8 fc2/9

fc2/10 fc2/11 fc2/12 fc2/14 fc2/15 fc2/16 fc7/1 fc7/2

fc7/3 fc7/4 fc7/5 fc7/6 fc7/7 fc7/8 fc7/9 fc7/10

fc7/11 fc7/12 fc7/13 fc7/14 fc7/15 fc7/16 fc7/17 fc7/18

fc7/19 fc7/20 fc7/21 fc7/22 fc7/23 fc7/24 fc7/25 fc7/26

fc7/27 fc7/28 fc7/29 fc7/30 fc7/31 fc7/32

vsan 4 interfaces:

fc2/5 fc2/13

vsan 4094(isolated_vsan) interfaces:

This sample output shows that all the interfaces on the switch belong to VSAN 3, with the exception of interface fc2/5 and fc2/13, which are part of VSAN 4.

Step 3 Use the vsan database vsan vsan-id interface command to move the ports into the same VSAN.

Resolving an Isolated ISL Using Fabric Manager

Trunking E ports (TE ports) are similar to E ports except that they carry traffic for multiple VSANs. E ports carry traffic for a single VSAN. Because TE ports carry traffic for multiple VSANs, ISL isolation can affect one or more VSANs. For this reason, on a TE port you must troubleshoot for ISL isolation on each VSAN.

To resolve VSAN isolation on a TE port using Fabric Manager, follow these steps:

Step 1 Choose Switches > Interfaces > FC Physical and check the FailureCause column on the TE port to verify that you have a trunk problem.

Step 3 Correct the problem listed in the FailureCause column. See the "DPVM Config Database Not Activating" section for domain misconfiguration problems. Choose Switches > Interfaces > FC Physical and use the PortVSAN field to correct the VSAN misconfiguration problems.

Step 4 Repeat this procedure for all isolated VSANs on this TE port.

Resolving an Isolated ISL Using the CLI

Trunking E ports (TE ports) are similar to E ports except that they carry traffic for multiple VSANs. E ports carry traffic for a single VSAN. Because TE ports carry traffic for multiple VSANs, ISL isolation can affect one or more VSANs. For this reason, on a TE port you must troubleshoot for ISL isolation on each VSAN.

To resolve VSAN isolation on a TE port using the CLI, follow these steps:

Step 1 Use the show interface command on the TE port to verify that you have an isolated VSAN.

switch# show interface fc2/14

fc2/14 is trunking

Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e

Port mode is TE

Speed is 2 Gbps

vsan is 2

Beacon is turned off

Trunk vsans (allowed active) (1-3,5)

Trunk vsans (operational) (1-3,5)

Trunk vsans (up) (2-3,5)

Trunk vsans (isolated) (1)

Trunk vsans (initializing) ()

475 frames input, 8982 bytes, 0 discards

0 runts, 0 jabber, 0 too long, 0 too short

0 input errors, 0 CRC, 3 invalid transmission words

0 address id, 0 delimiter

0 EOF abort, 0 fragmented, 0 unknown class

514 frames output, 7509 bytes, 16777216 discards

Received 30 OLS, 21 LRR, 18 NOS, 53 loop inits

The example shows the output of the show interface command with one or more isolated VSANs. Here, the TE port has one VSAN isolated.

Troubleshooting Interop Mode Issues

Dynamic Port VSAN Membership Issues

You can dynamically assign VSAN membership to ports is achieved by assigning VSANs based on the device WWN. Dynamic port VSAN membership (DPVM) offers flexibility and eliminates the need to reconfigure the VSAN to maintain fabric topology when a host or storage device connection is moved between two switches or between ports on the same switch. It retains the configured VSAN regardless of where a device is connected or moved.

Verify the following requirements when using DPVM:

•The interface through which the dynamic device connects to the Cisco MDS switch must be configured as an F port. FL ports do not support DPVM and no entries will be learned through an FL port.

•The static port VSAN of the F port should be valid (not isolated, not suspended, and in existence).

•The dynamic VSAN configured for the device in the DPVM database should be valid (not isolated, not suspended, and in existence).

Note The DPVM feature overrides any existing static port VSAN membership configuration. If the VSAN corresponding to the dynamic port is deleted or suspended, the port is shut down.

Note If you copy the DPVM database and fabric distribution is enabled, you must commit the changes.

To begin configuring DPVM, you must explicitly enable DPVM on the required switches in the fabric. By default, this feature is disabled in all switches in the Cisco MDS 9000 Family.

For more information on enabling DPVM, refer to one of the following guides:

Troubleshooting DPVM Using the CLI

Step 1 Use the show dpvm command in EXEC mode to verify that CFS distribution is enabled for DPVM.Optionally, use the dpvm distribute command in config mode to enable CFS distribution if required.

Step 2 Use the show dpvm status command in EXEC mode to verify that autolearning is disabled. Optionally, use the no dpvm auto-learn command in config mode if you need to disable autolearning before activating the database.

Step 3 Use the show dpvm pending-diff command in EXEC mode to compare the active and pending databases.Optionally use the dpvm commit command in config mode to commit any pending entries to the config database.

Step 4 Use the dpvm activate command in config mode to activate the database.

DPVM Configuration Not Available

Symptom DPVM configuration is not available on Fabric Manager or CLI.

Table 11-3 DPVM Configuration Not Available

Symptom

Possible Cause

Solution

DPVM configuration is not available on Fabric Manager or CLI.

DPVM has not been enabled.

DPVM must be enabled before it can be configured. Choose Fabricxx > All VSANs > DPVM and check the Status field in Fabric Manager or use the show dpvmstatus CLI command to verify that DPVM is not enabled. Set the Status field to enable in Fabric Manager and then click Apply Changes or use the dpvm enable CLI command to enable DPVM.

DPVM Database Not Distributed

Symptom DPVM databases are not distributed.

Table 11-4 DPVM Database Not Distributed

Symptom

Possible Cause

Solution

DPVM databases are not distributed.

DPVM distribution has not been enabled on the local switch.

Choose Fabricxx > All VSANs > DPVM and select the CFS tab. Check the Global field in Fabric Manager or use the show dpvm status CLI command to verify that DPVM distribution is not enabled. Set the Global field to enable in Fabric Manager and then click Apply Changes or use the dpvm distribute CLI command to enable DPVM.

DPVM distribution has not been enabled on one or more remote switches.

DPVM Autolearn Not Working

The DPVM autolearn feature allows you to automatically populate the DPVM configuration database with all devices currently in the fabric. This feature is best used when you first turn on DPVM in a stable fabric. Once the devices are learned, you disable autolearning to populate the configuration database with these autolearned entries.

When you add a new device, it is a best practice to manually add that device to the DPVM configuration database. If you turn on autolearning for a new device, you may add other devices that you did not intend to add.

Symptom DPVM autolearn does not work or is not getting enabled.

Table 11-5 DPVM Autolearn Not Working

Symptom

Possible Cause

Solution

DPVM autolearn does not work or is not getting enabled.

DPVM active database may be absent.

Choose Fabricxx > All VSANs > DPVM and select the ActiveDatabase tab in Fabric Manager or use the show dpvmdatabase CLI command to verify that DPVM is not enabled. Select the Actions tab and set the Action field to activate in Fabric Manager and then click Apply Changes or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database.

Note When DPVM distribution is enabled, you must do an explicit commit for DPVM activate and autolearn to take effect.

No Autolearn Entries in Active Database

Symptom There are no autolearn entries in the active database.

Table 11-6 No Autolearn Entries in Active Database

Symptom

Possible Cause

Solution

There are no autolearn entries in the active database.

Autolearn is not enabled.

Choose Fabricxx > All VSANs > DPVM and select the Actions tab in Fabric Manager or use the show dpvm status CLI command to determine if autolearn is enabled. Check the Auto Learn Enable check box in Fabric Manager and click Apply Changes or use the dpvm auto-learn enable and dpvm commit CLI commands to enable autolearning.

Port type is not supported.

Verify that the device you want to autolearn is connected to an F port. DPVM does not support FL, TE, FCIP, or PortChannels.

VSAN Membership not Added to Database

Symptom The VSAN membership of the port is not added to the database.

Table 11-7 VSAN Membership Not Added to Database

Symptom

Possible Cause

Solution

The VSAN membership of the port is not added to the database.

Entry may be present in the config database.

Choose Fabricxx > All VSANs > DPVM and select the Config Database tab in Fabric Manager or use the show dpvm database CLI command to determine if the entry is present in the config database.

DPVM distribution is enabled but a database change was not committed.

Choose Fabricxx > All VSANs > DPVM and select the CFS tab in Fabric Manager. Set the Config Action drop-down menu to commit.

Or use the show dpvm pending CLI command to determine if there are uncommitted changes. Use the dpvm database and dpvm commit CLI commands to commit any pending changes.

DPVM Config Database Not Activating

Symptom DPVM config database is not getting activated.

Table 11-8 DPVM Config Database Not Activating

Symptom

Possible Cause

Solution

DPVM config database is not getting activated.

Conflicting entries may be present between the DPVM config and active databases.

Determine if there are conflicting entries between the active and config databases. Use the dpvm database diff active conf CLI command.

Override the active database with the config database. Choose Fabricxx > All VSANs > DPVM and select the Actions tab in Fabric Manager. Set the Actions drop-down menu to forceActivate and click Apply Changes.

Or use the dpvm activate force and dpvm commit CLI commands to

Cannot Copy Active to Config DPVM Database

Symptom Cannot copy the active DPVM database to the config database.

Table 11-9 Cannot Copy Active to Config DPVM Database

Symptom

Possible Cause

Solution

Cannot copy the active DPVM database to the config database.

Active database may be absent.

Choose Fabricxx > All VSANs > DPVM and select the ActiveDatabase tab in Fabric Manager or use the show dpvmdatabase CLI command to verify that DPVM is not enabled. Select the Actions tab and set the Action field to activate in Fabric Manager and then click Apply Changes.

Or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database. Then copy the active database again.

Port Suspended or Disabled After DPVM Activation

Symptom A port in a static VSAN that was operational goes into suspended or disabled state after DPVM
database activation.

Table 11-10 Port Suspended or Disabled After DPVM Activation

Symptom

Possible Cause

Solution

A port in a static VSAN that was operational goes into suspended or disabled state after DPVM database activation.

DPVM database maps a connected device to a nonexistent VSAN.

Choose Switches > Interfaces > FC Physical in Fabric Manager or use the show interface CLI command to check the interface status for a dynamic VSAN-related failure. Create the VSAN or map the device to another VSAN.

DPVM Merge Failed

Symptom DPVM merge failed.

Table 11-11 DPVM Merge Failed

Symptom

Possible Cause

Solution

DPVM merge failed.

DPVM operational parameters in the two merging fabrics are different.

Choose Fabricxx > All VSANs > DPVM in Fabric Manager e or use the show dpvmCLI command to verify the DPVM configuration in both fabrics. Manually reconcile any differences before attempting to merge the fabrics. Use the show cfs merge status name dpvm CLI command to verify the merge status.

DPVM Process Terminates

DPVM has the following vulnerabilities that could result in a process termination and possible switch reload.

Symptom DPVM process terminates. A switch reload may also occur.

Table 11-12 DPVM Service Failure

Symptom

Possible Cause

Solution

DPVM process may terminate, causing a possible switch reload.

If more than 64 devices login to a single switch, and a logged in devicelogs out and immediately logs in, the DPVM process may terminate. This can occur with SAN-OS 3.0(1) or earlier.

Upgrade to SAN-OS 3.2(1) or later.

Or, deactivate the DPVM database, and then disable DPVM. See Disabling DPVM.

If multiple devices with the same NWWN login to a switch, and then one device logs out and immediately logs in, the DPVM process may terminate. This can occur with SAN-OS 3.0(2) or earlier.

If devices with NPIV / NPV enabled are connected to a switch, and one host behind this device logs out and the other performs a login, the DPVM process may terminate. This can occur with SAN-OS 3.1 or earlier.

Disabling DPVM

Before you disable DPVM on the switch, you must deactivate the DPVM database. You can deactivate the DPVM database on a switch without service disruption.

When you deactivate the database, the dynamic VSAN on a port is converted to a static VSAN on that port, ensuring that ports continue to be in the same VSANs and thus maintaining service continuity.

To disable DPVM, follow these steps:

Note No devices should login or logout during this entire process.

Step 1 Use the no dpvm activate command in config mode to deactivate the DPVM database.

Step 2 Use the dpvm commit command in config mode to commit the changes to the config database.

Step 3 Use the no dpvm enable command in config mode to disable DPVM on the switch.

Domain ID Conflict Troubleshooting

In a Fibre Channel network, the principal switch assigns domain IDs when a new switch is added to an existing fabric. However, when two fabrics merge, the principal switch selection process determines which one of the preexisting switches becomes the principal switch for the merged fabric.

The election of the new principal switch is characterized by the following rules:

•A switch with a populated domain ID list takes priority over a switch that has an empty domain ID list. The principal switch becomes the one in the fabric with the populated domain ID list.

•If both fabrics have a domain ID list, the priority between the two principal switches is determined by the configured switch priority. This is a user-settable parameter. The lower the value is, the higher the priority.

•If the principal switch cannot be determined by the two previous criteria, the principal switch is then determined by the WWNs of the two switches. The lower the value of the WWN the higher the switch priority.

When merging two fabrics, the administrator can expect the following behavior:

•In Cisco SAN-OS Release 2.1(1a) and later releases, when connecting a single-switch fabric to a multi-switch fabric, a build fabric (BF) occurs and the switch with the better priority becomes the principal switch. In earlier releases, when connecting a single-switch fabric to a multi-switch fabric, the multi-switch fabric always retains its principal switch status regardless of the principal switch priority setting on the single switch fabric.

•In Cisco SAN-OS Release 2.1(1a) and later releases, when powering up a new switch in a multi-switch fabric, a BF occurs and the switch with the better priority becomes the principal switch. In earlier releases, when powering up a new switch in a multi-switch fabric, the multi-switch fabric always retains its principal switch status regardless of the principal switch priority setting on the single switch fabric.

•When powering up a new switch that is connected to a standalone switch, the new principal switch is determined by the administratively assigned priority if both switches are running Cisco SAN-OS Release 2.0(x) or earlier. If no priority is assigned (where the default priority is used in every switch), the principal switch is determined by the WWN. This also applies to connecting to two single-switch fabrics.

•When connecting a multi-switch fabric to another multi-switch fabric, the principal switch is determined by the administratively assigned priority. If no priority is assigned (where the default value is used by every switch), the principal switch is determined by the WWN of the existing principal switches of the two fabrics.

Two switch fabrics might not merge. If two fabrics with two or more switches are connected, and they have at least one assigned domain ID in common, and the auto-reconfigure option is disabled (this option is disabled by default), then the E ports that are used to connect the two fabrics will be isolated due to domain ID overlap.

Switch Cannot See Other Switches in a VSAN

Symptom Switch cannot see other switches in a VSAN.

Table 11-13 Switch Cannot See Other Switches in a VSAN

Symptom

Possible Cause

Solution

Switch cannot see other switches in a VSAN.

Switch is isolated because of a domain ID overlap.

Either change the overlapping static domain ID by manually configuring a new static domain ID for the isolated switch, or disable the static domain assignment and allow the switch to request a new domain ID after a fabric reconfiguration.

FC Domain ID Overlap

To resolve an FC domain ID overlap, you can either change the overlapping static domain ID by manually configuring a new static domain ID for the isolated switch, or disable the static domain assignment and allow the switch to request a new domain ID after a fabric reconfiguration.

You may see the following system message in the message log when a domain ID overlap occurs:

Error Message PORT-5-IF_DOWN_DOMAIN_OVERLAP_ISOLATION: Interface [chars] is down
(Isolation due to domain overlap).

Explanation The interface is isolated because of a domain overlap.

Recommended Action Use the showfcdomain domain-list to determine which domain IDs are
overlapping. Use the fcdomaindomaindomain-id [static | preferred]vsanvsan-id CLI command
or similar Fabric Manager procedure to change the domain ID for one of the overlapping domain
IDs.

Assigning a New Domain ID Using Fabric Manager

All devices attached to the switch in the VSAN get a new FC ID when a new domain ID is assigned. Some hosts or storage devices may not function as expected if the FC ID of the host or storage device changes.

To verify FC domain ID overlap and reassign a new domain ID using Fabric Manager, follow these steps:

Step 2 Choose Fabricxx > VSANxx > Domain Manager to view which domains are currently in the VSAN.

Step 3 Repeat Step 2 on the other switch to determine which domain IDs overlap.

Step 4 Select the Configuration tab and setConfig Domain and Config Type to change the domain ID for one of the overlapping domain IDs.

•The static option tells the switch to request that particular domain ID. If it does not get that particular address, it will isolate itself from the fabric.

•The preferred option has the switch request a specified domain ID. If that ID is unavailable, it will accept another ID.

Step 5 Set the Restart drop-down menu to disruptive and click Apply Changes to restart the Domain Manager.

Note While the static option can be applied to runtime after a disruptive or nondisruptive restart, the preferred option is applied to runtime only after a disruptive restart.

Assigning a New Domain ID Using the CLI

All devices attached to the switch in the VSAN get a new FC ID when a new domain ID is assigned. Some hosts or storage devices may not function as expected if the FC ID of the host or storage device changes.

To verify FC domain ID overlap and reassign a new domain ID using the CLI, follow these steps:

Step 1 Issue the show interface command. The following example output shows the isolation error message.

switch# show interface fc2/14

fc2/14 is down (Isolation due to domain overlap)

Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e

vsan is 2

Beacon is turned off

192 frames input, 3986 bytes, 0 discards

0 runts, 0 jabber, 0 too long, 0 too short

0 input errors, 0 CRC, 3 invalid transmission words

0 address id, 0 delimiter

0 EOF abort, 0 fragmented, 0 unknown class

231 frames output, 3709 bytes, 16777216 discards

Received 28 OLS, 19 LRR, 16 NOS, 48 loop inits

Transmitted 62 OLS, 22 LRR, 25 NOS, 30 loop inits

Step 2 Use the show fcdomain domain-list vsan vsan-id command to view which domains are currently in your fabric.

switch1# show fcdomain domain-list vsan 2

Number of domains: 2

Domain ID WWN

--------- -----------------------

0x4a(74) 20:01:00:05:30:00:13:9f [Local]

0x4b(75) 20:01:00:05:30:00:13:9e [Principal]

--------- -----------------------

Step 3 Repeat Step 2 on the other switch to determine which domain IDs overlap.

switch2# show fcdomain domain-list vsan 2

Number of domains: 1

Domain ID WWN

--------- -----------------------

0x4b(75) 20:01:00:05:30:00:13:9e [Local][Principal]

--------- -----------------------

In this example, switch 2 is isolated because of a domain ID 75 overlap.

Step 4 Use the fcdomaindomaindomain-id [static | preferred]vsanvsan-id command to change the domain ID for one of the overlapping domain IDs.

•The static option tells the switch to request that particular domain ID. If it does not get that particular address, it will isolate itself from the fabric.

•The preferred option has the switch request a specified domain ID. If that ID is unavailable, it will accept another ID.

Note While the static option can be applied to runtime after a disruptive or nondisruptive restart, the preferred option is applied to runtime only after a disruptive restart.

Using Fabric Reconfiguration for Domain ID Assignments

You can use a fabric reconfiguration to reassign domain IDs and resolve any overlapping domain IDs. If you enable the auto-reconfigure option on both switches before connecting the fabric, a disruptive reconfiguration (RCF) occurs. The RCF functionality would automatically force a new principal switch selection and cause new domain IDs to be assigned to the different switches.

Caution A disruptive reconfiguration might affect data traffic.

Using Fabric Reconfiguration for Domain ID Assignments with Fabric Manager

To use fabric reconfiguration to reassign domain IDs for a particular VSAN using Fabric Manager, follow these steps:

Or use the show fcdomain domain-list to view the current allowed domain ID list. Compare this to any other switches in the VSAN to determine what domain IDs are missing. Use the fcdomain allowed CLI command to add any missing domain IDs.

Or use the show fcdomain domain-list to view the current allowed domain ID list. Compare this to any other switches in the VSAN to determine what domain IDs are missing. Use the fcdomain allowed CLI command to add any missing domain IDs.

FSPF Issues

The implementation of VSANs dictates that each configured VSAN support a separate set of fabric services. One such service is the FSPF routing protocol, which can be independently configured per VSAN. Therefore, within each VSAN topology, FSPF can be configured to provide a unique routing configuration and resulting traffic flow. Using the traffic engineering capabilities offered by VSANs allows greater control over traffic within the fabric and higher utilization of the deployed fabric resources.

This section describes how to identify and resolve Fabric Shortest Path First (FSFP) problems. It includes the following topics:

•The Age value is a 16-bit counter starting at 0x0000, incremented by one for each switch during flooding and by one for each second held in the database. This field is used as a tie-breaker if incarnation numbers are the same.

•The IncarnationNumber is a 32-bit value between 0x80000001 and 0x7FFFFFFF that is incremented by one each time the originating switch transmits an LSR. This is used before the Age value.

Step 2 Choose FC > Advanced > FSPF and select the LSDB Links tab to verify that each path is in the FSPF database.

Step 3 Choose FC > Advanced > FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and verify that the AdminStatus is up.

•The Cost column shows the cost of the path out of the interface.

•The Intervals column shows the configured FSPF timers for this interface, which must match on both sides.

•The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks. The port is now ready to route frames.

•The Neighbors column shows FSPF neighbor information.

Step 4 Choose FC > Advanced > FSPF and select the Statistics or InterfaceStats tab to verify that there are no excessive errors present.

Troubleshooting FSPF Using the CLI

To troubleshoot FSPF using the CLI, follow these steps:

Step 1 Use the show fspf database vsan command to verify that each path is in the FSPF database.

3. This is a 16-bit counter starting at 0x0000, incremented by one for each switch during flooding and by one for each second held in the database. This field is used as a tie-breaker if incarnation numbers are the same.

4. This is a 32-bit value between 0x80000001 and 0x7FFFFFFF. which is incremented by one each time the originating switch transmits an LSR. This is used before LSR Age.

5. The path to domainID 237, switch 1.

6. The path to domain ID 238, switch 5.

7. Switch 1, domain ID 237 is the owner.

8. The path to domain ID 239, switch 3.

9. The path to domain ID 1, switch 2.

Step 2 Use the show fspf vsan vsan-id interface command to verify that the FSPF parameters are correct for each interface and verify that the interface is in the FSPF active state.

The next hop (238) has two interfaces. This indicates that both paths will be used during load sharing. Up to sixteen paths can be used by FSPF with a Cisco MDS 9000 Family switch.

With the implementation of VSANs used with Cisco MDS 9000 Family switches, a separate instance of FSPF runs within each VSAN, and each instance is independent of the others. For this reason, FSPF issues affecting one VSAN have no effect on FSPF running in other VSANs.

Note For all FSPF configuration statements and diagnostic commands, if the vsan keyword is not specified, VSAN 1 is used by default. When making configuration changes or issuing diagnostic commands in a multi-VSAN environment, be sure to explicitly specify the target VSAN by including the vsan keyword in the statement or command

Loss of Two-Way Communication

If FSPF is misconfigured, then the switches will not reach the "two-way" state.

The following events occur when two-way communication is lost:

•The port enters Init state and removes its neighbor's domain ID from the Recipient Domain ID field and inserts 0xFFFFFFFF.

•FSPF removes the Inter-Switch Link (ISL) from the topology database.

•New link state records (LSRs) are flooded to adjacent switches to notify them that the FSPF database has changed.

Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.

Step 2 Use the undebug all command to turn off debugging.

Step 3 Use the show fspf internal route vsan commandto show FSPF information.

Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.

Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.

Step 2 Use the undebug all command to turn off debugging.

Step 3 Use the show fspf vsan vsan-id interface commandto show FSPF information.

Note To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.

Tip We recommend that you open a second Telnet or SSH session before entering any debug commands. If the debug output overwhelms the current session, you can use the second session to enter the undebug all command to stop the debug message output.

Step 3 Use the undebug all command to turn off debugging.

Step 4 Use the show fspf command to show FSPF configuration and check the autonomous region.

Step 5 Use the fspf config vsan command to enter the FSPF configuration mode and use the region command to change the region.