1.1 Network Troubleshooting Tests

Use loopbacks to test newly created circuits before running live traffic or to logically locate the source of a network failure. All ONS 15600 SDH optical (STM-N) cards allow loopbacks.

Caution On optical cards, a loopback can be applied only if the port state is Locked,maintenance for facility, terminal, and payload loopbacks, and the circuit state is Locked,maintenance for cross-connect loopbacks.

Note Do not use loopbacks to verify circuit switch times or traffic hits because it could exceed 60 msec. For switch times, a test set should be placed at both ends of the circuits.

Note When an entity is put in the administrative state, the ONS 15600 SDH suppresses all standing alarms on that entity. All alarms and events appear on the Conditions tab. You can change this behavior for the LPBKFACILITY, LPBKPAYLD, and LPBKTERMINAL alarms. To display these alarms on the Alarms tab, set the NODE.general.ReportLoopbackConditionsOnPortsInLocked,maintenance to TRUE on the NE Defaults tab.

1.1.1 Facility Loopbacks

The following sections give general information about facility loopback operations and specific information about ONS 15600 SDH card loopback activity.

1.1.1.1 General Behavior

A facility loopback tests the line interface unit (LIU) of an ASAP card or STM-16 card and related cabling. After applying a facility loopback on a port, use a test set to run traffic over the loopback. A successful facility loopback isolates the LIU or the cabling plant as the potential cause of a network problem. To test an STM-N port or Ethernet port, connect an optical test set to the port and perform a facility loopback. Alternately, use a loopback or hairpin circuit on a card that is farther along the circuit path.

Note CTC sometimes calls a facility loopback a facility (line) loopback. This is done to clarify the direction that the loopback signal travels, that is, out from the facility toward the span.

Caution Before performing a facility loopback on an STM-N port, be sure the ASAP card contains at least two data communications channel (DCC) paths to the node where the card is installed. A second DCC provides a nonlooped path to log into the node after the loopback is applied, enabling you to remove the facility loopback. Issuing a second DCC is not necessary if you are directly connected to the ONS 15600 SDH containing the loopbacked ASAP card.

1.1.1.2 Card Behavior

Loopbacks either terminate or bridge the loopback signal. When a port terminates a facility loopback signal, the signal only loops back to the originating port and is not transmitted downstream. When a port bridges a loopback signal, the signal loops back to the originating port and is also transmitted downstream.

The loopback itself is listed in the Conditions window. For example, the window would list the LPBKFACILITY condition for a tested port. (The Alarms window will show AS-MT, which means that alarms are suppressed on the facility during loopback.) In Software Release 8.0, an option in node defaults allows you to specify that loopback conditions be reported as alarms, even though the port or circuit is Locked,maintenance.

In addition to the Conditions window listing, the following behaviors occur:

•If an electrical or optical port is in the Locked-enabled, disabled service state, it injects an alarm indication signal (AIS) upstream and downstream.

•When an electrical or optical port is placed in the Locked-enabled,maintenance service state before loopback testing, the port clears the AIS signal upstream and downstream unless there is a service-affecting defect that would also cause an AIS signal to be injected. For more information about placing ports into alternate states for testing, refer to the "Change Card Settings" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Caution A lockout of protection must be executed before putting a two-fiber or four-fiber MS-SP Ring span into a facility loopback state. That is, a span lockout of one side (such as the east side) of a two-fiber MS-SP Ring is required before operating a facility loopback on the same (east) side of the ring. A span lockout of one protection side (such as the east protection side) of a four-fiber MS-SP Ring is required before operating a facility loopback on the same (east) side working line of the ring. If you do not execute the lockout prior to creating the loopback, the ring can become stuck in an anomalous state after you release the loopback.

1.1.2 Payload Loopbacks

The payload loopback is similar to a facility loopback but occurs on STM-64 cards. Another difference is that a payload loopback terminates and regenerates section and line overhead, while a facility loopback passes section and line overhead through, untouched. The STM-16 card executes a facility loopback by looping the signal back just before the framer chip. The STM-64 card cannot do this because of the differences in the design. To execute a loopback on an STM-64 card, the loopback signal passes through the framer chip and then terminates and regenerates line and section overhead. Since STM-64 card line and section overhead is terminated and regenerated, this type of loopback is called a payload loopback.

1.1.3 Terminal Loopbacks

The following sections give general information about ASAP card terminal loopback operations.

1.1.3.1 General Behavior

A terminal loopback tests a circuit path as it passes through the SSXC card and loops back from the card with the loopback. Figure 1-2 shows a terminal loopback on an ASAP card. The test-set traffic enters the optical or Ethernet port and travels through the cross-connect card to the optical port. A terminal loopback turns the signal around before it reaches the LIU and sends it back through the SSXC card to the card. This test verifies that the SSXC card and terminal circuit paths are valid, but does not test the LIU on the optical card.

Note CTC sometimes calls a terminal loopback a terminal (inward) loopback. This is done to clarify the direction that the loopback signal travels, that is, inward toward the facility.

Figure 1-2 Terminal Loopback Path on an ASAP Card

1.1.3.2 Card Behavior

If a port terminates a terminal loopback signal, the signal only loops back to the originating port and is not transmitted downstream. If the port bridges a loopback signal, the signal loops back to the originating port and is also transmitted downstream.

The loopback is listed in the Conditions window. For example, the window would list the LPBKTERMINAL condition or LPBKFACILITY condition for a tested port. (The Alarms window would show AS-MT, which indicates that all alarms are suppressed on the port during loopback testing.)

In addition to the Conditions window listing, the following behaviors occur:

•If an electrical or optical port is in the Locked-enabled,disabled service state, it injects an AIS signal upstream and downstream.

•When an optical or Ethernet port is placed in the Locked-enabled, maintenance service state before loopback testing, the port clears the AIS signal upstream and downstream unless there is a service-affecting defect that would also cause an AIS signal to be injected. For more information about placing ports into alternate states for testing, refer to the "Change Card Settings" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Caution A lockout of protection must be executed before putting a two-fiber or four-fiber MS-SP Ring span into a terminal loopback state. That is, a span lockout of one side (such as the east side) of a two-fiber MS-SP Ring is required before operating a facility loopback on the same (east) side of the ring. A span lockout of one protection side (such as the east protection side) of a four-fiber MS-SP Ring is required before operating a terminal loopback on the same (east) side working line of the ring. If you do not execute the lockout prior to creating the loopback, the ring can become stuck in an anomalous state after you release the loopback.

1.1.4 Cross-Connect (XC) Loopbacks

An XC loopback tests an SDH virtual container (VC) circuit path as it passes through an SSXC card and loops back to the port being tested without affecting other traffic on the optical port. Cross-connect loopbacks are less invasive than terminal or facility loopbacks. Testing with facility or terminal loopbacks often involve taking down the whole line; however, an XC loopback allows you to create a loopback on any embedded channel at supported payloads of VC3 granularity and higher. For example, you can place a loopback on a single VC3, VC4, VC4-2c, etc. on an optical facility without interrupting the other VC circuits. Figure 1-4 shows the XC loopback path.

Figure 1-4 Cross-Connect Loopback Path on an STM-N Port

This test can be conducted locally or remotely through the CTC interface without on-site personnel. It takes place on an STM-16, STM-64, or ASAP port and tests the traffic path on that VC circuit through the port and SSXC. The signal path is similar to a facility loopback.

The XC loopback breaks down the existing path and creates a new cross-connect—a hairpin—while the source of the original path is set to inject a line-side AIS-P. The signal path and AIS injection are shown in Figure 1-5.

Figure 1-5 Network Element with SDH Cross-Connect Loopback Function

Note If a terminal or facility loopback exists on a port, you cannot create an XC loopback on it.

Note When testing STM-64 signals with jitter analyzers, be sure to verify with the manufacturer that you are using the most current test equipment. Some test equipment has demonstrated false high jitter readings caused by accumulated jitter dependencies within the test equipment.

1.2 Troubleshooting Optical Circuit Paths With Loopbacks

Facility loopbacks or payload loopbacks, terminal loopbacks, and cross-connect (XC) loopback circuits are often used together to test the circuit path through the network or to logically isolate a fault. Performing a loopback test at each point along the circuit path systematically isolates possible points of failure.

The procedures in this section apply to STM-16, STM-64, and ASAP optical ports. (For instructions on ASAP Ethernet ports, go to the "Troubleshooting an Ethernet Circuit Path With Loopbacks" section.) The example in this section tests an STM-N circuit on a three-node MS-SPRing. Using a series of facility, cross-connect, and terminal loopbacks, the example scenario traces the circuit path, tests the possible failure points, and eliminates them. The logical progression contains seven network test procedures:

Note The test sequence for your circuits will differ according to the type of circuit and network topology.

1.2.1 Perform a Facility Loopback or Payload Loopback on a Source-Node Optical Port

The STM-16 card or ASAP card optical port facility loopback test is performed on the node source port in the network circuit. Likewise for the STM-64 payload loopback. In the testing situation used in this example, the source optical port in the source node. Completing a successful facility loopback on this port isolates the optical port as a possible failure point. Figure 1-6 shows an example of a facility loopback on a circuit source STM-N port.

Figure 1-6 Facility Loopback on a Circuit Source STM-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Create the Facility Loopback or Payload Loopback on the Source Optical Port

Step 1 Connect an optical test set to the port you are testing.

Note For specific procedures to use the test set equipment, consult the manufacturer.

Use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to the port you are testing. The Tx and Rx terminals connect to the same port. Adjust the test set accordingly. (Refer to manufacturer instructions for test-set use.)

b. Choose the appropriate state (Unlocked; Locked,disabled; Locked,maintenance; Unlocked,automaticInservice) from the Admin State column for the port being tested. If multiple ports are available, select the appropriate row for the desired port. (The new administrative state will override the loopback.)

Test the Optical Card

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the returned materials authorization (RMA) process. Contact Cisco Technical Support (1 800 553-2447).

b. Choose the appropriate state (Unlocked; Locked,disabled; Locked,maintenance; Unlocked,automaticInservice) from the Admin State column for the port being tested. If multiple ports are available, select the appropriate row for the desired port. (The new administrative state will override the loopback.)

1.2.2 Perform a Terminal Loopback on a Source-Node Optical Port

The terminal loopback test is only available on ASAP card optical and Ethernet ports. (This section will only address the optical ports; Ethernet ports are covered in Troubleshooting an Ethernet Circuit Path With Loopbacks.) Terminal loopbacks are not available on STM-16 or STM-64 cards.

To create a terminal loopback, create a bidirectional circuit originating on the node source optical port and looping back on the node source optical port. You then proceed with the terminal loopback test. Completing a successful terminal loopback to a node source port verifies that the circuit is good to the source port. Figure 1-7 shows an example of a terminal loopback on a source optical port.

Figure 1-7 Terminal Loopback on a Source-Node STM-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

b. If you are starting the current procedure without the optical test set hooked up to the source optical port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

c. Adjust the test set accordingly. (Refer to manufacturer instructions for test-set use.)

Step 2 Use CTC to set up the terminal loopback on the test port:

a. In node view, click the Circuits tab and click Create.

b. In the Circuit Creation dialog box, choose the type, such as VC_HO_PATH_CIRCUIT, and circuit number, such as 1.

c. Click Next.

d. In the next Circuit Creation dialog box, give the circuit an easily identifiable name such as Opt1toOpt2.

e. Leave the Bidirectional check box checked.

f. Click Next.

g. In the Circuit Creation source dialog box, select the same Node, card Slot, Port, and VC where the test set is connected.

h. Click Next.

i. In the Circuit Creation destination dialog box, use the same Node, card Slot, Port, and VC used for the source dialog box.

Test and Clear the Terminal Loopback Circuit

Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.

Step 2 Examine the test traffic being received by the test set. Look for errors or any other signal information that the test set is capable of indicating.

Step 3 If the test set indicates a good circuit, no further testing is necessary on the loopback circuit. Clear the terminal loopback state on the port:

a. Double-click the ASAP in the source node.

a. Click the Maintenance > Optical > Loopback > Port tabs.

b. Choose the appropriate state (Unlocked; Locked,disabled; Locked,maintenance; Unlocked,automaticInservice) from the Admin State column for the port being tested. If multiple ports are available, select the appropriate row for the desired port. (The new administrative state will override the loopback.)

c. Click Apply.

d. Click Yes in the confirmation dialog box.

Step 4 Clear the terminal loopback circuit:

a. Click the Circuits tab.

b. Choose the loopback circuit being tested.

c. Click Delete.

d. Click Yes in the Delete Circuits dialog box. Do not check any check boxes.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Step 5 Resend test traffic on the loopback circuit with a known-good card.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

Step 8 Clear the terminal loopback on the port before testing the next segment of the network circuit path:

a. Double-click the ASAP card in the source node with the terminal loopback.

a. Click the Maintenance > Optical > Loopback > Port tabs.

b. Choose the appropriate state (Unlocked; Locked,disabled; Locked,maintenance; Unlocked,automaticInservice) from the Admin State column for the port being tested. If multiple ports are available, select the appropriate row for the desired port. (The new administrative state will override the loopback.)

c. Click Apply.

d. Click Yes in the confirmation dialog box.

Step 9 Clear the terminal loopback circuit before testing the next segment of the network circuit path:

a. Click the Circuits tab.

b. Choose the loopback circuit being tested.

c. Click Delete.

d. Click Yes in the Delete Circuits dialog box. Do not check any check boxes.

1.2.3 Perform an XC Loopback on the Source Optical Port

Note This procedure is performed from an STM-N card or ASAP card optical port to test the cross-connect circuit connection.

Note You can perform an XC loopback on either the circuit source working or the protect port of a 1+1 protection group.

Note XC loopbacks do not require on-site personnel.

The XC loopback test is available for STM-16, STM-64, and ASAP cards and occurs on an optical circuit transiting the SSXC card in a network circuit. Completing a successful XC loopback from an optical port through the SSXC card eliminates the SSXC card as the source of trouble for a faulty circuit. Figure 1-8 shows an example of an XC loopback path on a source STM-N port.

b. If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. The Tx and Rx terminals connect to the same port.

Test the Alternate SSXC Card

Step 1 Do a manual data copy switch of the SSXC cards before retesting the XC loopback circuit:

a. In node view, select the Maintenance > Preferred Copy tabs.

b. In the Set Preferred drop-down list, select the alternate copy. (For example, if Copy B is preferred and in use, select Copy A.)

Note CTC Copy A refers to the SSXC card in Slot 6. Copy B refers to the SSXC card in Slot 8. Either copy might be chosen as the preferred copy SSXC. The other SSXC is called the alternate SSXC in this chapter.

c. Click Apply.

d. Click Yes in the confirmation dialog box.

Note If you attempt a preferred copy switch and the switch is unsuccessful, a problem is present with the alternate SSXC.

e. Click Refresh until the tab shows that the alternate copy you selected is now the preferred copy. The Currently Used field will show the newly-selected preferred copy.

Step 2 Resend test traffic on the XC loopback circuit.

The test traffic now travels through the alternate cross-connect card.

Step 3 If the test set indicates a faulty circuit, assume the cross-connect card is not causing the problem. Clear the XC loopback circuit:

a. Click the Circuits tab.

b. Choose the XC loopback circuit being tested.

c. Click Delete.

d. Click Yes in the Delete Circuits dialog box. Do not check any check boxes.

e. Confirm that the XC loopback circuit is deleted from the Circuits tab list. If the test set indicates a good circuit, the problem might be a defective cross-connect card.

Retest the Preferred SSXC Card

Step 1 Do a manual data copy switch of the SSXC cards before retesting the loopback circuit:

a. In node view, select the Maintenance > Preferred Copy tabs.

b. In the Set Preferred drop-down menu, select the alternate copy. (For example, if Copy B is preferred and in use, select Copy A.)

c. Click Apply.

d. Click Yes on the confirmation dialog box.

Note If you attempt a preferred copy switch and the switch is unsuccessful, a problem is present with the alternate SSXC.

e. Click Refresh until the tab shows that the alternate copy you selected is now the preferred copy. The Currently Used field will show the newly selected preferred copy.

Step 2 Resend test traffic on the loopback circuit.

Step 3 If the test set indicates a faulty circuit, the problem is probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447) and proceed to Step 4. If the circuit is not shown to be faulty and the card is not shown to be defective, you are finished with testing.

1.2.4 Perform a Facility Loopback or Payload Loopback on an Intermediate-Node Optical Port

Performing an STM-16 or ASAP card optical facility loopback (or STM-64 payload loopback) on an intermediate port isolates whether this node is causing circuit failure. In the situation shown in Figure 1-9, the test is being performed on an intermediate STM-N port.

Figure 1-9 Facility Loopback Path to an Intermediate-Node STM-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Create a Facility Loopback or Payload Loopback on an Intermediate-Node Optical Port

Step 1 Connect an optical test set to the port you are testing. If you are starting the current procedure without the optical test set hooked up to the source port port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

For specific procedures to use the test set equipment, consult the manufacturer.

d. Select locked,maintenance from the Admin State column. If multiple ports are available, select the row appropriate for the desired port.

e. For an STM-16 card or ASAP card optical port, select Facility from the Loopback Type column. For an STM-64 card, select Payload. If multiple ports are available, select the row appropriate for the desired port.

Test the Optical Card

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.2.5 Perform a Terminal Loopback on an Intermediate-Node Optical Port

In the next troubleshooting test, you perform a terminal loopback on the intermediate-node port to isolate whether the destination port is causing circuit trouble. In the example situation in Figure 1-10, the terminal loopback is performed on an intermediate optical port in the circuit. You first create a bidirectional circuit that originates on the source-node optical port and loops back on the intermediate-node port. You then proceed with the terminal loopback test. If you successfully complete a terminal loopback on the node, this node is excluded from possible sources of circuit trouble.

b. If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Test the Optical Card

Step 2 Resend test traffic on the loopback circuit with a known-good card.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support at (1 800 553-2447).

1.2.6 Perform a Facility Loopback or Payload Loopback on a Destination-Node Optical Port

You perform a facility loopback test at the destination port to determine whether this local port is the source of circuit trouble. The example in Figure 1-12 shows a facility loopback being performed on a destination-node STM-N port.

Figure 1-12 Facility Loopback Path to a Destination-Node STM-N Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Create the Facility Loopback or Payload Loopback on a Destination-Node Optical Port

Step 1 Connect an optical test set to the STM-N or ASAP optical port you are testing. If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Note For specific procedures to use the test set equipment, consult the manufacturer.

Test the Optical Card

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 3 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.2.7 Perform a Terminal Loopback on a Destination-Node Optical Port

The terminal loopback at the destination-node ASAP card optical port is the final local hardware error elimination in the circuit troubleshooting process. If this test is completed successfully, you have verified that the circuit is good up to the destination port.

Caution Performing a loopback on an in-service circuit is service-affecting.

Create the Terminal Loopback on a Destination-Node Optical Port

Step 1 Connect an optical test set to the ASAP card optical port you are testing: If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Note For specific procedures to use the test set equipment, consult the manufacturer.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 5 Resend test traffic on the loopback circuit with a known-good card.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

c. Select None from the Loopback Type column for the port being tested.

d. Select the appropriate state (Unlocked; Locked,disabled; Locked,maintenance; Unlocked,automaticInservice) in the Admin State column for the port being tested.

e. Click Apply.

f. Click Yes in the confirmation dialog box.

Step 9 Clear the terminal loopback circuit:

a. Click the Circuits tab.

b. Choose the loopback circuit being tested.

c. Click Delete.

d. Click Yes in the Delete Circuits dialog box. Do not check any check boxes.

The entire optical circuit path has now passed its comprehensive series of loopback tests. This circuit qualifies to carry live traffic.

1.3 Troubleshooting an Ethernet Circuit Path With Loopbacks

Facility loopbacks and terminal loopbacks are often used together to test the circuit path through the network or to logically isolate a fault. Performing a loopback test at each point along the circuit path systematically isolates possible points of failure.

You can use these procedures only on the ASAP card Ethernet ports in the ONS 15600 SDH system. The example in this section tests an Ethernet circuit on a three-node MS-SP Ring. Using a series of facility loopbacks and terminal loopbacks, the example scenario traces the circuit path, tests the possible failure points, and eliminates them. The logical progression contains six network test procedures:

Note The test sequence for your circuits will differ according to the type of circuit and network topology.

1. A facility loopback on the source-node Ethernet port

2. A terminal loopback on the source-node Ethernet port

3. A facility loopback on the intermediate-node Ethernet port

4. A terminal loopback on the intermediate-node Ethernet port

5. A facility loopback on the destination-node Ethernet port

6. A terminal loopback on the destination-node Ethernet port

Note Facility and terminal loopback tests require on-site personnel.

1.3.1 Perform a Facility Loopback on a Source-Node Ethernet Port

The facility loopback test is performed on the node source port in the network circuit. In the testing situation used in this example, the source is an ASAP Ethernet port in the source node. Completing a successful facility loopback on this port isolates the port as a possible failure point. Figure 1-13 shows an example of a facility loopback on a circuit source Ethernet port.

Note Facility loopbacks require on-site personnel.

Figure 1-13 Facility Loopback on a Circuit Source Ethernet Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.3.2 Perform a Terminal Loopback on a Source-Node Ethernet Port

The terminal loopback test is performed on the node source Ethernet port. For the circuit in this example, it is the source Ethernet port in the source node. You first create a bidirectional circuit that starts on the node destination Ethernet port and loops back on the node source Ethernet port.You then proceed with the terminal loopback test. Completing a successful terminal loopback to a node source port verifies that the circuit is good to the source port.

Caution Performing a loopback on an in-service circuit is service-affecting.

b. If you are starting the current procedure without the optical test set hooked up to the source Ethernet port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 5 Resend test traffic on the loopback circuit with a known-good card.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

Create a Facility Loopback on an Intermediate-Node Ethernet Port

Step 1 Connect an optical test set to the ASAP card Ethernet port you are testing: If you are starting the current procedure without the optical test set hooked up to the source ASAP card Ethernet port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Note For specific procedures to use the test set equipment, consult the manufacturer.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.3.4 Create a Terminal Loopback on an Intermediate-Node Ethernet Port

In the next troubleshooting test, you perform a terminal loopback on the intermediate-node ASAP Ethernet port to isolate whether the destination port is causing circuit trouble. In the example situation in Figure 1-15, the terminal loopback is performed on an intermediate Ethernet port in the circuit. You first create a bidirectional circuit that originates on the source-node Ethernet port and loops back on the intermediate-node port. You then proceed with the terminal loopback test. If you successfully complete a terminal loopback on the node, this node is excluded from possible sources of circuit trouble.

Figure 1-15 Terminal Loopback on an Intermediate-Node Ethernet Port

Caution Performing a loopback on an in-service circuit is service-affecting.

b. If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 5 Resend test traffic on the loopback circuit with a known-good card.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.3.5 Perform a Facility Loopback on a Destination-Node Ethernet Port

You perform a facility loopback test for ASAP card Ethernet port at the destination port to determine whether this local port is the source of circuit trouble. The example in Figure 1-16 shows a facility loopback being performed on an Ethernet port.

Figure 1-16 Facility Loopback on a Destination-Node Ethernet Port

Caution Performing a loopback on an in-service circuit is service-affecting.

Create the Facility Loopback on a Destination-Node Ethernet Port

Step 1 Connect an optical test set to the destination ASAP card optical port you are testing. If you are starting the current procedure without the optical test set hooked up to the source optical port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Note For specific procedures to use the test set equipment, consult the manufacturer.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

1.3.6 Perform a Terminal Loopback on a Destination-Node Ethernet Port

The terminal loopback at the destination-node ASAP card Ethernet port is the final local hardware error elimination in the circuit troubleshooting process, and is performed on the destination-node ASAP card Ethernet port. If this test is completed successfully, you have verified that the circuit is good up to the destination port. The example in Figure 1-17 shows a terminal loopback on a destination-node Ethernet port.

Figure 1-17 Terminal Loopback on a Destination-Node Ethernet Port

Caution Performing a loopback on an in-service circuit is service-affecting.

b. If you are starting the current procedure without the optical test set hooked up to the source port, use appropriate cabling to attach the Tx and Rx terminals of the optical test set to the port you are testing. Both Tx and Rx connect to the same port.

Test the ASAP Card

Step 1 Determine whether you are experiencing trouble on a single SFP (PPM), on all PPMs within a 4PIO (PIM), or on all 4PIO used in that ASAP card. If there is only partial failure, you might be able to replace this part rather than the entire card.

Step 2 If the errors are being observed on one port but not all ports of the ASAP, you might only need to replace that SFP (PPM). Remove the errored SFP (PPM) and replace it with a known-good SFP (PPM) by completing the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 3 If all SFPs (PPMs) on a particular 4PIO (PIM) are experiencing problems, the 4PIO (PIM) is indicated. Remove this 4PIO (PIM) and replace it with a known-good one using the procedures for this in the "Install Cards and Fiber-Optic Cable" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Step 4 If the trouble still is not located, complete the "Replace an I/O Card" procedure for the suspected bad ASAP card and replace it with a known-good one.

Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this, perform an external switch if a switch has not already occurred. See the procedures in the
"Protection Switching, Lock Initiation, and Clearing" section. For more information, refer to the "Maintain the Node" chapter of the
Cisco ONS 15600 SDH Procedure Guide.

Step 5 Resend test traffic on the loopback circuit with a known-good card.

Step 6 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective card to Cisco through the RMA process. Contact Cisco Technical Support (1 800 553-2447).

•Provision of a downloaded, machine-readable diagnostic log file to be used by Cisco Technical Support

Some of these functions, such as ASIC verification and standby card operation, are invisibly monitored in background functions. Change or problem notifications are provided in the Alarms and Conditions window. Other diagnostic functions—verifying card LED function or downloading diagnostic files for technical support—are available to the user in the node view Maintenance > Diagnostic tab. The user-operated diagnostic features are described in the following paragraphs.

1.4.1 Card LED Lamp Tests

A card LED lamp test determines whether card-level indication LEDs are operational. This diagnostic test is run as part of the initial ONS 15600 SDH turn-up, during maintenance routines, or any time you question whether an LED is in working order. Maintenance or higher-level users can complete the following tasks to verify LED operation.

Verify Card LED Operation

Note The LED test must be performed on the physical card. This test is not available in the CTC interface. For typical STM-N, SSXC, and TSC card LED behavior, see the "LED Behavior" section.

Step 1 Determine the active TSC card using the green ACT /STBY LED on the face of the card.

Step 2 Press the LAMP button on the face of the active TSC card.

Step 3 Ensure that all the LEDs on the cards in the shelf illuminate for several seconds.

1.4.2 Retrieve Tech Support Logs Button

When you click the Retrieve Tech Support Logs button in the Diagnostics tab of the Maintenance window, CTC retrieves system data that a Retrieve or higher level user can off-load to a local directory and sent to Technical Support for troubleshooting purposes. The diagnostics file is in machine language and is not human-readable, but can be used by Cisco Technical Support for problem analysis. Complete the following procedure to off-load the diagnostics file.

Note In addition to the machine-readable diagnostics file, the ONS 15600 SDH stores an audit trail of all system events such as user log-ins, remote log-ins, configuration, and changes. This audit trail is considered a record-keeping feature rather than a troubleshooting feature. Information about the feature is located in the "Maintain the Node" chapter of the Cisco ONS 15600 SDH Procedure Guide.

Off-Load the Diagnostics File

Note The diagnostics operation is performed at a shelf level. Only single-node-related diagnostic information can be downloaded at a time.

The diagnostic files retrieved by CTC depends on the user privilege levels. Table 1-1 lists the user privilege levels and the diagnostic retrieval operations they can perform.

Step 3 In the Select a Filename for the Tech Support Logs Zip Archive dialog box, add the diagnostics file name in the format TechSupportLogs_<node_name>.zip by default. Substitute the last 20 alphanumeric characters of the node name for <node_name>. Navigate to the directory (local or network) where you want to save the file.

A message appears asking you if you want to overwrite any existing diagnostics file in the selected directory.

Step 4 Click Save.

CTC performs the diagnostic tasks and writes the diagnostic files in a folder named TechSupportLogs_<node_name> under the location selected in Step 3. After all the diagnostic files are written to the TechSupportLogs_<node_name> folder, CTC archives the retrieved diagnostic files as TechSupportLogs_<node_name>.zip. CTC deletes the TechSupportLogs_<node_name> folder after the archiving process is successfully completed. CTC retains this folder if the archiving process fails. The retrieved diagnostic files can be accessed in the TechSupportLogs_<node_name> folder.

A progress bar indicates the percentage of the file that is being saved. The Save Tech Support Logs Completed dialog box appears when the file is saved. CTC logs any error during the retrieval and archiving of diagnostics file to the CTC Alerts Log.

1.4.3 Data Communications Network (DCN) Tool

In Software R8.0, CTC contains a DCN tool that assists with network troubleshooting for Open Shortest Path First (OSPF) networks. This tool, located in network view, is shown in Figure 1-18. It executes an internal dump command to retrieve information about all nodes accessible from the entry point.

Figure 1-18 DCN Tools OSPF Dump

The dump, which provides the same information as a dump executed by special networking commands, is available in the network view Maintenance > Diagnostic tab. You can select the access point node in the Select Node drop-down list. To create the dump, click Retrieve. (To clear the dump, click Clear.)

The contents of this file can be saved or printed and furnished to Cisco Technical Support for use in OSPF network support.

1.5 Restoring the Database to a Previous or Original Configuration

This section contains troubleshooting for node operation errors that might require restoring software data or restoring the node to the default setup.

1.5.1 Node is Functioning Improperly or Has Incorrect Data

Symptom One or more nodes are not functioning properly or have incorrect data.

Table 1-3 describes the potential cause of the symptom and the solution.

Table 1-3 Node is Functioning Improperly or Has Incorrect Data

Possible Problem

Solution

The node has an incorrect or corrupted database.

Complete the procedures in the "Maintain the Node" chapter of the Cisco ONS 15600 SDH Procedure Guide.

1.6 PC Connectivity Troubleshooting

This section contains information about system minimum requirements, supported platforms, browsers, and Java Runtime Environments (JREs) for Software Releases 9.1 and 9.2, and troubleshooting procedures for PC and network connectivity to the ONS 15600 SDH.

–Workstation: Solaris Version 9 or 10 on an UltraSPARC-III or faster processor, with a minimum of 1 GB RAM and 250 MB of available hard drive space

–Apple Mac OS X. CTC needs to be installed using the CacheInstaller available on the CCO or the ONS CD

Check with the vendor for the latest patch/Service Pack level

Java Runtime Environment

JRE 5.0 (Release 9.1)

JRE 1.6 (Release 9.2.1)

The appropriate JRE version is installed by the CTC Installation Wizard included in the Cisco ONS 15454 software CD. JRE installation provides enhancements to CTC performance, especially for large networks with numerous circuits.

Cisco recommends that you use JRE 5.0 for networks with Software R8.5 nodes. If CTC must be launched directly from nodes running software R7.0 or R7.2, Cisco recommends JRE 1.4.2 or JRE 5.0. If CTC must be launched directly from nodes running software R5.0 or R6.0, Cisco recommends JRE 1.4.2. If CTC must be launched directly from nodes running software earlier than R5.0, Cisco recommends JRE 1.3.1_02.

Web browser

•Release 9.1

–PC: Internet Explorer 6.x or Netscape 7.x

–UNIX Workstation: Mozilla 1.7, Netscape 4.76, Netscape 7.x

•Release 9.2.1

–PC: Internet Explorer 6.x, 7.x, 8.x

–UNIX Workstation: Mozilla 1.7

–Mac OS X PC: Safari

For the PC, use JRE 5.0 or JRE 1.6 with any supported web browser. For UNIX, use JRE 5.0 with Netscape 7.x or JRE 1.3.1_02 with Netscape 4.76.

Netscape is available at this site: http://channels.netscape.com/ns/browsers/default.jsp

Internet Explorer is available at this site: http://www.microsoft.com

Mozilla is available at this site: http://www.mozilla.com

Safari is available at this site:http://www.apple.com

Cable

Use a crossover or straight-through LAN (CAT-5) cable to connect:

•The ONS 15600 to a hub using the backplane RJ-45 ports, or to connect through a LAN.

•The ONS 15600 to a PC using the backplane RJ-45 ports.

•The active TSC RJ-45 port to a laptop or hub.

A direct PC-to-ONS 15600 connection means your computer is physically connected to the ONS 15600. This is most commonly done by connecting a LAN (CAT-5) straight-through cable from your PC to the RJ-45 port on the TSC. However, direct connections include connections to switches or hubs where the ONS 15600 is physically connected.

Note Use only the active TSC connector for connectivity. If you connect to the standby or switch TSCs, you will lose connectivity. Cisco recommends that you use the RJ-45 connector on the Customer Access Panel (CAP/CAP2) so that connection to the ONS 15600 will not be lost during a TSC switch.

1.6.2 Retrieve the Node Information

If you do not know the IP address of your ONS 15600 SDH network element (NE), you can obtain and view the NE information using a TL1 session.

Step 17 Verify that the computer does not have two network interface cards (NICs) installed. If the computer does have two NICs, remove one.

Step 18 Restart the browser and log into the ONS 15600 SDH.

1.6.5 Unable to Verify the NIC Connection on your PC

Symptom When connecting your PC to the ONS 15600 SDH, you are unable to verify that the NIC connection is working properly because the link LED is not illuminated or flashing.

Table 1-7 describes the potential causes of the symptom and the solutions.

Table 1-7 Unable to Verify the NIC Connection on Your PC

Possible Problem

Solution

The CAT-5 cable is not plugged in properly.

Confirm that both ends of the cable are properly inserted. If the cable is not fully inserted because of a broken locking clip, replace the cable.

The CAT-5 cable is damaged.

Ensure that the cable is in good condition. If in doubt, use a known-good cable. Often, cabling is damaged due to pulling or bending.

Incorrect type of CAT-5 cable is being used.

•CAP connection: To connect an ONS 15600 SDH directly to your laptop/PC or a router, use a cross-over CAT-5 cable. To connect the ONS 15600 SDH to a hub or a LAN switch, use a straight-through CAT-5 cable.

•TSC card connection: To connect an ONS 15600 SDH active TSC card directly to your laptop/PC, you might use either a straight-through or cross-over CAT-5 cable because the RJ-45 port on the faceplate is auto sensing.

•If you are using a PCMCIA-based NIC, remove and reinsert the NIC to make sure the NIC is fully inserted.

•If the NIC is built into the laptop/PC, verify that the NIC is not faulty.

The NIC is faulty.

Confirm that the NIC is working properly. If you have no issues connecting to the network (or any other node), the NIC should be working correctly.

If you have difficulty connecting to the network (or any other node), the NIC might be faulty and needs to be replaced.

1.6.6 TCP/IP Connection is Lost

Symptom The TCP/IP connection was established and then lost, and a DISCONNECTED alarm appears on CTC.

Table 1-8 describes the potential cause of the symptom and the solution.

Table 1-8 TCP/IP Connection is Lost

Possible Problem

Solution

Your PC lost TCP/IP connection with the ONS 15600 SDH.

Use a standard ping command to verify the TCP/IP connection between the PC and the ONS 15600 SDH TSC card. A ping command will work if the PC connects directly to the TSC card or uses a LAN to access the TSC card. A ping command will also work if the CTC is connected through a gateway network element (GNE) and DCC if the node and CTC are in the same subnet or the required static routes are configured.

Ping the ONS 15600

a. If you are using a Microsoft Windows operating system, from the Start Menu choose Run, type command in the Open field of the Run dialog box, and click OK.

b. If you are using a Sun Solaris operating system, from the Common Desktop Environment (CDE) click the Personal Application tab and click Terminal.

Step 2 For both the Microsoft and Sun operating systems, type the following at the prompt:

pingONS 15600 SDH IP address

For example

ping 192.1.0.2

If the workstation has connectivity to the ONS 15600 SDH, the ping is successful and displays a reply from the IP address. If the workstation does not have connectivity, a "Request timed out" message appears.

Step 3 If the ping is successful, it demonstrates that an active TCP/IP connection exists. Restart CTC.

Step 4 If the ping is not successful, and the workstation connects to the ONS 15600 SDH through a LAN, verify that the workstation's IP address is on the same subnet as the ONS node.

If the ping is not successful and the workstation connects directly to the ONS 15600 SDH, verify that the link light on the workstation NIC is illuminated.

The cache file location is usually C:\ProgramFiles\Netscape\Users\<yourname>\cache. The <yourname> segment of the file location is often the same as the user name.

1.7.4 Java Runtime Environment Incompatible

Symptom The CTC application does not run properly.

Possible Cause The compatible Java 2 JRE is not installed.

Recommended Action The JRE contains the Java virtual machine, runtime class libraries, and Java application launcher that are necessary to run programs written in the Java programming language. The ONS 15600 SDH CTC is a Java application. A Java application, unlike an applet, cannot rely completely on a web browser for installation and runtime services. When you run an application written in the Java programming language, you need the correct JRE installed. The correct JRE for each CTC software release is included on the Cisco ONS 15600 SDH software CD and on the Cisco ONS 15600 SDH documentation CD. If you are running multiple CTC software releases on a network, the JRE installed on the computer must be compatible with the different software releases. Table 1-12 shows JRE compatibility with ONS 15600 SDH software releases.

Note To avoid network performance issues, Cisco recommends managing a maximum of 50 nodes concurrently with CTC. To manage more than 50 nodes, Cisco recommends using Cisco Transport Manager (CTM). Cisco does not recommend running multiple CTC sessions when managing two or more large networks.

Delete the CTC Cache File Manually

Step 1 To delete the JAR files manually, from the Windows Start menu choose Search > For Files or Folders.

Step 2 Enter ctc*.jar or cms*.jar in the Search for files or folders named field on the Search Results dialog box and click Search Now.

Step 3 Click the Modified column on the Search Results dialog box to find the JAR files that match the date when you downloaded the files from the TSC card.

Step 4 Highlight the files and press the keyboard Delete key.

Step 5 Click Yes in the confirmation dialog box.

1.7.5.1 Set the CTC_HEAP and CTC_MAX_PERM_SIZE_HEAP Environment Variables for Windows

Note Before proceeding with the following steps, ensure that your system has a minimum of 1 GB of RAM. If your system does not have a minimum of 1 GB of RAM, contact the Cisco Technical Assistance Center (TAC).

Step 1 Close all open CTC sessions and browser windows.

Step 2 From the Windows Start menu, choose Control Panel > System.

Step 3 In the System Properties window, click the Advanced tab.

Step 4 Click the Environment Variables button to open the Environment Variables window.

Step 5 Click the New button under the System variables field.

Step 6 Type CTC_HEAP in the Variable Name field.

Step 7 Type 512 in the Variable Value field, and then click the OK button to create the variable.

Step 8 Again, click the New button under the System variables field.

Step 9 Type CTC_MAX_PERM_SIZE_HEAP in the Variable Name field.

Step 10 Type 128 in the Variable Value field, and then click the OK button to create the variable.

Step 11 Click the OK button in the Environment Variables window to accept the changes.

Step 12 Click the OK button in the System Properties window to accept the changes.

Set the CTC_HEAP and CTC_MAX_PERM_SIZE_HEAP Environment Variables for Solaris

Usually accompanied by Ethernet-specific alarms. Verify the Ethernet connections between nodes.

A lost DCC connection.

Usually accompanied by an EOC alarm. Clear the EOC alarm and verify the DCC connection as described in the "EOC" alarm on page 2-41.

OSPF not properly configured.

Usually accompanied by a HELLO failure. Reconfigure the OSPF on the system to proper settings.

CTC launched from ONS 15454 SDH node.

You can manage an ONS 15600 SDH from CTC launched on the same release or higher CTC session from an ONS 1545 SDH node. Restart CTC and log into an ONS 15600 SDH node to enable node management.

1.7.7 Cisco Transport Controller Does Not Recognize the Node

Symptom This situation is often accompanied by the INCOMPATIBLE-SW alarm.

Table 1-15 describes the potential cause of the symptom and the solutions.

Table 1-15 Cisco Transport Controller Does Not Recognize the Node

Possible Problem

Solution

The software loaded on the connecting workstation and the software on the TSC card are incompatible.

Incompatibility occurs when the TSC card software is upgraded but the PC has not yet upgraded to the compatible CTC JAR file. It also occurs on login nodes with compatible software that encounter other nodes in the network that have a newer software version.

In mixed platform networks (ONS 15600 SDH and ONS 15454 SDH), you must log into the same or higher CTC software release as the one loaded on the ONS 15600 SDH node to enable OAM&P of all nodes.

Note You cannot access other nodes over DCC (the gray nodes) when the PC is connected to the active TSC card unless that ONS 15600 SDH is configured as a gateway NE.

1.7.8 Username or Password Mismatch

Symptom A mismatch often occurs concurrently with a NOT-AUTHENTICATED alarm.

Table 1-16 describes the potential cause of the symptom and the solution.

Table 1-16 Username or Password Mismatch

Possible Problem

Solution

The username or password entered does not match the information stored in the TSC card.

All ONS nodes must have the same username and password created to display every ONS node in the network. You can also be locked out of certain ONS nodes on a network if your username and password were not created on those specific ONS nodes.

For initial login to the ONS 15600 SDH, type the CISCO15 user name in capital letters, type the otbu+1 password, and click Login.

The username or password does not match the information stored in the RADIUS server database.

If the node has been configured for RADIUS authentication, the username and password are verified against the RADIUS server database rather than the security information in the local node database. For more information about RADIUS security, refer to the "Security" chapter in the Cisco ONS 15600 SDH Reference Manual.

Verify Correct Username and Password

Step 1 Ensure that your keyboard Caps Lock key is not turned on and affecting the case-sensitive entry of the user name and password.

Step 2 Contact your system administrator to verify the user name and password.

Reset the ONS 15600 SDH Password

Step 2 Press in and hold down the recessed button labelled LAMP TEST for five seconds.

Step 3 Release the LAMP TEST button for approximately two seconds.

Step 4 Again press in and hold down the button labelled LAMP TEST for five seconds.

Step 5 Again release the LAMP TEST button.

Step 6 Start a normal CTC session. At the login screen, CTC accepts the default username and password set when the ONS 15600 SDH node shipped. The default username is CISCO15 and the password is otbu+1. CISCO15 has Superuser rights and privileges, which allow you to create a user name and assign a password.

Note Other existing usernames and passwords are not affected by the reset. The Superuser reset applies only to the local node where the procedure is performed.

Step 7 If you need to create another user name and password, complete the following steps:

a. Click the Provisioning > Security tabs and click create.

b. Fill in the fields with a new user name and password and assign a security level.

c. Click OK.

Note After new user names and passwords are set up, including at least one Superuser, log in as a newly created Superuser and delete the default CISCO15 username and otbu+1 password to ensure that security is not compromised.

1.7.10 No IP Connectivity Exists Between Nodes

Symptom The nodes have a gray icon which is usually accompanied by alarms.

Table 1-18 describes the potential causes of the symptom and the solutions.

Table 1-18 No IP Connectivity Exists Between Nodes

Possible Problem

Solution

The node has lost DCC connection.

Usually is accompanied by DCC termination alarms, such as EOC or EOC-L. Clear the EOC (or EOC-L) alarm and verify the DCC connection as described in the "EOC" alarm on page 2-41.

The nodes are in different subnetworks and required static routes that are not provisioned.

Usually is accompanied by DCC termination alarms. Properly provision required static routes and nodes in the same subnets. Refer to the procedure for setting up CTC access in the Cisco ONS 15600 SDH Procedure Guide.

1.7.12 Loss of IP Communication Between Nodes on an OSPF LAN

Symptom The CTC session on an ONS 15600 SDH connected to Router 1 loses communication with the ONS 15600 SDH connected to Router 2 on the same LAN in OSPF backbone Area 0.

Table 1-20 describes the potential causes of the symptom and the solutions.

Table 1-20 Loss of IP Communication in Segmented OSPF Area

Possible Problem

Solution

The OSPF backbone Area 0 has segmented into multiple gateway network elements (GNEs_.

If multiple ONS 15600 SDH nodes and routers are connected to the same LAN in OSPF backbone Area 0 and a link between two routers breaks, the backbone OSPF area 0 could divide into multiple GNEs. If this occurs, the CTC session on the ONS node connected to Router 1 will not be able to communicate with the ONS 15600 SDH connected to Router 2. This is standard behavior for an OSPF network.

To resolve this problem, you must repair the link between the routers or provide another form of redundancy in the network. Refer to the Cisco ONS 15600 SDH Procedure Guide for procedures to repair the link between the routers.

A broken link between two routers on the LAN in OSPF backbone Area 0.

1.8 Circuits and Timing

This section provides solutions to circuit creation and reporting errors, as well as common timing reference errors and alarms.

1.8.1 ONS 15600 SDH Switches Timing Reference

Symptom Timing references switch when one or more problems occur.

Table 1-21 describes the potential causes of the symptom and the solutions.

Table 1-21 ONS 15600 Switches Timing Reference

Possible Problem

Solution

The optical or building integrated timing supply (BITS) input is receiving loss of signal (LOS), loss of frame (LOF), or AIS from its timing source.

Set up the timing input to a reliable timing source. Refer to the procedure for setting up timing in the Cisco ONS 15600 SDH Procedure Guide.

The input clock wanders and has more than three slips in 30 seconds.

1.8.2 Holdover Synchronization Alarm

Symptom The clock is running at a different frequency than normal and the HLDOVRSYNC alarm appears. Holdover occurs when the node is provisioned for external or line timing and both of the provisioned references fail. The timing switches to the internal Stratum 3E clock on the TSC card.

Table 1-22 describes the potential cause of the symptom and the solution.

1.8.3 Free-Running Synchronization Mode

Symptom The clock is running at a different frequency than normal and the FRNGSYNC alarm appears. Free Running is reported when the node is running on the internal clock after a failure of the primary and secondary clock references.

Table 1-23 describes the potential cause of the symptom and the solution.

Table 1-23 Free-Running Synchronization Mode

Possible Problem

Solution

No reliable reference input is available.

The clock is using the internal oscillator as its only frequency reference. This occurs when no reliable, prior timing reference is available. See the "FRNGSYNC" condition on page 2-64 for a detailed description.

1.8.4 Daisy-Chained BITS Not Functioning

Symptom You are unable to daisy-chain the BITS.

Table 1-24 describes the potential cause of the symptom and the solution.

Table 1-24 Daisy-Chained BITS Not Functioning

Possible Problem

Solution

Daisy-chaining BITS is not supported on the ONS 15600 SDH.

Daisy-chaining BITS causes additional wander buildup in the network and is therefore not supported. Instead, use a timing signal generator to create multiple copies of the BITS clock and separately link them to each ONS 15600 SDH.

You cannot use BITS Out A and/or BITS Out B outputs when providing a clock source from BITS In A and/or BITS In B inputs. To provide BITS Out A and/or BITS Out B external outputs, the clock source must be derived from an optical input.

1.8.5 Circuits Remain in PARTIAL Status

Symptom Circuits remain in the PARTIAL status.

Table 1-27 describes the potential cause of the symptom and the solution.

Note The CTC session freezes until all circuits are repaired. Circuit repair can take up to five minutes or more depending on the number of circuits provisioned.

When the circuit repair is complete, the Circuits Repaired dialog box appears.

Step 6 Click OK.

Step 7 In node view of the new node, click the Circuits tab. Note that all circuits listed are DISCOVERED. If all circuits listed do not have a DISCOVERED status, call the Cisco TAC (1 800 553-2447) to open an RMA.

1.9 Fiber and Cabling

This section explains problems typically caused by cabling connectivity errors. It also includes instructions for crimping CAT-5 cable and lists the optical fiber connectivity levels.

1.9.1 Bit Errors Appear for an Optical Traffic Card

Symptom An optical traffic card has multiple bit errors.

Table 1-26 describes the potential causes of the symptom and the solutions.

Table 1-26 Bit Errors Appear for a Traffic Card

Possible Problem

Solution

Faulty cabling

Low optical-line power

High optical-line power

Bit errors on line (traffic) ports usually originate from cabling problems or low or high optical-line power levels. The errors can be caused by synchronization problems, especially if PJ (pointer justification) errors are reported. Troubleshoot cabling problems using the "Network Troubleshooting Tests" section. Troubleshoot low or high optical-line power levels using the "Faulty Fiber-Optic Connections" section. Use a test set whenever possible to check for errors.

Warning Invisible laser radiation may be emitted from disconnected fibers or connectors. Do not stare into beams or view directly with optical instruments. Statement 1051

Warning Invisible laser radiation may be emitted from the end of the unterminated fiber cable or connector. Do not view directly with optical instruments. Viewing the laser output with certain optical instruments (for example, eye loupes, magnifiers, and microscopes) within a distance of 100 mm may pose an eye hazard. Statement 1056

•If the power level is within tolerance, the problem is with the fan-out cables or the optical (STM-N) card.

•If the power level is too high, add the appropriate attenuation.

Step 4 If the power level falls below the specified range:

Note When this condition occurs, the far-end node is usually an ONS 15454 SDH.

a. Clean or replace the OGI fiber fan-out cables. If possible, do this for the optical (STM-N) card you are working on and the far-end card. Refer to the Cisco ONS 15600 SDH Procedure Guide for fiber cleaning procedures.

b. Clean the optical connectors on the card. If possible, do this for the optical (STM-N) card you are working on and the far-end card. Refer to the Cisco ONS 15600 SDH Procedure Guide for fiber cleaning procedures.

c. Replace the far-end transmitting optical (STM-N) card to eliminate the possibility of a degrading transmitter on the far-end optical (STM-N) card.

d. If the power level still falls below the specified range with the replacement fibers and replacement card, check for one of these three factors that attenuate the power level and affect link loss (LL):

•Excessive number or fiber connectors; connectors take approximately 0.5 dB each.

•Excessive number of fiber splices; splices take approximately 0.5 dB each.

Note These are typical attenuation values. Refer to the specific product documentation for the actual values or use an optical time domain reflectometer (OTDR) to establish precise link loss and budget requirements.

Step 5 If no power level shows on the fiber, the fiber is bad or the transmitter on the STM-N port failed.

a. Check that the Tx and Rx fibers are not reversed. LOS and EOC alarms normally accompany reversed Tx and Rx fibers. Fixing reversed Tx and Rx fibers clears the alarms and restores the signal.

b. Clean or replace the OGI fiber fan-out cables. If possible, do this for both the STM-N port you are working on and the far-end STM-N port. Refer to the Cisco ONS 15600 SDH Procedure Guide for fiber cleaning procedures.

c. Retest the fiber power level.

d. If the replacement fiber still shows no power, replace the optical (STM-N) card.

Tip To prevent overloading the receiver, use an attenuator on the fiber between the STM-N port transmitter and the receiver. Place the attenuator on the receive transmitter of the STM-N ports. Refer to the attenuator documentation for specific instructions.

Crimp Replacement CAT-5 Cables

You can crimp your own CAT-5 cables for use with the ONS 15600 SDH. To connect the CAP of an ONS 15600 SDH directly to your laptop/PC or a router, use a straight-through CAT-5 cable. To connect the CAP of an ONS 15600 SDH to a hub or a LAN switch, use a cross-over CAT-5 cable. To connect an ONS 15600 SDH active TSC card directly to your laptop/PC, you might use either a straight-through or cross-over CAT-5 cable because the RJ-45 port on the faceplate is autosensing.

Use a straight-through or cross-over cable to connect to the backplane Ethernet connections of an ONS 15600 SDH. Use a straight-through cable to connect to the faceplate connector of the ONS 15600 SDH TSC card. Use CAT-5 cable RJ-45 T-568B, Color Code (100 Mbps), and a crimping tool. Figure 1-20 shows the layout of an RJ-45 connector.

Note Odd-numbered pins always connect to a white wire with a colored stripe.

1.9.3 Optical Traffic Card Transmit and Receive Levels

Each optical traffic card has connectors on its faceplate that contain both transmit and receive ports. Table 1-30 shows the optical power levels for the transmit and receive ports of the optical traffic cards.

The CTC Maintenance > Transceiver tab shows the optical power transmitted (OPT) and optical power received (OPR) levels.

Note CTC might show OPT levels at 1 dBm more or less than the actual card OPT level.

1.10 Power Supply Problems

This section provides the a procedure for troubleshooting power supply difficulties.

Note For information about power consumption for nodes and cards, refer to the Cisco ONS 15600 SDH Reference Manual.

Symptom Loss of power or low voltage, resulting in a loss of traffic.

Table 1-31 describes the potential causes of the symptom and the solutions.

Table 1-31 Power Supply Problems

Possible Problem

Solution

A loss of power or low voltage reading.

The ONS 15600 SDH requires a constant source of DC power to properly function. Input voltage range is from -40.5 VDC to -72 VDC.

A newly installed ONS 15600 SDH that is not properly connected to its power supply will not operate. Power problems can be confined to a specific ONS 15600 SDH or affect several pieces of equipment on the site.

Caution Operations that interrupt power supply or short the power connections to the ONS 15600 SDH are service-affecting.

Warning The power supply circuitry for the equipment can constitute an energy hazard. Before you install or replace the equipment, remove all jewelry (including rings, necklaces, and watches). Metal objects can come into contact with exposed power supply wiring or circuitry inside the equipment. This could cause the metal objects to heat up and cause serious burns or weld the metal object to the equipment. Statement 207