This chapter describes how to replace a Sun Blade 6048 Switched InfiniBand Network ExpressModule (IB NEM) in a powered-on Sun Blade 6048 Series Chassis. This chapter also includes instructions to verify that the replacement IB NEM has been installed correctly.

Caution - Damage to the IB NEM can occur as the result of careless handling or electrostatic discharge (ESD). Always handle an IB NEM with care to avoid damage to electrostatic sensitive components. To minimize the possibility of ESD-related damage, Sun strongly recommends using both a workstation antistatic mat and an ESD wrist strap. You can get an ESD wrist strap from any reputable electronics store or from Sun as part number 250-1007.

You can install the IB NEM in the following Sun Blade 6048 Series Chassis:

Sun Blade 6000 Chassis

Sun Blade 6000 P Chassis

2.1 Replacing IB NEM Hardware

If an IB NEM fails or if you choose to change the I/O configuration, you will need to replace the IB NEM. You can replace an IB NEM in a powered-on Sun Blade 6048 Series Chassis using a hot-plug operation.

If you are removing but not replacing the IB NEM, you must install both IB NEM filler panels to meet FCC limits for electromagnetic interference (EMI) and to ensure proper airflow and cooling.

Note - If you are installing a IB NEM in a Sun Blade 6048 Series Chassis that has not been powered on, see the Sun Blade 6048 Series Installation Guide (Sun Part number:
820-2312).

The IB NEMs are customer-replaceable units (CRUs).

2.1.1 Replace IB NEM in a Powered-On Chassis

1. Identify which IB NEM to replace.

If the amber Service Action Required LED is lit, this indicates a problem with a specific IB NEM. Otherwise, you can choose any IB NEM to replace if, for example, you want to change the I/O configuration.

2. Prepare the IB NEM for a hot-plug procedure. Use either of these methods:

Press the Attention button on the IB NEM to initiate the hot-plug removal.

The green OK LED will blink for up to one minute, indicating that the IB NEM is being prepared for removal.

To abort the operation, press the Attention button again within five seconds.

Once the green LED goes dark and the blue LED is illuminated, you can safely remove the IB NEM.

Use the ILOM web interface or the command-line interface (CLI) to initiate the hot-plug removal.

After you physically install the IB NEM, the Chassis Monitoring Module (CMM) automatically detects the presence of the IB NEM. The green OK indicator on the IB NEM transitions from Standby Blink to Steady On when the IB NEM is operational.

Note - If you are replacing an IB NEM, you do not need to install the InfiniBand software packages. The appropriate software package will have been installed and configured as part of the initial IB NEM installation.

2.2 Verifying Installation

If you have not installed the IB NEM in the chassis and connected it to an operational InfiniBand switch, do so before you attempt to verify the installation. The InfiniBand switch should automatically recognize InfiniBand servers when the servers are connected to the fabric.

2.2.1 Verify Hardware Installation

1. Once you have physically installed the IB NEM and ensured that the cables are connected to the IB NEM and switches, ensure that an IB subnet manager is running on the connected InfiniBand fabric (network).

If the green port LED is illuminated, you have successfully completed the hardware installation and you can proceed to verification through the ILOM interfaces. The green LED indicates that the port is enabled, that is, that a physical link to a remote switch (or, possibly an HCA) has been established.

If the port LEDs are not illuminated, one possible cause might be that the InfiniBand drivers are not installed. You cannot verify a complete installation on Linux until you install these drivers.

2. You can now examine hardware status through one of the ILOM (Integrated Lights Out Manager) interfaces. Use one of the following procedures:

The last entry in the sample output (InfiniBand: Mellanox Technologies) verifies the hardware installation and confirms the IB NEM’s availability to the Linux host.

2.4 Troubleshooting a Hot-Remove Operation

Because IB NEMs are shared resources, all Sun Blade Server Modules must respond favorably to the PCI hot-remove request. However, a blade might not relinquish the link to a IB NEM if, for instance, there are busy NFS mounted volumes, file transfers, and so on.

To determine the state of the IB NEM-to-blade connections, you can use the ILOM web interface or the ILOM command-line interface, as described in the following procedures.

2.4.1 Troubleshoot Using the ILOM Web Interface

1. To verify IB NEM-to-blade connections, log in to the ILOM web interface for the CMM.

2. In the left navigation pane, select CMM.

The ILOM Version Information page appears.

3. Select the System Information tab and then select the Components tab.

The Component Management page appears.

4. Click on the IB NEM component name.

A page displaying properties and values for the selected IB NEM appears.

As shown, the system responds with bladen_link_status entries for each blade (where n is the blade module number). Any blade not reporting a Not_present status needs intervention from the host OS on that blade. This intervention from the host OS depends entirely on the OS that is active on the blade. Each supported OS has a different method for managing attached devices.

5. Perform the appropriate host OS procedure for releasing the IB NEM from the blade.

As shown, the system responds with bladen_link_status entries for each blade (where n is the blade module number). Any blade not reporting a Not_present status needs intervention from the host OS on that blade. This intervention from the host OS depends entirely on the OS that is active on the blade. Each supported OS has a different method for managing attached devices.

3. Perform the appropriate host OS procedure for releasing the IB NEM from the blade.