Lenovo is open and ready for business with System xIBM x86 products are now products of Lenovo in the U.S. and other countries. IBM will continue to host x86-related content on ibm.com until migrated to Lenovo.Learn more about the acquisition

Abstract

The IBM® Flex System™ PCIe Expansion Node provides the ability to attach additional PCI Express cards, such as High IOPS SSD adapters, fabric mezzanine cards, and next-generation graphics processing units (GPU), to supported IBM Flex System compute nodes. This capability is ideal for many applications that require high performance I/O, special telecommunications network interfaces, or hardware acceleration using a PCI Express GPU card. The PCIe Expansion Node supports up to four PCIe adapters and two additional Flex System I/O expansion adapters.

Changes in the November 21 update:* NVIDIA GRID K1 and K2 currently not supported with the x240 M5

Contents

The IBM® Flex System™ PCIe Expansion Node provides the ability to attach additional PCI Express cards, such as High IOPS SSD adapters, fabric mezzanine cards, and next-generation graphics processing units (GPU), to supported IBM Flex System compute nodes. This capability is ideal for many applications that require high performance I/O, special telecommunications network interfaces, or hardware acceleration using a PCI Express GPU card. The PCIe Expansion Node supports up to four PCIe adapters and two additional Flex System I/O expansion adapters.

The PCIe Expansion Node is ideal for application environments that are written to take advantage of acceleration and visualization performance using GPUs that are connected to Flex System Compute nodes. It is also useful for environments that require specific PCIe adapter connectivity to a Flex System Compute node.

Support for one full-length, full-height double-wide adapter (consuming the space of the two full-length, full-height adapter slots)

Support for PCIe cards with higher power requirements

The Expansion Node provides two auxiliary power connections, up to 75W each for a total of 150W of additional power using standard 2x3, +12V six-pin power connectors. These connectors are placed on the base planar so that they both can provide power to a single adapter card (up to 225W), or to two adapters (up to 150W each). Power cables are used to connect from these connectors to the PCIe adapters and are included with the PCIe Expansion Node.

Figure 4 shows the architecture of the PCIe Expansion Node when connected to a compute node.

Figure 4. Architecture of the PCIe Expansion Node

The Expansion Node connects to a standard-width compute node using the interposer cable which plugs into the expansion connector and interposer connector on the compute node and Expansion Node respectively. This link forms a PCIe 2.0 x16 connection between the compute node and the PCIe switch in the Expansion Node. The PCIe switch has connections to the six PCIe slots in the Expansion Node:

PCIe 2.0 x16 connectors to the two Flex System I/O expansion slots (labeled I/O 3 and I/O 4 in the figure).

Notes:

In compute nodes, such as the x220 and x240, I/O expansion slot 1 and 2 in the server operate at PCIe 3.0 speeds. However, I/O expansion slots 3 and 4 in the PCIe Expansion Node (and also the four standard PCIe slots) operate at PCIe 2.0 speeds.

The expansion connector in the compute node is routed through processor 2. Therefore, processor 2 must be installed in the compute node.

When adapters are installed in the two I/O expansion slots, they connect to the chassis midplane and provide additional connections to I/O module bays in the chassis. Table 3 shows the connections between the expansion slots and the module bays. Note that the ports available from the adapter vary, depending on whether the adapter is a 2-port or 4-port adapter. Similarly, the number of ports to the I/O module depend on the number of ports activated in the switch.

Table 3. Adapter-to-I/O bay correspondence

I/O expansion slot

Port on the adapter

Corresponding I/O module bayin the chassis

Slot 1(Compute Node)

Port 1

Module bay 1

Port 2

Module bay 2

Port 3*

Module bay 1**

Port 4*

Module bay 2**

Slot 2(Compute Node)

Port 1

Module bay 3

Port 2

Module bay 4

Port 3*

Module bay 3**

Port 4*

Module bay 4**

Slot 3(Expansion Node)

Port 1

Module bay 1

Port 2

Module bay 2

Port 3*

Module bay 1**

Port 4*

Module bay 2**

Slot 4(Expansion Node)

Port 1

Module bay 3

Port 2

Module bay 4

Port 3*

Module bay 3**

Port 4*

Module bay 4**

* Ports 3 and 4 require that a four-port card be installed in the expansion slot.** Might require one or more port upgrades to be installed in the I/O module.

Supported PCIe adapter cards

The Expansion Node supports the following general adapter characteristics:

The front-facing bezel of the Expansion Node is inset from the normal face of the compute nodes. This inset is to allow for the use of cables connected to PCIe adapter cards that support external connectivity. The Expansion Node provides up to 80 mm of space in the front of the PCIe adapter cards to allow for the bend radius of these cables.

Table 4 lists the PCIe adapters that are supported in the Expansion Node. Some adapters must be installed in one of the full-height slots as noted. If the NVIDIA Tesla M2090 is installed in the Expansion Node, then an adapter cannot be installed in the other full-height slot. The low-profile slots and Flex System I/O expansion slots can still be used, however.

* Withdrawn from marketing** If the NVIDIA Tesla M2090 is installed in the Expansion Node, then an adapter cannot be installed in the other full-height slot. If installed, only this adapter is supported in the expansion node. No other PCIe adapters may be selected.# NVIDIA GRID K1 and K2 not supported in the x240 M5.† The K1, K2, K20, K40 and 5110P adapters are double-wide cards and occupy the two full-height PCIe slots. If installed, the adjacent slot is unavailable, however adapters can be installed in the two low-profile slots. § NVIDIA GRID Kx and Tesla Kxx adapters supported only in servers with 1 TB or less memory installed.

Note: Although the design of Expansion Node allows for a much greater set of standard PCIe adapter cards, the preceding table lists the adapters that are specifically supported. If the PCI Express adapter that you require is not on the ServerProven web site, use the IBM ServerProven Opportunity Request for Evaluation (SPORE) process to confirm compatibility in the desired configuration.

Supported I/O expansion cards

Table 5 lists the Flex System I/O expansion cards that are supported in the PCIe Expansion Node.

Table 5. Supported I/O expansion cards

Part number

Feature code

Description

Supported in the PEN

Networking

90Y3482

A3HK

IBM Flex System EN6132 2-port 40Gb Ethernet Adapter

Supported

88Y5920

A4K3

IBM Flex System CN4022 2-port 10Gb Converged Adapter

Supported

90Y3554

A1R1

IBM Flex System CN4054 10Gb Virtual Fabric Adapter

Supported*

00Y3306

A4K2

IBM Flex System CN4054R 10Gb Virtual Fabric Adapter

Supported†

90Y3558

A1R0

IBM Flex System CN4054 Virtual Fabric Adapter (SW Upgrade)

Supported

49Y7900

A10Y

IBM Flex System EN2024 4-port 1Gb Ethernet Adapter

Supported

90Y3466

A1QY

IBM Flex System EN4132 2-port 10Gb Ethernet Adapter

Supported

90Y3454

A1QZ

IBM Flex System IB6132 2-port FDR InfiniBand Adapter

Supported

Storage

88Y6370

A1BP

IBM Flex System FC5022 2-port 16Gb FC Adapter

Supported

95Y2386

A45R

IBM Flex System FC5052 2-port 16Gb FC Adapter

Supported

95Y2391

A45S

IBM Flex System FC5054 4-port 16Gb FC Adapter

Supported

69Y1942

A1BQ

IBM Flex System FC5172 2-port 16Gb FC Adapter

Supported

69Y1938

A1BM

IBM Flex System FC3172 2-port 8Gb FC Adapter

Supported

95Y2375

A2N5

IBM Flex System FC3052 2-port 8Gb FC Adapter

Supported

* The CN4054 is only supported in the x220 and the x240 with E5-2600 v1 processors
† The CN4054R is not supported in the x220 nor in the x240 with E5-2600 v1 processors

Special Notices

This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.