One or more network adapters that support RDMA (Remote Direct Memory Access)

2. Configuration

2.1. Installing

SMB Multichannel is enabled by default. There is no need to install components, roles, role services or features. The SMB client will automatically detect and use multiple network connections if a proper configuration is identified.

2.2. Disabling

SMB Multichannel is enabled by default and there is typically no need to disable it. However, if you want to disable SMB Multichannel (for testing purposes, for instance), you can use the following PowerShell cmdlets:

On the SMB server side:

Set-SmbServerConfiguration -EnableMultiChannel $false

On the SMB client side:

Set-SmbClientConfiguration -EnableMultiChannel $false

Note: Disabling the feature on either the client or the server prevent the systems from using it.

2.3. Re-enabling

You can re-enable SMB Multichannel after you disabled it by using:

On the SMB server side:

Set-SmbServerConfiguration -EnableMultiChannel $true

On the SMB client side:

Set-SmbClientConfiguration -EnableMultiChannel $true

Note: You need to enable the feature on both the client or the server to start using it again.

3. Sample Configurations

This section provides details on how SMB Multichannel works with a few different configurations using a variety of Network Interface Cards (NIC). Please note that these are only samples and many other configurations not detailed here are possible.

3.1. Single RSS-capable NIC

This typical configuration involves an SMB client and SMB Server configured with a single 10GbE NIC. Without SMB multichannel, if there is only one SMB session established, SMB uses a single TCP/IP connection, which naturally gets affinitized with a single CPU core. If lots of small IOs are performed, it’s possible for that core to become a performance bottleneck.

Most NICs today offer a capability called Receive Side Scaling (RSS), which allows multiple connections to be spread across multiple CPU cores automatically. However, when using a single connection, RSS cannot help.

With SMB Multichannel, if the NIC is RSS-capable, SMB will create multiple TCP/IP connections for that single session, avoiding a potential bottleneck on a single CPU core when lots of small IOs are required.

3.2. Multiple NICs

When using multiple NICs without SMB multichannel, if there is only one SMB session established, SMB creates a single TCP/IP connection using only one of the many NICs available. In this case, not only it’s not possible to aggregate the bandwidth of the multiple NICs (achieve 2Gbps when using two 1GbE NICs, for instance), but there is a potential for failure if the specific NIC chosen is somehow disconnected or disabled.

With Multichannel, SMB will create multiple TCP/IP connections for that single session (at least one per interface or more if they are RSS-capable). This allows SMB to use the combined NIC bandwidth available and makes it possible for the SMB client to continue to work uninterrupted if a NIC fails.

3.3. Teamed NICs

Windows Server 2012 supports the ability to combine multiple NICs into one using a new feature commonly referred to as NIC teaming. Although a team always provides fault tolerance, SMB without Multichannel will create only one TCP/IP connection per team, leading to limitations in both the number of CPU cores engaged and the use of the full team bandwidth.

SMB Multichannel will create multiple TCP/IP connections, allowing for better balancing across CPU cores with a single SMB session and better use of the available bandwidth. NIC Teaming will continue to offer the failover capability, which will work faster than using SMB Multichannel by itself. NIC Teaming is also recommended because it offers failover capabilities to other workloads that do not rely on SMB and therefore cannot benefit from the failover capabilities of SMB Multichannel.

Note: A team of RDMA-capable NICs is always reported as non-RDMA capable. If you intend to use the RDMA capabilities of the NIC, do not team them.

3.4. Single or Multiple RDMA NICs

SMB Multichannel is the feature responsible for detecting the RDMA capabilities of NICs to enable the SMB Direct feature (SMB over RDMA). Without SMB Multichannel, SMB will use regular TCP/IP with these RDMA-capable NICs (they all provide a TCP/IP stack side-by-side with the new RDMA stack).

With SMB Multichannel, SMB will detect the RDMA capability and create multiple RDMA connections for that single session (two per interface). This allows SMB to use the high throughput, low latency and low CPU utilization offered by these RDMA NICs. It will also offer fault tolerance if you’re using multiple RDMA interfaces.

Note 1: A team of RDMA-capable teams is reported as non-RDMA capable. If you intend to use the RDMA capability of the NIC, do not team them.

Note 2: After at least one RDMA connection is created, the TCP/IP connection used for the original protocol negotiation is no longer used. However, that connection is kept around in case the RDMA connections fail.

3.5. Multichannel, RDMA and NIC Teaming compatibility

Here’s a table summarizing the different capabilities available when combining SMB Multichannel, RDMA (SMB Direct) and NIC Teaming:

For non-RDMA NICs, your best bet is combining NIC Teaming with SMB Multichannel. This will give you the best throughput, plus fault tolerance for applications using SMB and other protocols.

When using RDMA NICs, LBFO is not a good option, since it disables the RDMA capability of the NIC.

3.6. Sample Configurations that do not use SMB Multichannel

The following are sample network configurations that do not use SMB Multichannel:

Single non-RSS-capable network adapters. This configuration would not benefit from multiple network connections, so SMB Multichannel is not used.

Network adapters of different speeds. SMB Multichannel will choose to use the faster network adapter. Only network interfaces of same type (RDMA, RSS or none) and speed will be used simultaneously by SMB Multichannel, so the slower adapter will be idle.

4. Operations

Here are a few common scenarios for testing SMB Multichannel:

4.1. Compare a file copy with and without SMB Multichannel

To measure the increased throughput provided by SMB Multichannel, follow these steps:

Setup SMB Multichannel in one the configurations described earlier

Measure the time to perform a long-running file copy using SMB Multichannel

Disable SMB Multichannel (see instructions in previous topic).

Measure the time it takes to perform the same file copy without SMB Multichannel

Re-enable SMB Multichannel (see instructions in previous topic).

Compare the two results.

Note: to avoid the effects of caching, you should:

Copy a large amount of data (more data than would fit on memory).

Perform the copy twice, using first copy as a warm-up and timing only the second copy.

Restart both the server and the client before each test to make sure they operate under similar conditions.

4.2. Fail one of multiple NICs during a file copy with SMB Multichannel

To confirm the failover capability of SMB Multichannel, follow these steps:

Make sure SMB Multichannel is operating in a multi-NIC configuration.

Perform a long-running file copy.

While the file copy is running, simulate a failure one of the network paths by disconnecting one of the cables (or by disabling one of the NICs)

Confirm that the file copy continues using the surviving NIC, without any file copy errors.

Note: Make sure there are no other workloads using the disconnected/disabled path, to avoid failures in workloads that do not leverage SMB Multichannel.

5. Troubleshooting

Here are troubleshooting tips for SMB Multichannel.

5.1. Verifying if you’re using SMB Multichannel

You can use the following steps to verify you are using SMB Multichannel.

Step 1: Verify network adapter configuration

Use the following PowerShell cmdlets to verify you have multiple NICs and/or to verify the RSS and RDMA capabilities of the NICs. Run on both the SMB server and the SMB client.

Get-NetAdapter

Get-NetAdapterRSS

Get-NetAdapterRDMA

Get-NetAdapterHardwareInfo

Step 2: Verify SMB configuration

Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RSS and RDMA capabilities are being properly identified.

On the SMB client, run the following PowerShell cmdlets:

Get-SmbClientConfiguration | Select EnableMultichannel

Get-SmbClientNetworkInterface

On the SMB server, run the following PowerShell cmdlets:

Get-SmbServerConfiguration | Select EnableMultichannel

Get-SmbServerNetworkInterface

Step 3: Verify the SMB connection

On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right version of SMB and that SMB Multichannel is working:

Get-SmbConnection

Get-SmbMultichannelConnection

Get-SmbMultichannelConnection -IncludeNotSelected

5.2. View SMB Multichannel Events (Optional)

SMB 3.0 offers a “Object State Diagnostic” event log that can be used to troubleshoot Multichannel (and therefore RDMA) connections. Keep in mind that this is a debug log, so it’s very verbose and requires a special procedure for gathering the events. You can follow the steps below:

After the log is enabled, perform the operation that requires an RDMA connection. For instance, copy a file or run a specific operation. If you’re using mapped drives, be sure to map them after you enable the log, or else the connection events won’t be properly captured.

Because of feedback during the Beta, we added in the Release Candidate a new set of cmdlets using the SmbMultichannelConstraint noun to allow you to control the SMB Multichannel behavior in more complex network configurations.

For instance, if you have 4 NICs in your system and you want the SMB client to use only two specific NICs when you access server FS1, you can accomplish that with the following PowerShell cmdlets:

This is an advanced setting, available only via PowerShell (or WMI). Please be careful when configuring these SMB Multichannel constraints, because you have the potential to apply a less-than-ideal configuration and end up with something that does not provide you the best level of network fault tolerance (also known as “shooting yourself in the foot”).

This is specially true if you add/remove network adapters from your system and forget to adjust your constraints accordingly. We highly recommend that you stick to the default, let-SMB-Multichannel-figure-it-out configuration if at all possible, since this will let SMB 3.0 dynamically and automatically adjust to any changes in network configuration shortly after it happens.

8. Conclusion

I hope this blog post has helped you understand the basics of SMB Multichannel. I would encourage you to try it out in any of the configurations shown.

Windows 7 implements SMB 2.1.There are way too many new features in SMB 3.0 to back port to previous Windows versions.

Obviously Windows 8 and Windows Server 2012 will be able to talk SMB to Windows 7, but they will negotiate down to SMB 2.1 in that case. The protocol clearly describes the scenario of negotiating with previous versions.

That sounds very intruiging! Ok, so I'm redirecting the desktop and document folders for ~250 staff/faculty and ~3000 students. On a high use day I could have as many as 750 users online at a given time. So assuming that I have properly configured a win2012 fail-over cluster with SMB 3, if node1 fails, redirected folders failover to node 2, my users won't lose their sessions?

Nice! Can you go over how this would impact using win2012 as a fail-over cluster for file serving? Specifically, i'm grasping at straws, if the node goes down that hosts your file share, currently that session is lost and you have to re-establish against the new server, it almost sounds like that may not be an issue now?

If you have dual network adapters for every node of your file server cluster, you can certainly configure it to use SMB Multichannel. If only one network path fails, you will continue to use the same node if you configure your network resources with "OR" dependencies.

We also have a feature in SMB 3.0 called Transparent Failover. This allows file shares to continue to work from another cluster node when the entire node fails. SMB 3.0 essentially resumes the sessions on one of the surviving nodes, without any impact to the application.

All RDMA adapters that support SMB Direct in Windows Server 2012 also have a side-by-side TCP/IP stack. From an IT Administrator's perspective, it's a regular NIC with an IP address, subnet mask, default gateway, DNS, etc… You can use your usual techniques to verify connectivity, like pinging one system from the other. SMB will, however, detect the RDMA capability and switch to RDMA connections as soon as it can verify the other side is able to do RDMA as well. However, SMB will always start the negotiation process of TCP/IP, since it does not know beforehand if the other side is SMB 3.0, capable of using SMB Multichannel or capable of using SMB Direct.

In 3.4 with multiple RDMA adapters on both source and destination servers, how should IP settings be configured on those adapters? Should both adapters on each server be in the same VLAN and both configured with a default gateway? We would like to do RDMA over 10 Gb and have fault tolerance.

That server name is usually associated with the cluster internal network.
The fact that it happens every 10 minutes probably means that it failed to connect and it’s retrying after a while.
You should probably check if the cluster nodes can talk to each other via this 192.168.120.* network.

The Cluster nodes can ping each other node via this Network without any Problems. Why the SMBClient from one node will reach the other node via Cluster communication Network? SMB should use other more faster Networks.
Andreas

Do you have any problems after disabling Multichannle, I had have the same situation bit with emulex ROCE and Win 2016 TP5 storage spaces. But I’m not sure do I have solution for my problems. But in logs don’t have errors about RDMA adapters

– The switch has the latest firmware installed and has been reset to factory defaults to avoid any past LAG settings from interfering

– All Intel NICs have the latest Intel drivers installed (from the ethernet connections package 21.0, I extracted the drivers only to avoid issues with the separate Intel management software)

– Neither at the PCs nor the switch are any LAG/teaming settings enabled manually

– Each IPV4 address is set manually, each NIC can ping every other NIC (IPv6 is disabled)

– Large file transfers via Windows Explorer (accessed the shares via \\PCxy, not their IP addresses) are stuck at 113 MB/s (118 MB/s with Jumbo Frames enabled), during which only one NIC on each PC is at 99 %, the other one is at 0 % activity

Even if IP addresses of different segments are assigned to each NIC, is this multi-channel function used and communication is to be carried out？
Or, as for the NIC that can not be reached by the opponent, it is not used beforehand.

1. That NICS need to be the same for SMB to work seamlessly
2. That you should leave one NIC out of the scope of constraint for out of band management of the server were the SMB multichannel may be saturated with traffic.