Flow Control in Fibre Channel (BB_Credits)

Similar presentations

1 Flow Control in Fibre Channel (BB_Credits)All presenters should use this section (Notes Pages) to provide additional information and details about the bullet points above. These notes will help remind students of key points when reviewing materials in their office. It will also help students conduct more standardized and effective train-the-trainer sessions when they return to their offices.Lincoln Dale

2 Topology Hosts connected to a single switchStorage connected to a second switchSwitches connected to each otherHostsSAN SwitchesStorageFC

4 Buffer Credits: The Good and The BadSince BB_Credits are per-hop for FC Class 3:Any speed mismatch (1G/2G) between senders & receiversand/orInsufficient numbers of BB_Credits in a devicecan result in Congestion due to Head of Line (HoL) BlockingThis Congestion can impact other ports & devices in a SANFCDisk shelf capable of sustaining 200MB/secTape capable of sustaining 15MB/sec

5 Buffer Credits (BB_Credits): UtopiaDO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!!Buffer Credits (BB_Credits): UtopiaFrames to switchA111122223333In an ideal FC network, there is no blocking in any device connected to the fabric. (all devices can process frames at the same rate and negotiate equal levels of BB_Credits)444455556666FC77778888Disk shelf capable of sustaining 200MB/secFrames to deviceFrames across switchA12/1616/1613/1614/1615/1611/16111111Available BB_Credits22222213/1616/1610/1611/168/167/1615/1612/1614/169/16Available BB_Credits33333344444455555511/1614/1615/1613/1612/1616/16666666Available BB_Credits777777Tape capable of sustaining 15MB/sec888888Frame towards Disk shelfFrame towards Tape/Return BB_Credit token

6 Buffer Credits (BB_Credits): Real world impact on performanceDO NO DELETE THIS STUFF OFF THE LEFT OF THE SLIDE. IT IS USED FOR THE ANIMATIONS!!Buffer Credits (BB_Credits): Real world impact on performanceFrames to switchA11112222In reality, not all devices can process frames at the same rate and different devices negotiate different BB_CreditsThis means that a single device is capable of causing HOL blocking across a FC switch or even across a SAN!3333Frames to other devices/ports backlog behind blocked Frame at Head-of-Queue444455556666FC77777 Frames Blocked2 Frames BlockedUncongested Fabric8 Frames Blocked1 Frame Blocked3 Frames Blocked6 Frames Blocked4 Frames Blocked5 Frames Blocked8888Disk shelf capable of sustaining 200MB/secFrames to deviceFrames across switchA15/1614/1613/1616/16111111Available BB_Credits22222211/1612/1616/1614/1613/167/1615/165/168/169/166/1610/16While this example isn’t perhaps the most realistic case (flooding FC frames at a device with only 2 BB_Credits), it does show the ramifications of Fibre Channel standards mandating “frames must not be dropped”.Available BB_Credits333333FC Frame blocks at head of Ingress port dueto 0 available BB_Credits on egress port4444445555552/21/20/2666666Available BB_Credits777777Tape capable of sustaining 15MB/sec888888Frame towards Disk shelfFrame towards Tape/Return BB_Credit token

7 Buffer Credits (BB_Credit) impact on performanceWhile Head-of-Line Blocking is a transitory event (until some BB_Credits are returned on the blocked port), performance can be adversely affected across an entire multi-switch FC Fabric by a single blocking portThe Cisco MDS 9000 Series has multiple features to help alleviate the problem:Virtual Output Queueing (VoQ) on all portsDeep Buffers – 255 BB_Credits per portFibre Channel Congestion-Control (FCC) – Detects congested ports and throttles the port causing the congestion at its origin

8 Virtual Output Queueing (VoQ)In a typical FC switch, each port has an input queue for frames arriving on a portAll frames, regardless of destination, are queued in the order they’re receivedThis can potentially cause Head-of-Line blocking should one of the destination ports become congestedFrames will begin to backup in the queue and the sourcing devices will eventually have to stop transmitting data.Switch without VoQFrame to Port 5Frame to Port 6Frame to Port 4Input Queue at Port 1Top of Queue

9 Virtual Output Queueing (VoQ)Virtual Output Queuing ModelFrame to Port 5Frame to Port 6Frame to Port 4Input Queue at Port 1Top of Virtual QueueThe MDS 9000 series utilizes Virtual Output Queues (VoQ) to eliminate Head-of-Line blocking.Each Destination Port is given its own Virtual Queue on the Input Port.Congestion on any Destination Port has no effect on traffic destined for other ports.Every Source Port on the MDS 9000 series has four Virtual Output Queues for every Destination Port – to cover 4 QoS levels per-port across a chassis

10 Deep Buffers – 255 Buffer-to-Buffer Credits (BB_Credits) per portTypical rule-of-thumb is:2 BB_Credits are required for every 1gbit/s4 BB_Credits are required for every 2gbit/sOther Fibre Channel switches in the marketplace support up to 60 Buffer-to-Buffer Credits (BB_Credits)Performance is limited to 2gbit/s maximum over <15kmPerformance is limited to 1gbit/s maximum over <30kmThe MDS 9000 Series has up to 255 Buffer-to-Buffer Credits (BB_Credits) per portWire-rate (2gbit/s) achievable for 64+km1gbit/s achievable for 127+km

11 Fibre Channel Congestion Control (FCC)Consider the following topology:Host A is issuing write operations to 100MB/secHost B is issuing writes operations to 50MB/secCongestion on Switch B’s port connecting the tape can cause a performance degredation for Host A writing to DiskFCC mitigates the congestion by throttling the traffic at the originating port1. Switch B detects congestion on port connecting to TapeFCDisk shelf capable of sustaining 200MB/sec2. Switch B signals Switch A to quench the initiatorA3. Switch A rate-limits incoming trafficSwitch ASwitch BCongestionCongestionTape capable of sustaining 15MB/secPort Rate-Limited on VoQLimit HostBTape to 15MB/secB