~ We put the 'P' in the Internet

So we have been working diligently to resolve an issue in relation to a vendor deployed Cisco Telepresence Show and Share setup. We ended up with through the troubleshooting process with two requirements. Basically what we were seeing was if the access port attached to the receiver was configured or negotiated at 100Mbps as opposed to Gigabit that the video stream would not play with any measure of predictability.

When everyone would go home or be out for lunch, the stream would play okay, but when the switch became even slightly busy, the video would most of the time not even provide a single key-frame and hence no video. Keep in mind that this was a 768kbps stream, so it was insane to think this could actually be the switch causing the issue.

With Auto-QoS defaults, we were dropping around 500 packets in 10 seconds. We actually forced this work by modifying marking the multicast traffic generated at the server on the server’s access switch and by modifying the output buffers for QoS on the client attached 2960S Access Switch. This required giving 70% of the buffer to the priority Queue, making sure CS5 marked traffic was getting inserted into the Priority Queue, and setting the thresholds for the priority queue to allow oversubsciption at 3200%.

It seemed insane that we had to provide this level of buffer assignment on the priority queue to enable delivery of a 768kbps multicast stream. We grabbed a few packet captures and what they showed was that when you zoom in close enough, there was just too much traffic showing up at the receiver to make it down the interface without exceeding the buffers. For example, in the following picture you can see a graph of a key-frame (I Frame) just after 13:19:48.049 and 13:19:48.449. The scale is now horizontally 1 pixel = 1 millisecond of time which makes the vertical scale in Kbps. Keep in mind this is on a 100 Mbps link. You’ll quickly notice that somehow at an instance within the key-frame receipt the interface appears to be exceeding 100 Mbps.

This isn’t actually possible, but its very telling of a problem. Wireshark is showing this impossibility because there is a buffer on our NIC and depending on when WINSOCK is notified of the arriving packet, time can get skewed and mislead wireshark as to the actual time the packet arrived resulting in miscalculated interface peak in that time slot.

Back to the issue at hand. The problem is the NIC on the server is connected to a switch at Gigabit, while the NIC on the receiver is connected at 100 Mbps. The Windows Media Server has a buffer that it stores packets in before dropping it into the WINSOCK stack for delivery. If you are using the Unicast Data Sink, Microsoft provides the ability to not buffer the packet at the server before delivery. (page 17) This spreads out the packets generated by the steam considerably in time. This to my knowledge is not modifiable for multicast. Since multicast rides UDP and UDP doesn’t provide a packet recovery mechanism by itself, all of these packets show up at the same time and the queue is exceeded on the switchport resulting in packet loss.

However, Microsoft does have a (rather old) document with an insinuating reference in it. According to the Optimizing Windows Media Services Document provided by Microsoft, in order to provide Multicast Windows Media Services 9 recommends:

Considering at the time the referenced document was written 10-Gigabit Ethernet wasn’t considerably available, we made the assumption they were expecting you to utilize 100BaseTX interfaces for multicast transmission. After weeks of trying to resolve this issue, configuring the switch the Windows Media Server was plugged into to 100/Full fixed it.

After speaking with Cisco TAC and reading various blogs and threads on CCO, We have come to the conclusion that the 2960S/3560 provides inadequate buffers in the ASICs. When attached to a 3750, 4500, or 6500 series Catalyst Switch this issue does not present itself at least until the WMS Multicast stream approaches 6 Mbps.

We also noticed, Microsoft has pulled support for MS-MSB in Windows Media Services on Server 2012. The entire framework has actually moved to IIS Smooth Streaming. So no more Multicast from WMS. This is a damned shame.