I don't need no stinking 10 Gigabit Ethernet, I have a Thunderbolt cable!

Share this story

If you open your network settings in the System Preferences after upgrading to OS X 10.9 "Mavericks", you'll be informed that a new "Thunderbolt Bridge" network interface was added to the system. So it's now possible to network two Macs over Thunderbolt. Let's take our new network for a spin.

Obviously you need a Thunderbolt cable to connect two Thunderbolt-equipped Macs. I got the 0.5 meter one from Apple, which is quite short, and connected my mid-2011 MacBook Air to a brand new MacBook Pro ("with Retina Display", but that almost goes without saying now). The MacBook Pro has 20 Gbps Thunderbolt 2 while the Air has to make do with the regular 10 Gbps Thunderbolt—but on both the Network Utility reports a link speed of 10 Gbps.

On the Air, the network interface now came up, but not on the Pro. The reason for that is that the Thunderbolt Bridge interface is actually a virtual bridge interface that allows network packets to be passed from one physical network interface to another. This also means that packets toward those interfaces should now go through the bridge interface. The Air's lone Thunderbolt port was part of the Thunderbolt Bridge out of the box, but on the Pro only one of the ports was part of the bridge—and of course I had connected my cable to the other port. This was easily fixed.

With the Thunderbolt Bridge interface up and running on both computers, I turned off Wi-Fi on the Pro to make sure that I wouldn't accidentally connect wirelessly. I then connected from the Air to the Pro through the Finder, mounting the Pro's SSD as a network share. Now I was in the position to copy files back and forth and time how long that took.

The result: much longer than expected, in the Pro-to-Air direction, at least. However, by monitoring network throughput in the Activity Monitor, it turned out that when copying from the MacBook Pro to the MacBook Air, the transfers could be extremely fast, sometimes showing transfer speeds of a gigabyte per second. But at random intervals, the transfer would stall for a number of seconds, and then continue at either very low or very high speed.

In the other direction, from the older, slower MacBook Air to the brand new, much faster MacBook Pro, there were no such issues. Here, Activity Monitor fairly consistently showed around 500MB per second worth of network traffic. However, that number doesn't correspond to the transfer times and file sizes. Based on these, the transfer speed was 200MB/sec. Some of the discrepancy is explained by network overhead and TCP taking some time to ramp up to multi-Gbps speeds, but I'm pretty sure what's really going on here is that the system counts the network activity of both the physical Thunderbolt interface and the virtual bridge interface, thus reporting twice as much network activity as what really took place.

It remains unclear why the transfers from the Pro to the Air were so inconsistent. An obvious reason would be that the Pro sends packets faster than the Air can handle, but looking at the interface statistics (netstat -i en1 -ss), it turns out that the difference between packets sent and packets received was only 0.02 percent—not really enough to cause the observed behavior. Getting out tcpdump and inspecting the data traffic also didn't show any reason for the slowdown: I couldn't find a single instance of SACK blocks, which would indicate lost packets in the middle of a transfer. The only indication that something is going on is that once in a while, there are simply no packets for 10 milliseconds or so, for no reason that I could find.

Maybe there is a software issue, or maybe there is a bottleneck in one of the Thunderbolt implementations—the mid-2011 MacBook Air was the first model to gain Thunderbolt. With Mavericks, Apple switched to SMB as the default protocol to transfer files locally, but using AFP didn't make a difference.

Although the interfaces report using the standard Ethernet packet size or MTU (maximum transfer unit) of 1500 bytes, in reality the SMB transfers typically happened as four 65212-byte packets and then a 1556-byte packet—which makes for a 256kB payload. However, sometimes the packet sizes would be different.

The Thunderbolt network interface also indicates that it supports TCP segmentation offloading for both IPv4 and IPv6 (TSO4 and TSO6), but presumably, there's no actual network hardware in the Thunderbolt interface that could perform this function. The idea behind TSO is that the network software creates one large packet or segment, and the networking hardware splits that packet into pieces that conform to the MTU limit. This allows gigabit-scale networks to operate without using excessive amounts of CPU time. What seems to be happening here is that the system maintains an outward appearance of using the standard MTU size so nothing unexpected happens, but then simply transmits the large TCP segment over Thunderbolt without bothering with the promised segmentation.

As Apple actually implemented Thunderbolt networking as Ethernet over Thunderbolt rather than IP over Thunderbolt, it's possible to add other Ethernet and Ethernet-like interfaces (such as Wi-Fi) to the Thunderbolt Bridge: use "manage virtual interfaces" in the little gear menu under the list of interfaces in the network settings.

I had the Pro bridge its Wi-Fi interface to the Air over Thunderbolt, and from the Air's perspective it was just like it was connected to the Wi-Fi network: other computers showed up in the Finder and everything. However, the Pro had a kernel panic and the Air had a hard time copying files or loading web pages. Could be a side effect of the creative take on TSO, or maybe something else is going on.

Assuming stability and consistency will be improved, is Thunderbolt networking useful in the first place?

Obviously, outfitting Macs with Thunderbolt 10 Gigabit Ethernet adapters and hooking them up to a 10GE switch is the superior solution. With an equally superior price tag. In situations where a fast network is only needed to copy files between two computers, using a $30 or $40 Thunderbolt cable is much more convenient at a fraction of the price. I can also imagine a Mac Pro being outfitted with a 10GE adapter, and then one or two other Macs connecting to the 10GE network through a Thunderbolt connection to that Mac Pro. And if Thunderbolt networking catches on, we may even see Thunderbolt ports on NAS devices.

Say what you will about Apple, but they're not afraid to try something new. As this new feature shows, it really pays off to have a generic high speed port in every Mac. I can't wait to see what else Thunderbolt has in store for us.

IP over Thunderbolt looks like it has lots of potential, but on the other hand, IP over Firewire was pretty cool a decade ago, too. Windows actually handled IP over Firewire in much the same way as OS 10.9 handles Thunderbolt networking, allowing you to bridge between Firewire and Ethernet ports. But then it was dropped in Vista. So enjoy your fast network while it lasts.

Update: by popular demand, I also tested with iPerf, which tests raw TCP performance, without reading or writing to disk. iPerf transfers data without the overhead incurred by actual file-sharing protocols, and these numbers won't be representative of actual performance in most cases. Sending from the MacBook Pro to the MacBook Air happened at 5.3Gbps, in the other direction slightly faster at 5.7Gbps.

Share this story

Iljitsch van Beijnum
Iljitsch is a contributing writer at Ars Technica, where he contributes articles about network protocols as well as Apple topics. He is currently finishing his Ph.D work at the telematics department at Universidad Carlos III de Madrid (UC3M) in Spain. Emaililjitsch.vanbeijnum@arstechnica.com//Twitter@iljitsch

76 Reader Comments

During file copy operations, wouldn't the limiting factor between a couple of connected 10Gbps machines be the disk read/write speed on each machine? While the latest SSDs are quite fast, they are still much slower than a 10Gbps network connection. Can we re-run this test with iperf and get some real benchmarks?

IP over Firewire was ultimately killed by the fact that it was never well tested and tended to be unstable and inconsistent if you actually tried to use it. Last time I tried to use it, the interfaces would crash about once a day and I would have to reboot to bring them back to life. It was just not worth the hassle.

The weird stalls described in this article are not a good sign. I would think twice before using this for more than a quick connect/transfer of some file.

However, it's possible that it's not the network adapters fault. Maybe it's a problem with Apple's file sharing setup. I certainly wouldn't be surprised if it is not designed to keep up with a 10Gbps link. Maybe if you tried iperf instead?

During file copy operations, wouldn't the limiting factor between a couple of connected 10Gbps machines be the disk read/write speed on each machine? While the latest SSDs are quite fast, they are still much slower than a 10Gbps network connection. Can we re-run this test with iperf and get some real benchmarks?

In my own experience with 10 Gb/s Ethernet and iperf, I had to run multiple threads using the -p option. A single core (using iperf anyway) isn't enough to generate enough traffic to saturate a 10 Gb/s link.

During file copy operations, wouldn't the limiting factor between a couple of connected 10Gbps machines be the disk read/write speed on each machine? While the latest SSDs are quite fast, they are still much slower than a 10Gbps network connection. Can we re-run this test with iperf and get some real benchmarks?

The late 2013 models of the MacBook Air and the retina MacBook Pro come with PCI-e based SSD's that are rated up to 1.2 GB/s read bandwidth and 1.0 GB/s write. With the typical networking overhead, those speeds would match the bandwidth 10 Gbit networking.

Next year the PCI-e SSD's could move to PCI-e 3.0 for double the theoretical performance but that'd match nicely with the 20 GBit Thunderbolt 2 speeds.

Would be interesting to see some of these tests done with just the raw network speed. As others have mentioned protocols or SSD speed could get in the way. Doing a test with something like netcat dumped to /dev/null would be more valid.

A thunderbolt switch as a replacement for a fibre channel switch in a SAN is pretty damn appealing. That's one more thing that a Mac Pro could handle natively without a PCI card. And it would make laptops first class citizens on a SAN too. I'm curious if something like this is on the drawing board.

During file copy operations, wouldn't the limiting factor between a couple of connected 10Gbps machines be the disk read/write speed on each machine? While the latest SSDs are quite fast, they are still much slower than a 10Gbps network connection. Can we re-run this test with iperf and get some real benchmarks?

In my own experience with 10 Gb/s Ethernet and iperf, I had to run multiple threads using the -p option. A single core (using iperf anyway) isn't enough to generate enough traffic to saturate a 10 Gb/s link.

I find that odd. Transferring a file over a smb network that is getting close to the 1Gb limit uses maybe 5% of a single core on my file server (Haswell Pentium). It seems like the performance would scale pretty linearly (unless I am misunderstanding something), so a single core could still saturate a 10Gb link pretty easily. Obviously this is with pretty newish hardware, but old hardware isn't going to be getting 10Gb ports in the first place (it requires a x8 slot I think, and most old computers don't have an extra one. Newer ones can use embedded graphics and leave the main slot available for this).

It remains unclear why the transfers from the Pro to the Air were so inconsistent. An obvious reason would be that the Pro sends packets faster than the Air can handle, but looking at the interface statistics (netstat -i en1 -ss), it turns out that the difference between packets sent and packets received was only 0.02 percent. That's not really enough to cause the observed behavior. Getting out tcpdump and inspecting the data traffic also doesn't show any reason for the slowdown. I couldn't find a single instance of SACK blocks, which would indicate lost packets in the middle of a transfer. The only indication that something is going on is that once in a while, there are simply no packets for 10 milliseconds or so for no discernable reason.

Off the top of my head this might simply imply the faster Pro was pushing at a faster rate than the Air could keep up w/ - idle time might suggest the Air having to play catch up w/ the overload of throughput. So the Pro ends up waiting around until the Air is ready for more.

Hell - you could technically hook your garden hose up to a fire hydrant but you are still only going to get as much water out of the hydrant as the garden hose will allow at one time (hypothetically assuming the garden hose doesn't explode / rupture from the water pressure).

During file copy operations, wouldn't the limiting factor between a couple of connected 10Gbps machines be the disk read/write speed on each machine? While the latest SSDs are quite fast, they are still much slower than a 10Gbps network connection. Can we re-run this test with iperf and get some real benchmarks?

The late 2013 models of the MacBook Air and the retina MacBook Pro come with PCI-e based SSD's that are rated up to 1.2 GB/s read bandwidth and 1.0 GB/s write. With the typical networking overhead, those speeds would match the bandwidth 10 Gbit networking.

Next year the PCI-e SSD's could move to PCI-e 3.0 for double the theoretical performance but that'd match nicely with the 20 GBit Thunderbolt 2 speeds.

Iljitsch specifically says he "connected [his] mid-2011 MacBook Air to a brand new MacBook Pro". The mid-2011 MBA was definitely not rated at 1GB/s read/write, so any file copy operation involving the MBA would be irrelevant. If it was two new MBPs, then that would have been fine.

Regardless, Iljitsch updated the article to show the iperf results. Thank you, Iljitsch!

edit: the mid-2011 MBA tops out around 270MB/s in a 4K sequential write according to Ars' own review here.

Hell - you could technically hook your garden hose up to a fire hydrant but you are still only going to get as much water out of the hydrant as the garden hose will allow at one time (hypothetically assuming the garden hose doesn't explode / rupture from the water pressure).

Hydrants generally are at the same water pressure as the rest of the water lines, at least in suburbia they are placed on the main water lines, so are the rest of the water lines. Hooking up a normal water hose is no problem at all. They do it at work all the time. At home the water company would probably be a bit concerned why I was using the hydrant to water my lawn and bypassing the water meter. The point still stands that the amount of water coming out the end of a 100 ft hose hooked to the house spigot and to the fire hydrant is probably pretty close.

If you ask your city, most have a provision for hooking directly to a Fire Hydrant. Generally you have to pay some fee for the privilege and the license only lasts a few days, but it is legal and can be convenient when you need a lot of water at some site and the only plumbing nearby is a hydrant.

In my own experience with 10 Gb/s Ethernet and iperf, I had to run multiple threads using the -p option. A single core (using iperf anyway) isn't enough to generate enough traffic to saturate a 10 Gb/s link.

I suspect this will probably yield more than 5Gb/s. Also tcp vs. udp should alter the numbers somewhat as udp has much less overhead.

I wonder what happens when more than two devices are involved. How does the bus negotiate who connects to what?

IP over FireWire suffered because its topology made device associations unpredictable. If you had a handful of machines connected via FW for IP purposes and then connected a hard disk to one of them, the disk might actually mount on a different system, with no way we could find to force it to go to one system or another. As a result, if you wanted to use IP over FW at all, you had to use FW exclusively for IP.

So what happens if you connect up two Macs for Ethernet over Thunderbolt, and then add a Thunderbolt display into the mix? Is it visible to both Macs? Which one would it attach to?

Networking over a DMA bus (Firewire and Thunderbolt) needs very, very stable drivers. It is a security and stability nightmare. IMO a better solution is cheaper 10Ge cards, not software 10Ge .

On the plus side, if you are feeling creative, having a very fast DMA-capable bus should allow all sorts of wacky shenanigans. Most of them would be geek/demoscene stunts (or messy physical attacks); but some might actually be useful...

Most obviously, if you know where the framebuffer is, it should be possible to use part or all of any thunderbolt-equipped computer's display as a thunderbolt monitor (or quietly grab screenshots directly out of video memory...) With some suitably brutal maltreatment of other devices on the PCIe bus, it might also be possible to use the peripherals of one machine on another (in the same way that, with supported chipsets, VMs can grab pieces of physical hardware on the host).

Now, if it turns out the the primary use for thunderbolt turns out to be dumping flying toasters into the victim's framebuffer, to confound them Classic OS style, or to rewrite the password validation portions of the system, in memory, in order to defeat access control (already done with firewire), that would be an issue; but you could also do some cute things with it.

Networking over a DMA bus (Firewire and Thunderbolt) needs very, very stable drivers. It is a security and stability nightmare. IMO a better solution is cheaper 10Ge cards, not software 10Ge .

On the plus side, if you are feeling creative, having a very fast DMA-capable bus should allow all sorts of wacky shenanigans. Most of them would be geek/demoscene stunts (or messy physical attacks); but some might actually be useful...

Most obviously, if you know where the framebuffer is, it should be possible to use part or all of any thunderbolt-equipped computer's display as a thunderbolt monitor (or quietly grab screenshots directly out of video memory...) With some suitably brutal maltreatment of other devices on the PCIe bus, it might also be possible to use the peripherals of one machine on another (in the same way that, with supported chipsets, VMs can grab pieces of physical hardware on the host).

Now, if it turns out the the primary use for thunderbolt turns out to be dumping flying toasters into the victim's framebuffer, to confound them Classic OS style, or to rewrite the password validation portions of the system, in memory, in order to defeat access control (already done with firewire), that would be an issue; but you could also do some cute things with it.

Ethernet now supports RDMA, But the DMA buffer is created in user space, then registered with the OS for the Layer2 Ethernet session. You can't just start writing anywhere in memory.

I wonder what happens when more than two devices are involved. How does the bus negotiate who connects to what?

IP over FireWire suffered because its topology made device associations unpredictable. If you had a handful of machines connected via FW for IP purposes and then connected a hard disk to one of them, the disk might actually mount on a different system, with no way we could find to force it to go to one system or another. As a result, if you wanted to use IP over FW at all, you had to use FW exclusively for IP.

So what happens if you connect up two Macs for Ethernet over Thunderbolt, and then add a Thunderbolt display into the mix? Is it visible to both Macs? Which one would it attach to?

The difference is that FireWire does the forwarding between ports at the hardware level; as a result, logically, it all ends up being one large FireWire bus. In contrast, what happens here is that Ethernet frames (not TB packets!) are forwarded between separate (virtual) Ethernet-over-TB interfaces on separate TB buses. That is why the bridge device is necessary for TB, but wasn't necessary for IP-over-FW.

Device associations will be predictable here because there's always only a single upstream TB controller for each TB device; for the TB display, that would be the Mac connected to the cable, not the one connected to the port in the back of the display.

EDIT: I guess that last bit falls apart if you connect two hosts to a device that has multiple ports but no clear notion of which is up- vs. downstream, such as disks. Not sure how that would be handled; perhaps one port would have priority? In any case, unlike FW, at least you won't get unexpected association across ports.

I don't know about TCP ramp up, but between my desktop and server using SMB multichannel (Windows 8.1 on the desktop, 8 on the server) it takes less than a second for it to ramp up to either the limits of the RAID array speed or the two GbE links (around 220-225MB/sec read/write when it is link limited and around 200-220MB/sec when it is array limited, just kind of depends on where I am pushing/pulling from the arrays in the two machines, a 2x1TB 7200rpm array in my desktop and a 2x2TB 5400rpm array in the server).

Speeds also stay consistent unless I run in to a bunch of small files that the disks have to churn though. No hicups either.

I could enable the on-board GbE ports instead of using my nice Intel CT GbE NICs...but I don't really want to mix in different manufacturer's adapters and hope Windows gets it right...that and I'd be wholely disk limited then (and at best my arrays aren't going to be able to push/pull faster than about 240-250MB/sec at best, based on testing on the arrays locally, barely faster than what I am limited to by the dual GbE links).

I really want to see 10GbE come down to the consumer level. I doubt it is more than 3 years off for enthusiast home networking though. A couple of 10GbE RJ45 based cards and an 8-port 10GbE switch only total a bit over $1,000. Not cheap, but I'd be suprised if in 3 years that cost wasn't closer to $300-400, putting it in range of a lot of enthusiasts. Another 2-3 years after that and you'll probably start seeing 10GbE ports on higher end motherboards and laptops and maybe only $100-200 for a decent 10GbE switch.

We do kind of need it though. 1.3Gbps 802.11ac is starting to move awfully close to actually saturating a GbE port under ideal conditions. Unless they are going to add channel bonding to Wifi routers, much more improvement in Wifi speeds and ideal condition use is going to be capped by wire speeds. Add-in that most spinning disks, even laptop 5400rpm drives, these days can saturate a single GbE link transfering large files...a faster wired network interface is really needed.

Cat5e can support roughly 45m 10GbE runs...so most peoples installed cabling can support 10GbE speeds without having to be upgraded. The longest wire in my house is the run from my FIOS OTN box in my garage opposite the inside of my house, to the other side of my house in the basement to the router there. Roughly a 120ft run, or about 36 meters). So even the very longest cable in my house is theoretically capable of running 10GbE today. That one might be a stretch, but when the heck am I going to have faster than 1000Mbps internet service to my house? Heck I am subsisting at 75/35 speeds today (I half joke, if it wasn't so expensive, I wouldn't complain at all). I'd be suprised if by the end of the decade I had 300/100 speeds. My next longest run is probably about 25m and most are in the 12-20m range, well within the capabilities of Cat5e and 10GbE.

One bug in the rug is that sooo much is going to SSD storage these days. Many laptops have only SSDs in them, a lot of desktops have them and so on. Likely you aren't moving movies or big backups between two SSDs, but you might be. Even ignoring than, most brand spanking new 7200rpm 2, 3 and 4TB drives are capable north of 150MB/sec transfer speeds (so long as you aren't getting close to the inside tracks), so a single disk can hit about 140% of a single GbE link speed and if you happen to be a crazy monkey who makes even a two disk RAID array, you are looking at almost 3 times the speed of a single link for the array...getting damned impractical to be doing SMB multichannel with all those wires and NICs.

Please, Ghu, I know you prefer being ignored, but lord and God of the ceiling, please let us have commodity 10GbE before the decade is out and preferably something affordable within the next 3-5 years. Once 4k h.264 (even h.265!) files start rolling around in a couple of years, it is going to get rather tiring waiting for files to transfer around (I mean, crap, that 6GB 1080p file took almost 30 whole seconds to transfer from my desktop to my server over my doubled up GbE link...I'd really prefer being able to have the disk array and a single link be able to do that in more like 10 or 12 seconds. K, thanks, bi).

On the Air, the network interface now came up, but not on the Pro. The reason for that is that the Thunderbolt Bridge interface is actually a virtual bridge interface that allows network packets to be passed from one physical network interface to another. This also means that packets toward those interfaces should now go through the bridge interface. The Air's lone Thunderbolt port was part of the Thunderbolt Bridge out of the box, but on the Pro only one of the ports was part of the bridge—and of course I had connected my cable to the other port. This was easily fixed.

Sorry, I guess I'm confused by this description about "bridging". Neither the MacBook Pro nor the MacBook Air have physical ethernet ports anymore, and the article says WiFi was disabled so what exactly is being "bridged"?

How is this different than just "Ethernet over Thunderbolt"?

EDIT: Oh, is it that the two Thunderbolt ports on the MBP are bridged?

EDIT: Oh, is it that the two Thunderbolt ports on the MBP are bridged?

Yes. On the Air, the bridge doesn't really do anything useful (unless you actually add other interfaces), but on the MBP, it allows Ethernet frames to be forwarded between the two Ethernet-over-TB interfaces (one per port).

The, "choppiness" is actually very common in networking. You have one side that outperforms another and you get output overruns. This is analagous to a low-end cisco 3750x with a 10g uplink to a non-blocking Nexus 7000. Within an environment like that you will see output queue drops on the 3750 uplink port if you don't increase the output threshold which effectively increases memory for port buffering. The problem with buffering is that it can increase jitter and delay.

You can then do a "sho mls qos interface tenGigabitEthernet 1/1/1 statistics" and you will see that you will no longer get output queue drops. Assuming you can apply egress port buffering in addition to flow control on the machines as needed you would probably get the same results bridging via thunderbolt.

The, "choppiness" is really just the way TCP manifests lots of drops with windowing and scaling.

From Article:<snip>What seems to be happening here is that the system maintains an outward appearance of using the standard MTU size so nothing unexpected happens, but then simply transmits the large TCP segment over Thunderbolt without bothering with the promised segmentation.</snip>

This could be the key to the performance issues. Suppose the tcp window is just open enough to allow only one of these large tcp segments. If the Air is not fast enough at receiving it, and drops it, there is no way it can generate a sack block(or even a duplicate ack, for that matter), because there is nothing else coming.The tcp sender is stuck until its retransmit timer expires. Which could easily be 10 ms (I don't know what the min retransmit timeout is on Mavericks, but it might be in that range, given the tricks Apple is doing with timers to save power).