Christian Kellner

Thunderbolt security has been in the news recently: researchers presented a set of new vulnerabilities involving Thunderbolt which they named Thunderclap1. The authors built a "fake" network card2 and performed various DMA attacks and were able to temper with memory regions that their network card should have no access to whatsoever. In some way this is not all that surprising because the foundation of Thunderbolt are PCIe tunnels to external hardware and one of the reasons that PCIe is fast is because it can do direct memory access (DMA).

Use boltctl domains to inspect the current security level.

The current primary defense against DMA attacks for Thunderbolt 3 are the security levels: if enabled (the default on most systems) it gives the software the ability to decide on a per device level to allow or deny PCIe tunnels (and with that potentially access to all the memory via DMA)3. While not protecting from DMA attacks per se it protects from some — maybe the most — prominent threat scenarios4: 1) somebody plugging that evil device into your computer while you are away or 2) you have to plug in a device into your computer that you don't trust, i.e. a projector at a conference. On GNU/Linux boltd will authorize a plugged-in device only if an admin user is logged in and the screen is unlocked. For untrusted environments the authorization by boltd can be disabled entirely, e.g. when you go to a conference, via the GNOME settings panel. The toggle is called "Direct Access" (see screenshot below).

The panel has a global switch to toggle all authorization performed by bolt.

This is not the full story though, because there is a second way to prevent DMA attacks utilizing the input–output memory management unit (IOMMU). The general idea is to assign a specific memory region to a device which is the only area the device can directly access. Mika Westerberg from Intel has worked on kernel patches (will be in 5.0) to use firmware supported IOMMU virtualization (Intel calls this VT-d) which should make DMA attacks harder. Having said that, Mika was pointing out (thanks!) that there are still two possible issues mentioned in the paper that are not yet addressed (I paraphrase Mika): 1) IOMMU window size granularity is such that it may open a "too big" IOMMU window and 2) IOMMU mappings are not immediately torn down, leaving memory exposed for some time. Intel (Lu Baolu) is working on patches to mitigate those issues. On the bolt side of things, I have recently merged MR!137 which adds the necessary IOMMU plumbing. The big caveat for this is that it needs hardware/firmware support and for most (all?) currently shipping systems we are out of luck.

Security levels and IOMMU based DMA protection are orthogonal and the way support is implemented in Linux 5.0 is that userspace (bolt) is still required to authorize devices (if the security level demands it, i.e. user or secure). This means that you still have the ability to globally disable authorization of any device, e.g. for when you go to a conference.

To sum it all up: For now, make sure you have Thunderbolt 3 security enabled in the BIOS and whenever you are in an untrusted environment (e.g. conference) disable device authorization completely (via the Settings panel). In the future, with Linux 5.0 and new hardware Linux will also get IOMMU based DMA protection that should greatly reduce the risk of DMA attacks and work is underway to plug the remaining known issues.

How they did it is pretty cool: "[W]e extracted a software model of an Intel E1000 from the QEMU full-system emulator and ran it on an FPGA". More details in the paper on page 5. The whole platform is also available at github.com/thunderclap-io

The paper states that situation for Linux is: "patches for approval of hotplug devices have been produced by Intel and distributions are beginning to implement user interfaces." So this is not super up to date (long paper review process I guess). The current situation is: kernel level support landed in 4.13; bolt was included in Fedora 27, RHEL 7.6 and is now included in many distributions. IOMMU support will land in 5.0 (but see the text for remaining issues).

Assuming that you trust manufacturers of hardware (attacks can also happen at the "supply chain").

With the release of bolt 0.61 "Make the firmware do it!"boltd gained support for preboot access control list (bootacl). The quick summary of that feature is best given by explaining what problem it solves: You are using a Thunderbolt 3 dock that has the main keyboard and mouse connected to it and you want to use the mouse and keyboard during pre- or early-boot, e.g. to enter a password to unlock the hard-drive. If the "pre-boot ACL" feature is enabled in the BIOS, boltd will now do the right thing and Thunderbolt 3 peripherals will be functioning from early on. It comes with one big security caveat though: no device verification is possible, which means the effective security level is user mode during pre-boot (see below for details). The requirements checklist is: bolt version >= 0.6, a kernel >= 4.17 as well as recent enough thunderbolt firmware. The feature usually needs to be explicitly enabled in the BIOS as well (it will be called something like "pre-boot ACL").

The pre-boot BIOS settings

To explain in more detail how all of this works let me first quickly recap Thunderbolt 3 security levels: Thunderbolt tunnels PCIe to connect peripherals and thus allows for direct memory access (DMA); this makes in fast but can be used as an attack vector. In order to prevent DMA attacks Thunderbolt 3 introduced different security levels: none (SL0), user (SL1)2, secure (SL2) and dponly (SL3).3 In both user and secure mode the device needs to authorized before the PCIe tunnels to the peripherals are established. This is done by boltd, a system daemon that manages a list of trusted devices. After Thunderbolt devices are enrolled, i.e. stored in boltd trust device database, the daemon will automatically authorize those devices as soon as they appear. Devices are identified via a unique identifier (UUID). But since the device itself is reporting the UUID to the system, a malicious devices could pretend to have the UUID of a already trusted device. To prevent this secure (SL2) mode uses a per-device key that is generated by the host and stored in the device so it can be used to verify the device identity.

As mentioned above, boltd is the user space component that will authorize the devices. As a consequence if boltd is not running, nobody is authorizing the device and the device will not be able to function. Two of important instances where boltd is NOT running but you might want use the mouse and keyboard in BIOS setup and probably way more important during early boot, to enter the LUKS password.

Recent BIOS versions enabled support for storing a limited list of UUIDs directly in the thunderbolt controller. This is called the pre-boot access control list (or preboot ACL), in bolt simply called "bootacl". The devices corresponding to the devices in the bootacl will be authorized during pre-boot (and only then) by the firmware. One big caveat about this feature should be become obvious now: No device verification can happen because only the UUIDs are stored but not the key, so if you are using SECURE mode but enable preboot ACL in the BIOS you effectively will get USER mode during boot.

The kernel exposes the bootacl via a per-domain sysfs attribute boot_acl. Every time a device is enrolled, boltd will automatically add it to the bootacl as well. Conversely if the device is forgotten and it is in the bootacl, boltd will automatically remove it from the bootacl. There are is small complication to these seemingly straight forward operations: in BIOS assist mode, the thunderbolt controller is powered down by the firmware if no device is connected to it. Therefore when devices are forgotten boltd might not be able to directly write to the boot_acl sysfs attribute. In a dual boot scenario this is complicated by the fact that another operating system might also modify the bootacl and thus we might be out of sync. As the solution to this boltd will write individual changes to a journal file if the thunderbolt controller is powered down and re-apply these changes (as good as possible) the next time the controller is powered up.

boltctl showing the list of domains including the bootcl

An easy way to check the current bootacl entries is via boltctl domains. Currently this is read only but issue #139 is open for write support. This would be a perfect issue to start if anyone would like to hack a bit bolt.

TL;DR: Not every USB-C port is a Thunderbolt 3 port. Watch out for the logos!

One thing that I have learned in conversations about Thunderbolt is that it is important to distinguish between Thunderbolt the I/O technology and the physical connector that is used by it, because this seems to be a constant source of confusion. Over the course of its history, Thunderbolt used different types of connectors; Version 1 and 2 used Mini DisplayPort. USB as the older and more ubiquitous I/O tech also uses a myriad of different connectors ([{mini, micro}-]{A, B, AB}, SuperSpeed, ...).

USB-C connector on a Thunderbolt 3 cable. NB: The flash logo

In order to simplify this and make things generally better, a new universal connector was designed: the USB Type-C connector. A universal connector for a universal bus, ha! But besides USB, this connector is also used by Thunderbolt 3. It has some nice properties (it is symmetrical!) and is quite versatile, i.e. it can be used to deliver power via USB-PD to peripherals.
On the technical side, this can be done because USB-C supports different Alternate Modes: e.g. DisplayPort, HDMI and Thunderbolt 3, which itself then can also carry DisplayPort. Now the important bit: all of the alternate modes (and USB-PD) are optional. It depends on what kind of controller the USB type C port is connected to. Ergo it is not clear what data can be transported by just looking at the port alone: maybe DisplayPort or it might even support Thunderbolt 3. The crucial bit of information is conveyed via the little logo that is printed next to the port. Since physically different pins are used for different modes, the logos are also important on the USB-C cables. Logo usage is regulated by the USB logo usage guidelines.

What this all boils down to is the tl;dr from the top of the article: not every USB-C port is also a Thunderbolt 3 port. If you want to connect a TB3 device to a computer, make sure all involved ports (and cables) have the right logo:

Thunderbolt 3 logo

Some people who own a T480s learned this the hard way when they were wondering why Thunderbolt was not working for them when plugging things into the leftmost USB-C port. I had a few reports of bolt not working, followed by noticeable embarrassment, although it is arguably the design of the port itself which is at fault.

NB: Since there are also different types of Thunderbolt 3 cables (active, passive), you might get different maximum speeds (20Gbit/s vs 40Gbit/s) depending on the cable you use. For the few different cables that I have seen so far, they could not be distinguished based on their appearance alone.

bolt 0.5 "You've got the power"

In related news: bolt 0.5 is out (since about a month now) and will be shipped with Fedora 29. Have a look at the release notes for a complete list of changes, but the most important one I want to highlight here is the new force power D-Bus API. What is it and why do we need it? The Thunderbolt controller can be in two different modes: one in which it is constantly powered (native enumeration mode) and one in which it is controlled by the BIOS. In the latter mode, if nothing is plugged into the Thunderbolt port the controller is completely powered down and it looks as if there is no Thunderbolt hardware present at all. This is great because it saves battery, but there are two problems: 1) boltd wants to know what security level the Thunderbolt controller is in, and more importantly 2) the firmware update daemon (fwupd) wants to know the firmware version of the Thunderbolt controller, so that it can check if there are updates available (and if so, show them in GNOME Software). Luckily, newer kernel versions have (on supported platforms) a sysfs interface that can be used to "force-power" the Thunderbolt controller. Both boltd and fwupd have support for that, which is great, but also the root of a race: the force-power interface is not reference counted and also write only (you cannot ask for the current status). Now if boltd force-powers the controller, uevents will be generated which, in turn, will be processed by fwupd and it will try to read the firmware version. If, in the meantime, boltd is done with its thing and powers the controller down again but fwupd is not yet done reading the firmware, then that read will fail. Or the other way around: fwupd powers the controller, boltd gets started due to the uevents, but meanwhile fwupd is powering the controller down again, boltd might e.g. hang reading the boot-acl.

boltctl power --help for more information

The solution to the problem that Mika, Mario and I came up with was to have only boltd talk to the "force-power" interface and provide a D-Bus API for clients (fwupd) to use. Internally, boltd keeps track of open force-power requests and only when there is none left does the controller get powered down. If you have bolt 0.5 installed, you query the force power status via boltctl power -q or actually request force-powering with boltctl power.

What's next?

I am currently working on boot acl support (#77), which will be the main feature of bolt 0.6. It is still work in progress, but it is very close to being finished and could use some testing. The code is in the bootacl branch and there is a merge request !119 for potential feedback.

Thunderbolt allows for peer-to-peer network connections by connecting two computers directly via a thunderbolt cable. Mika from Intel added support for this to the 4.15 kernel. Recently, Thomas Haller from NetworkManager and I worked together to figure out what needs to be done in userspace to make it work. As it turns out, it was not that hard and the pull-request was merged swiftly.

GNOME Settings' Network Panel

So now if you connect your GNU/Linux box to another machine, NetworkManager will configure the connection to be a Link-Local one.

ping Linux from macOS

I have successfully tested Fedora to Fedora, Fedora to macOS and Fedora to Windows 10. The other operating systems also use a Link-Local address by default.

Copying from Linux to Windows

The technical details

When another host is connected via thunderbolt, we get a new device in sysfs that resembles a normal thunderbolt device but is of DEVTYPEthunderbolt_xdomain. Like a normal thunderbolt device, it has a unique_id and a device_name, but is missing the authorized and key attributes. A new udev rule in NetworkManager was added to load the thunderbolt-net kernel module. As a result, we get a new network device (thunderbolt0).

nmcli showing the thunderbolt networking device

Another udev rule will set a new udev property (NM_AUTO_DEFAULT_LINK_LOCAL_ONLY) to all network devices where the driver is thunderbolt-net to instruct NetworkManager to use the Link-Local configuration on these interfaces.

The dev snapshot release 1.11.3-dev has the thunderbolt networking support. Grab it, build it, and start playing around with it! Feel free to post your record transfer speeds in the comments ⚡ 🏆.

Fedora 28 is around the corner and I wanted to highlight what we did to make the Thunderbolt 3 experience as smooth as possible. Although this post focuses on Fedora 28 for what is currently packaged and shipping, all changes are of course available upstream and should hit other distributions in the future.

The new Control Center Thunderbolt Panel

bolt

First of all, the home of bolt has moved from Github to the new Gitlab instance at freedesktop.org1. File issues and spread your love in the form of stars ★ there now ;)

I released version 0.3 of bolt about a week ago, with a lot of internal changes, small D-Bus API changes and support for future kernel changes2. For example it already supports the new USBONLY security level of the upcoming Titan Ridge controller, and the boot attribute; kernel level support for both landed in 4.17.

GNOME Shell

The GNOME shell bits to enable Thunderbolt 3 support were merged some time ago and made it into 3.28. This means that the GNOME shell will act as authorization agent and will automatically authorize new thunderbolt 3 devices — if the user has administrator rights, is currently logged in and the session is unlocked. It will also show a little indicator (called the cable_snake by some) in the status indicator section while devices are being authorized because the whole authorization dance can take quite a while.

Activity indicator, aka The Cable Snake

GNOME Control Center

We have a new panel to manage Thunderbolt 3 devices, woohoo 🎉! It landed in master not too long ago and will be in 3.30, but we backported it to Fedora 28. It lets you remove currently enrolled devices and shows if devices need authorization (they will be listed under "Pending Devices"). The latter might be happening, for example, because the devices were connected while the session was locked. The device detail dialog can then be used to authorize them.

Authorizing a device

Another important feature is the possibility to temporarily disable authorization of Thunderbolt devices. In the panel this is called "Direct Access", because disabling it will stop the authorization of Thunderbolt 3 peripherals, but the DisplayPort and "pure" USB functionality of the thunderbolt port will be preserved. In technical terms, it means that without authorization no PCIe tunnels are allowed to be created, but due to a feature called alternate modes, the thunderbolt port will still allow USB devices and DisplayPort streams to connect.

Switching "Direct Access" on and off

This feature might come in handy if you are traveling, say to defcon: while you don't want malicious devices (like specially prepared thunderbolt cables) to read your main memory, you still want attached projectors to work correctly.

Firmware updates

Support for updating the firmware of the Thunderbolt host controller and devices landed in fwupd quite some time ago and is of course available in F28 as well. On supported platforms, like Dell, it is therefore only one click away in GNOME Software. Mario also added support to force power the host controller during firmware updates, so updating the host controller is possible without the need to have something plugged into the Thunderbolt port.

Outlook

All in all I am quite happy with the current state of Thunderbolt on GNU/Linux and what we will be shipping in Fedora 28. For most people, connecting Thunderbolt 3 devices should just work. In the next few weeks I will focus on adding support for authorizing devices during pre- and early boot.

Footnotes

The bolt Github repository will continue to exist and be a mirror of the main Gitlab repo. Release tarballs will also be available in both locations at least for the next few releases, but not forever, so please update your links.

If you are upgrading from bolt 0.1: the database default location has changed to /var/lib/boltd so you might have to re-authorize your devices. Also, devices names are now cleaned-up and stored in the clean form in the DB. If you were running earlier versions of bolt, you might see "Vendor Vendor Device-Name" in the Control Center (see #89). The easiest way to fix this is to forget and re-authorize the device.

Thunderbolt 3 security levels

Thunderbolt is an I/O technology that can be used to connect external peripherals to a computer — similar to USB and FireWire. It works by bridging PCIe between the controllers on each end of the connection, which in turn means that devices connected via Thunderbolt are ultimately connected via PCIe. Therefore thunderbolt can achieve very high connection speeds, fast enough to even drive external graphics cards. The downside is that it also makes certain attacks possible (e.g. Thunderstrike, DMA attack).
To mitigate these security problems, the latest version — known as Thunderbolt 3 — supports different security levels:

none: No security. The behavior is identical to previous Thunderbolt versions.

dponly: No PCIe tunnels are created at all, but DisplayPort tunnels are allowed and will work.

user: Connected devices must be authorized by the user. Only then will the PCIe tunnels be activated.

secure: Basically the same as user mode, but additionally a key will be written to the device the first time the device is connected. This key will then be used to verify the identity of the connected device.

The active security level can normally be selected prior boot via a BIOS option, but it is interesting to note that in the future the none option is likely to go away. This of course means connected thunderbolt devices wont work at all unless they are authorized by the user from with the running operating system.

Intel has added support for the different security levels to the kernel and starting with Linux 4.13. The interface to interact with the devices is via files in sysfs. Since July we have been working on the userspace bits to make Thunderbolt 3 support "just work" 😉. The UX design design was drafted by Jimmac. The solution that we came up with to implement it consists of two parts: a generic system daemon and for GNOME a (new) component in gnome-shell. The latter will use the daemon to automatically authorize new devices. This will happen if and only if the currently active user is an administrator and the session is not locked.

bolt 0.1 Accidentally Working

Today I released the first version 0.1 (aka "Accidentally Working") of bolt, a system daemon that manages Thunderbolt 3 devices. It provides a D-Bus API to list devices, enroll them (authorize and store them in the local database) and forget them again (remove previously enrolled devices). It also emits signals if new devices are connected (or removed). During enrollment devices can be set to be automatically authorized as soon as they are connected. A command line tool, called boltctl, can be used to control the daemon and perform all the above mentioned tasks (see the man page of boltctl(1) for details).

I hope that other desktop environments will also find that daemon useful and use it. As this is not a stable release yet, we still have room for API changes, so feedback is welcome.

botlctl example output

New software needs testers, so everybody who has a computer with Thunderbolt 3 and feels courageous enough is welcome to give it a try. I created a copr with builds for Fedora 27 & rawhide and Jaroslav created a PKGBUILD file so Arch users can find it already in the AUR. As this is very fresh software it will contain bugs and those can be filed at the issue tracker of the github repo.

what's next: gnome-shell integration

I am locally running a Proof-of-Concept gnome-shell extension that implements the user session bits to complete the aforementioned : It uses bolt's D-Bus interface and listens for new Thunderbolt devices and then enrolls them, if the user is logged in. Since it can take a while until all the devices that are attached via thunderbolt are properly connected the daemon has a Probing property that is used to display a little icon as a way to inform the user that something is happening on the thunderbolt bus. All of this is already working quite well here on my test machine. In the next few days (weeks?) I will be working on integrating that code into gnome-shell. There are a few open UX questions that need to be addressed, but all in all things looking good.

thunderbolt activity indicator (aka cable snake)

Special thanks to Alberto Ruiz, Benjamin Berg, Hans de Goede, Harald Hoyer, Javier Martinez Canillas, Jaroslav Lichtblau, Jakub Steiner, Richard Hughes who all helped and supported this project during the last few months! ❤️

Ever since we (Red Hat's Desktop Hardware Enablement Team) received the 2017 models from Lenovo for testing (e.g. the T470s), we experienced an issue (rhbz#1480844) where sometimes the fan would run at 100% after resuming from suspend. A warm reboot alone would not make the fan go back to normal and a hard reboot was required. It seems the behavior is the result of firmware bug and kernel ACPI changes. Patches for 4.13 reduced the likelihood for the appearance of the noisy issue. Additionally, we have been working together with Lenovo to fix the firmware side and I am happy to report that for the T470s Lenovo recently released a new firmware that should completely fix the issue. Since Lenovo is not yet(!) part of Linux Vendor Firmware Service updating the BIOS is currently not super straight-forward. Thankfully, Jeff has provided detailed instructions how to do this from GNU/Linux only.

FOSDEM 2018 is approaching fast. There will be a Hardware Enablement Devroom, among many other very interesting ones. We invite everybody to come and participate:

Important dates

Conference date: 3 & 4 February 2018 in Brussels, Belgium

Devroom date: Sunday 4 February 2018

Submission deadline: Sunday 26 November 2017

Speaker notified: Sunday 10 December 2017

About

In this devroom we want to discuss topics surrounding hardware enablement. Subjects can range from the firmware running on the bare metal machine, drivers and plumbing all the way to the user interface.
We welcome a board range of presentations, including but not limited to technical talks, state of union summaries as well as discussions that facilitate the collaboration between community members, software vendors and OEMs. A particular emphasis will be given to talks covering a significant part of the software stack involved in hardware enablement, with an obvious focus on using open source throughout the whole stack.

I joined Red Hat's Desktop Hardware Enablement team almost a year ago. One of the things that I have been looking into recently is Thunderbolt 3. With kernel 4.13 we got a completely new kernel interface for interacting with it from userspace (the work was done by Intel). One of the two big things this interface provides is updating the firmware (the non-volatile memory, or NVM in short) of the host controller and attached thunderbolt devices. With help from Dell's Mario Limonciello, Intel's Yehezkel Bernat, and of course our own Richard Hughes I created a thunderbolt 3 plugin for fwupd, which device and host firmware updates should show up in GNOME Software (or any other fwupd userspace clients) and updating them should be a breeze. The code landed already in fwupd 0.9.7.

On the technical side this is done all done via sysfs. Thunderbolt devices (including the host) are exposed via /sys/bus/thunderbolt/devices and for the NVM we get two additional subdevices for each devices, i.e. nvm_activeX and nvm_non_activeX (where X is a global nvm device counter). The latter one contains a file called nvmem where we can write new firmware binaries to. The upgrade process is triggered by authenticating the NVM via a write to the nvm_authenticate file of the device, like # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate. That in brief is what the thunderbolt 3 fwupd does to apply firmware updates. It uses udev to monitor for changes of attached devices. Discovery of updates is all done by the existing fwupd code, and the fwupd plugin API was actually nice to work with and made writing the plugin straight-forward and, yes even fun.

Once vendors start uploading firmware to the LVFS it will be easy, safe and reliable to update the NVM. If you're interested in helping out with fwupd or the Thunderbolt stuff then please join the mailing list where we discuss this kind of thing.

Just a very quick announcement. I created two new packages for Fedora:

rtags - A source code indexer for the C language family. I use it together with Emacs, to get IDE-like functionality. The package contains systemd (user) unit files that should socket-activate the rtags daemon on demand. Check via systemctl status --user rtags.socket.

renderdoc - a stand-alone graphics debugger; it is a wonderful tool to help developers when doing anything 3D (OpenGL, Vulkan) related.
Thanks goes to @baldurk for a patch to install renderdoc's library into a private location.

Install on Fedora 26, 27 and rawhide via dnf install rtags renderdoc.

I also created my first flatpak:

NixView - as the name suggests a viewer for the free and open NIX scientific data format. It is available via flathub. Open data formats are the necessary basis for data sharing, which is one of the most important problems in science that is still not solved properly.

All this was done during the workshop & hackathon on open data formats and data sharing in neuroscience (in Japan!). Thanks goes to the German Neuroinformatics Node and Red Hat (my employer), that made it possible for me to attend it; and also to the Japanese Node for organizing it. I had a great and productive time in Japan.