Posted
by
kdawson
on Tuesday November 23, 2010 @03:11PM
from the burrowing-in-deep dept.

KindMind notes coverage in The Register on a researcher who has developed a firmware-based rootkit that resides in a network card. Here is the developer's blog entry. "Guillaume Delugré, a reverse engineer at French security firm Sogeti ESEC, was able to develop proof-of-concept code after studying the firmware from Broadcom Ethernet NetExtreme PCI Ethernet cards... Using the knowledge gained from this process, Delugré was able to develop custom firmware code and flash the device so that his proof-of-concept code ran on the CPU of the network card."

An attacker would then be able to communicate remotely with the rootkit in the network card and get access to the underlying operating system thanks to DMA."

Not if the CPU had IOMMU hardware that was configured to only allow the network card to write to the proper memory area.

However, this still would not protect against the network card forging data, manipulating packets before passing them to the OS,
for example manipulating packets to be malformed so to exploit an OS security vulnerability,
emitting packets the OS did not generate (such as ICMP pings, or other packets for a hardware-based DDoS emitted without assistance from host OS.. or connecting to a P2P network of compromised NICs
to form a spam-sending botnet, without host involvement.

The possibility also exists of capturing packets crossing the NIC and forwarding samples to an outside address, or manipulating aspects of packets
to create an "open proxy" the host does not know about, enabling IP spoofing, cache poisoning, or opening other vulnerabilities that don't require manipulation of the host itself.

Yes, but wouldn't the network card's limited hardware be a problem? I mean if you want to make a spam bot / P2P, etc., the code+data will have to fit in whatever RAM or EEPROM capacity the network card has.

the code+data will have to fit in whatever RAM or EEPROM capacity the network card has.

Or a downloader/backdoor will have to fit on the card to allow a remote load of any code that can't be stored on the PROM.

It could be a simple stub, executing exactly instructions carried in magic data packets.
Downloaders can pull more code than is stored by using sources found outside the NIC, such as sources on the internet.

the hacked firmware could remove standard features like Wake on Lan, and use that space to implement features the malware author wants, like "Flood on LAN".

Most NICs nowadays support things like PXE boot.
Either that part of the option ROM could be completely hijacked, OR in fact the PXE boot function could be used as a way of booting the system to a 'boot sector infection' routine next boot after the NIC is infested.

Think about it... Phase 1, your NIC gets infected,
Phase 2, next boot a vulnerability will be opened in your system, thanks to the ability of every PCI card to include an option ROM in the BIOS, or code will run to use blue pill against your OS and introduce malicious code, the hypervisor above your OS downloads code from the attacker.

Depending on the payload downloaded, the malware could be anything from a keylogger to a spam node

Quite true. But I would be willing to bet that most NICs don't have a very big program in EEPROM, but have at least 8 to 32 megabits of the stuff. After all, flash prices have dropped a ton and it's probably a better idea when building something to go with the 1.0078 cent flash rom that gives you lots of space you probably don't need than the 1.0072 cent one that gives you a constraint and may be hard to source next year due to it's small size.

the code+data will have to fit in whatever RAM or EEPROM capacity the network card has.

Or a downloader/backdoor will have to fit on the card to allow a remote load of any code that can't be stored on the PROM.

This solution is defeated by a proper IOMMU.

the hacked firmware could remove standard features like Wake on Lan, and use that space to implement features the malware author wants, like "Flood on LAN".

Yes, that's space in the adapter ROM which you're reusing as was suggested previously.

Most NICs nowadays support things like PXE boot. Either that part of the option ROM could be completely hijacked, OR in fact the PXE boot function could be used as a way of booting the system to a 'boot sector infection' routine next boot after the NIC is infested.

Anything with user-upgradeable firmware and enough hardware to insert itself into the boot sequence is a potential threat.

If necessary, the spammers could just have the card do NAT so you get blamed for the spam.

Interestingly.. they wouldn't even have to do that much. Just provide the spammer a 'magic packet' to tell the NIC to start replacing and forwarding either all packets, or packets destined to certain ports to the spammer's destination IP.

Dumb rewriting is fine as long as the spammer gets the packet otherwise unchanged, as the spammer can implement all the 'NAT logic' in their own software.

That is what the Intel VT-d extension is for, and qubes-os.org is building a secure Hypervisor to operate it from a higher privilege than the normal root privilege, so the DMA can not break out via the normal driver level hacks.

But wouldn't he first have to sneak into your home and flash your NIC firmware, then sneak out again, and start the electronic attack?

There are plenty of people with access to your network card before it reaches you. People at the factory, those doing transportation, people working in customs, people near all the places the card is stored before it reaches you, etc. You may also take the card to repair once you got it. And if your computer already is compromised the malware can flash itself to the network card so it will survive a reinstall of the operating system and boot loader.

say you're a front for the chinese military making these things. you install the rootkit. broadcom or whoever will do an audit of retail boxes to make sure the cards are being produced to spec. how do you hide what you did?

but it will be done randomly so to get value from your virus you have to know who to sell the virus cards to. and since the chinese don't control the serial numbers you somehow have to produce and sneak them into the market with the right numbers

run your in house tools to verify that the code on the card is the same as your in house code you developed

And a properly hacked card outputs to the in-house tool the exact code it's supposed to,
because the hack contains a bit of code to remove all the patches and return itself to pristine state, when a debug connection is detected

You're assuming the NIC manufacturer is conducting audits in the first place. If they are, there's probably single person who maintains a list of good hash values for the firmware. Bribe that person and the audits won't matter.

The easier solution is to simply buy the cards from the OEM, flash them with a malicious firmware, then resell those cards at discount prices. Are NIC manufacturers purchasing off-the-shelf goods and conducting audits on those? Probably not.

Another attack level could be if you already rooted an OS, and want to protect your root kit against reinstall. Someone already mentioned PXE boot, as well as option ROM. In short, as soon as the PC gets rebooted (which is required for a wipe/reinstall), you get complete control.

say you're a front for the chinese military making these things. you install the rootkit. broadcom or whoever will do an audit of retail boxes to make sure the cards are being produced to spec. how do you hide what you did?

One way is to operate completely within spec. The 'retail box audit' normally includes hardware components, not the actual firmware,
so an audit is not likely to detect.
It is not like they're going to audit NICs with a $100,000 logic analyzer, and spend thousands of skilled man hours verifying every bit on the programmable chip service matches their master. Hacked firmware can be designed to lie about its own contents when inquired, and these things can be designed to lie dormat for months on average.

The hacked firmware might open a backdoor only periodically, not every time. Each box will probably be audited once, not 50 times.
When an end user gets the thing, they will eventually trigger the malicious code, because they'll use their machine for a long time.

Isolating the NIC as a cause would be extremely difficult, if the malicious code is sensitive to network activity, and specific kinds of network activity,
for example keywords.

Perhaps the hack is configured only to activate if the computer sends something to an IP address in certain ranges, or containing a certain keyword.
There are innumerable criteria that auditing won't detect

Don't have to turn it on for all cards, just like one of the prime vectors for malware are ad infected ad rotators where the ads just show to a small percentage -- just one in every several thousand cards with a bongoed ROM can bring in a superb ROI for blackhats.

That's pretty frightening. I would think this would be a pain in the ass to discover, and you'd end up replacing motherboards on servers/workstations trying to figure out why they kept crashing. I mean, who would flash their network card as a troubleshooting step?

That's pretty frightening. I would think this would be a pain in the ass to discover, and you'd end up replacing motherboards on servers/workstations trying to figure out why they kept crashing. I mean, who would flash their network card as a troubleshooting step?

last week they almost made me upgrade to the latest RAID controller firmware to replace a few drives showing predictive failure. i was one version behind and this new firmware was a week old. but generally if you're a few months behind they will make you upgrade.

and i've seen a lot of mysterious reboots and other problems thought to be MS's fault fixed by HP firmware/driver upgrades

Windows 7 will require the last know controller mode in BIOS that it was installed under. For example, if you switch it to AHCI or SATA from whatever mode it was installed under will cause a BSOD. That's because the service isn't flagged to be started.

I read these security reports and have to wonder how much, if any, driver experience these security specialists have.

When we talk about patents, we like to drone on and on about prior art and how obvious something is to someone skilled in the art. But these security reports about flashing the EEPROM and running code on the NIC CPU and using DMA to corrupt the OS are all things that are done daily by embedded systems and driver developers.

Mainly because the security experts, for the most part, don't know what they are doing and spend most of their time reinventing bugs that developers have already grappled with and overcome.

It's a lot like how a lot of teachers have a Masters in Education but not in anything specific to the courses they teach. Basically, all they have is a bunch of random ideas without any expertise to show them the right way.

That's a flamebait, but unfortunately it's usually true, at least from my limited experience (as a security person you aren't likely to encounter a lot of colleauges unless you work in the business). However, all "real" "security researchers" I've encountered have been programmers as well - and certainly level enough to consult the technical documentation/research backgrounds of whatever they're trying to break. You also have to remember that a lot of stuff is already known since a decade or more, but since

Then why hasn't someone gotten to it and embedded a firmware rootkit like this before?

How do you know someone hasn't done it already? The whole point of rootkits is that they're undetected for as long as possible. And firmware rootkits are most likely employed by people who really know what they're doing and thus it's not likely the rootkits are found.

I suspect that they are (reasonably) well aware that somebody, presumably an embedded system/driver dev had to produce the blobs and loaders and other structures they are monkeying with in the first place. However, from their perspective as security guys, the point isn't "Wow, nobody has ever written an embedded device firmware, burned it to a device, and done some stuff with it" it is "Hey, it is possible for a third party of some(but by no means unique) skill and experience to, wholly without the cooperation of the manufacturer, work out everything that is necessary to get an ill documented or undocumented piece of hardware up and running with a new firmware that is both compatible with the original driver and capable of non-malicious operation and also capable of additional malicious functions".

Anybody who gives the matter a moment's thought, even pure amateurs, must conclude by simple logic that somebody can do it; what the security people are pointing out is that not only can somebody do it, potentially hostile third parties with reasonably available skills and no manufacturer support or collaboration can do it....

I suspect that they are (reasonably) well aware that somebody, presumably an embedded system/driver dev had to produce the blobs and loaders and other structures they are monkeying with in the first place.

Most of so called "hackers" are incompetent, barely script kiddies. I consider myself quite incompetent, and I find it unbelievable that they can get a job anywhere. They're the security worlds mirror of the people who can't pass the FizzBuzz test. But then there's people who actually have half a brain. Thing is, for these people, shutting the fuck up on a semi-permanent basis might be a good idea. I'm sure you can imagine a few reasons why.

You're assuming intelligence. An intelligent person would come to the same conclusions as you have. The same caution has come out for the Intel microcode uploader, flash-based BIOSes of all kinds and intelligent devices that can handle uploadable programs. It's not new, it's not even that dramatic, but it is (sooner or later) going to be highly significant. And all those who failed to take any action now will deny that they were ever told it was a possibility, and all those manufacturers who opted for point

I'm sure people are familar with LoJack for Laptops, where either due to a hook in BIOS (Dells and HPs have an option that will reinstall the LoJack software even if the BIOS is reflashed and all disks are zapped) or other means it gets loaded.

I can see this happening with malware, especially on a NIC with DMA access. Even if a machine is completely DBAN-ed, the botnet client will silently reinstall itself. As more devices (keyboards and such) have ROMs that can be flashed, we will see more and more devices have this avenue for compromise.

How to fix? The obvious fix would be signing the flash BIOS, but this completely locks out homebrewers wanting to do something different. Another fix would be having the flash process be offline, such as only though a USB port with a usb flash drive. However, NICs won't have USB ports present. Still another possible avenue would be a slot for a MicroSD card, but that adds complexity to the device. So, this isn't something easy to deal with. The only thing that might come close would be a DIP switch toggle to allow for unsigned images to be flashed (which is shipped off), and all updates signed.

I'm sure people are familar with LoJack for Laptops, where either due to a hook in BIOS (Dells and HPs have an option that will reinstall the LoJack software even if the BIOS is reflashed and all disks are zapped) or other means it gets loaded.

It's not a hook, LoJack comes with every BIOS. That's why it survives reflashing, you don't have the option of a BIOS without it. I co-wrote some article [coresecurity.com] about this not long ago.

How to fix? The obvious fix would be signing the flash BIOS, but this completely locks out homebrewers wanting to do something different. Another fix would be having the flash process be offline, such as only though a USB port with a usb flash drive. However, NICs won't have USB ports present. Still another possible avenue would be a slot for a MicroSD card, but that adds complexity to the device. So, this isn't something easy to deal with. The only thing that might come close would be a DIP switch toggle to allow for unsigned images to be flashed (which is shipped off), and all updates signed.

None of this would work. Maybe it will make it more difficult, but can't protect you against a logical flaw in the firmware that allows you to execute code. Firmware is like any other software, what happens if you sign code that executes any code? then all code is automatically "signed".

The solution IMHO is complex, expensive and involves signing+software protections in the NIC and in the OS (I.E. iommu, etc.) and WILL fail with a sufficiently resourceful attacker.

One might also go the avenue of adding a system-wide mechanism, designed from the ground up for maximum simplicity(so it doesn't itself need potentially malicious patching), for reading and writing all persistent memory in a system using an external piece of hardware in a special non-operating debug type mode(jtag-esque; but designed for lower complexity and this single purpose).

Some vendors would, no doubt, cry about the security of their precious binary blobs; but the customer, and security must ultima

Perhaps even just having a standard connector and method for accessing the JTAG ports might be the way to go. Plug a connector in, check on a second device if the code stored matches what it should be. If not, copy over a version that does. This could be automated so the NIC maker can make a security tool with a green/yellow/red light about the size of a 1/8 to 1/4" audio jack adapter that plugs into cards, reads a green light if the ROM matches a known good one, red if it doesn't, and yellow to tell the

My concern would be that any verification interface that doesn't have raw, independent, access to the persistent storage(doesn't have to be fast, I2C would cut it for all but the biggest blobs, does have to be independent) could theoretically be subverted by a malicious firmware.

In effect, unless you can take the system offline and scan the raw memory, you are really just asking the (potentially compromised) firmware running on the embedded CPU "Dear sir, are you compromised?" to which the answer will, i

Perhaps a separate, burned ROM (that can't be tampered with) that boots if a button is pressed? This ROM would scan the other BIOS storage and do exactly as you say -- compare everything to known hashes, and if there is an issue, zero out the BIOS and slap a "1.0" image that originally shipped with it, or perhaps have another mechanism for writing a BIOS to the storage. This is similar to booting a Linux machine from a Knoppix CD, running a hash of all files, then permissions and comparing the two to a kn

The only thing that might come close would be a DIP switch toggle to allow for unsigned images to be flashed (which is shipped off), and all updates signed.

How about a special cable?

Have say a USB port with an extra 'notch' at the bottom.

When a special proprietary flash drive is plugged in that has an extra plastic notch attached to the bottom, the 'button' will be pushed and held down while it is plugged in, enabling a "hardware maintenance" signal line.

How to fix? The obvious fix would be signing the flash BIOS, but this completely locks out homebrewers wanting to do something different.

Why not just have the hardware detect an unsigned BIOS and print a message on every boot that says "Modified firmware detected, press F7 for ten seconds to restore to factory default"? Then you can modify it if you like and you just ignore the message.

When a PC or Server is booted for the first time, force the user to create a password to password protect all firmware. The key here is to create a hierarchy of protection starting with the motherboard down to the peripherals installed on it. This could be vendor proprietary and eventually made into an industry standard. Any software that needs to change or update them will require this password in the future.

1st: Passwords are an acceptable form of authentication as long as you provide proper complexity requirements (better that way).

2nd: What? You can't trust a brand spanking-new PC/Server to be free from rooted firmware? Why buy from vendor X then? Besides, I said that one would be created from the first time it was booted. That means YOU or someone you trust unboxing it first.

3nd: Passwords, or the information to verify authentication is always stored someplace. Locality isn't all that important. It also

IMHO, I think you're over analyzing this way to much. At some point, you have to find your link in a chain of trust. How and where I leave up to you and the company you may or may not represent. At that level, it's really a risk assessment.

As for vendors, I trust Dell, HP, and IBM. At least the major brands would be HIPAA and PCI complaint. Also, some people run entire server farms using thousands of 1U servers from them. Each having a NIC or two.

I'd say the simple switch or button located on the device, like you propose, would be the best option. Just add couple steps, "Find paperclip." "Find the little hole you wondered about in the plastic." "Stick paperclip into hole to press tiny button." The device is now flashable for x period and will revert back on its own after x or it is flashed."

But still completely and utterly fascinating and relevant, especially since no one seemed to pay to much attention back at CANSECWEST (yet another computer security/tool/hacker/exploit research convention) this year in March when the same group shared their research and did a live demonstration of getting root (or system level, I forget if they hacked a windows or linux box) over the network by taking over the NIC, and not doing anything at all through the host OS.

I think it's utterly fascinating how blind we are about computer security unless we take matters into our own hands and look. It's like being in a series of twisty little passages, all alike. And some people are groping the walls instead of each other.

I take it back, this is new but related stuff. The old stuff was a hack to gain control of a NIC and then the host computer over the network (only affected 1 model of NIC that they knew of). There new stuff is firmware that would require them to first have root level access on the target system so that they could flash the attached network card. The upside to this is that they could remove all tools on the system itself and traces that they had been there, and be very very stealthy.

I imagine that the bigger risk would be contamination of the supply chain. Having a box rooted and NIC flashed(especially if said NIC(s) are embedded on a motherboard and the malicious flash includes a mechanism for silently eating all reflashes while reporting success...) is a downer; but learning that 45% of counterfeit Cisco gear, and 20% of the real used stuff, is also loaded with firmware level malice would be a real downer...

Firmware based rootkits aren't anything new, there has been lots of them already before. Like for example, last year there was several demonstrations of someone writing firmware rootkit for certain Apple-branded keyboards; there simply was enough space in the ROM for a complete keylogger and a bit of heuristics there and several kilobytes of space where to store the log. And network card base rootkits? I remember having read about them and seeing a demonstration already 5-6 years ago.

The thing is, as long as the user has actual physical access to the computer in question he or she can do lots of different kinds of small modifications, and for example the keyboard rootkit is easiest to do, doesn't require admin rights, and is undetectable unless you verify the actual firmware.

"However, the attack presented only applies to a specific network card model (Broadcom NetXtreme) whenever a remote administration functionality (called ASF for Alert Standard Format 2.0) is turned on (it is off by default) and configured. According to vendors, this functionality is far from being widely used. As a consequence, this vulnerability is really likely to have a very limited impact in practice."

When did bricklayer become something you are and something you do? Or arson, for that matter? There's a crime defined in the penal code in the country where I live called approximately "general devastation", if you happen to cause earthquakes, landslides, train accidents or large explosions. Why aren't the ones penalized under it refered to as "devastators?" That's a big linguistic miss in my opinion.

You don't do "bricklayer", you lay bricks, or perhaps you do a layer of bricks. His point was that you aren't "reverse engineer"; "reverse engineer" is a process someone does, but the person doing it isn't even necessarily an engineer at all.

My point was that the instincts of human language has no regards for the finer points of naming. If reverse engineering is a major activity in your life, you're going to get the title reverse engineer.

My point was that the instincts of human language has no regards for the finer points of naming. If reverse engineering is a major activity in your life, you're going to get the title reverse engineer.

Doubtful. It sounds kind of... wrong to me, so I would avoid the term. I might use it now and then in Slashdot summaries etc for humorous effect.

If we haven't been concerned over all of the cheap manufacturing going on in China, I would say this clearly illustrates what can really be done in a hard-to-detect way.

I have been repeating how "fear beats facts" lately, but there is one thing that beats fear... that would be greed. Not a lot beats greed and that is what is at the core so much. In this case, greed over the low cost of manufacturing in China to save a few bucks and to boost that bottom line.

I recall this article [ksplice.com] that hypothetically starts by using the BIOS extension ROM function to hook into GRUB and modify it, then the modified GRUB loads and patches the kernel to host a rootkit, then runs that.

So instead of a smart peripheral with onboard processor and firmware, the dumb ones are affected as well (which only requires the BIOS extension ROM interface).

Even though BIOS is on its way out (we can't MBR-boot >2TiB drives anymore, so we have to use GPT) and EFI is on its way in, we're still stuck because EFI has similar features. Apple's video cards for Mac Pros have both BIOS extension ROMs and EFI ROMs.

People running consumer routers are already very vulnerable for the most part. Reflashing the NIC is too much work. What you need to worry about is if you are doing everything else right, running full disk encryption, with encrypted swap, and a nice long passphrase. Let your computer out of sight for a bit and it's been flashed with firmware that will tftp your encryption key to hostile intelligence agencies (foreign or domestic, take your pick). Hell, they could even intercept your equipment before you