Thursday, July 16, 2015

In the previous part, we moved away from emulation to working with physical hardware. We identified a UART header inside the Netgear R6200 that can be used for console access. I demonstrated how to access the CFE bootloader's recovery mode to reflash a working firmware over TFTP. This makes it possible to iteratively modify and test firmware images that will be used in the SetFirmware UPnP exploit.

In this part, I'll talk about regenerating the filesystem portion of the firmware image. I'll also walk through shrinking the filesystem in order to avoid crashing upnpd.

Updated Exploit Code

I last updated the exploit code for part 9, when we filled out the "janky" ambit header enough to satisfy upnpd. In this part I've updated the code to add an additional header field that must be filled in order to boot. If you've previously cloned the repository, now would be a good time to do a pull. You can clone the git repo from:

Regenerating the Filesystem

Recall from before that the firmware image for the R6200 consists of four parts:

Proprietary "Ambit" header

TRX header, which is well documented

Compressed Linux kernel

Squashfs filesystem

We reverse engineered the ambit header by analyzing the httpd and upnpd binaries. The TRX header is well documented and did not need to be reversed. We can reuse the Linux kernel from an existing firmware; no changes to it are required. All that remains is regenerating the SquashFS filesystem.

Generating a SquashFS filesystem is relatively straightforward; there are existing tools to turn a root filesystem directory into a filesystem image. The problem lies in the many different variations of SquashFS. In addition to the various official versions, vendors tweak it further for their own motivations. As a result of this proliferation of SquashFS variations, it can be hard to know which SquashFS tool will work with a given device. For this project, we're in luck. Netgear makes available open source GPL archives for most of its consumer products, including the R6200.

You'll find the source code, as well as a precompiled mksquashfs binary. Oddly, there are even intermediate objects from compilation. I always get the feeling that GPL releases from router vendors are just someone's workspace that got tarred up and posted online. Anyway, the mksquashfs binary is the one that I used. It's 32-bit, so I had to install lib32stdc++6 in my 64-bit Ubuntu VM. In theory, you should be able to rebuild the tools from source as well, but I didn't try. I put the executable in my path (~/bin in my case) so I can easily call it from scripts. I also gave it a unique name to differentiate from other SquashFS utilities.

In order to regenerate a filesystem image, you run mksquashfs on the root directory and give it the -noappend and -all-root options:

The first argument is the name of the root directory to convert to an image. The "rootfs.bin" is the name of the image to generate. The "-noappend" option means to not append to an existing image, and the "-all-root" option means to set ownership of all files to root.

Shrinking the Filesystem.

When we generate the root filesystem, it comes out to be over 7MB. There are additional options to mksquashfs that affect compression and block size and can impact the resulting image size. I wasn't able to get the resulting image to come out any smaller regardless of what options I used. In some cases, it ended up larger.

We can't change the size of the headers or of the kernel. So that leaves us with only the filesystem. If the total firmware size is to come in under 4MB, we need to get the filesystem down to around 2,700 KB or less. That's down from 7,400 KB. Obviously, there's no way to get a full firmware to fit in this size, or even one that approximates a full firmware.

So what can we do with such a small firmware? Is there even a point in this exercise? My strategy was to strip down the firmware as much as possible to come in under the limit, but still have the router do the following:

boot successfully

have a functioning userspace, including shell

have network connectivity, including to the internet

This first stage firmware should have some sort of agent that phones home to a predetermined server to download a second stage image. It should flash that image, and reboot. The second stage firmware will be a full blown firmware that looks identical to the stock firmware, but contains whatever additional tools and remote access capability we want.

Our goal is to figure out what we can strip out of this firmware while leaving it with a minimum level of functionality to bootstrap the second stage. The uncompressed filesystem takes up 28MB.

$ du -hs rootfs28M rootfs

There are a number of executables that are tempting to remove, as they seem noncritical. Before doing so, be sure they aren't links to /bin/busybox. Removing a link won't save significant space. The only way to save space with these executables is to rebuild busybox with fewer personalities.

The first thing that can go is the HTTP server and its resources. The www directory takes 4.6 MB on disk, and httpd takes 1.6 MB.

Removing a system service can be risky. On embedded devices such as this one, the boot sequence can be pretty brittle. Unlike a general purpose Ubuntu or Red Hat server, these are designed with the assumption that no components will be added or removed. If a service is removed that is critical to the boot process, the device may be rendered unusable. To reduce this risk, I replaced any removed system executables with a shell script of the same name that terminates with a successful exit status. This should trick whatever init or rc program is kicking off boot processes into thinking the service started successfully, thereby allowing the boot sequence to proceed uninterrupted.

Here's a script that replaces a given system binary with a dummy script:

As I removed each service, I generated a new, complete firmware image and installed it through the R6200's web interface to be sure the device would still boot and had network connectivity. Of course even it it does boot and run, you've now removed the web interface. This means there's no facility to reinstall the factory firmware. You'll need to recover via the serial interface I described in part 10. Using the serial console, you can recover using the bootloader's TFTP server.

For each service you remove, there may be shared libraries that are no longer needed. Those can be removed as well. An easy trick is to grep all the remaining executables for a given library's name. Here's a script you can paste into the terminal as a one-liner that will use grep to discover what executables link what shared libraries.

The libvolume_id.so shared library evidently isn't linked by anything and can be removed. The libvorbis.so shared library is linked by the DLNA service and may be removed once that service is removed. Re-run the script to generate a new list of library references each time you remove a service. This was a lengthy, iterative process for me. You may remove a critical service by accident or you may remove a library that is critical but not linked directly. It's important to test that each change results in a firmware which will still boot.

After we remove httpd and the www directory, the new root file system is just over 7000KB. That leaves 4300 KB to go. Keep repeating this process of removing services from /usr/sbin and /sbin, and corresponding libraries that have no references. Make your changes a few at a time so you know what to put back if the device is no longer functional after rebooting.

With all of these libraries and executables removed, the root filesystem directory was down to 9.8MB from 28 MB. The compressed SquashFS filesystem was down to 2,228KB! That's from a starting point 7.2MB. After building the complete ambit image (with ambit header, TRX header, kernel, and filesystem), it came to 4121918 bytes, or 0x3EE53E in hex. Recall the undersized malloc() for Base64 decoding was 0x400000. That's 70KB to spare! Kick ass[1].

Checksum Mismatch!

Now we can try uploading our minimized firmware to the R6200 using the UPnP SetFirmware exploit code[2]. At this point, you definitely need to connect to the UART serial interface if you haven't already. Even if the firmware boots, we've stripped out all the essential services, so there's no other way to see what's happening or what state the device is in after boot. And if it doesn't boot, well, you'll be glad you have the serial connection and CFE's recovery mode.

When I built a firmware and pushed it to the device over UPnP, exploiting the SetFirmware vulnerability, I was able to see the updating progress over the serial console. And then the R6200 rebooted. So close! After the reboot, I saw CFE initializing. And then this.

The CFE bootloader detects a firmware checksum mismatch.

On boot we see:

Image chksum: 0x61354161
Calc chksum: 0x9F3FAE72

Then the boot sequence halts, and CFE helpfully starts up a TFTP server for us.

The image checksum, 0x61354161, looks familiar. Let's go back to the firmware generating script and find_offset().

Oh look. That's the 4-byte value at offset 16. We discovered what field is when for when reversing httpd. From part 7:

We're not done with checksums just yet. The basic block at 0x0043643C is another checksum operation. Once again the data points to "HDR0", but the size is only the value from offset 28. The size from offset 24 is not used this time. The checksum result is the same as before, but this time compared to the value at offset 16. We now know the checksum we compute and store at offset 32, must also be stored at offset 16. Presumably, this would be to calculate a separate checksum without including the mysterious extra section I speculated about above.

So, even though this field is never validated in upnpd (which is why we didn't find it the second time around), it does get checked by CFE at boot. In fact if we had gone a little farther with static analysis, there is a section where sa_parcRcvCmd() seeks to the end of the flash partition, unlocks and erases the last erase-size (65536) bytes, seeks to 8 bytes the end, then writes the values from field 24, the TRX image size, and from field 16, the TRX image checksum.

Writing the TRX image size and checksum to the end of the flash partition.

This problem is easily solved. We already have the TRX image size at offset 24. That's the size that got checked against a limit of 4MB. It's also the size that is used to determine how much data to write to flash. We just need to add the TRX checksum at offset 16:

SC.gadget_section(self.TRX_IMG_CHECKSUM_OFF_1,self.trx_image_checksum,description="Checksum of TRX image. This gets verified by CFE on boot.")

With that done (and the router recovered back to a stock firmware), we can try again. And when we do, success! The router boots up completely to an interactive console.

Let's take a break for a second and reflect on where we are. We've successfully exploited a broken, abandoned, and forgotten capability in order to upload a firmware that we control to the Netgear R6200 over the network without authentication. We had to overcome the following challenges to get here:

Reverse engineer the UPnP daemon

Come up with silly timing games necessary to work around the broken networking code.

Binary patch, emulate, and debug upnpd and httpd.

Work out what the SOAP request should look like since the “parsing” is just bunch of strstr()’s against the *entire* HTTP request, and spread across a whole bunch of different functions

Reverse engineer the legitimate firmware format, as parsed by httpd.

Reverse engineer how upnpd parses the firmware format.

A few things remain before we can declare victory. We need to:

Embed a tool in the minimized stage 1 firmware image that will download the larger stage 2 firmware.

Successfully write the downloaded firmware to flash so that CFE is satisfied and will boot it.

Embed some sort of backdoor in the larger firmware. After all, that's the point of the exercise, right?

Before wrapping up the series, I'll discuss all three of these things. Before that, though, I'll discuss an intermittent crasher due to an invalid free() that you may or may not have encountered. Avoiding it is necessary to ensure the router reboots into the stage 1 firmware. I'll talk about how we can abuse the firmware header in such a way as to prevent crashing.

------------------------------
[1] When I did this project the first time around, back in December 2013 and January 2014, I hadn't discovered samba hiding out in /usr/local/samba. After deleting all the nonessential stuff from /usr/sbin and /lib, the SquashFS filesystem was still about a MB over the 2.7MB we need. What I ultimately did back then was to delete the (huge!) 4.1MB wl.ko from /lib/modules/2.6.22/kernel/drivers/net/wl. This, unfortunately, is the kernel module for the wireless hardware. Deleting this meant when the system booted there would be no WiFi. The system still worked and had network connectivity, but this was a very intrusive modification that I was never really happy with. Fortunately, finding the Samba installation in a non-standard directory means we don't need to remove the wireless driver.

[2] This is in the git repository linked earlier. The exploit script is setfirmware.py.

Thursday, July 09, 2015

Debugging and De-bricking the Netgear R6200 via UART

Update: I forgot to credit my former colleague, Tim (@bjt2n3904), for helping me locate the UART header. This project would have been way more challenging without the serial connection. It would have involved desoldering the flash memory chip, probably replacing it with a ZIF socket, and then removing and reprogramming the chip for each iteration of testing.

In the previous installment, we filled out the ambit firmware header just enough to satisfy Netgear's broken UPnP server. We also patched out several ioctl() calls in upnpd in order to test the SetFirmware exploit in emulation.

We're now at the point that emulation is no longer adequate; we need to start testing against actual hardware. There are subtle and not-so-subtle differences between emulation and hardware that affect how the exploit works. Some exploits, such as command injections and even buffer overflows, can be tested and developed entirely in emulation. Since this exploit writes a firmware image to flash memory, we need to ensure it is written to physical storage properly and will successfully boot and run.

Experimentation with modifying a device's firmware calls for some sort of connectivity at a lower level than just a Linux shell. If the operating system fails to boot, there is no shell. We'll need to connect to the device in order to diagnose the problem and recover. The iterative process of developing the small, bootstrap firmware that I will describe later entails many incomplete builds that will leave the device in a semi-broken state. Knowing that you can recover by restoring a good firmware makes the project much less risky.

3 male-to-female jumper wires of different colors (black, orange, and yellow are ideal)

Hunting for UART Header

Fortunately the R6200 has a UART header you can connect to using a serial terminal application such as Minicom. With Minicom, you can interact with the bootloader to see diagnostic messages and even drop into a recovery console.

To interface with the R6200's UART, you can use a cable like the FTDI 3.3V USB to Serial cable, (part number TTL-232R-3V3-2MM). It's available from Allied Electronics, Amazon, SparkFun, and others.

USB to UART cable for serial debugging

The UART connection isn't exactly set up and ready for you to use, though. This means taking apart your router and heating up your soldering iron.

There are couple of torx screws that hold the base on.

Then there are a couple more torx screws that hold the outer shell together. These are the same size as the previous ones, but different length. Keep them organized if you plan to put the router back together.

More screws.

With the outer screws removed, you can start separating the front and back half of the clamshell. There are plastic tabs all the way around that hold it together. I broke a few trying to get it open. Once you get the front half off, you'll find the PCB held in by more torx screws.

Once you remove the PCB, you can locate the UART header, which is exposed as four solder pads.

The solder pads, from left to right, are VCC, ground, transmit, and receive. You don't need VCC; it's +3.3V power. The USB adapter is powered by your computer's USB port, instead. That leaves ground, TX, and RX. The transmit and receive are relative to the device, so transmit from the device connects to receive of your cable and vice versa. Solder short leads to the appropriate pads, and connect your jumper wires to them. Then, route the jumpers out of the case so you can access the UART once you reassemble your router. I drilled a small hole in the top for a passthrough.

Here's how the UART header maps to the USB adapter's pinout:

Device GND <-> Adapter GND (black)

Device TX <-> Adapter RX (yellow)

Device RX <-> Adapter TX (orange)

If you have orange, yellow, and black jumpers, connecting them up so the colors match the USB adapter will save you some trouble. Sadly, I had green, pink, and blue on hand, so mine is exciting and confusing every time I hook it up.

Then, I zip-tied the leads to reduce stress on them.

Connecting Using Minicom

You may want to test the serial connection before reassembling. The baud rate is 115,200 and serial port settings should be 8,N,1. Here's my mincom configuration for the R6200. Obviously adjust your ttyUSB device as appropriate, but it's usually /dev/ttyUSB0.

When you connect with Minicom and power on the R6200, you can see the boot text scrolling across the console. If you let it boot, and hit return in the console, it gives you a root prompt. It's not a great terminal environment, though. There's no scrollback, for example. Once you have a serial console, use netgear-telnetenable[1] to fire up the telnet backdoor.

Shitty terminal environments aside, the serial console is great for restoring to a non-broken firmware. As long as nothing trashed the flash partition that contains the CFE boot loader, you can break in to a debug prompt and do a restore.

When you first power on the device and see CFE loading, break in with ctrl+c. You need to break in right after CFE starts, but before it finishes loading the kernel and operating system from flash. Incidentally, this gets trickier after we shrink the firmware down from nearly 9MB to under 4MB because the load time shortens dramatically, narrowing the window when you can break in.

Recovering a Bricked Router

If you break in at just the right time (I just mash ctrl+c repeatedly), you should get a CFE> prompt. Once you've got the prompt you can start up CFE's TFTP server with the tftpd command to restore a factory firmware.

The router's network configuration is 192.168.1.1/24. There's no DHCP server in this mode, so you'll need to configure your own network interface manually. You'll need a tftp client to upload the firmware image. TIP: Be sure to switch your client to binary mode. This gets me every time.

When you reboot, the router should be back to normal. Now you can iteratively test custom firmware knowing that it only takes a minute or two to restore back to a good one.

In the next part, we'll regenerate the SquashFS filesystem. We'll also work on shrinking the firmware down to 4MB to avoid crashing upnpd during exploitation. We'll need to hunt down and eliminate nonessential services, while avoiding breaking the boot sequence. Stay tuned!

------------------------------
[1] Did you know that nearly every one of Netgear's consumer devices has a well-known but unacknowledged backdoor? It's true. What the fuck are we even doing here. Who needs trojaned firmware when Netgear devices already have a backdoor. http://wiki.openwrt.org/toh/netgear/telnet.console