Perverting embedded devices - Lexmark N4000e Print Server (Part - I)

During an engagement it's normal to find a number of embedded devices, access controllers, faxes, routers, print servers, etc. Their presence is often overlooked but from an attacker perspective it's ideal for persistence and stealth activity on a network.

This devices are full blown computers and they live in the core networks of every organization ,without hardening, no firmware updates and no maintenance. They are not critical in the network ecosystem but that's mainly why these devices are an Achilles heel of every networking environment.

Network Printers can be found everywhere, grab a rock, throw it and you'll hit one. They can be found exposed on the Internet and even in a full segmented network their VLAN have access to most of the other segments making it perfect as entry or pivot point device.

This subject has been discussed a number of times but this blog post is aimed into auditing a specific device using different approaches. This is mainly a work in progress so feel free to comment.

Introduction

We will analyse a Lexmark N4000E which we had laying around in our office. This print server at first sight has the following hardware features:

1 x RJ45 10/100 Mbps

1 x USB

In order to see all the potential of this device, we will have to see what's under the hood. One of our goals is to be able to access the OS to execute arbitrary commands in the device, and maybe replace the default firmware and transform it to a pen test drop box during our work as a Red Team. To do so we will have to do some of the following:

Network Services

During the audit of a device, sometimes we look for the complicated way to it but I have seen many time that this kind of systems are vulnerable to simple OS command injection flaws in their administration services or sometimes it is possible to upload a binary using TFTP to an arbitrary directory. Either way it is always recommended to spend some time checking for these kind of vulnerabilities.

A quick port scan, reveals a number of exposed services which is good, since our chances to find flaws are more likely.Host is up (0.021s latency).Not shown: 992 closed portsPORT STATE SERVICE21/tcp open ftp79/tcp open finger80/tcp open http515/tcp open printer631/tcp open ipp9000/tcp open net-config9100/tcp open jetdirect10000/tcp open lexdebug
This Lexmark print server has a number of administration services, three to be exact. One of those is located in the 10000/TCP, which has a hand full of commands which can be useful during the recon phase.

If we autenticate ourselves using the enable command a couple of new options are given, one of them is the familiar ps:

LXK: ps

USER PID %CPU %MEM SIZE RSS TTY STAT START TIME COMMAND

root 1 0.0 1.9 356 136 ? S 00:00 0:00 init

root 2 0.0 0.0 0 0 ? SW 00:00 0:00 (khubd)

root 3 0.0 0.0 0 0 ? SW 00:00 0:00 (svcerrd)

root 4 0.0 0.0 0 0 ? SW 00:00 0:00 (kswapd)

root 5 0.0 0.0 0 0 ? SW 00:00 0:00 (kflushd)

root 6 0.0 0.0 0 0 ? SW 00:00 0:00 (kupdate)

root 7 0.0 0.0 0 0 ? SW 00:00 0:00 (khubd)

root 27 0.0 1.5 368 108 ? S 00:00 0:00 /bin/dbprint -in -id

root 30 5.7 2.3 408 164 ? S 00:00 0:11 /bin/Nvram -e -c

root 32 0.0 1.1 356 80 ? S 00:00 0:00 ./flashmon

root 40 0.0 2.2 392 156 ? S 00:00 0:00 ./lexutils -d

root 41 0.0 1.5 432 108 ? S 00:00 0:00 flashsrv

root 47 0.1 3.2 520 228 ? S 00:00 0:00 NVRamServer

root 48 0.0 3.3 440 232 ? S 00:00 0:00 ErrorExit

root 54 0.0 3.2 448 228 ? S 00:00 0:00 StringsServer

root 55 0.1 4.3 564 304 ? S 00:00 0:00 VacuumServer

root 57 0.0 4.3 564 304 ? S 00:00 0:00 VAC_Slave_56

root 58 0.0 4.3 564 304 ? S 00:00 0:00 VAC_Slave_56

root 61 0.0 4.8 624 340 ? S 00:00 0:00 StatusServer

root 62 0.0 4.8 624 340 ? S 00:00 0:00 \_ VAC_Slave_61

root 63 0.0 4.8 624 340 ? S 00:00 0:00 \_ VAC_Slave_61

root 64 0.0 4.8 624 340 ? S 00:00 0:00 \_ VAC_Slave_61

root 65 0.0 4.8 624 340 ? S 00:00 0:00 \_ VAC_Slave_61

root 67 0.0 4.8 624 340 ? S 00:00 0:00 \_ SA_Slave_61

root 68 0.0 4.8 624 340 ? S 00:00 0:00 SA_Slave_66

root 70 0.0 2.7 444 196 ? S 00:00 0:00 enaid

root 74 0.0 2.7 444 196 ? S 00:00 0:00 \_ SA_Slave_70

root 76 0.0 4.4 508 312 ? S 00:00 0:00 NPAP_Server

root 80 0.0 4.4 508 312 ? S 00:00 0:00 \_ NPAP_DSA

root 81 0.0 4.4 508 312 ? S 00:00 0:00 \_ NPAP_Upd_Config

root 82 0.0 4.4 508 312 ? S 00:00 0:00 \_ NPAP_Reverse

root 84 0.0 4.4 508 312 ? S 00:00 0:00 | \_ NPAP_Slave_82

root 85 0.0 4.4 508 312 ? S 00:00 0:00 \_ NPAP_Internal

root 86 0.0 4.4 508 312 ? S 00:00 0:00 \_ SA_Slave_76

root 77 0.0 3.9 544 276 ? S 00:00 0:00 NPA_Port_MUX

root 78 0.0 3.9 544 276 ? S 00:00 0:00 \_ VAC_Slave_77

root 79 0.0 3.9 544 276 ? S 00:00 0:00 \_ SA_Slave_77

root 87 0.0 4.3 588 304 ? S 00:00 0:00 SA_Slave_83

root 107 0.0 4.3 588 304 ? S 00:00 0:00 VAC_Slave_83

root 108 0.0 4.3 588 304 ? S 00:00 0:00 VAC_Slave_83

root 116 0.0 2.3 428 168 ? S 00:00 0:00 LinkMonitor 0

root 117 0.0 3.8 488 272 ? S 00:00 0:00 Hbn

root 118 0.0 3.8 488 272 ? S 00:00 0:00 \_ SA_Slave_117

root 166 0.0 3.8 488 272 ? S 00:00 0:00 \_ VAC_Slave_117

root 121 0.0 3.0 444 212 ? S 00:00 0:00 addrconf

root 122 0.0 3.0 444 212 ? S 00:00 0:00 \_ SA_Slave_121

root 124 0.0 2.7 436 192 ? S 00:00 0:00 SA_Slave_123

root 130 0.0 3.0 452 216 ? S 00:00 0:00 wins

root 134 0.0 3.0 452 216 ? S 00:00 0:00 \_ SA_Slave_130

root 132 0.0 2.6 368 188 ? S 00:00 0:00 inetd

root 172 0.0 3.0 456 212 ? S 00:02 0:00 \_ lexdebug

root 176 0.0 3.9 412 280 ? R 00:03 0:00 \_ ps auxf

root 139 0.0 2.7 432 196 ? S 00:00 0:00 Ntp

root 140 0.0 5.0 756 356 ? S 00:00 0:00 snmpd

root 167 0.0 5.0 756 356 ? S 00:00 0:00 \_ VAC_Slave_140

root 169 0.0 5.0 756 356 ? S 00:00 0:00 \_ NPAP_Slave_140

root 170 0.0 5.0 756 356 ? S 00:00 0:00 \_ SA_Slave_140

root 141 0.0 3.7 696 260 ? S 00:00 0:00 http

root 156 0.0 3.2 436 228 ? S 00:00 0:00 tftp

root 157 0.0 3.5 440 252 ? S 00:00 0:00 host-config

root 158 0.0 4.2 476 300 ? S 00:00 0:00 ZeroConfig

root 159 0.0 4.2 476 300 ? S 00:00 0:00 \_ SA_Slave_158

As you can see, it definitely looks like a Linux system. Another hint is that all the services are running as root because if we find a vulnerability in any of these services we won't have to do any privilege escalation.
After reviewing all the services we still haven't found any flaw that would allow us to execute commands in the system but we found some things that can be useful in your next Penetration Test.

Lexmark's kinky secrets

Lexmark decided to make our devices full of undocumented features, specially for debugging and troubleshooting. On the port 79/TCP we have to access the good old finger service, normally used to check which users are logged in a UNIX system but Lexmark decided to use it as a status service of the back end printer, but there was more behind that.

The finger service when requested replies:

Nothing interesting. I tried a couple of usernames, so I used the common ones and between them there was the user setup:Escape character is '^]'.setupEthernet 10/100Network Card Status: Connected Speed, Duplex: 100 Mbps, Full Duplex (Auto) Current Date and Time: 1971-02-01 07:01 End-of-Job Timeout: 90
[...]

So this service supports commands? We decided to check a little bit more as you may know: "When in doubt, use brute force"

So what's the nasty part? The dump of the NVRAM includes the SNMP community used in the device and there is no way to password protect or disable this service.

SNMP Community leak

There are a couple of commands supported in this service, after a little bit of googling someone has published some other commands, if you are interested you can check this post. If we had access to this binary we might be able to discover other undocumented commands.

The web administration panel has also some "features" for us. If we set our browser to the /se directory a engineering menu is shown, there is no way to password protect it either. The amount of commands depends on the printer version.

Lexmark n4000e

An attacker could take advantage of this information, the SNMP community leak also works using this panel. If we check the same directory in an expensive printer there are a lot of debugging logs that may include usernames and if it is used as a fax machine, the callers are also shown.

Lexmark Optra MFP

We had discovered a couple interesting flaws, but none of these are useful enough for our goal of having access to it. These services have a lot of interesting options but it's a little bit out of the scope of this blog post.

What's next? It's time to break this thing apart and hook some cable into it.

Searching for a UART serial port

Most of the embedded devices hide a hardware debugging port. If you have never done it before the first idea seems like this, but detecting it is quite simple and this provides a great tool during the assessment of a embedded device since if we succeed we might get:

Access to a shell (could be password protected)

Access to the boot loader

Access to a non-interactive debugging console

All of the possible outcomes are great so, how do we detect a UART port? First of all a visual inspection of the device. Normally we should look for unsoldered test pads/probe points, they should be aligned, but this changes from device to device.

Initial Inspection

If take a closer look, we can notice a number of interesting things:

Debug UART

JT1 - 2 pins - 3,3v and GND

JT2 - Extra unsoldered USB connector, could be useful to add storage

Unamed - 22 pins - JTAG ? Possible expansion bus?

J4 - Debug - 4 pins - Seems quite interesting!

A UART serial normally has this elements:

VCC

GND

TX

RX

Ground Pin

The first thing we need to do is to detect the Ground (GND) pin and to do so the easy way is to use a continuity test of a multimeter. With the device unplugged, connect one of the probes to the metallic shielding of the RJ45 or the USB shield and with the other probe we will cycle around the suspected pins. When you get an audible tone confirms that those two points of the circuit are connected between each other.

VCC Pin

Normally this is used to power up the serial cable interface. We will not be using it but we will need to identify it. First we have to power up the device and mesure using the previously identified GND and cycling around the other pins using the other probe. The reading should say 3,3v.

RX/TX Pins

Using the multimeter to detect VCC we should detect one of the pins with no voltage for RX and one pin with 3,3v. But in this case the other two had the same voltage and connecting both TX and RX blindly to this device could lead in having a burned adaptor. If you really want to connect the RX you should use a 1k OHM to prevent that.

The easiest way is detecting TX first. For it I normally use a BusPirate which is a great tool and is quite useful but you could use any other UART adapter, one of my favorites is the CP2102 which not only works out of the box with Linux. We could setup custom baudrate using this which could be quite handy.

Bus Pirate cheatsheet

With the device off we connect GND and the MISO cable from the bus pirate to the suspected TX pin, then we configure the serial settings such as baudrate, I first applied 115200 which normally works with most of this kind of devices.

After everything is connected we power on the device to see what happens... nothing. Just a couple of garbage characters, so I switched to the other suspected pin and power the device again.

Garbage, a lot of it which is quite promising. This normally happens when using the wrong baudrate. There are many ways to detect the correct speed. We now know that the pinout is something like this.

DEBUG

1 - 3,3v(UNKOWN)

2 - 3,3v (TX)

3 - 3,3v (UNKOWN)

4 - GND

To detect the correct speed we could do manually and see if it works. We could automate this process using baudrate or if you are lazy and have a bus pirate, the auto-baud detection macro could save the day.

boom

I now know the possible baudrate, using my CP2102 serial adapter connected to the Lexmark debug port and I powered on:

Voila! Now there is no doubt that this is a Linux system. We now have a serial debug console, the main issue is that it is not an interactive console as we hoped. The RX pin seems to be inactive or could be something from the Serial Debug configuration that prevent us to send commands to the Lexmark print server. Another possibility is that a resistance could be in place filtering all the information being transmitted.

Uncover and exploit one of the exposed services

At this stage, we now have plenty of information about the device. It's a Linux ARMv920. All the services run as root and we have a realtime debug serial console, Source code review could be an option since they are using GPL licensed code so Lexmark is forced to publish any modded source. In the user manual they claim to have the sources in a FTP located in ftp.lexmark.com/swlabs but the service seems down...for the past year

Since we don't have source, fuzzing any of the exposed services is a good option, we could be able to exploit a flaw and finally have command execution.

If you never fuzzed before, I recommend using the Sulley framework which is quite easy to build your own fuzzer modules, but you could use many other tools to do this job.

We hook to the serial console and we start fuzzing the HTTP service, after a little bit we start to see some progress:

Related Post

5
comments

Hello! You might enjoy some of this code which should relate to the n4000e: http://opensource.dell.com/releases/Dell_Wireless_Printer_Adapter_3300/

The 3300 is a modified version of the n4000e, and there are references to the latter in some of the files in that archive.

I came upon your site while investigating the 3300 I have. I believe the software is very similar on the two devices. The Lexmark sources seem to have disappeared from the internet, but I contacted Dell and they were able to recover their GPL package for me! I just wanted to let you know about this in case you would be interested.

It would appear at first glance that this information also relates to at least some lexmark laser printers, I just bought a Dell 3335dn, which appears to be a rebadged Lexmark X464de and certainly the network commands and webpage information works although the nvram dump returns:Block ID: 0x01 Sub ID: 0x01 Size 248*** HIDDEN ***I made a quick paste of the process list here: http://pastebin.com/RwVGFZ6h