Project Hydra: the USB Multiheaded Monster

You have a bunch of servers to administer, not enough consoles and a small budget. Roll your own serial console server with USB.

When it comes to server consoles, users
have
two major options, graphical or text. I personally prefer a serial console to a
graphical one, and this preference isn't all rooted in snobbery.
Serial console switches, unlike KVM switches, are cross-platform and
relatively inexpensive. The problem with serial console switches, though,
is they aren't expandable. As we buy additional servers,
we often find ourselves short of available ports. Typically, I've punted in these
situations by using an available
serial port on a nearby machine and a null modem cable to create a
temporary console server.

This situation got us thinking. It would be really cool to have a
dedicated, remotely accessible console server with a lot of serial ports
connected to all of our servers. One way to do this would be to use
multiport serial cards, but they are comparable in price to serial
console switches and have a similar scalability problem. Instead, we
figured out how to use the readily available USB bus with USB-to-serial
adapters. This solution works well even in our mixed environment,
and it is also scalable and inexpensive.

Before embracing this idea as the console access cure-all, however, we should
discuss the limitations. The first is the USB bus, which is
limited to 127 devices per controller. This seems like a lot, but it is
important to remember that hubs count as one device, and under
Linux the root hub also counts as a device. So, consider a tree of USB
hubs and USB-serial adapters. Let n be the number of ports per hub,
k be the number of hubs in the tree and r be the number of USB-serial
adapters. This gives us a total of n * k ports. To connect the k
hubs in a tree we require a minimum of k – 1 interconnects, so
the maximum number of USB-serial adapters is r = nk – (k –
1). Because there is a maximum of 126 devices (127 minus one for the
root hub), in the best case we have k + r = 126. Solving for nk yields two
equations, nk = 125 and r = 126 – k. When nk = 125 does not have an
integer solution, it is best to round k up; otherwise, less
than the maximum number of ports will be available. Consequently, with
four-port
hubs (n = 4) we have k = 32 and r = 94, for a maximum of 94 USB-serial
adapters on a single controller. With seven-port hubs, k = 18 and r = 108.

We can do a little better than 127 devices by using additional USB
controllers. Although most PCs are equipped with multiple USB ports, these
ports often are on the same controller, and the 127 device limit
constrains both ports. Even if we add more controllers,
there is an upper limit of 256 USB-serial devices allowed,
or r </= 256. This is a limitation of the number of allowable
minor numbers for the assigned USB-serial major number, which is 188.
More industrious users probably could modify the driver to allow
additional majors. Adding additional controllers also may be required in
cases where the serial ports have a high baud rate; keep in mind
the USB bus is limited to 12Mbps. Timing issues also may
keep the usable number of USB-serial adapters below the theoretical
limit, but adding controllers is relatively easy and inexpensive.

The other limitation of using USB devices is the interconnect length;
cables are limited to a maximum length of five meters. Effective cable
length can be extended by using an active device, such as a powered hub or
powered extension cable, though even an active extension cable counts as
a device. The total depth is limited to seven tiers, counting the root and the
bottom device. This means there is a maximum of six 5m interconnects
for a total length of 30m. Although 30m is sufficient to reach each corner
of our server room, the reach can be increased another 15m by using
shielded RS-232 cable to connect the USB-serial adapter to the server
console port. Other ways of extending the RS-232 signal are available;
for example, use a pair of RS-422 adapters with an effective range of about 1.3km.

The choice of hub is irrelevant; however, all hubs must be powered to limit
signal degradation. However, the selection of USB-serial adapters is
important. The first adapter we tried was a Xircom/Entrega, which was
listed as experimental in the kernel driver list. As it turns out, the
vendor never provided source code for its drivers, so the Linux driver
was developed through reverse engineering. Though this driver works with
some models, it didn't work with the one we purchased. To
avoid a similar fate, spend some time looking through the USB-users
mailing list archive or the USB device database (see Resources).

The Keyspan line of adapters often is recommended, but they are fairly
expensive, around $50 each. We managed to find some Maxxtro adapters
based on the pl2303 chip for about $15 each, which have worked well. These
adapters are essentially a cable with a USB connector on one end, some
circuitry in the middle and a male DB9 on the other. The only other
thing needed was a DB9f-DB9f serial cable to connect the adapter to the
console port; the serial cable needs to be a null modem cable.

We spent about $16 for each hub, $5 for each 5m USB A/B cable, $15 for each
USB-serial adapter and about $2 for each 1m null modem cable.
If we assume that each four-port hub connects three adapters, leaving
the fourth port for descendant hubs, and that each hub requires one
5m interconnect, then each console port costs (16 + 5 + 3 * 15 + 3 * 2) / 3, or
$24/console on average. Consequently, 16 USB-serial console ports will
cost about $384 US.

Of course, there is the expense of the console server itself, but
I'm discounting this because we simply used a machine that
otherwise would
have been given away. Secondly, most serial console switches
do not come with the necessary cabling hardware I've included
in the per-port cost of the USB solution.

Now that you are ready to buy some hardware and try it out, how do you
make it work? First, configure your kernel with USB support. At a minimum,
you need CONFIG_USB, CONFIG_USB_SERIAL and CONFIG_USB_SERIAL_PL2303
(or the driver for the adapter you choose) set to y or m. You also need one of
the USB controller drivers, EHCI, UHCI or OHCI. I recommend building
all these drivers as modules to simplify any troubleshooting you
may need to perform. Once the modules are built, plug in a USB-serial
adapter, load the modules and check the output of
dmesg. You should
see several lines of output, ending with:

Here I connected one of the USB-serial adapters to a hub.
The adapter has been assigned the device name ttyUSB0, as it was the
first USB-serial adapter detected. If you were to connect this adapter
to a console port and point minicom at /dev/ttyUSB0, you should be able
to establish a connection. Although the process is that simple, this is where we
encounter a problem. Once I connect a server to this adapter, there is no
assurance this server is always available at /dev/ttyUSB0. The
ttyUSB device numbers are assigned in the order that the devices are
detected on the bus. Although the order in which devices are detected is a
well defined algorithm, the problem is the USB-serial driver always
assigns the first available ttyUSB device number. As a simple example,
consider two devices, ttyUSB0 and ttyUSB1. If we disconnect the adapter
assigned to ttyUSB1, disconnect ttyUSB0 and then reconnect the
adapter that was ttyUSB1, it is now ttyUSB0 because that is the
first available ttyUSB device number.

Of course, we could avoid this problem by not plugging and unplugging
devices, though arguably this ability is a strength of USB. However, there
is still a problem: devices may be detected at times other than at
module load. Consider adding a new USB-serial adapter to an existing
tree of devices, perhaps to connect a new server. Because the new device
is now the last detected device, it receives the next available ttyUSB
device number. This probably will be the highest ttyUSB device number,
assuming no devices have been disconnected previously. However, if this
device is added close to the root hub of the tree, then the next time
the console server is rebooted or the USB-serial module is reloaded,
this device may be assigned a low ttyUSB device number, as it probably will
be one of the earliest detected devices.

A possible solution: what if we could locate the desired USB-serial
adapter by its position in the device tree? This option would be more
reliable, because it is less likely that the existing structure will be
modified. That is, we can choose to preserve the position of existing devices in the
tree, even when adding new devices. Inspection of /proc/bus/usb/devices
reveals that each device detected on the bus has a topology field
of the form:

T: Bus=# Lev=# Prnt=# Port=# Cnt=# Dev#=#

The fields of interest are Port, which indicates
the position of this device on its parent device (usually a hub), and
Prnt, which indicates the USB device ID of the device,
again usually a hub, to which this device is connected. Working
backward from a specific device to the root, the path of the device
can be determined recursively. Although this does connect the USB
path to a specific device that represents its position in the tree,
the information about which ttyUSB device number was assigned is not
available from /proc/bus/usb/devices. The only connection we have to the
USB path is the assigned USB device number from the Dev
field. Unfortunately, USB device numbers are assigned in the order
the devices are detected. As such, it has the same problem as
before; if a device is added to an existing tree it will be assigned the
next available USB device number, which may not be the same assignment
when the USB-serial module is reloaded.

What is needed is a way to associate the USB path directly to the assigned
ttyUSB device number. As it turns out, the USB developers already have
solved this problem. Starting with kernel 2.4.20-pre7, a new proc entry
for the USB-serial driver exists: /proc/tty/driver/usb-serial. This
driver contains entries that, at least as of pre7, look like:

Comment viewing options

modern motherboards seem to have multiple controllers, 1 port per controller or in the worst case 2 ports per controller. this could be further extended with pci cards.

furthermore you could use multiport usb-to-serial converters that provide 2,4 or 8 serial ports on a single usb device. At some point usb bandwidth may limit your system - the converters quite often require several usb transactions for a single-byte transfer.

there is a more serious limitation in the older linux kernels, like those you mention - all usb-to-serial converters share the same major device number, thus limiting number of accessible converters to 256. current kernels solve this entire mess with udev where rules are used to name the devices by location in the device tree or by the serial number, which may be available on FTDI converters, for example.

I would like to know as well. I will designing a new production environment for my company. We will have 20+ linux systems. We probably are going to use some type of kvm setup for a few machines. I would much rather have a many to one USB system giving me console access to all machines from on central computer. Isn't there some mechanism already build into linux that will give us this functionality?

can you provide your source for "Maxxtro adapters based on the pl2303 chip for about $15 each". I went looking for that on froogle/google and managed to land a heap of sites in .ru, which I'm not in the mood to buy from, really.

Let's just say that the serial adapters are only going to cost us $25.00 each, and let's say we are only going to do 16 ports. For the $400 we are going to spend, it might be cheaper to get a used terminal server (16 or 32 ports) for the same price and have a dedicated hardware device.

Personally I am using the lantrontix series products and have several doing all my serial consoles.

I think so. First, it seems a little unfair to compare used prices of lantronix hardware to "new" USB hardware prices. The list price of a 16-port lantronix is roughly $2,200 new (though admitedly you could beat that by shopping around a little). And the condition that you will only purchase used hardware can create a deficit when you need a new terminal right away. That is, the ease of extensibility of using USB means that I can drive down the street and spend $40 and have enough hardware to hook up the two new machines we just received; The hardware is common and available everywhere and I don't have to buy a 16-port server for two new machines. Also, the prices quoted in this article are not the best you can do. I have on occassion spotted good 7-port USB hubs for $12, and 4-port hubs for $5.

Finally, there is a hidden value. This solution is all open. You can do other things with it, like monitor UPS, control other serial devices, etc. It's completely flexible. One of the things I've been working on lately is using our old "useless" laptops as remote consoles. All of the virtual terminals are sessions back on the main console server, so we can have as many "heads" spread around the server room as we like, and we can move them to any location with a network port without having to log in twice...

Of course, the lantronix units provide a nice clean solution. One of the drawbacks of using USB for serial consoles is that you can end up with a "spiderweb" of cables. So if you're planning a new rack of 32 servers and you've got the money then a lantronix unit is a great solution. But if you've got a server room like ours with every different kind of UNIX hardware imaginable and non-rackable cases spread all over the place, you might find that the flexibility of using a USB solution is desireable.

About the software redirection: one of the kewl things bootloaders can do is give serial-console access. Hence, an option to pass parameters to the OS kernel before bootstraping (boot into initrd.img on ramdisk, and (try to) fix stuff from there) .

Trending Topics

Upcoming Webinar

Getting Started with DevOps - Including New Data on IT Performance from Puppet Labs 2015 State of DevOps Report

August 27, 2015
12:00 PM CDT

DevOps represents a profound change from the way most IT departments have traditionally worked: from siloed teams and high-anxiety releases to everyone collaborating on uneventful and more frequent releases of higher-quality code. It doesn't matter how large or small an organization is, or even whether it's historically slow moving or risk averse — there are ways to adopt DevOps sanely, and get measurable results in just weeks.