Reducing OS Boot Times for In-Car Computer Applications, Part III

The final installment in this series--did they meet their goal of a five-second boot?

This is the final article in a three-part series (see Resources) on reducing boot times
for in-car computers. The impetus for the project has been my company,
CarBot, where we're trying to get
computer hardware to behave as it should in a car. In the previous two
articles, I discussed a range of solutions for solving the problem
of slow boots. In the months since those articles were written, we've
thoroughly explored the possibility of using a BIOS replacement on the
VIA Epia-M
boards in an attempt to reduce boot times to the theoretical minimum. During this
time, we've also discovered some hacks that can be used to minimize the effect
of boot times on the user experience.

In our earlier articles, we compared the unnecessarily slow boot process
to that of a car radio. The car radio boot times have climbed from nearly
instant in the 1980s to several seconds today, but they still are rapid
enough to be hardly noticeable. If you watch a modern radio right after you
start the engine, you may see it do a small power-on self test, flash all
the lights on the unit--much as dashboards in many cars do--and then power
on the amplifier, producing sound within a second or two.

We decided to use the radio as our benchmark of rapid usability and
appliance-like behavior. We attempted to minimize the following two
variables: time from computer power-on to video and time from computer
power-on to audio.

Our testing rationale was if we can minimize the time before the arrival of
audio and video in the boot process, we later can optimize our
software so that whatever the user was seeing and hearing last can be
restored. This is what a radio does, this is what a television does, this
is what a computer should do.

It takes users a couple of seconds to orient themselves to the task at
hand, such as configuring a car computer. If the screen looked
like it did the last time they shut it off, it at least would give
the illusion of being on and would be a little less annoying. In the book
The
Humane Interface, Jef Raskin, father of the Macintosh computer,
asserts that there are no real technical reasons why a computer should
take any boot time at all.

Reducing boot times can be done by brute force or through finesse. The
solutions we have found or been told about can be broken down into
three basic categories:

Reduce the boot time to its theoretical
minimum.

Distract the user while the computer is booting
up.

Anticipate the user, and boot before the user
has a chance to wait.

Minimizing Boot Time

Long ago, I saw a comment on the Linux BIOS page that
EPIA-M support existed but was unstable. That fact had not changed since
July 2003. I e-mailed the maintainer of that branch, and he said "The
fastest time that I have is 5s with some optimization of [the] init script."

Wow, five seconds. We wanted to get that.

Initially, I tried to find a commercial vendor to replace the BIOS.
General Software was kind enough to send
me a sample of its BIOS replacement for the EPIA-M, but the company did not
yet have a similar version for the EPIA M2, another motherboard we use in
our products. Also, the entry price was surprisingly consistent among the several vendors
I asked--around $10,000 dollars. Recouping the effort of making a custom BIOS
requires this sort of pricing, I suppose. Part of the problem was these BIOS
implementations did all sorts of great stuff, and what we were in search of was
a BIOS that did almost nothing.

So we set out to see how much time we could recapture by throwing away
the BIOS and booting Linux ourselves. The idea was to throw away all
the useless BIOS functionality, show video at the boot loader (that is, GRUB
splash screens) and start playing audio as early as we possibly could
in the boot sequence.

Replacing the BIOS

Our Linux engineer, James, was sent off with a pair of Epia-M boards,
a bootable 256MB Flash drive, the first two boot articles and a pointer
to the LinuxBIOS site. I initially thought that we could get enough
Linux into the BIOS chip itself to do something useful, such as squeak
and Flash the monitor. That would have satisfied our light and sound
objective, but James discovered that we did not have enough
space without going to the Flash drive. So the new goal was to reduce
the BIOS boot as much as possible and then launch a stripped
ZipSlack
distribution
on the compact Flash IDE drive.

The main benefit of LinuxBIOS for our application also was the
main drawback. The primary development focuses on making clusters of
supercomputers that need to be able to reboot quickly. These clusters
were designed for network booting, however; we wanted to boot
off a disk or a Flash device.

The EPIA-M boards, although quite popular in the
hardware hacking scene are not
popular as clustered supercomputing nodes. The EPIA is not the fastest
motherboard, running between 600MHz and 1GHz with a comparatively
feeble CPU.

As the LinuxBIOS project has progressed, some of the code to do things
other than bootstrap supercomputer nodes has fallen out of sync with
the documentation. Downloading and reading most of the LinuxBIOS
mailing list
archives helped to reconcile the discrepancies.
For instance, for a while it seemed as though the Etherboot modules would be
adaptable to booting off of other devices, such as the compact flash/IDE
adapter combo we were using. Eventually, James switched to using the
FILO bootloader.

The hardware aspects of flashing a BIOS were the most
adventurous. The first few compiles of a new BIOS image naturally
produce a machine unable to boot. The essential tools for BIOS
programming work is two of the same motherboard and the
correct chip
tweezers.

Restoring a fried BIOS seems wrong when you're doing it. When you fry
your BIOS in computer A, you boot computer B with the fixed BIOS and then
carefully extract its chip with the computer still running. Then,
you put the other BIOS chip in its place and burn it with a backup of
the original BIOS. Then, you can put the chip back in the other computer and
continue on your way.

Getting a usable BIOS was no small feat. Etherboot would fail in many
charming ways, for any reason, such as not finding the Ethernet card,
not finding the kernel on the disk and so on. After a bad flash, either there would
be nothing at reboot or LinuxBIOS would fail to load the BIOS itself or FILO.
After a dozen or so failed attempts, a bootable BIOS finally was flashed,
and ZipSlack was booted successfully off the CF card.

The following several sections, written by James, detail what was
discovered along the way.

- You can in theory use many different PLCC32 flash ICs. I am using
an AT49F040A-55JI (512kB) from DigiKey ($1.68 qty 1) in place of the
SST39F020A (256kB) VIA installed. The AT49F040A required some minor
hacking inside flash_and_burn/

- If you do use a 512kB flash device, set PAYLOAD_SIZE to 458752,
VGABIOS_START to 0xfff80000, and ZKERNEL_START to 0xfff90000.

- You MUST edit src/northbridge/via/vt8601/raminit.inc to configure
the DRAM (specifically "DIMM_PC133" and "DIMM_CL2"). If these
options are incorrect, LinuxBIOS will freeze immediately after the
initial banner.

- raminit.inc hints that you must set the FSB jumpers to 133 MHz.
Fortunately that is not the case. The V8000 runs LinuxBIOS fine at
66MHz (which reduces heat output enough to safely disconnect the CPU
fan)

- See HOWTO/EPIA for VGA BIOS support on the V8000. The extraction
procedure differs from the newer EPIA-M.

- You will probably need "ideN=noprobe" on the Linux kernel command
line to prevent IDE probe delays.

- If you need the parallel port, add "parport_init_mode=spp" on the
kernel command line to enable it.

- CONFIG_COMPRESS is responsible for the huge 3+ second delay after
"Copying LinuxBIOS to ram" and before "Jumping to LinuxBIOS". The
decompression is extremely slow, and is unnecessary, as LinuxBIOS
easily fits in the 64KB area without it.

Add "option CONFIG_COMPRESS=0" to the LinuxBIOS config file and
it will disable compression in favor of a fast, plain memory copy.
With this option, LinuxBIOS transfers control to Linux 2-3 seconds
after power on (even at 66MHz FSB with VGA enabled)!

Something is wrong with your linuxBIOS setup. I'm the maintainer of the Intel 440bx port for LinuxBIOS V1. I've got 2 different 440BX boards with 400Mhz Celerons that get to your 9 second mark in under a second.

Oh and FYI. The proper name of LinuxBIOS is indeed LinuxBIOS. Not FreeBIOS. FreeBIOS is the project that LinuxBIOS came from. The reason its still in CVS as freebios is historcial and that its a real PITA to change the CVS setup.

Here's the first clue something is wrong. The only thing that LB does between reset and this message is enable the RAM. Taking 300mS to turn on your RAM is _WAY_ slow.

3.492 Copying LinuxBIOS to ram.
6.292 Jumping to LinuxBIOS.

2.8 seconds to copy the code into RAM?! Thats crazy. You need to subscribe up to the LinuxBIOS mailing list and post your time stamped serial log there and ask for help to get your settings correct. Something is very hosed.

You may want to check out the work that has been done by the
CE Linux Forum on bootup time reduction. I presented a paper
at the 2004 Ottawa Linux Symposium, listing some techniques
for bootup time reduction.