Upgrading Embedded Linux Without Bricking – ELCE 2012

Arnout Vandecappelle, senior embedded software architect at Essensium/Mind, talks about ways to greatly decrease the risk of bricking your board/device during upgrade thanks to gubies scripts and tools at the Embedded Linux Conference in Barcelona, Spain, on November 6, 2012.

Abtract:

Embedded systems are often modified remotely, e.g. to upgrade the firmware or change the configuration. This may however break the system and render it inaccessible, which is a major problem if the device is hard to reach physically. Unfortunately, no catch-all failsafe solution exists to make sure that the device stays accessible remotely even if a modification goes wrong. Instead, the possible failures have to be anticipated and covered. This talk discusses some of the frequently occurring failures, how they can be detected and handled. These include power failure, kernel crashes, network failure and data corruption. We include examples of concrete use cases. Finally, there is room for discussion about possible alternative or more generic solutions than the ones proposed.

This talk is geared towards system architects and developers who want to improve the quality of their product.

Failure mechanisms

Power failure during firmware update. Solutions:

Add fail-safe firmware

Detect failed power

Atomic update of firmware images

Use journalling filesystem for writable data

Bad firmware. Solutions:

Fall back on previous (known good) firmware

Fail-safe firmware that can do upgrades

Upgrade script included in upgrade image

Watchdog reboot + boot fail-safe after bad boot

Communication errors. Solution: verify data before writing (e.g. GnuGP). If memory is lacking to store the full firmware before upgrade, split in chunks for streaming upgrade

Boot loader upgrade

if boot loader is broken, no recovery is possible => avoid bugs and features in the bootloader.

Upgrade of boot loader with (bootable) backup media

Package-based upgrade

Advantage – results in smaller upgrade files

Drawbacks – Difficult to predict what is installed exactly, more places where something can go wrong, and no package manager is truly atomic (but nixos package manager comes close).

You can download the slides for this presentation, as well as checkout gupies (Generic UPgrade Infrastructure for Embedded Systems), a set of set of scripts and source code that can be used for upgrading the firmware of embedded systems maintained by Arnout.