With the proliferation of small highly-distributed IOCs and remote I/O
we are seeing a rapid increase in the number of "intelligent" devices
connected to our control systems. One outcome of this is the widespread
use of vendor specified 'firmware' within these devices. Often times
this firmware is not quite... as firm... as one would hope, and numerous
updates are released over the lifespan of our embedded designs.

This concerns me in a number of ways.

First; without delving into the minutia of a specific manufacturer's
software engineering processes (which I am usually not privy to), I am -
in effect - (re)installing software in a (presumably) mature system with
little knowledge of the ramification of this upgrade. Take for example a
PLC based system which I've deployed in the past. The software on this
PLC (ladder logic) was written 8 years ago, and a portion of that
software was a "workaround" for a race condition which exhibited itself
in a predictable (and documented) manner. A recent firmware update of
that PLC apparently corrected this error, and hence made the workaround
code unnecessary. I can envision a case in which a designer
inadvertently "takes advantage" of an undocumented feature in a
subsystem which is then removed or deprecated via firmware update. A
routine or tangential upgrade may make that system fail altogether, or
(more dangerous) act in an unexpected manner, with the culprit hard to
detect.

Second (similar to #1, but subtly different); do I just trust a vendor
that their firmware will not introduce problems to my current system? I
have seen on more than one occasion firmware updates with little or no
change logs accompanying them. An easy answer would be "don't fix it if
it ain't broke," but that is not reasonable. We have systems here that
require periodic updates to keep up with newer or updated commercial
products - for example, the latest I/O modules for the Koyo PLCs will
not function with the processors we have without a firmware update
first. Another easy answer is "test test and test," but I believe that
is not necessarily practical nor exhaustive.

Finally; assuming firmware updates are an accepted price to pay for COTS
devices, how do we manage and track them? More often than not I'm going
to receive not only a proprietary (binary) firmware image, but also a
custom loader and procedure. Here at the APS, a PLC firmware update has
the potential to take many hours of effort to physically walk the
machine and connect a windows based laptop to each installation to
perform the update. Note that this also (typically) will shut the system
down as the firmware is applied so cannot be done during beam
operations. What about "rolling back" to a previous version? I have seen
systems which do not allow this with current loaders.