Unix Standards

The 1973 rewrite of Unix in C made it unprecedentedly easy to port
and modify. As a result, the ancestral Unix diverged into a family of
operating systems early on. Unix standards originally developed to
reconcile the APIs of the different branches of the family
tree.

The Unix standards that evolved after 1985 were quite successful
at this — so much so that they serve as valuable documentation
of the API of modern Unix implementations. In fact, real-world Unixes
follow published standards so closely that developers can (and
frequently do) lean more on documents like the POSIX specification
than on the official manual pages for the Unix variant they happen to
be using.

In fact, on the newer open-source Unixes (such as
Linux), it is common for operating-system
features to have been engineered using published standards as the
specification. We'll return to this point when we examine the RFC
standards process later in this chapter.

Standards and the Unix Wars

The original motivation for the development of Unix standards
was the split between the
AT&T and
Berkeley lines of
development that we examined in Chapter 2.

The 4.x BSD Unixes were descended from the 1979 Version 7.
After the release of 4.1BSD in 1980 the BSD line quickly developed a reputation as the
cutting edge of Unix. Important additions included the
vi
visual editor, job control facilities for managing multiple foreground
and background tasks from a single console, and improvements in
signals (see Chapter 7). By far the most important addition
was to be TCP/IP
networking, but though Berkeley got the contract to do it in 1980,
TCP/IP was not to ship in an external release for three years.

But another version, 1981's System III, became the basis of AT&T's later
development. System III reworked the Version 7 terminals interface
into a cleaner and more elegant form that was completely incompatible
with the Berkeley enhancements. It retained the older (non-resetting)
semantics of signals (again, see Chapter 7 for discussion of this point). The
January 1983 release of System V Release 1 incorporated some
BSD utilities (such
as
vi(1)).

The first attempt to bridge the gap came in February 1983 from
UniForum, an influential Unix user group. Their Uniforum 1983 Draft
Standard (UDS 83) described a “core Unix System” consisting of a
subset of the System III kernel and libraries plus a file-locking
primitive. AT&T
declared support for UDS 83, but the standard was an inadequate subset
of evolving practice based on 4.1BSD. The problem was exacerbated by the July
1983 release of 4.2BSD, which added many new features (including
TCP/IP networking) and
introduced some subtle incompatibilities with the ancestral Version
7.

The 1984 divestiture of the Bell operating companies and the
beginnings of the Unix wars (see Chapter 2) significantly complicated matters. Sun
Microsystems
was leading the workstation industry in a BSD direction; AT&T was
trying to get into the computer business and use control of Unix as a
strategic weapon even as it continued to license the operating system
to competitors like Sun. All the vendors were making business
decisions to differentiate their versions of Unix for competitive
advantage.

During the Unix wars, technical standardization became something
that cooperating technical people pushed for and most product managers
accepted grudgingly or actively resisted. The one large and important
exception was
AT&T,
which declared its intention to cooperate with user groups in setting
standards when it announced System V Release
2 (SVr2) in January
1984. The second revision of the UniForum Draft Standard, in 1984,
both tracked and influenced the API of SVr2. Later Unix standards
also tended to track System V except in areas where
BSD facilities were
clearly functionally superior (thus, for example, modern Unix
standards describe the System V terminal controls rather than the BSD
interface to the same facilities).

In 1985,
AT&T released
the System V Interface Definition (SVID). SVID
provided a more formal description of the SVr2 API, incorporating UDS
84. Later revisions SVID2 and SVID3 tracked the interfaces of System V
releases 3 and 4. SVID became the basis for the POSIX standards, which
ultimately tipped most of the Berkeley/AT&T disputes over system
and C library calls in
AT&T's favor.

But this would not become obvious for a few years yet;
meanwhile, the Unix wars raged on. For example, 1985 saw the
release of two competing API standards for file system sharing over
networks: Sun's Network File System (NFS) and
AT&T's Remote
File System (RFS). Sun's NFS prevailed because Sun was willing to
share not merely specifications but open-source code with
others.

The lesson of this success should have been all the more pointed
because on purely logical grounds RFS was the superior model. It
supported better file-locking semantics and better mapping among user
identities on different systems, and generally made an effort to get
the finer details of Unix file system semantics precisely right,
unlike NFS. The lesson was ignored, however, even when it was repeated
in 1987 by the open-source X windowing system's victory over Sun's proprietary
Networked Window System (NeWS).

After 1985 the main thrust of Unix standardization passed to the
Institute of Electrical and Electronic Engineers (IEEE). The IEEE's
1003 committee developed a series of standards generally known as
POSIX.[144] These went beyond describing
merely systems calls and C library facilities; they specified
detailed semantics of a shell and a minimum command set, and also
detailed bindings for various non-C programming languages. The first
release in 1990 was followed by a second edition in 1996. The
International Standards Organization adopted them as ISO/IEC
9945.

Key POSIX standards include the following:

1003.1 (released 1990)

Library procedures. Described the C system call API, much like
Version 7 except for signals and the terminal-control interface.

1003.2 (released 1992)

Standard shell and utilities. Shell semantics strongly resemble those
of the System V Bourne shell.

In the 1996 Second Edition, 1003.4 was split into 1003.1b
(real-time) and 1003.1c (threads).

Despite being underspecified in a couple of key areas such as
signal-handling semantics and omitting BSD sockets, the original POSIX standards
became the basis of all later Unix standardization work. They are
still cited as an authority, albeit indirectly through references like
POSIX Programmer's Guide [Lewine]. The de facto Unix API standard is still
“POSIX plus sockets”, with later standards mainly adding
features and specifying conformance in unusual edge cases more
closely.

The next player on the scene was X/Open (later renamed the Open
Group), a consortium of Unix vendors formed in 1984. Their X/Open
Portability Guides (XPGs) initially developed in parallel with the
POSIX drafts, then after 1990 the XPGs incorporated and extended
POSIX. Unlike POSIX, which attempted to capture a safe subset of all
Unixes, the XPGs were oriented more toward common practice at the
leading edge; even XPG1 in 1985, spanning SVr2 and 4.2BSD,
included
sockets.

XPG2 in 1987 added a terminal-handling API that was essentially
System V
curses(3).
XPG3 in 1990 merged in the X11 API. XPG4 in 1992 mandated full
compliance with the 1989 ANSI C standard. XPG2, 3, and 4 were heavily concerned
with support of internationalization and described an elaborate API
for handling codesets and message catalogs.

In reading about Unix standards you might come across references
to “Spec 1170” (from 1993), “Unix 95” (from
1995) and “Unix 98” (from 1998). These were
certification marks based on the X/Open standards; they are now of
historical interest only. But the work done on XPG4 turned into
Spec 1170, which turned into the first version of the Single Unix
Specification (SUS).

In 1993 seventy-five systems and software vendors including
every major Unix company put a final end to the Unix
wars when they
declared backing for X/Open to develop a common definition of Unix.
As part of the arrangement, X/Open acquired the rights to the Unix
trademark. The merged standard became Single Unix Standard version 1.
It was followed in 1997 by a version 2. In 1999 X/Open absorbed the
POSIX activity.

In 2001, X/Open (now The Open Group) issued the Single Unix Standard version 3. All the threads of Unix API standardization were finally
gathered into one bundle. This reflected facts on the ground; the
different varieties of Unix had re-converged on a common API. And, at
least among old-timers who remembered the turbulence of the 1980s,
there was much rejoicing.

The Ghost at the Victory Banquet

There was, unfortunately, an awkward detail — the
old-school Unix vendors who had backed the effort were under severe
pressure from the new school of open-source Unixes, and were in some
cases in the process of abandoning (in favor of
Linux)
the proprietary Unixes for which they had gone to so much effort to
secure conformance.

The conformance testing needed to verify Single Unix
Specification conformance is an expensive proposition. It would need
to be done on a per-distribution basis, but is well out of the reach
of most distributors of open-source operating systems. In any case,
Linux changes so fast that any given release of a distribution would
probably be obsolete by the time it could get
certified.[145]

Standards like the Single Unix Specification have not entirely lost their
relevance. They're still valuable guides for Unix implementers. But
how The Open Group and other institutions of the old-school Unix
standardization process will adapt to the rapid tempo of open-source
releases (and to the low- or zero-budget operation of open-source
development groups!) remains to be seen.

Unix Standards in the Open-Source World

In the mid-1990s, the open-source community began
standardization efforts of its own. These efforts built on the
source-code-level compatibility secured by
POSIX and its
descendants. Linux, in particular, had been written from
scratch in a way that depended on the availability of Unix API
standards like POSIX.[146]

In 1998 Oracle ported its market-leading database product to
Linux, in a move that was rightly seen as a major breakthrough in
Linux's mainstream acceptance. The engineer in charge of the port
provided a definitive demonstration that API standards had done their
job when he was asked by a reporter what technical challenges Oracle
had had to surmount. The engineer's reply was “We typed
‘make’.”

The problem for the new-school Unixes, therefore, was not API
compatibility at the source-code level. Everybody took for granted
the ability to move source code between different
Linux,
BSD, and
proprietary-Unix distributions without more than a trivial amount of
porting labor. The new problem was not source compatibility but
binary compatibility. For the ground under Unix had shifted in a
subtle way as a consequence of the triumph of commodity PC
hardware.

In the old days, each Unix had run on what was effectively its
own hardware platform. There was enough variety in processor
instruction sets and machine architectures that applications had to be
ported at source level to move at all. On the other hand, there were
a relatively few major Unix releases, each with relatively long
service lifetimes. Application vendors like Oracle could afford the
cost of building and shipping separate binary distributions for each
of three or four hardware/software combinations, because they could
amortize the low cost of source-code porting over large customer
populations and a long enough product life cycle.

But then the minicomputer and workstation vendors were swamped by
inexpensive 386-based supermicros, and open-source Unixes changed the
rules. Vendors found they no longer had a stable platform to ship
their binaries to.

The superficial problem, at first, was the large number of Unix
distributors — but as the Linux distribution market
consolidated, it became clear that the real issue was the rate of
change over time. APIs were stable, but the expected locations of
system administrative files, utility programs, and things like the
prefix of the paths to user mailbox names and system log files kept
changing.

The first standards effort to develop within the new-school
Linux
and BSD community
itself (beginning in 1993) was the Filesystem Hierarchy Standard (FHS).
This was incorporated into the Linux Standards Base (LSB), which also
standardized an expected set of service libraries and helper
applications. Both standards became activities of the Free Standards Group,
which by 2001 developed a role similar to X/Open's position amidst the
old-school Unix vendors.

[144] The
original 1986 trial-use standard was called IEEE-IX. The name
‘POSIX’ was suggested by Richard Stallman. The
introduction to POSIX.1 says: “It is expected to be pronounced
pahz-icks as in positive, not poh-six, or other variations. The
pronounciation has been published in an attempt to promulgate a
standardized way of referring to a standard operating system
interface”.

[145] One Linux distributor, Lasermoon in Great
Britain, did achieve POSIX.1 FIPS 151-2 certification — and went
out of business, because potential customers didn't
care.