David A. Wheeler

2009-04-13 (revised 2013-08-23)

If you’ve written (or started to write) some
Free/Libre/Open Source Software (FLOSS), please follow
the time-tested community standards for releasing FLOSS software
when you want people to be able to install it from source code.
This article will briefly explain why you should follow
standard release conventions, and what they are.

Why Follow Standard Release Conventions?

First, the why.
The quick answer is, “because it makes it easy to install your software -
including creating packages to do so”.
Modern Linux and BSD systems (as well as MacOS and MS Windows) normally install
programs via “packages”; on Linux and BSD systems, this automatically
installs all the necessary dependencies too.
These packages are much easier to
create and maintain if you follow standard release practices.

I’ve written a number of FLOSS programs (such as
flawfinder and
SLOCCount), and participated
in other development, but I didn’t really understand the importance of the
release standards until I started to create packages as part of my work on
open proofs.
I quickly learned that the packaging concept was simple, but
some programs were hideously painful to package because they hadn’t
followed the usual release standards.
I knew some of these practices before, but I sometimes skimped on them
before I understood the nightmare I was (unintentionally) inflicting on others.

I intend to modify my future releases to better meet usual release practices,
now that I’ve “learned better”... and
I hope to spread the word so that others will do the same.

What are the Standard Release Conventions?

Now, the what. These documents in particular describe
good release practices:

A Quick Checklist

I know that the above is a lot information, so here is a
quick checklist of some of the most important guidelines
(in my opinion), based on problems that I’ve had:

Pick a good, simple, and unique Google-able name for the package
and its executables.
Limit your package name to the lower case letters
a-z, digits 0-9, and the four punctuation characters “-._+”.
The
Software Release Practice HOWTO’s chapter on
good project and archive naming practice and the
Fedora naming guidelines may help.
Make sure the uppercase/lowercase distinction is irrelevant; some filesystems
can’t distinguish them, and package names tend to be all lowercase anyway.
Try to pick a name that’s unique —
at least unique among packages, and
preferably, one that doesn’t have any Google hits at all
(so that people can find your package easily).
A good example of a bad name is
Why, because it is
completely un-Googleable.
If you don’t pick a good name, then in the best possible case,
people will rename your program, often leading to confusion.
At worst, they will never find your program at all.
Similarly, try to make to make the name of the executable(s) unique;
they will probably be installed in /usr/bin with all other user programs.

Identify the version in MAJOR.MINOR.SUBMINOR format
or by using dates in YYYYMMDD format.
I especially recommend using the MAJOR.MINOR.SUBMINOR form
defined as
semantic versioning
(with the added restriction: don't add "-..." at the end of a version name,
because the rpm tool used by many Linux distributions
does not allow hyphens in version names).
People and software must be able to easily determine if a version
is older or newer than another one.
Using version numbers (e.g., 2.2.2) or date values in YYYYMMDD format
(e.g., 20090413) makes that clear.
The filenames of major releases should include the version number, e.g.,
“myprogram-1.2.7.tar.gz”, so that even after downloading there
are no questions.
Please use a version control system (such as git),
but please make occasional “real” releases that
are single-file archives; that way, people can easily know what version
is ready to use, and how to get it.

Use a standard, widely-used, GPL-compatible FLOSS license — and say so.
One of the worst mistakes a FLOSS project can make is to use
a non-standard license (especially a home-grown one) or fail to
state a license at all.
“Home-grown” licenses are often not even FLOSS, impede reuse
of other code, can delay packaging or use by years,
and will dissuade potential co-developers.
Many organizations and distributions explicitly forbid using software
that fails to include a license statement, because they create a
legal hazard.
As a result, failing to state a license
makes the software useless to many.
Use one of the standard FLOSS licenses (e.g., MIT, BSD-new,
LGPL, or GPL), and
be sure your license is GPL-compatible.
Fedora’s licensing
page has more info.
You do not need to use the GPL itself, but using a GPL-incompatible
license is a mistake, in part because most FLOSS is GPL’ed.
Do not write your own FLOSS license.
Clearly state in your documents - and code header files - what the license is.

Follow good distribution-making practice, in particular,
make sure tarballs
always unpack into a single new subdirectory (of the current directory)
named NAME-VERSION.
The
Software Release Practice HOWTO’s “Good distribution-making practice”
goes into this.
Make sure that tarballs unpack into a subdirectory, or unwary users will
end up with unwanted junk in directories they use.
By doing this, you’ll fulfill people’s
expectation that the first two steps to do are
(1) unpack (untar) the file, and
(2) “cd” into its directory.

Support the standard invocation to configure, build, and install it:
./configure --prefix=/usr ; make; make install.
Many people expect that, after the unpack-and-cd steps,
you’ll perform the normal configure, make, make install process.
Please implement configuring, building, and installing this way,
because this presumption is widespread (e.g.,
here,
here,
here,
and
here).
You do not need to use the GNU autotools (autoconf, automake, and libtool)
to implement this - just conform to the usual external interface.
See the
GNU Coding Standards’ “Managing the Release Process” for more.
That said, you might consider using autotools, since it automates this;
see the notes below on autotools.
Don’t interleave the configuring, building, and installation steps;
in particular, the build (“make”) step should create all the files to be
installed, and the install step should do nothing but install the files
created by the build step.
The first two steps should not require root privileges; the latter
one may require privileges depending on where files are being
installed.
The next three bullets discuss the assumptions of each of these
three steps.

Support the standard ./configure options like --prefix,
--exec-prefix, --bindir, --libdir, and so on.
By supporting the standard “configure” options you make it easy to
install the program wherever the user wants it.
Supporting “prefix” is especially important;
“prefix” is prepended to all filenames to be installed, and
should default to “/usr/local”
(for a system-wide installation that’s not part of the system).
When the program is packaged for a given system, the prefix is typically
changed to “/usr” instead.
You should support installation by a single unprivileged user, in which
case the prefix would typically be “$HOME”.
There is a reason that people need control over these values,
for example, never assume that “libdir” is the same as “$(prefix)/lib” —
on many 64-bit systems it is “$(prefix)/lib64”.
Here’s how configuration should work.
It’s nice to support --srcdir, but I wouldn’t worry about that;
--srcdir can be awkward to support, and is far less important
(storage space is plentiful nowadays,
so you can just make a copy and then build).
But options like --prefix and --libdir are vital.

Create a makefile that can rebuild everything and
uses makefile variables (including applicable standard
standard makefile variable names and targets).
By default, invoking “make” should rebuild/recompile everything
(by tradition this target is named “all”).
Makefiles should be able to rebuild everything;
many distributions forbid using pre-built binaries or libraries in packages
(since they can’t then fix problems).
It’s okay to include pre-built binaries/libraries in a source
distribution, as long as “make clean”
(or “make maintainer-clean” or
“make really-clean”) can get rid of them.

Makefiles should maximally use makefile variables so they can be easily
overridden where necessary, e.g., “make CC=mygcc”.
GNU publishes useful makefile conventions.
Overriding “make” values is very portable (it’s defined by the POSIX standard),
with this precedence: (1) variables set on the “make” command line,
(2) MAKEFLAGS, (3) variables set in the Makefile unless “-e” is set,
(4) environment values, (5) default Makefile rules.
in particular, variables set on the make line or via MAKEFLAGS take precedence
over the rest (make will use the environment values if their values are
not set in a makefile; “-e” makes the environment values take precedence).

You don’t need to create a makefile directly; many programs
use makefile generators, such as the one in
Cmake
and
autoconf/automake.
Just make sure that the generated makefile(s) support these conventions.
It’s nice if you can create a portable makefile by sticking with
just the capabilities in the
POSIX standard, but the 2008 POSIX specification for “make”
lacks so many capabilities that many people find it unacceptable
(others have come to the same conclusion, e.g.,
MozillaLLVMMad Scientist).
If you will not create a portable makefile, then use GNU make and its
extensions; GNU make is open source software, free (in both senses),
available everywhere, actively maintained, and full of useful
capabilities.
(Hopefully someday the POSIX standard for make will get fixed; I have
submitted some comments to try to get things better.)

Note: be sure to mark phony targets (like “all”, “clean”, and “install”)
with the .PHONY mark, like this:

.PHONY : clean

This is particularly important for “install”; without it,
“make install” will probably do nothing on filesystems that
ignore case distinctions (because the
make will see that the “install” file already exists).
In general,
marking phony targets (targets that aren’t really files)
will eliminate mysterious errors and document more clearly what is what.

Even if you use a completely different build system that does not use
make, at least include a tiny makefile that invokes the
“real” make system.
There’s no reason you have to use make directly
if you don’t want to.
For example, today’s FLOSS Java applications tend to use
Ant or
Maven.
Cmake is
a very useful cross-platform build system, with a lot of users
(Cmake generates makefiles, so it automatically meets the requirement
to provide a makefile).
However, make sure you include a small makefile that can invoke your
build and installation system.
That way, people don’t need to figure out how to invoke your make system,
and programs can be automatically built and installed regardless
of your preferred build system.

“make install” should not rebuild anything; simply copy files to locations as specified by DESTDIR, prefix, and so on.
Most packaging systems, including Ubuntu, Debian, and Fedora’s,
expect programs to be completely built by the steps above, and then split
“installation” into two steps:
An “install” to an intermediate directory
(where the files will then be collected),
and a final “install” done
on the user’s actual system.
The program will not be given root privileges in the first step, and
in the second step it will not normally run commands at all, it will just
copy files into predetermined positions.
This is easily done if your installation program is invoked using
“make install” and
supports a few standard conventions (such as DESTDIR).
The install rule should
“not modify anything in the directory where the program was built,
provided ‘make all’ has just been done...
The commands should create all the directories in which files are
to be installed, if they don’t already exist.”
In most cases using “cat” in “make install” is a mistake;
“make install” should simply make directories and copy already-built
files into the appropriate directories, so think carefully before
adding any command other than mkdir, install, cp, and ln.
Please support the
DESTDIR convention, that is, they should always install into
the directory $(DESTDIR)$(prefix) where DESTDIR can be set by the user.
This implies that you should always create directories before installing
into them; $(DESTDIR)/usr/bin might not exist!
Packaging can be very painful if you don’t support DESTDIR.
Automating
DESTDIR can be a pain, so it’s best if the program supports it
to start with.
If your package doesn’t support DESTDIR, at least make sure that
the “make install” target only uses a few simple commands to place the files
in their final positions (e.g., mkdir, install, cp, and ln), so that
the package
Auto-DESTDIR can
automatically support DESTDIR.
The GNU documents recommend not depending on mkdir supporting “-p”,
but I think this is obsolete advice; “mkdir -p” has become very widespread,
and it is even included in the Single Unix Standard version 3,
so I think it’s okay to depend on it now.
It’s often helpful to
support other make targets like “clean” and “uninstall”.

Do not require web access or interactivity for building, testing,
or installing the software.
Often these aren’t permitted or possible.
If you must interact with a user before installation can complete,
install all the files without interaction,
and detect when a user tries to start the program for the first time.
Normally you should do this by looking for a file or directory named
$HOME/.PACKAGE_NAME.
If that file/directory doesn’t exist,
display information on how to complete the installation or
interactively complete the installation, and in the process create the
file or directory so this won’t happen next time.

Let users easily use and update the system/local versions of external tools and libraries needed for building and running it, and document what those are..
Document in your README or INSTALL file the external tools/libraries you
need for building and running your software, including a URL
for each uncommon one.
Most importantly, make it easy for users to separately install and
update those tools/libraries!

It’s okay if you include a copy of an externally-developed
tool or library, as a convenience for people who don’t want to download it
separately.
But do not require people to use these embedded versions.
Over time, separate tools/libraries get updated, and whatever version
is embedded in your release will become obsolete.
Even if you sent the latest version at the time, external programs
will be upgraded later.
This is especially an issue for security updates; people want to be able
to download and install them once, and then have that upgrade work for
everything.
Try to ensure that users can use dynamic libraries to load the latest versions,
if that’s practical.
Unfortunately, many Java programs are especially bad at this.
Many distribution guidelines forbid using
bundled libraries in their packages, for example, see
Fedora’s “No bundled libraries” rule
and
Debian’s policy on convenience copies explicitly forbid them.
For example,
Tom “spot” Callaway gave up packaging the Chromium web browser
for Fedora
for this reason.
Debian works to try to identify packages that
embed code from other projects, saying,
“This is considered bad for fixing security flaws because the fix needs
to be applied in multiple source packages.”
Similar problems mean that the math system Sage is not easily installed
(via their package managers) on
Fedora or
Ubuntu.

If you support plugs-in/extensions, use a directory for them.
If your application can be extended with “plug-ins” (variously
called “extensions” or other terms),
do not require plug-ins to modify the contents of any files.
Instead, just create a directory for plug-ins to be placed in
(the names of such directories often end in “.d” but that’s not required).
Then, when a plug-in is to be added or removed, the relevant file(s) can
simply be added or removed.
Your program should simply use the contents of this directory to
determine what plug-ins are available.

If you patch an external library/tool, get the patch upstream.
If you don’t, your “local” version will stagnate while the original tool
gets many improvements.

Use pkg-config when finding or installing libraries.Freedesktop.org’s pkg-config
is relatively new but has swept the world of POSIX systems because
it’s an easily and simple solution to a more complicated problem.
If you’re compiling an application, it can tell you which flags to use.
If you’re creating a library, please create and install the pkg-config .pc
files, so others can use pkg-config.

Use standard user interfaces.
For command line tools, use “-” single-letter options, “--” long-name
options, and “--” by itself to signal “no more options”.
For GUI tools, provide a .desktop file.
Make it easy for people to start up your application.
GNU standards for command line interfaces
notes standard option conventions for command line interfaces, including
some standard long names like “--version” and “--help”.
Once installed, GUI programs should be runnable from the main menu,
which requires that a .desktop entry file be created and correctly installed;
see
freedesktop.org’s desktop entry specification.

Where sensible, make the application useable as a library
or via the command line.
This makes it possible to build a larger program
(using libraries or scripts) that uses your program/library.

Include tests as “make check” and return its success or failure as an exit status.
Users and packagers want to know if the program is working as intended,
and anyone who makes an improvement to it will want to know if the program
is still working.
So, include an automated test suite,
where you send in inputs, produce outputs, and check
that the outputs are correct.
It should be invocable with “make check”, it should
“exit 0” if the tests succeeded, and it should
“exit nonzero” if a test failed
(so users will know that the program is not working properly).
Tests that cover a significant amount of the program’s functionality
and implementation are best, but even a cursory test
(a “smoke test”) is better than nothing.
Set up your test suite so that it’s easy to add new tests.
Make sure the test will test the program built even if another
version of the software is installed.

Include documentation.
Typically you’ll have a brief README (what is this program?) and INSTALL
(how to install it and any requirements for tools).
It’s a good idea to have a man page and additional separate documentation,
sufficient to answer the questions:
What does your program do (sufficient so the user can determine if they
want it)?
What are some sample invocations?
What are its options?
What is its input format or API?
What is output format or result?
What are its strengths or weaknesses?
What is the project’s URL?

Make it portable.
If your software can port to different architectures and platforms,
then it’s probably flexible enough to handle likely future changes.
Follow formal and informal standards, such as the
Single Unix Specification.
Consider using checking tools like the
Linux Foundation’s AppChecker.

Support internationalization/locationalization (i18n/L10n).
Use tools (like gettext) for any command-line or GUI interface
so that people can use their native language in such interfaces.
Do not assume that characters all fit in one or two bytes (they do not).
I recommend using UTF-8 to encode characters in most cases,
but in some cases (e.g., Java, Windows APIs)
it may be easier to use a different encoding;
if you do, beware of their gotchas
(some glyphs take more than 2 bytes in UTF-16, and you have to worry
about byte ordering and the byte-order mark).

Test by packaging

By far the best way to see if you’ve done a poor job of packaging
is to create a package.
If it isn’t trivial to create a package, then you haven’t
released it properly.

Or, you can do the same thing with Debian or Ubuntu; create a package.
If you have to do anything nonstandard, you are probably
doing it wrong.

Yes, packagers can work around a lot, but in the long term that
will create problems.
If you release your software in a format that is easy to package,
you will speed the release of your software out to end-users, and
reduce the risk that there will be unintended problems.

The GNU autotools suite can help you automate creating a good build
process for your users.
In particular, autotools will automatically implement many of the conventions
listed above.

Autotools is not the only way to create source code releases
that are easily built and packaged.
But it’s one of the most widely-used, especially
for programs that use C or C++ (though it’s not limited to that).
Anyone releasing software as non-Windows source code should
know a little about autotools,
so that they can determine if they should use it.

There isn’t actually a program called “autotools” —
instead, this is the
conventional name for a suite of GNU tools that can be used together
for this purpose, including autoconf, automake, and libtool.
The process of changing a program to use autotools is called
autoconfiscation.

Years ago, autotools was hard for developers to use and it
had lousy documentation.
GNU autotools has been significantly improved over the years,
and much better documentation is available too.
Unfortunately, there’s a lot of really obsolete documentation
(and a lot of obsolete complaints about autotools).

See my
autotools (autoconf, automake, and friends) if you’re interested in more information.
This page includes an autotools tutorial,
pointers to other useful information, and
warnings that may help you avoid wasting your time.

Recursive make considered harmful

Historically, when programs got big, people would divide the source
code into directories, and create a makefile in each directory that
would build that one directory.
Then you would have makefiles recursively call down to other makefiles.
This is the obvious way to scale up, and many projects have historically
done this.

I think Miller is quite right that recursive make, though widely used, is
a bad idea.
I would add some caveats.
Miller correctly notes the problems of “=” and the advantages of “:=”,
but “:=” is not in the POSIX standard (it’s a GNU make extension).
I worked with the POSIX standards folks to get “::=” into the POSIX standard;
hopefully in the future this kind of assignment will be more portable.
Still, you can learn a lot about correctly using make from this paper.

Quagmire
is intended to replace automake and libtool, and eventually some of
autoconf. Unlike these tools it requires GNU make and is
written solely using GNU make features -- no preprocessing is required.
“The GNU Configure and Build System” by Ian Lance Taylor
explains the goals of quagmire, by an author of a book about
the GNU configure and build system (autoconf, automake, and libtool).

Automating
DESTDIR can be a pain, so it’s best if the program supports it
to start with;
my package
Auto-DESTDIR can
automatically support DESTDIR in some cases if the program installation
does not support it to begin with.