Compiling Made Easy

From LXF Wiki

'''There’s a new version of your favorite program available, but no packages for your distro. What do you do? Compile its source code, of course! Mike Saunders shows you all you need to know...'''

'''There’s a new version of your favorite program available, but no packages for your distro. What do you do? Compile its source code, of course! Mike Saunders shows you all you need to know...'''

-

''Wiki post by Captain_Tux. Original article written by Mr. Mike Saunders and published in LXF 111 (November 2008). For the original document in PDF [[click here]].''

+

''Wiki post by Captain_Tux. Original article written by Mr. Mike Saunders and published in LXF 111 (November 2008). For the original PDF document [[click here]].''

-

Building software from source – that’s a bit old-school, isn’t it? Who wants to wrestle with the command line, hunting down dependencies and coaxing the ''GCC'' compiler into running properly? Well, it does sound like a strange thing to do in this world of binary packages and online repositories. We have thousands of packages available via the internet, all neatly compiled for our distros, thereby nullifying the need to get down and dirty with a Makefile. Or so it seems. As great as they are, binary packages have a lot of limitations that can only be overcome by compiling a program from its source code. Here are some of the benefits of building by hand:

+

Building software from source – that’s a bit old-school, isn’t it? Who wants to wrestle with the command line, hunting down dependencies and coaxing the ''GCC'' compiler into running properly? Well, it does sound like a strange thing to do in this world of binary packages and online repositories. We have thousands of packages available via the internet, all neatly compiled for our distros, thereby nullifying the need to get down and dirty with a Makefile. Or so it seems. As great as they are, binary packages have a lot of limitations that can only be overcome by compiling a program from its source code. Here are some of the benefits of building by hand...

* '''Timeliness''' Get software as soon as it’s available. When CoolApp 2.1 is released, your distro will probably have only version 2.0, and (unless it’s a rolling-upgrade distro like Gentoo) you’ll have to wait until the next major release of your distro to get the new app. But by building from source, we can get the newest version of anything when it’s released.

* '''Timeliness''' Get software as soon as it’s available. When CoolApp 2.1 is released, your distro will probably have only version 2.0, and (unless it’s a rolling-upgrade distro like Gentoo) you’ll have to wait until the next major release of your distro to get the new app. But by building from source, we can get the newest version of anything when it’s released.

Line 9:

Line 9:

* '''Features''' In the interests of stability, distro vendors often disable experimental features when making a package, leaving you with a rather plain (albeit reliable) piece of software. When we go down the source-code route, however, we can turn on extra goodies – it’s our choice!

* '''Features''' In the interests of stability, distro vendors often disable experimental features when making a package, leaving you with a rather plain (albeit reliable) piece of software. When we go down the source-code route, however, we can turn on extra goodies – it’s our choice!

-

* '''Optimisations''' When you compile from source, you get the chance to boost the program for your machine. Binary packages are built for a wide range of x86 processor types. Using certain compilation options, you can build apps specifically for your exact model of CPU, helping to wring extra performance from your PC.

+

* '''Optimizations''' When you compile from source, you get the chance to boost the program for your machine. Binary packages are built for a wide range of x86 processor types. Using certain compilation options, you can build apps specifically for your exact model of CPU, helping to wring extra performance from your PC.

To prove that it’s not as difficult as it sounds, let’s see how this works. The skills you learn here will put you in good stead for your Linux (and Unix-using) career, as the vast majority of programs can be compiled in this manner. We’re going to take a stock, unmodified Ubuntu 8.04 installation and install the ''Audacity'' sound editor, enabling some extra features along the way. ''Audacity'' is a good choice because it can throw up some strange errors during the compilation process, so we’ll learn how to deal with every eventuality.

To prove that it’s not as difficult as it sounds, let’s see how this works. The skills you learn here will put you in good stead for your Linux (and Unix-using) career, as the vast majority of programs can be compiled in this manner. We’re going to take a stock, unmodified Ubuntu 8.04 installation and install the ''Audacity'' sound editor, enabling some extra features along the way. ''Audacity'' is a good choice because it can throw up some strange errors during the compilation process, so we’ll learn how to deal with every eventuality.

Line 16:

Line 16:

== Get prepared ==

== Get prepared ==

-

First off, we need to get hold of the Audacity source code. You can find this on the project’s website at '''http://audacity.sf.net''', or on the '''LXF'''DVD in the Audio section. Save '''audacity-src-1.2.6.tar.gz''' on to your desktop, then open a command line window (Applications > Accessories > Terminal) and enter the following:

+

First off, we need to get hold of the ''Audacity'' source code. You can find this on the project’s website at '''http://audacity.sf.net''', or on the '''LXF'''DVD in the Audio section. Save '''audacity-src-1.2.6.tar.gz''' on to your desktop, then open a command line window (Applications > Accessories > Terminal) and enter the following:

<pre>

<pre>

Line 40:

Line 40:

This executes the configure script (the '''./''' is required, as we’re running it in the current directory), telling it to spew out all available compilation options. If you scroll up through the terminal, you can see that there’s a huge range of options available to us. But don’t fret – we can happily accept the defaults. We’re going to make a couple of small changes though: firstly we’re going to enable the ''SoundTouch'' library, which provides extra pitch and tempo manipulation tools that aren’t included in the default Ubuntu package.

This executes the configure script (the '''./''' is required, as we’re running it in the current directory), telling it to spew out all available compilation options. If you scroll up through the terminal, you can see that there’s a huge range of options available to us. But don’t fret – we can happily accept the defaults. We’re going to make a couple of small changes though: firstly we’re going to enable the ''SoundTouch'' library, which provides extra pitch and tempo manipulation tools that aren’t included in the default Ubuntu package.

-

Another tweak we can make is to set the location of the installed program when it has been built. To do this, we use the '''--prefix''' option with the configure script. For instance, to install ''Audacity'' in the /usr/local directory (away from other software and easier to maintain), we use '''--prefix=/usr/local'''.

+

Another tweak we can make is to set the location of the installed program when it has been built. To do this, we use the '''--prefix''' option with the configure script. For instance, to install ''Audacity'' in the '''/usr/local''' directory (away from other software and easier to maintain), we use '''--prefix=/usr/local'''.

-

As mentioned earlier, another great benefit of compiling software from source is the choice of optimisations. By setting the CFLAGS (for C code) and CXXFLAGS (for C++) environment variables before we run the '''configure''' script, we can select the exact GCC compiler optimisations we want to use. This is a complicated subject, but we can get an extra speed boost very easily. Consider these commands:

+

As mentioned earlier, another great benefit of compiling software from source is the choice of optimizations. By setting the CFLAGS (for C code) and CXXFLAGS (for C++) environment variables before we run the '''configure''' script, we can select the exact ''GCC'' compiler optimizations we want to use. This is a complicated subject, but we can get an extra speed boost very easily. Consider these commands:

<pre>

<pre>

Line 52:

Line 52:

</pre>

</pre>

-

Here we’re setting up the aforementioned environment variables with optimisations suitable for the Intel Core 2 CPU architecture. The '''-O3''' bit at the start has a letter '''O''', not a zero, and requests the third level of optimisation. (Most distributions compile their packages with '''-O2''' for slightly smaller executables and easier debugging.) If you have a machine with a AMD64 chip, use '''-march=athlon64''', and if you’re not sure, just use '''-march=i686''' for wide compatibility. You can consult the ''GCC'' manual page ('''man gcc''') for a complete list of supported CPU architectures, but it’s heavy reading!

+

Here we’re setting up the aforementioned environment variables with optimizations suitable for the Intel Core 2 CPU architecture. The '''-O3''' bit at the start has a letter '''O''', not a zero, and requests the third level of optimization. (Most distributions compile their packages with '''-O2''' for slightly smaller executables and easier debugging.) If you have a machine with a AMD64 chip, use '''-march=athlon64''', and if you’re not sure, just use '''-march=i686''' for wide compatibility. You can consult the ''GCC'' manual page ('''man gcc''') for a complete list of supported CPU architectures, but it’s heavy reading!

-

So, we have our '''optimisations''' set up, and we’re ready to run the configure script. Here we go:

+

So, we have our optimizations set up, and we’re ready to run the '''configure''' script. Here we go:

<pre>

<pre>

Line 152:

Line 152:

Development packages will

Development packages will

almost always end with a

almost always end with a

-

''-dev'' or ''-devel'' extension.)

+

'''-dev''' or '''-devel''' extension.)

Now re-run the configure

Now re-run the configure

Line 227:

Line 227:

need '''libqt-dev'''. Often the README file will provide a full list.

need '''libqt-dev'''. Often the README file will provide a full list.

-

The command we need to kick off the GCC compiler is (drum

+

The command we need to kick off the ''GCC'' compiler is (drum

roll please)...

roll please)...

Line 254:

Line 254:

ourselves. You may recall, back at the start, that we passed the

ourselves. You may recall, back at the start, that we passed the

'''--with-soundtouch=system''' option to the '''configure''' script,

'''--with-soundtouch=system''' option to the '''configure''' script,

-

instead of '''=local''', because the version supplied with Audacity is

+

instead of '''=local''', because the version supplied with ''Audacity'' is

broken. (You would’ve found out that '''=local''' is broken, so we’ve

broken. (You would’ve found out that '''=local''' is broken, so we’ve

saved you a bit of time!)

saved you a bit of time!)

Line 260:

Line 260:

So, '''configure''' let us down a bit there. But no big deal: open

So, '''configure''' let us down a bit there. But no big deal: open

''Synaptic'', search for ''libsoundtouch'' and install it (along with its

''Synaptic'', search for ''libsoundtouch'' and install it (along with its

-

-dev sibling). We don’t need to re-run the configure step again, as

+

'''-dev''' sibling). We don’t need to re-run the configure step again, as

that succeeded, so enter '''make''' again and watch the output. Now

that succeeded, so enter '''make''' again and watch the output. Now

you’ll see that it can’t find FLAC. D’oh! So fire up '''Synaptic''' and

you’ll see that it can’t find FLAC. D’oh! So fire up '''Synaptic''' and

Line 300:

Line 300:

'''gettext''' and a '''gettext-base''' packages. The latter was installed as

'''gettext''' and a '''gettext-base''' packages. The latter was installed as

part of the '''build-essential''' bundle, whereas the former includes

part of the '''build-essential''' bundle, whereas the former includes

-

our essential '''msgfmt''' tool. Maybe one day distro developers will

+

our essential ''msgfmt'' tool. Maybe one day distro developers will

stop fragmenting programs into a squillion ambiguously named

stop fragmenting programs into a squillion ambiguously named

packages, but at least the solution was a quick Google away here.

packages, but at least the solution was a quick Google away here.

Line 351:

Line 351:

that can crop up, so now you’re trained and ready for any

that can crop up, so now you’re trained and ready for any

eventuality. Now go forth and compile!

eventuality. Now go forth and compile!

+

+

[[Category:LXF Articles]]

+

[[Category:Programming]]

Current revision

There’s a new version of your favorite program available, but no packages for your distro. What do you do? Compile its source code, of course! Mike Saunders shows you all you need to know...

Wiki post by Captain_Tux. Original article written by Mr. Mike Saunders and published in LXF 111 (November 2008). For the original PDF document click here.

Building software from source – that’s a bit old-school, isn’t it? Who wants to wrestle with the command line, hunting down dependencies and coaxing the GCC compiler into running properly? Well, it does sound like a strange thing to do in this world of binary packages and online repositories. We have thousands of packages available via the internet, all neatly compiled for our distros, thereby nullifying the need to get down and dirty with a Makefile. Or so it seems. As great as they are, binary packages have a lot of limitations that can only be overcome by compiling a program from its source code. Here are some of the benefits of building by hand...

Timeliness Get software as soon as it’s available. When CoolApp 2.1 is released, your distro will probably have only version 2.0, and (unless it’s a rolling-upgrade distro like Gentoo) you’ll have to wait until the next major release of your distro to get the new app. But by building from source, we can get the newest version of anything when it’s released.

Features In the interests of stability, distro vendors often disable experimental features when making a package, leaving you with a rather plain (albeit reliable) piece of software. When we go down the source-code route, however, we can turn on extra goodies – it’s our choice!

Optimizations When you compile from source, you get the chance to boost the program for your machine. Binary packages are built for a wide range of x86 processor types. Using certain compilation options, you can build apps specifically for your exact model of CPU, helping to wring extra performance from your PC.

To prove that it’s not as difficult as it sounds, let’s see how this works. The skills you learn here will put you in good stead for your Linux (and Unix-using) career, as the vast majority of programs can be compiled in this manner. We’re going to take a stock, unmodified Ubuntu 8.04 installation and install the Audacity sound editor, enabling some extra features along the way. Audacity is a good choice because it can throw up some strange errors during the compilation process, so we’ll learn how to deal with every eventuality.

Get prepared

First off, we need to get hold of the Audacity source code. You can find this on the project’s website at http://audacity.sf.net, or on the LXFDVD in the Audio section. Save audacity-src-1.2.6.tar.gz on to your desktop, then open a command line window (Applications > Accessories > Terminal) and enter the following:

cd Desktop

tar xfvz audacity-src-1.2.6.tar.gz

cd audacity-src-1.2.6

The first command switches us into the Desktop folder, and the second command extracts the compressed .tar.gz archive. (Note that for archives with a .tar.bz2 suffix, you should use tar xfvj instead.) Finally, we switch into the newly created source code directory.

Enter ls to see the list of files in the expanded archive. Along with various scripts (highlighted in green) and sub-directories (blue), you’ll see a README.txt file. It’s always worth having a quick peek at any file called README.txt or INSTALL.txt, as it may have useful information on building the program. Enter less README.txt to view the file, and hit the Q key to quit. In this case, we don’t need to read through it all – we can get straight to work on the compilation process.

The configure script will prepare the Audacity source code for compilation. Enter this command:

./configure --help

This executes the configure script (the ./ is required, as we’re running it in the current directory), telling it to spew out all available compilation options. If you scroll up through the terminal, you can see that there’s a huge range of options available to us. But don’t fret – we can happily accept the defaults. We’re going to make a couple of small changes though: firstly we’re going to enable the SoundTouch library, which provides extra pitch and tempo manipulation tools that aren’t included in the default Ubuntu package.

Another tweak we can make is to set the location of the installed program when it has been built. To do this, we use the --prefix option with the configure script. For instance, to install Audacity in the /usr/local directory (away from other software and easier to maintain), we use --prefix=/usr/local.

As mentioned earlier, another great benefit of compiling software from source is the choice of optimizations. By setting the CFLAGS (for C code) and CXXFLAGS (for C++) environment variables before we run the configure script, we can select the exact GCC compiler optimizations we want to use. This is a complicated subject, but we can get an extra speed boost very easily. Consider these commands:

export CFLAGS=”-O3 -march=core2”

export CXXFLAGS=”-O3 -march=core2”

Here we’re setting up the aforementioned environment variables with optimizations suitable for the Intel Core 2 CPU architecture. The -O3 bit at the start has a letter O, not a zero, and requests the third level of optimization. (Most distributions compile their packages with -O2 for slightly smaller executables and easier debugging.) If you have a machine with a AMD64 chip, use -march=athlon64, and if you’re not sure, just use -march=i686 for wide compatibility. You can consult the GCC manual page (man gcc) for a complete list of supported CPU architectures, but it’s heavy reading!

So, we have our optimizations set up, and we’re ready to run the configure script. Here we go:

./configure --prefix=/usr/local --with-soundtouch=system

This tells the script that we want to install into the /usr/local directory, and enable the SoundTouch option that was listed when we ran ./configure --help before. (The ‘system’ part for the soundtouch option indicates that we want to use the version of SoundTouch available in the distro, as the one supplied with Audacity is broken.)

Why compile at all?

If you haven’t even glimpsed at a chunk of source code before, you may be perplexed by some of the terminology in use here, so let’s clarify the
terms and processes.

Compiling is the process of
converting human-readable
source code – a software
recipe that could be written in
any high-level programming
language – into the binary
instructions used by a specific
computer’s processor. For the
most popular programming
languages, C and C++, the
GNU Compiler Collection (GCC) does the hard work of
converting the source code
into a binary executable file. It
reads the source files,
translating the C or C++ code
listing into machine code
instructions for your CPU.

Different CPU families have
different types of instructions;
for instance, if you compile an
app’s source code to run on an
Intel processor, the resulting
program won’t run on a
PowerPC. You need to compile
it for each type of processor
on which it will be used.

Note that for interpreted
languages, such as Python,
Perl and Ruby, each line of the
source code is translated on
the fly. With those languages,
there is no compilation
process – the language
interpreter steps through the
program line by line.

Grab dependencies

Oops – something’s not quite right. After you’ve run the ./configure command, you’ll get this error back: ‘C compiler cannot create executables’. We need to install a development toolchain to build binaries from source, so enter:

sudo apt-get install build-essential

On Ubuntu, this installs a meta-package that pulls in all the
tools we need to compile bare-bones command-line programs. If
you’re running another distro, look for GCC, G++ and Binutils in
your package manager, and go ahead and install them. Now run
the previous configure script
again. (A handy time-saver for
those new to the command
line: hit the up cursor key and
you’ll be able to retrieve
previously entered commands.)

Now the script will proceed
more healthily, but it will stop
and inform us when it can’t find a program on the system called
wx-config. Audacity uses a graphical toolkit called WXWidgets to
render its interface, and we don’t have it installed, hence the error
message that you’ll see. So we need to grab WXWidgets from the
Synaptic package manager.

Before you run off with the mouse, though, it’s important to
note that many Linux programs, especially libraries and toolkits on
which other programs depend, are supplied in two versions. One
version is the run-time library that a program can link to; the other
is a bundle of development libraries and header files used for
compiling software. So, open Synaptic (under the System >
Administration menu) and search for WXWidgets.

Scroll down the list of results and you’ll see libwxgtk2.6-0.
This is the run-time library on which WXWidgets-using programs
depend – it provides a GTK layer on top of the WXWidgets
framework. Check that for installation, and also check
libwxgtk2.6-dev, which will provide the necessary development
bits and bobs for our source code build, and then apply the
changes. (In other distros, these packages will have similar names,
so search for WXWidgets
or wxgtk to find them.
Development packages will
almost always end with a
-dev or -devel extension.)

Now re-run the configure
script. You’ll see that Audacity
requires an older version of
WXWidgets – namely, 2.4.x!
This is pretty much the
standard process for building
from source code, which is why
we’ve illustrated it for you here; it
often takes a bit of trial and error
to work out exactly what library
dependencies and versions are
required – and it can sometimes be frustrating. So, head back into
Synaptic, remove WXWidgets 2.6, and install version 2.4 (don’t
forget the -dev package).

The magic of Checkinstall

If you’ve got several PCs, or
you’re an administrator in
charge of a large number of
corporate machines, you
probably won’t have time to
follow the whole compilation
procedure for every single
computer. Thankfully then,
there’s an RSI-eliminating tool
that enables you to create
binary packages of your
source-built programs.
Checkinstall, available from http://tinyurl.com/2uodas
(or the LXFDVD in the
Essentials section) generates
Deb, RPM and .tgz packages
for Slackware.

After you’ve installed
Checkinstall, run the
./configure script for the
software you want to build as
described in the main guide.
Then run make to compile the
software, but don’t run the
usual make install command
afterwards. Instead, as root,
run checkinstall.

You’ll then be asked a
variety of questions, such as
the type of package you want
to build, and whether you
want to include any
documentation or not. When
Checkinstall has finished its
work, you’ll have a binary
package that you can then
install on as many other
machines as you like
(providing they’re running the
same distro and have the
same libraries installed).

Build the code

Run the configure script once more and you’ll see that
everything goes smoothly this time. For other programs,
you may have more dependencies to sort out before you
can build – as mentioned, it’s all a trial-and-error process. For
instance, if a program’s configure script complains that it can’t
find the PNG development libraries, look for a package named
something like libpng-dev. If you’re building a Qt-based app, you’ll
need libqt-dev. Often the README file will provide a full list.

The command we need to kick off the GCC compiler is (drum
roll please)...

make

Simple as that. The ‘make’ utility consults a newly created file
called Makefile, which details the order in which the program’s
source code files are to be compiled. Depending on the size of
your program – and speed of your machine – the compilation
process can take anywhere between 10 seconds (for a small
command line app) to several hours (for a leviathan like
OpenOffice.org).

During the compilation phase, you’ll see that GCC is being
executed using the CFLAGS and CXXFLAGS settings we defined
before. You may also see some warning messages about nonstrict
code, but those are nothing to worry about. You will, however,
get a bit of a shock when the compilation process halts abruptly
with an error. You’ll see that SoundTouchEffect.o couldn’t be
compiled – does that mean SoundTouch is missing?

Well, yes, and the configure script should have alerted us to
that. But not all is perfect in this world, so we have to fix it
ourselves. You may recall, back at the start, that we passed the
--with-soundtouch=system option to the configure script,
instead of =local, because the version supplied with Audacity is
broken. (You would’ve found out that =local is broken, so we’ve
saved you a bit of time!)

So, configure let us down a bit there. But no big deal: open
Synaptic, search for libsoundtouch and install it (along with its
-dev sibling). We don’t need to re-run the configure step again, as
that succeeded, so enter make again and watch the output. Now
you’ll see that it can’t find FLAC. D’oh! So fire up Synaptic and
install the libflac-dev library package.

Enter ‘make’ once more, and you’ll see that the compilation
progresses a bit further, but then halts with a rather terrifying
sequence of errors, including this:

undefined reference to ‘soundtouch::SoundTouch::SoundTouch()’

Well, it looks like the compiler still can’t find SoundTouch, even
though we’ve installed it and it didn’t complain during the
configure phase. So during the last part of the compilation, when
all of the program’s files are being linked together into the
executable, GCC doesn’t know where SoundTouch is.

We need to alter the Makefile which controls the compilation
process. Enter:

gedit src/Makefile

Scroll down to line 31, where you’ll see ‘LIBS’ – that is,
the libraries with which Audacity needs to be linked. Add
-libSoundTouch to the end of the line (it’s case-sensitive), save the
file and exit the editor. Run make again. You’ll see that the main
Audacity binary compiles correctly, but the process hits another
wall: it can’t find the msgfmt utility to handle documentation.

This time we need to rely on Google: search for ‘msgfmt’ and
‘Ubuntu’ (both terms in the same search, but not in quotes as it’s
not a phrase) and you’ll see that other users have had similar
problems when compiling programs. You’ll discover that it’s in the
gettext package – but, tediously, the Ubuntu team has both
gettext and a gettext-base packages. The latter was installed as
part of the build-essential bundle, whereas the former includes
our essential msgfmt tool. Maybe one day distro developers will
stop fragmenting programs into a squillion ambiguously named
packages, but at least the solution was a quick Google away here.

So, install gettext
to get msgformat,
and run make again.
In the words of Borat,
great success! You’ll
land back at the
command line
prompt with no
errors. All we need to do now is install the newly built Audacity into
the filesystem, so enter:

sudo make install

This action needs to be performed as the root (admin) user,
hence sudo. On non-Ubuntu distros, you may need to switch to
root with su and then enter make install. And now you can run
the program:

/usr/local/bin/audacity

If you want to uninstall the program, do sudo make uninstall.
You can also remove the source code directory whenever you want
– the final program is now in /usr/local/.

Font of knowledge

You’ll notice that this version of Audacity isn’t looking so pretty on
the font front. That’s because the version of WXWidgets 2.4
included with Ubuntu is built against GTK 1, which doesn’t have
anti-aliased fonts. Recompiling WXWidgets 2.4 to use GTK 2 is
another exercise that you can try when you’ve finished this guide!
Anyway, load up a sound sample and click on the Effect menu.
You’ll see two new options that weren’t available in the stock
Ubuntu version: Change Pitch and Change Tempo.

Thanks to our geektastic compiling powers, we’ve got one up
on Ubuntu. Eat it, Shuttleworth! Given that it’s quite fiddly to add
SoundTouch support and get those two extra features, it’s not
surprising that the Ubuntu team has steered clear. Most of the
programs you compile will go completely smoothly or just require
a few obvious dependencies, but we’ve looked at lots of issues
that can crop up, so now you’re trained and ready for any
eventuality. Now go forth and compile!