Main menu

Category Archives: development

Post navigation

There has been gradual movement towards CMake in Canonical projects. I have inspected quite a lot of build setups written by many different people and certain antipatterns and inefficiencies seem to pop up again and again. Here is a list of the most common ones.

Clobbering CMAKE_CXX_FLAGS

Very often you see constructs such as these:

set(CMAKE_CXX_FLAGS "-Wall -pedantic -Wextra")

This seems to be correct, since this command is usually at the top of the top level CMakeLists.txt. The problem is that CMAKE_CXX_FLAGS may have content that comes from outside CMakeLists. As an example, the user might set values with the ccmake configuration tool. The construct above destroys those settings silently. The correct form of the command is this:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -pedantic -Wextra")

This preserves the old value.

Adding -g manually

People want to ensure that debugging information is on so they set this compiler flag manually. Often several times in different places around the source tree.

It should not be necessary to do this ever.

CMake has a concept of build identity. There are debug builds, which have debug info and no optimization. There are release builds which don’t have debug info but have optimization. There are relwithdebinfo builds which have both. There is also the default plain type which does not add any optimization or debug flags at all.

The correct way to enable debug is to specify that your build type is either debug or relwithdebinfo. CMake will then take care of adding the proper compiler flags for you. Specifying the build type is simple, just pass -DCMAKE_BUILD_TYPE=debug to CMake when you first invoke it. In day to day development work, that is what you want over 95% of the time so it might be worth it to create a shell alias just for that.

Using libraries without checking

This antipattern shows itself in constructs such as this:

target_link_libraries(myexe -lsomesystemlib)

CMake will pass the latter one to the compiler command line directly so it will link against the library. Most of the time. If the library does not exist, the end result is a cryptic linker error. The problem is worse still if the compiler in question does not understand the -l syntax for libraries (unfortunately those exist).

The solution is to use find_library and pass the result from that to target_link_libraries. This is a bit more work up front but will make the system more pleasant to use.

Adding header files to target lists

Suppose you have a declaration like this:

add_executable(myexe myexe.c myexe.h)

In this case myexe.h is entirely superfluous. It can just be dropped. The reason people put that in is probably because they think it is required to make CMake rebuild the target in case the header is changed. That is not necessary. CMake will use the dependency information from Gcc and add this dependency automatically.

The only exception to this rule is when you generate header files as part of your build. Then you should put them in the target file list so CMake knows to generate them before compiling the target.

Using add_dependencies

The second line is unnecessary. CMake knows there is a dependency between the two just based on the first line. There is no need to say it again, so the second line can be deleted.

Add_dependencies is only required in certain rare and exceptional circumstances.

Invoking make

Sometimes people use custom build steps and as part of those invoke “make sometarget”. This is not very clean on many different levels. First of all, CMake has several different backends, such as Ninja, Eclipse, XCode and others which do not use Make to build. Thus the invocation will fail on those systems. Hardcoding make invocations in your build system prevents other people from using their preferred backends. This is unfortunate as multiple backends are one of the main strengths of CMake.

Second of all, you can invoke targets directly in CMake. Most custom commands have a DEPENDS option that can be used to invoke other targets. That is the preferred way of doing this as it works with all backends.

Assuming in-source builds

Unix developers have decades worth of muscle memory telling them to build their code in-source. This leaks into various place. As an example, test data file may be accessed assuming that the source is built in-tree and that the program is executed in the directory it resides in.

Out-of-source builds provide many benefits (which I’m not going into right now, it could fill its own article). Even if you personally don’t want to use them, many other people will. Making it possible is the polite thing to do.

Inline sed and shell pipelines

Some builds require file manipulation with sed or other such shell tricks. There’s nothing wrong with them as such. The problem comes from embedding them inside CMakeLists command invocations. They should instead be put into their own script files which are then called from CMake. This makes them more easily documentable and testable.

Invoking CMake multiple times

This last piece is not a coding antipattern but a usage antipattern. People seem to run the CMake binary over again after doing changes to their build systems. This is not necessary. For any given build directory, you only ever need to run the cmake binary once: when you first configure your project.

After the first configuration the only command you ever need to run is your build command (be it make or ninja). It will detect changes in the build system, automatically regenerate all necessary files and compile the end result. The user does not need to care. This behaviour is probably residue from being burned several times by Autotools’ maintainer mode. This is understandable, but in CMake this feature will just work.

Errata: 2013/11/7

Since writing this post I have discovered that there is a valid reason for adding header files to targets. Certain IDEs such as Qt Creator and Visual Studio will only display files that belong to a target. Thus headers need to be added in the source list or otherwise they can’t be edited.

A second issue is that apparently MSBuild does not always reload a changed build file. The simple workaround for this is to re-run CMake. However it should be noted that this is an issue in MSBuild and not in CMake. Thanks to Yuri Timenkov for this information.

For a few times now Thomas Bushnell of Google has given a presentation at UDS about Google’s private Ubuntu fork. One of the interesting tidbits he mentions is that deploying a system update that requires rebooting costs the company one million dollars in lost productivity.

This gives us a nice metric to evaluate how much other operations at googleplex might cost. If we assume that a reboot takes five minutes per person, then it follows that taking one minute of all Google staff costs two hundred thousand dollars. Let’s use this piece of information to estimate how much money configure scripts are costing in lost productivity.

The duration of one configure script varies wildly. It is rare to go under 30 seconds and several minutes is not uncommon. Let’s use a round lowball estimate of one minute on average.

That million dollars accounts for every single Google employee. What percentage of those have to build source code with Autotools is unknown, so let’s say half. It is likewise hard to estimate how often these people run configure scripts, but let’s be on the safe side and say once per day. Similarly let’s assume 200 working days in one year.

Crunching all the numbers gives us the final result. That particular organisation is paying $20 million every year just to have their engineers sit still watching text scroll by in a terminal.

We have all ran into software inconveniences. These are things that you can basically do but for some reason or another are unintuitive, hard or needlessly complex. When you air your concerns on these issues, sometimes they get fixed. At other times you get back a reply starting with “Well in general you may have a point, but …”.

The rest of the sentence is something along the lines of these examples:

“… you only have to do it once so it’s no big deal.”
“… there are cases where [e.g. autodetection] would not work so having the user [manually do task X] is the only way to be reliable.”
“… I don’t see any problem, in fact I like it the way it is.”
“… replacing [the horrible thing in question] with something better is too much work.”
“… fixing that would change things and change is bad.”
“… that is the established standard way of doing things.”
“… having a human being write that in is good, it means that the input is inspected.”

These are all fine and acceptable reasonings under certain circumstances. In fact, they are great! Let’s see what life would be like in a parallel universe where people had followed them slavishly.

Booting Linux: an adventure for the brave

You arrive to your work computer and turn it on. The LILO boot prompt comes up as usually. You type in the partition you want to boot from. This you must do every time because you might have changed partition settings and thus make LILO go out of sync. You type in your boot stanza sure in the knowledge that you get 100% rock solid boot every time.

Except when you have a typo in your boot command but a computer can’t work around that. And that happens only rarely anyways and why would you boot your computer more than once per month?

Once the kernel has loaded, you type in the kernel modules you need to use the machine. You also type in all extra parameters those modules require because some chipsets may work incorrectly sometimes (or so you have been told). So you type in some dozen strings of hexadecimal numbers and really enjoy it in a stockholmesque way.

Finally all the data is put in and the system will boot itself. Then it is time to type in your network settings. In this universe there is no Protocol to Configure network Host settings Dynamically. And why would there be? Any bug in such a system would render the entire network unusable. No, the only way to ensure that things work is to configure network settings by hand every time. Errors in settings cause only one machine to break, not the entire network. Unless you mix gateway/netmask/IP addresses but surely no-one is that stupid? And if they are, it’s their own damn fault! Having things fail spectacularly is GOOD because it shames people into doing the right thing.

After this and a couple of other simple things (each of which you only need to do once, remember) you finally have a working machine. You log on.

Into a text console, naturally. Not all people need X so it should not be started by default. Resources must be used judiciously after all.

But you only need to start X once per session so no biggie. Just like you only need to write in your monitor modeline once per X startup because autodetection might fail and cause HW failure. The modeline can not be stored in a file and used automatically because you might have plugged in a different monitor. Typing it in every time is the only way to be sure. Or would you rather die horribly in a fire caused by incorrect monitor parameters?

After all that is done you can finally fire up an XTerm to start working. But today you feel like increasing the font size a bit. This is about as simple as can get. XTerm stores a list of font sizes it will display in XResources. All you have to do is to edit them, shut down X and start it up again.

Easy as pie. And the best part: you only have to do this once.

Well, once every time you want to add new font sizes. But how often is that, really?

Summary

The examples listed above are but a small fraction of reality. If computer users had to do all “one time only” things, they would easily take the entire eight hour work day. The reason they don’t is that some developer Out There has followed this simple rule:

Almost every task that needs to be done “one time only” should, in fact, be done exactly zero times.

Using libraries in C++ is simple. You just do #include<libname> and all necessary definitions appear in your source file.

But what is the cost of this single line of code?

Quite a lot, it turns out. Measuring the effect is straightforward. Gcc has a compiler flag -E, which only runs the preprocessor on the given source file. The cost of an include can be measured by writing a source file that has only one command: #include<filename>. The amount of lines in the resulting file tells how much code the compiler needs to parse in order to use the library.

Here is a table with measurements. They were run on a regular desktop PC with 4 GB of RAM and an SSD disk. The tests were run several times to insure that everything was in cache. The machine was running Ubuntu 12/10 64 bit and the compiler was gcc.

It should be noted that the elapsed time is only the amount it takes to run the code through the preprocessor. This is relatively simple compared to parsing the code and generating the corresponding machine instructions. I ran the test with Clang as well and the times were roughly similar.

Even the most common headers such as vector add almost 10k lines of code whenever they are included. This is quite a lot more than most source files that use them. On the other end of the spectrum is stuff like Boost.Python, which takes over three seconds to include. An interesting question is why it is so much slower than Nux, even though it has less code.

This is the main reason why spurious includes need to be eliminated. Simply having include directives causes massive loss of time, even if the features in question are never used. Placing a slow include in a much used header file can cause massive slowdowns. So if you could go ahead and not do that, that would be great.

A long time ago in the dawn of the new millennium lived a man. He was working as a software developer in a medium sized company. His life was pretty good all things considered. For the purposes of this narrative, let us call him Junior Developer.

At his workplace CVS was used for revision control. This was okay, but every now and then problems arose. Because it could not do atomic commits, sometimes two people would check in things at the same time, which broke everything. Sometimes this was immediately apparent and caused people to scramble around to quickly fix the code. At other times the bugs would lay dormant and break things at the worst possible times.

This irritated everyone but since the system mostly worked, this was seen as an unfortunate fact of life. Then Junior Developer found out about a brand new thing called Subversion. It seeemed to be just the thing they needed. It was used in almost exactly the same way, but it was so much better. All commits were atomic, which made mixing commits impossible. One could even rename files, a feature thus far unheard of.

This filled the heart of Junior Developer with joy. With one stroke they could eliminate most of their tooling problems and therefore improve their product’s quality. Overjoyed he waited for the next team meeting where they discussed internal processes.

Once the meeting started, the Junior Developer briefly reminded people of the problems and then explained how Subversion would fix most of the problems that CVS was causing.

At the other end of the table sat a different kind of man, who we shall call the Senior Developer. He was a man of extraordinary skill. He had personally designed and coded many of the systems that the company’s products relied on. Whenever anyone had a difficult technical issue, he would go ask the Senior Developer. His knowledge on his trade had extensive depth and breadth.

Before anyone else had time to comment on the Junior Developer’s suggestion, he grabbed the stage and let out a reply in a stern voice.

- This is not the sort of discussion we should be getting into. CVS is good and we’ll keep using it. Next issue.

The Junior Developer was shocked. He tried to form some sort of a reply but words just refused to come out of his mouth. Why was his suggestion for improvement shot down so fast? Had someone already done a Subversion test he had not heard about? Had he presented his suggestion too brazenly and offended the Senior Developer? What was going on?

These and other questions raced around in his head for the next few days. Eventually he gathered enough willpower and approached the Senior Developer during a coffee break.

- Hey, remember that discussion on Subversion and CVS we had a few days ago? Why did you declare it useless so quickly? Have you maybe tried it and found it lacking?

- I figured you might mention this. Look, I’m sure you are doing your best but the fact of the matter is that CVS is a good tool and it does everything we want it to do.

- But that’s just the thing. It does not do what we want. For example it does not have atomic commits. It would prevent check-in conflicts.

The Senior Programmer lifted his coffee cup to his lips and took a swig. Not to play for time, but simply to give emphasis to his message.

- CVS is a great tool. You just have to be careful when using it and problems like this don’t happen.

The Junior Programmer then tried to explain that even though they were being careful, breakages still happened and they had been for as long as anyone could remember. He tried explaining that in Subversion one could rename files and retain their version control history. He tried to explain how these and other features would be good, how they would allow for developers to spend more of their time on actual code and less on working around features of their tools.

The Senior Developer countered each of these issues with one of two points. Number one was the fact that you could achieve roughly the same with CVS if that is really what was wanted (with an unspoken but very clear implication that it was not). Number two was that functionality such as renames could be achieved by manually editing the repository files. This was considered a major plus, since one would not be able to do this kind of maintenance work on Subversion repositories because they were not plain text files. Any database runs the risk of corruption, which was unacceptable for something as important as source code.

The discussion went on but eventually the Senior Developer had finished his coffee and walked over to the dishwasher to put his cup away.

- Look, I appreciate the thinking you have obviously put into this but let me tell you a little something.

He put his cup away and turned to face the Junior Developer who at this point was majorly frustrated.

- I have used CVS for over 10 years. It is the best system there is. In this time there have been dozens of revision control systems that claim to have provided the same benefits that this Subversion of yours does. I have tried many of them and none have delivered. Some have been worse than the systems CVS replaced if you can believe that. The same will happen with Subversion. The advantages it claims to provide simply are not there, and that’s the sad truth.

The Junior Developer felt like shouting but controlled himself realizing that no good thing could come out of losing his temper. He slouched in his chair. The Senior Developer looked at the clearly disillusioned Junior Developer and realized the argument had ended in his favor. He left back to his office.

At the coffee room door he turned around and said his final words to the Junior Developer.

- Besides, even if the atomic commits you seem to hold so dear would work, the improvements they provide would be minimal. They are a nice-to-have toy and will never account for anything more than that.

Epilogue

This story is not true. But it could be.

Variants of this discussion are being held every single day in software development companies and communities around the world.

Software quality has received a lot of attention recently. There have been tons of books, blog posts, conferences and the like on improving quality. Tools and practices such as TDD, automatic builds, agile methods, pair programming and static code analysers are praised for improving code quality.

And, indeed, that is what they have done.

But one should never mix the tool with the person using it. All these wonderful tools are just that: tools. They are not the source of quality, only facilitators of it. The true essence of quality does not flow from them. It comes from somewhere else entirely. When distilled down to its core, there is only one source of true quality.

Caring.

The only way to get consistently high quality code is that the people who generate it care about it. This means that they have a personal interest in their code tree. They want it to succeed and flourish. In the best case they are even proud of it. This is the foundation all quality tools lie on.

If caring does not exist, even the best of tools can not help. This is due to the fact that human beings are very, very good at avoiding work they don’t want to do. As an example, let’s look at code review. A caring person will review code to the best of their abilities because he wants the end result be the best it can be. A non-caring one will shrug, think “yeah, sure, fine, whatever” and push the accept button, because it’s less work for him and he knows that his merge requests will go in easier if there is a general (though unspoken) consensus of doing things half-assed.

Unfortunately caring is not something you can buy, it is something you must birth. Free food and other services provided by companies such as Valve and Google can be seen as one way of achieving quality. If a company sincerely cares about its employees, they will in return care about the quality of their work.

Every day computer users do a variation of a simple task: selecting an element from a list of choices. Examples include installing packages with ‘apt-get install packagename’, launching an application from the Dash with its name, selecting your country from a list on web pages and so on.

The common thing in all these use cases is intolerance for errors. If you have just one typo in your text, the correct choice will not be found. The only way around is to erase the query and type it again from scratch. This is something that people have learned to do without thinking.

It’s not very good usability, though. If the user searches for, say, Firefox by typing “friefox” by accident, surely the computer should be able to detect what the user meant and offer that as an alternative.

The first user-facing program in Ubuntu to offer this kind of error tolerance was the HUD. It used the Levenshtein distance as a way of determining user intent. In computer science terminology this is called approximate string matching or, informally, fuzzy matching.

Once the HUD was deployed, the need to have this kind of error correction everywhere became apparent. Thus we sent out to create a library to make error tolerant matching easy to embed. This library is called libcolumbus.

Technical info

Libcolumbus has been designed with the following goals in mind:

it must be small

it must be fast

it must be easy to embed

it is optimized for online typing

The last of these means that you can do queries at any time, even if the user is still typing.

What sets Columbus apart from these are two things, both of which are well known and documented but less often used: a fast search implementation and custom errors.

The first feature is about performance. The fast Levenshtein implementation in libcolumbus is taken almost verbatim from this public domain implementation. The main speedup comes from using a trie to store the words instead of iterating over all items on every query. As a rough estimate, a brute force implementation can do 50-100 queries a second with a data set of 3000 words. The trie version can do 600 queries/second on a data set of 50000 words.

The second feature is about quality of results. It is best illustrated with an example. Suppose there are two items to choose from, “abc” and “abp”. If the user types “abo”, which one of these should be chosen? In the classical Levenshtein sense both of the choices are identical: they are one replace operation away from the query string.

However from a usability point of view “abp” is the correct answer, because the letter p is right next to the letter o and very far from the letter c. The user probably meant to hit the key o but just missed it slightly. Libcolumbus allows you to set custom errors for these
kinds of substitutions. If the standard substitution error is 100, one could set the error for substitution error for adjacent keys to a smaller value, say 20. This causes words with “simple typos” to be ranked higher automatically.

There are several other uses for custom errors:

diacritical characters such as ê, é and è can be mapped to have very small errors to each other

fuzzy number pad typing can be enabled by assigning mapping errors from the number to corresponding letters (e.g. ’3′ to ‘d’, ‘e’ and ‘f’) as well as adjacent letters (i.e. those on number keys ’2′ and ’6′)

spam can be detected by assigning low errors for letters and numbers that look similar, such as ’1′ -> ‘i’ and ’4′ -> ‘a’ to match ‘v14gr4′ to ‘viagra’

Libcolumbus contains sample implementations for all these except for the last one. It also allows setting insert and delete errors at the beginning and end of the match. When set to low values this makes the algorithm do a fuzzy substring search. The online matching discussed above is implemented with this. It allows the library to match the query term “fier” to “firefox” very fast.

Get the code

Our goal for the coming cycle is to enable error tolerant matching in as many locations as possible. Those developers who wish to try it on their application can get the source code here.

The library is implemented in C++0x. The recommended API to use is the C++ one. However since many applications can not link in C++ libraries, we also provide a plain C API. It is not as extensive as the C++ one, but we hope to provide full coverage there too.

The main thing to understand is the data model. Libcolumbus deals in terms of documents. A document consists of a (user provided) document ID and a named collection of texts. The ID field is guaranteed to be large enough to hold a pointer. Here’s an example of what a document could look like:

id: 42
name: packagename
description: This package does something.

Each line is a single word field name followed by the text it contains. A document can contain an arbitrary number of fields. This is roughly analogous to what MongoDB uses. It should be noted that libcolumbus does not read any data files. The user needs to create document objects programmatically. The example above is just a visualisation.

When the documents are created and passed to the main matcher object for processing, the system is ready to be queried. The result of queries is a list of document IDs and corresponding relevancies. Relevancy is just a number whose meaning is roughly “bigger relevancy means better”. The exact values are arbitrary and may change even between queries. End-user applications usually don’t need to bother with them.

There is one thing to be mindful, though. The current implementation has a memory backend only. Its memory usage is moderate but it has not yet been thoroughly optimized. If your data set size is a few hundred unique words, you probably don’t have to care. A few thousand takes around 5 MB which may be a problem in low memory
devices. Tens of thousands of words take tens of megabytes which may be too much for many use cases. Both memory optimizations and a disk backend are planned but for now you might want to stick to smallish data sets.

There have been several posts in this blog about compile speed. However most have been about theory. This time it’s all about measurements.

I took the source code of Scribus, which is a largeish C++ application and looked at how much faster I could make it compile. There are three different configurations to test. The first one is building with default settings out of the box. The second one is about changes that can be done without changing any source code, meaning building with the Ninja backend instead of Make and using Gold instead of ld. The third configuration adds precompiled headers to the second configuration.

The measurements turned out to have lots of variance, which I could not really nail down. However it seemed to affect all configurations in the same way at the same time so the results should be comparable. All tests were run on a 4 core laptop with 4 GB of ram. Make was run with ‘-j 6′ as that is the default value of Ninja.

Default: 11-12 minutes
Ninja+Gold: ~9 minutes
PCH: 7 minutes

We can see that a bit of work the compile time can be cut almost in half. Enabling PCH does not require changing any existing source files (though you’ll get slightly better performance if you do). All in all it takes less than 100 lines of CMake code to enable precompiled headers, and half of that is duplicating some functionality that CMake should be exposing already. For further info, see this bug.

Is it upstream? Can I try it? Will it work on my project?

The patch is not upstreamed, because it is not yet clean enough. However you can check out most of it in this merge request to Unity. In Unity’s case the speedup was roughly 40%, though only one library build time was measured. The total build time impact is probably less.

Note that if you can’t just grab the code and expect magic speedups. You have to select which headers to precompile and so on.

Finally, for a well tuned code base, precompiled headers should only give around 10-20% speed boost. If you get more, it probably means that you have an #include maze in your header files. You should probably get that fixed sooner rather than later.

A relatively large portion of software development time is not spent on writing, running, debugging or even designing code, but waiting for it to finish compiling. This is usually seen as necessary evil and accepted as an unfortunate fact of life. This is a shame, because spending some time optimizing the build system can yield quite dramatic productivity gains.

Suppose a build system takes some thirty seconds to run for even trivial changes. This means that even in theory you can do at most two changes a minute. In practice the rate is a lot lower. If the build step takes only a few seconds, trying out new code becomes a lot faster. It is easier to stay in the zone when you don’t have to pause every so often to wait for your tools to finish doing their thing.

Making fundamental changes in the code often triggers a complete rebuild. If this takes an hour or more (there are code bases that take 10+ hours to build), people try to avoid fundamental changes as much as possible. This causes loss of flexibility. It becomes very tempting to just do a band-aid tweak rather than thoroughly fix the issue at hand. If the entire rebuild could be done in five to ten minutes, this issue would become moot.

In order to make things fast, we first have to understand what is happening when C/C++ software is compiled. The steps are roughly as follows:

Configuration

Build tool startup

Dependency checking

Compilation

Linking

We will now look at each step in more detail focusing on how they can be made faster.

Configuration

This is the first step when starting to build. Usually means running a configure script or CMake, Gyp, SCons or some other tool. This can take anything from one second to several minutes for very large Autotools-based configure scripts.

This step happens relatively rarely. It only needs to be run when changing configurations or changing the build configuration. Short of changing build systems, there is not much to be done to make this step faster.

Build tool startup

This is what happens when you run make or click on the build icon on an IDE (which is usually an alias for make). The build tool binary starts and reads its configuration files as well as the build configuration, which are usually the same thing.

Depending on build complexity and size, this can take anywhere from a fraction of a second to several seconds. By itself this would not be so bad. Unfortunately most make-based build systems cause make to be invocated tens to hundreds of times for every single build. Usually this is caused by recursive use of make (which is bad).

It should be noted that the reason Make is so slow is not an implementation bug. The syntax of Makefiles has some quirks that make a really fast implementation all but impossible. This problem is even more noticeable when combined with the next step.

Dependency checking

Once the build tool has read its configuration, it has to determine what files have changed and which ones need to be recompiled. The configuration files contain a directed acyclic graph describing the build dependencies. This graph is usually built during the configure step. Suppose we have a file called SomeClass.cc which contains this line of code:

#include "OtherClass.hh"

This means that whenever OtherClass.hh changes, the build system needs to rebuild SomeClass.cc. Usually this is done by comparing the timestamp of SomeClass.o against OtherClass.hh. If the object file is older than the source file or any header it includes, the source file is rebuilt.

Build tool startup time and the dependency scanner are run on every single build. Their combined runtime determines the lower bound on the edit-compile-debug cycle. For small projects this time is usually a few seconds or so. This is tolerable.

The problem is that Make scales terribly to large projects. As an example, running Make on the codebase of the Clang compiler with no changes takes over half a minute, even if everything is in cache. The sad truth is that in practice large projects can not be built fast with Make. They will be slow and there’s nothing that can be done about it.

There are alternatives to Make. The fastest of them is Ninja, which was built by Google engineers for Chromium. When run on the same Clang code as above it finishes in one second. The difference is even bigger when building Chromium. This is a massive boost in productivity, it’s one of those things that make the difference between tolerable and pleasant.

If you are using CMake or Gyp to build, just switch to their Ninja backends. You don’t have to change anything in the build files themselves, just enjoy the speed boost. Ninja is not packaged on most distributions, though, so you might have to install it yourself.

If you are using Autotools, you are forever married to Make. This is because the syntax of autotools is defined in terms of Make. There is no way to separate the two without a backwards compatibility breaking complete rewrite. What this means in practice is that Autotool build systems are slow by design, and can never be made fast.

Compilation

At this point we finally invoke the compiler. Cutting some corners, here are the approximate steps taken.

Merging includes

Parsing the code

Code generation/optimization

Let’s look at these one at a time. The explanations given below are not 100% accurate descriptions of what happens inside the compiler. They have been simplified to emphasize the facets important to this discussion. For a more thorough description, have a look at any compiler textbook.

The first step joins all source code in use into one clump. What happens is that whenever the compiler finds an include statement like #include “somefile.h”, it finds that particular source file and replaces the #include with the full contents of that file. If that file contained other #includes, they are inserted recursively. The end result is one big self-contained source file.

The next step is parsing. This means analyzing the source file, splitting it into tokens and building an abstract syntax tree. This step translates the human understandable source code into a computer understandable unambiguous format. It is what allows the compiler to understand what the user wants the code to do.

Code generation takes the syntax tree and transforms it into machine code sequences called object code. This code is almost ready to run on a CPU.

Each one of these steps can be slow. Let’s look at ways to make them faster.

Faster #includes

Including by itself is not slow, slowness comes from the cascade effect. Including even one other file causes everything included in it to be included as well. In the worst case every single source file depends on every header file. This means that touching any header file causes the recompilation of every source file whether they use that particular header’s contents or not.

Cutting down on interdependencies is straightforward. Only #include those headers that you actually use. In addition, header files must not include any other header files if at all possible. The main tool for this is called forward declaration. Basically what it means is that instead of having a header file that looks like this:

#include "SomeClass.hh"
class MyClass {
SomeClass s;
};

You have this:

class SomeClass;
class MyClass {
SomeClass *s;
}

Because the definition of SomeClass is not know, you have to use pointers or references to it in the header.

Remember that #including MyClass.hh would have caused SomeClass.hh and all its #includes to be added to the original source file. Now they aren’t, so the compiler’s work has been reduced. We also don’t have to recompile the users of MyClass if SomeClass changes. Cutting the dependency chain like this everywhere in the code base can have a major effect in build time, especially when combined with the next step. For a more detailed analysis including measurements and code, see here.

Faster parsing

The most popular C++ libraries, STL and Boost, are implemented as header only libraries. That is, they don’t have a dynamically linkable library but rather the code is generated anew into every binary file that uses them. Compared to most C++ code, STL and Boost are complex. Really, really complex. In fact they are most likely the hardest pieces of code a C++ compiler has to compile. Boost is often used as a stress test on C++ compilers, because it is so difficult to compile.

It is not an exaggeration to say that for most C++ code using STL, parsing the STL headers is up to 10 times slower than parsing all the rest. This leads to massively slow build times because of class headers like this:

As we learned in the previous chapter, this means that every single file that includes this header must parse STL’s vector definition, which is an internal implementation detail of SomeClass and even if they would not use vector themselves. Add some other class include that uses a map, one for unordered_map, a few Boost includes and what do you end up with? A code base where compiling any file requires parsing all of STL and possibly Boost. This is a factor of 3-10 slowdown on compile times.

Getting around this is relatively simple, though takes a bit of work. It is known as the pImpl idiom. One way of achieving it is this:

Now the dependency chain is cut and users of SomeClass don’t have to parse vector. As an added bonus the vector can be changed to a map or anything else without needing to recompile files that use SomeClass.

Faster code generation

Code generation is mostly an implementation detail of the compiler, and there’s not much that can be done about it. There are a few ways to make it faster, though.

Optimizing code is slow. In every day development all optimizations should be disabled. Most build systems do this by default, but Autotools builds optimized binaries by default. In addition to being slow, this makes debugging a massive pain, because most of the time trying to print the value of some variable just prints out “value optimised out”.

Making Autotools build non-optimised binaries is relatively straightforward. You just have to run configure like this: ./configure CFLAGS=’O0 -g’ CXXFLAGS=’-O0 -g’. Unfortunately many people mangle their autotools cflags in config files so the above command might not work. In this case the only fix is to inspect all autotools config files and fix them yourself.

The other trick is about reducing the amount of generated code. If two different source files use vector<int>, the compiler has to generate the complete vector code in both of them. During linking (discussed in the next chapter) one of them is just discarded. There is a way to tell the compiler not to generate the code in the other file using a technique that was introduced in C++0x called extern templates. They are used like this.

This instructs the compiler not to generate vector code when compiling file B. The linker makes it use the code generated in file A.

Build speedup tools

CCache is an application that stores compiled object code into a global cache. If the same code is compiled again with the same compiler flags, it grabs the object file from the cache rather than running the compiler. If you have to recompile the same code multiple times, CCache may offer noticeable speedups.

A tool often mentioned alongside CCache is DistCC, which increases parallelism by spreading the build to many different machines. If you have a monster machine it may be worth it. On regular laptop/desktop machines the speed gains are minor (it might even be slower).

Precompiled headers

Precompiled headers is a feature of some C++ compilers that basically serializes the in-memory representation of parsed code into a binary file. This can then be read back directly to memory instead of reparsing the header file when used again. This is a feature that can provide massive speedups.

Out of all the speedup tricks listed in this post, this has by far the biggest payoff. It turns the massively slow STL includes into, effectively, no-ops.

So why is it not used anywhere?

Mostly it comes down to poor toolchain support. Precompiled headers are fickle beasts. For example with GCC they only work between two different compilation units if the compiler switches are exactly the same. Most people don’t know that precompiled headers exist, and those that do don’t want to deal with getting all the details right.

CMake does not have direct support for them. There are a few modules floating around the Internet, but I have not tested them myself. Autotools is extremely precompiled header hostile, because its syntax allows for wacky and dynamic alterations of compiler flags.

Faster Linking

When the compiler compiles a file and comes to a function call that is somewhere outside the current file, such as in the standard library or some other source file, it effectively writes a placeholder saying “at this point jump to function X”. The linker takes all these different compiled files and connects the jump points to their actual locations. When linking is done, the binary is ready to use.

Linking is surprisingly slow. It can easily take minutes on relatively large applications. As an extreme case, linking the Chromium browser on ARM takes 3 gigs of RAM and takes 18 hours.

Yes, hours.

The main reason for this is that the standard GNU linker is quite slow. Fortunately there is a new, faster linker called Gold. It is not the default linker yet, but hopefully it will be soon. In the mean time you can install and use it manually.

A different way of making linking faster is to simply cut down on these symbols using a technique called symbol visibility. The gist of it is that you hide all non-public symbols from the list of exported symbols. This means less work and memory use for the linker, which makes it faster.

Conclusions

Contrary to popular belief, compiling C++ is not actually all that slow. The STL is slow and most build tools used to compile C++ are slow. However there are faster tools and ways to mitigate the slow parts of the language.

Using them takes a bit of elbow grease, but the benefits are undeniable. Faster build times lead to happier developers, more agility and, eventually, better code.

Say you start work on a new code base. Would you, as a user, rather have 90% or 10% of its API functions commented with Doxygen or something similar?

Using my psychic powers I suspect that you chose 90%.

It seems like the obvious thing. Lots of companies even have a mandate that all API functions (or >90% of them) must be documented. Not having comments is just bad. This seems like a perfectly obvious no-brainer issue.

But is it really?

Unfortunately there are some problems with this assumption. The main one being that the comments will be written by human beings. What they probably end up being is something like this.

This something I like to call documentation by word order shuffle. Now we can ask the truly relevant question: what additional information does this kind of a comment provide?

The answer is, of course, absolutely nothing. It is only noise. No, actually it is even worse: it is noise that has a very large probability of being wrong. When some coder changes the function, it is very easy to forget to update the comments.

On the other hand, if only 10% of the functions are documented, most functions don’t have any comments, but the ones that do probably have something like this:

/**
* The Foobar argument must not have been initialized in a different
* thread because that can lead to race conditions.
*/
int frobnicate_foobar(Foobar f, int strength)

This is the kind of comment that is actually useful. Naturally it would be better to check for the specified condition inside the function but sometimes you can’t. Having it in a comment is the right thing to do in these cases. Not having tons of junk documentation makes these kinds of remarks stand out. This means, paradoxically, that having less comments leads to better documentation and user experience.

As a rough estimate, 95% of functions in any code base should be so simple and specific that their signature is all you need to use them. If they are not, API design has failed: back to the drawing board.