Understanding CROSSTOOL

Overview

CROSSTOOL is a text file containing a
protocol buffer
that provides the necessary level of granularity for configuring the behavior of
Bazel’s C++ rules. By default, Bazel automatically configures CROSSTOOL for
your build, but you have the option to configure it manually. You reference the
CROSSTOOL file in your BUILD file(s) using a cc_toolchain target and check
it into source control alongside your project. You can share a single
CROSSTOOL file across multiple projects or create separate per-project files.

When a C++ target enters the analysis phase, Bazel selects the appropriate
cc_toolchain target based on the BUILD file, then reads the corresponding
toolchain definition from the CROSSTOOL file. The cc_toolchain target
passes information from the CROSSTOOL proto to the C++ target through a
CcToolchainProvider.

For example, a compile or link action, instantiated by a rule such as
cc_binary or cc_library, needs the following information:

The compiler or linker to use

Command-line flags for the compiler/linker

Configuration flags passed through the --copt/--linkopt options

Environment variables

Artifacts needed in the sandbox in which the action executes

All of the above information except the artifacts required in the sandbox is
specified in the CROSSTOOL proto.

The artifacts to be shipped to the sandbox are declared in the cc_toolchain
target. For example, with the cc_toolchain.linker_files attribute you can
specify the linker binary and toolchain libraries to ship into the sandbox.

CROSSTOOL proto structure

The CROSSTOOL proto has the following structure:

Map from --cpu to toolchain (to be used when --compiler is not specified
or when cc_toolchain_suite.toolchains omits the cpu entry)

Toolchain for a particular --cpu and --compiler combination (1)

Static toolchain:

compiler_flags

linker_flags

compilation_mode_flags

linking_mode_flags

Dynamic toolchain:

features

Toolchain for a particular --cpu and --compiler combination (2)

Toolchain for a particular --cpu and --compiler combination (3)

…

Toolchain selection

The toolchain selection logic operates as follows:

User specifies a cc_toolchain_suite target in the BUILD file and points
Bazel to the target using the
--crosstool_top option.
The CROSSTOOL file must reside in the same directory as the
BUILD file containing the cc_toolchain_suite target.

The cc_toolchain_suite target and the CROSSTOOL
file reference multiple toolchains. The values of the --cpu and
--compiler flags determine which of those toolchains is selected,
either based only on the --cpu flag value, or based on a joint
--cpu | --compiler value. The selection process is as follows:

If the --compiler option is specified, Bazel selects the
corresponding entry from the cc_toolchain_suite.toolchains
attribute with --cpu | --compiler. If Bazel does not find
a corresponding entry, it throws an error.

If the --compiler option is not specified, Bazel selects
the corresponding entry from the cc_toolchain_suite.toolchains
attribute with just --cpu.

However, if Bazel does not find a corresponding entry and the
--incompatible_disable_cc_toolchain_label_from_crosstool_proto
option is disabled, Bazel iterates through default_toolchains
in the CROSSTOOL file until it finds an entry where the
default_toolchain.cpu value matches the specified --cpu
option value. Bazel then reads the toolchain_identifier
value to identify the corresponding toolchain, and selects the appropriate
entry in the cc_toolchain_suite.toolchains attribute using
toolchain.target_cpu | toolchain.compiler.

If no flags are specified, Bazel inspects the host system and selects a
--cpu value based on its findings. See the
inspection mechanism code.

Once a toolchain has been selected, corresponding feature and action_config
messages in the CROSSTOOL file govern the configuration of the build (that is,
items described earlier in this document). These messages allow the
implementation of fully fledged C++ features in Bazel without modifying the
Bazel binary. C++ rules support multiple unique actions documented in detail
in the Bazel source code.

CROSSTOOL features

A feature is an entity that requires non-default command-line flags, actions,
constraints on the execution environment, or dependency alterations. A feature
can be something as simple as allowing BUILD files to select configurations of
flags, such as treat_warnings_as_errors, or interact with the C++ rules and
include new compile actions and inputs to the compilation, such as
header_modules or thin_lto.

Ideally, a toolchain definition consists of a set of features, where each
feature consists of multiple flag groups, each defining a list of flags that
apply to specific Bazel actions.

A feature is specified by name, which allows full decoupling of the CROSSTOOL
configuration from Bazel releases. In other words, a Bazel release does not
affect the behavior of CROSSTOOL configurations as long as those
configurations do not require the use of new features.

A feature is enabled only when both Bazel and the CROSSTOOL configuration
support it.

Features can have interdependencies, depend on command line flags, BUILD file
settings, and other variables.

Feature relationships

Dependencies are typically managed directly with Bazel, which simply enforces
the requirements and manages conflicts intrinsic to the nature of the features
defined in the build. The toolchain specification allows for more granular
constraints for use directly within the CROSSTOOL file that govern feature
support and expansion. These are:

Constraint

Description

requires: ['feature1', 'feature2']

Feature-level. The feature is supported only if the specified required
features are enabled. For example, when a feature is only supported in
certain build modes (features opt, dbg, or
fastbuild). Multiple `requires` statements are
satisfied all at once if any one of them is satisfied.

implies: 'feature'

Feature-level. This feature implies the specified feature. For example, a
module compile implies the need for module maps, which can be implemented
by a repeated implies string in the feature where each of
the strings names a specific feature. Enabling a feature also implicitly
enables all features implied by it (that is, it functions recursively).

Also provides the ability to factor common subsets of functionality out of
a set of features, such as the common parts of sanitizers. Implied
features cannot be disabled.
</td>
</tr>

provides: 'feature'

Feature-level. Indicates that this feature is one of several mutually
exclusive alternate features. For example, all of the sanitizers could
specify provides: "sanitizer".

This improves error handling by listing the alternatives if the user asks
for two or more mutually exclusive features at once.
</td>
</tr>

with_feature: {feature: 'feature1', not_feature: 'feature2']

Flag set-level. A feature can specify multiple flag sets with multiple
with_feature statements. When with_feature is
specified, the flag set will only expand to the build command if all of the
feature in the specified feature: set are enabled, and all the
features specified in not_feature: set are disabled.

</table>
## `CROSSTOOL` actions
`CROSSTOOL` actions provide the flexibility to modify the circumstances under
which an action executes without assuming how the action will be run. An
`action_config` specifies the tool binary that an action invokes, while a
`feature` specifies the configuration (flags) that determine how that tool
behaves when the action is invoked.
[Features](#crosstool-features) reference `CROSSTOOL` actions to signal which
Bazel actions they affect since `CROSSTOOL` actions can modify the Bazel action
graph. The `CROSSTOOL` file includes actions that have flags and tools
associated with them, such as `c++-compile`. Flags are assigned to each action
by associating them with a feature.
Each `CROSSTOOL` action name represents a single type of action performed by
Bazel, such as compiling or linking. There is, however, a many-to-one
relationship between `CROSSTOOL` actions and Bazel action types, where a Bazel
action type refers to a Java class that implements an action (such as
`CppCompileAction`). In particular, the "assembler actions" and "compiler
actions" in the table below are `CppCompileAction`, while the link actions are
`CppLinkAction`.
### Assembler actions

Action

Description

preprocess-assemble

Assemble with preprocessing. Typically for .S files.

assemble

Assemble without preprocessing. Typically for .s files.

### Compiler actions

Action

Description

cc-flags-make-variable

Propagates CC_FLAGS to genrules.

c-compile

Compile as C.

c++-compile

Compile as C++.

c++-header-parsing

Run the compiler's parser on a header file to ensure that the header is
self-contained, as it will otherwise produce compilation errors. Applies
only to toolchains that support modules.

## `CROSSTOOL` `action_config`
A `CROSSTOOL` `action_config` is a proto message that describes a Bazel action
by specifying the tool (binary) to invoke during the action and sets of flags,
defined by features, that apply constraints to the action's execution. A
`CROSSTOOL` action takes the following attributes:

Attribute

Description

action_name

The Bazel action to which this CROSSTOOL action corresponds.
Bazel uses this attribute to discover per-action tool and execution
requirements.

tool

The executable to invoke. This can depend on a feature. A default must be
provided.

flag_set

A set of flags that applies to a group of actions. Same as for a feature.

env_set

A set of environment constraints that applies to a group of actions. Same
as for a feature.

A `CROSSTOOL` `action_config` can require and imply other features and
action_configs as dictated by the
[feature relationships](#feature-relationships) described earlier. This behavior
is similar to that of a feature.
The last two attributes are redundant against the corresponding attributes on
features and are included because some Bazel actions require certain flags or
environment variables and we want to avoid unnecessary `action_config`+`feature`
pairs. Typically, sharing a single feature across multiple `action_config`s is
preferred.
You can not define more than one `CROSSTOOL` `action_config` with the same
`action_name` within the same toolchain. This prevents ambiguity in tool paths
and enforces the intention behind `action_config` - that an action's properties
are clearly described in a single place in the toolchain.
### `tool` messages
A `CROSSTOOL` `action_config` can specify a set of tools via `tool` messages.
A `tool` message consists of the following fields:

Field

Description

tool_path

Path to the tool in question (relative to the CROSSTOOL
file).

with_feature

A set of features that must be enabled for this tool to apply.

For a given `CROSSTOOL` `action_config`, only a single `tool` message applies
its tool path and execution requirements to the Bazel action. A tool is selected
by sequentially parsing `tool` messages on an `action_config` until a tool with
a `with_feature` set matching the feature configuration is found
(see [Feature relationships](#feature-relationships) earlier in this document
for more information). We recommend that you end your tool lists with a default
tool that corresponds to an empty feature configuration.
### Example usage
Features and `CROSSTOOL` actions can be used together to implement Bazel actions
with diverse cross-platform semantics. For example, debug symbol generation on
macOS requires generating symbols in the compile action, then invoking a
specialized tool during the link action to create compressed dsym archive, and
then decompressing that archive to produce the application bundle and `.plist`
files consumable by Xcode.
With Bazel, this process can instead be implemented as follows, with
`unbundle-debuginfo` being a Bazel action:
```
toolchain {
action_config {
config_name: "c++-link-executable"
action_name: "c++-link-executable"
tool {
with_feature { feature: "generate-debug-symbols" }
tool_path: "toolchain/mac/ld-with-dsym-packaging"
}
tool {
tool_path: "toolchain/mac/ld"
}
}
feature {
name: "generate-debug-symbols"
flag_set {
action: "c-compile"
action: "c++-compile"
flag_group {
flag: "-g"
}
}
implies: { feature: "unbundle-debuginfo" }
}
}
```
This same feature can be implemented entirely differently for Linux, which uses
`fission`, or for Windows, which produces `.pdb` files. For example, the
implementation for `fission`-based debug symbol generation might look as
follows:
```
toolchain {
action_config {
name: "c++-compile"
tool {
tool_path: "toolchain/bin/gcc"
}
}
feature {
name: "generate-debug-symbols"
requires { feature: "dbg" }
flag_set {
action: "c++-compile"
flag_group {
flag: "-gsplit-dwarf"
}
}
flag_set {
action: "c++-link-executable"
flag_group {
flag: "-Wl"
flag: "--gdb-index"
}
}
}
}
}
```
### Flag groups
`CROSSTOOL` allows you to bundle flags into groups that serve a specific purpose.
You can specify a flag within the `CROSSTOOL` file using pre-defined variables
within the flag value, which the compiler expands when adding the flag to the
build command. For example:
```
flag_group {
flag: "%{output_file_path}
}
```
In this case, the contents of the flag will be replaced by the output file path
of the action.
Flag groups are expanded to the build command in the order in which they appear
in the `CROSSTOOL` file, top-to-bottom, left-to-right.
For flags that need to repeat with different values when added to the build
command, the flag group can iterate variables of type `list`. For example, the
variable `include_path` of type `list`:
```
flag_group {
iterate_over: "include_paths"
flag: "-I%{include_paths}"
}
```
expands to `-I` for each path element in the `include_paths` list. All
flags (or `flag_group`s) in the body of a flag group declaration are expanded as
a unit. For example:
```
flag_group {
iterate_over: "include_paths"
flag: "-I"
flag: "%{include_paths}"
}
```
expands to `-I ` for each path element in the `include_paths` list.
A variable can repeat multiple times. For example:
```
flag_group {
iterate_over: "include_paths"
flag: "-iprefix=%{include_paths}"
flag: "-isystem=%{include_paths}"
}
```
expands to:
```
-iprefix= -isystem= -iprefix= -isystem=
```
Variables can correspond to structures accessible using dot-notation. For
example:
```
flag_group {
flag: "-l%{libraries_to_link.name}"
}
```
Structures can be nested and may also contain sequences. To prevent name clashes
and to be explicit, you must specify the full path through the fields. For
example:
```
flag_group {
iterate_over: "libraries_to_link"
flag_group {
iterate_over: "libraries_to_link.shared_libraries"
flag: "-l%{libraries_to_link.shared_libraries.name}"
}
}
```
### Conditional expansion
Flag groups support conditional expansion based on the presence of a particular
variable or its field using the `expand_if_all_available`, `expand_if_none_available`,
`expand_if_true`, `expand_if_false`, or `expand_if_equal` messages. For example:
```
flag_group {
iterate_over: "libraries_to_link"
flag_group {
iterate_over: "libraries_to_link.shared_libraries"
flag_group {
expand_if_all_available: "libraries_to_link.shared_libraries.is_whole_archive"
flag: "--whole_archive"
}
flag_group {
flag: "-l%{libraries_to_link.shared_libraries.name}"
}
flag_group {
expand_if_all_available: "libraries_to_link.shared_libraries.is_whole_archive"
flag: "--no_whole_archive"
}
}
}
```
**Note:** The `--whole_archive` and `--no_whole_archive` options are added to
the build command only when a currently iterated library has an
`is_whole_archive` field.
## `CROSSTOOL` reference
This section provides a reference of build variables, features, and other
information required to successfully configure `CROSSTOOL`.
### `CROSSTOOL` build variables
The following is a reference of `CROSSTOOL` build variables.
**Note:** `[action]` indicates the relevant action type.

Variable

Description

source_file

[compile] Source file to compile.

input_file

[strip] Artifact to strip.

output_file

[compile] Compilation output.

output_assembly_file

[compile] Emitted assembly file. Applies only when the
compile action emits assembly text, typically when using the
--save_temps flag. The contents are the same as for
output_file.

output_preprocess_file

[compile] Preprocessed output. Applies only to compile
actions that only preprocess the source files, typically when using the
--save_temps flag. The contents are the same as for
output_file.

includes

[compile] Sequence of files the compiler must
unconditionally include in the compiled source.

include_paths

[compile] Sequence directories in which the compiler
searches for headers included using #include<foo.h>
and #include "foo.h".

quote_include_paths

[compile] Sequence of -iquote includes -
directories in which the compiler searches for headers included using
#include<foo.h>.

system_include_paths

[compile] Sequence of -isystem includes -
directories in which the compiler searches for headers included using
#include "foo.h".

dependency_file

[compile] The .d dependency file generated by
the compiler.

preprocessor_defines

[compile] Sequence of defines, such as
--DDEBUG.

pic

[compile] Compiles the output as position-independent code.

gcov_gcno_file

[compile] The gcov coverage file.

per_object_debug_info_file

[compile] The per-object debug info (.dwp)
file.

stripotps

[strip] Sequence of stripopts.

legacy_compile_flags

[compile] Sequence of flags from legacy
CROSSTOOL fields such as compiler_flag,
optional_compiler_flag, cxx_flag, and
optional_cxx_flag.

user_compile_flags

[compile] Sequence of flags from either the
copt rule attribute or the --copt,
--cxxopt, and --conlyopt flags.

unfiltered_compile_flags

[compile] Sequence of flags from the
unfiltered_cxx_flag legacy CROSSTOOL field or the
unfiltered _compile_flags feature. These are not filtered by
the nocopts rule attribute.

sysroot

The sysroot.

runtime_library_search_directories

[link] Entries in the linker runtime search path (usually
set with the -rpath flag).

library_search_directories

[link] Entries in the linker search path (usually set with
the -L flag).

libraries_to_link

[link] Flags providing files to link as inputs in the linker
invocation.

def_file_path

[link] Location of def file used on Windows with MSVC.

linker_param_file

[link] Location of linker param file created by bazel to
overcome command line length limit.

output_execpath

[link] execpath of the output of the linker.

generate_interface_library

[link] "yes"|"no" depending on whether interface library
should be generated.

interface_library_builder_path

[link] Path to the interface library builder tool.

interface_library_input_path

[link] Input for the interface library ifso
builder tool.

interface_library_output_path

[link] Path where to generate interface library using the
ifso builder tool.

legacy_link_flags

[link] Linker flags coming from the legacy
CROSSTOOL.

user_link_flags

[link] Linker flags coming from the --linkopt
or linkopts attribute.

symbol_counts_output

[link] Path to which to write symbol counts.

linkstamp_paths

[link] A build variable giving linkstamp paths.

force_pic

[link] Presence of this variable indicates that PIC code
should be generated.

strip_debug_symbols

[link] Presence of this variable indicates that the debug
symbols should be stripped.

is_cc_test

[link] Truthy when current action is a cc_test
linking action, false otherwise.

is_using_fission

[link] Presence of this variable indicates that files were
compiled with fission. Debug info is in .dwo files instead
of .o files and the linker needs to know this.

### CROSSTOOL features
The following is a reference of `CROSSTOOL` features and their activation
conditions.

Feature

Activation Condition

opt | dbg | fastbuild

Enabled by default based on compilation mode.

static_linking_mode | dynamic_linking_mode

Enabled by default based on linking mode.

random_seed

Enabled by default.

dependency_file

Enabled by default.

per_object_debug_info

Enabled if the supports_fission attribute is set in the
`CROSSTOOL` file and the current compilation mode is specified in the
--fission flag.