But…​ why?

First things first: why dibs? The main need is to pack a Docker
container image starting from some code, much like what can be done with
a Dockerfile… let’s
look at a few problems and how dibs addresses them.

Trimming Container Images

… it might need additional pre-requisites in form of other software or
libraries, often provided by the underlying Linux distribution in the form of
packages

… it might need to have some parts compiled or undergo a build
process

… it will probably need the container to be set so that the invocation
of the program is eased.

The build process usually requires tools (like a C compiler, development
versions of libraries, etc.) that are rarely needed during the runtime phase.
To cope with this, release 17.05 of Docker introduced a new feature where a
Dockerfile can include
multiple
stages[1]
(each marked with an initial FROM section), where it’s possible to perform
e.g. compilation within a specific track and later use those artifacts inside
another container started from a leaner image.

dibs addresses this issue with the idea that the path from the code to the
Docker image is not necessarily linear and might be walked through different
lanes, or phases:

each phase runs a sequence of operations that stack upon a container image
much like a Dockerfile
does;

different phases share artifacts around through shared directories; only the

containers of interest are saved as images, others are thrown away.

As an example, a common patter is to divide the composition of an image into
two phases, one where artifacts are compiled (through compilation and build
tools) and one where these artifacts are installed along with runtime
components. For this reason, two base images can be setup.[2] and used in two different phases, one for
building and one for bundling the final artifact.

In this way, build operations can be performed from a heavier container
image (the one with the build tools), while the bundle operation relies on a
leaner starting container image (the one with just the runtime), providing a
leaner result.

The difference with respect to plain Dockerfiles is in how the whole process
is configured: in dibs it is much easier to reuse both operations performed
inside a container (the next section is about this), as well as fragments of
the configuration itself to avoid repetitions (which, much like code, can lead
to bugs).

Reuse of operations

Another common problem with
Dockerfiles has to do
with how container images are customized, e.g. to execute build/bundle
operations.

The RUN directive in
the Dockerfile is very
powerful, but it allows executing either direct commands or some
script/program that are available inside the container during its
construction. While this is strictly all that is needed, it’s quite difficult
to reuse operations and not repeat them.

dibs allows defining packs of operations[3] in many different ways, which include grouping them
in a git repository that can be shared across multiple projects. An example of
such a repository is
dibspack-basic.

Basics

The basic metaphor used in Dibs is inspired to drawing (because the aim is to
generate…​ an image). To generate an image, multiple actions are available:

setting the starting point for an image is to prepare it, much like the
FROM directive in
the Dockerfile;

a single operation step performed inside a container is a stroke. It can
be thought as a single 'RUN`
RUN directive in the
Dockerfile;

finalizing an image is to frame it (there is no equivalent operation in
the Dockerfile);

collecting multiple actions together is to create a sketch (it can contain
other sketches itself).

When run, dibs goes through two main phases, namely the setup and the
execution of actions:

in the setup phase collects all available information for the specific run.
It merges together different sources (command line parameters, environment
variables, and the configuration file that will be described below), making
also sure that the source code is fetched and made ready for operating the
different steps;

in the execution phases, dibs executes the required actions. This may end
up in an image or not, depending on the needs.

The operations in a stroke are performed through dibspacks (or simply
packs). They are the specification of where something can be found, most
probably a program or a group of programs that can be executed inside the
container.[4]

All actions (and something more) are defined inside a configuration
file,[5], whose
structure will be described below.

Dibs elects a directory as a base camp for its operations, called the project
directory. Depending on the specific configuration, it defaults to a dibs
sub-directory of the current directory, or to the current directory itself.
The configuration file and other directories shared across all stroke
executions are related to the project directory, and are:

src: where the source (code, prerequisites, etc.) is made available,
read-write;

cache: a convenience area where build artifacts can be stored, e.g. to pass
them across different strokes or entire phases. Available in read-write mode
inside the container

env and envile: read-only directories where data is passed from the
outside. The former can be used to set up data from the user, the latter is
used by dibs itself to set them in the form of keys (filename) associated
to data (the contents of the files);

pack, auto/open: where dibpacks are stored (the former local to the
specific project, the latter generated automatically by dibs from
remote/dynamic dibspacks).

Examples

It’s better to start looking at a couple of examples to better understand how
dibs works.

Getting Started

The basic mode of operations of dibs is development mode. As the name
implies, it is best used when developing the software and generating the
container image during development itself (e.g. as a developer).

alpine.build and alpine.bundle are two programs that, when executed
inside a container, make sure to install the OS packages needed by app.pl
or any of the modules that will be installed by cpanfile. Each program
installs the requirements for a specific phase, in this case build and
bundle represent the build phase (where artifacts are generated) and the
bundle phase (where the artifacts are put in place along with the runtime
environment).

The dibs.yml configuration file in this example is the following (note: this
is quite simple at this stage, additional features will be shown later):

Running dibs in this case is as simple as going in the root directory of the
code and run:

$ dibs

This will execute the defaultsketch, which is comprised of two actions
build and bundle. They will be executed both, in the specific order. They
are both sketches themselves (they both contain a list of actions).

Sketch build starts from a basic image (an Alpine Linux, release 3.6) and
executes three RUN-like actions on top of it, in the specific order:

installation of pre-requisites (calling the prereqsstroke defined
above). The script that install pre-requisites uses the variable
DIBS_PREREQS to select the right prerequisites script, which will be
prereqs/alpine.build in this case.

"compilation" of the Perl code. This reduces to the installation of modules
as specified in file cpanfile

save of app.pl (main program) and local (where installed modules are
placed) inside the cache directory (in particular, in the app
sub-directory)

Each step is executed "on top" of the previous one, just like several RUN
directives in a Dockerfile are executed.

Sketch build does not include a frame action, so the final container is
removed and not saved.

Sketch bundle is similar to build, but also different:

starts from the same base image alpine:3.6

install pre-requisites. In this case DIBS_PREREQS is set to bundle, so
the prerequisites program that will be run is prereqs/alpine.bundle. This
is an example of reuse, because the same script (prereqs in the basic
pack) is used to obtain different results in different conditions;

artifacts are copied from the cache to the final target destination (in
/app). This is the last "layer" that is added to the image, so there is
also the specification of a commit section to set the entrypoint and the
cmd to be executed by default.

the last action of the sketch is a frame that saves the final container as
an image with two tags: exadev:latest and exadev:0.3.

Explicit Pinpointing

The previous example showed an example where build and bundle are
separated, but as a matter of fact it does not provide a real advantage in
terms of execution time, because the installation of prerequisites on top of a
basic image is always performed.

From this point of view, dibs performs worse than plain
Dockerfiles[6] because it does not
come with implicit caching/pinpointing of intermediate containers. This is
meant as a feature though, because the implicit pinpointing and reuse of
previously built layers can bite when things change around and docker is not
aware of it[7]; this is a likely scenario in dibs because there is much more
space for using remote stuff.

It’s possible to expand the example to limit the amount of repeated work, like
shown in the following example.

Former build is divided into parts, this is the first and yields an
image that is saved permanently as builder:1.0

The image is then used as a base for the build stroke.

In this example, former build sketch has been broken down into two sketches,
the first one (builder) installing the pre-requisites and saving a base
image that is suitable for building (builder:1.0) and is thus used as the
starting point for sketch build. A similar split has been performed onto
bundle, extracting the pre-requisites part into bundler.

To generate the new base images for building and bundling the following
command is run:

$ dibs builder bundler
# generates builder:1.0 and bundler:1.0

After this step has been run, these images are used as bases for the new
build and bundle steps, so when the following command is run:

$ dibs build bundle

the prerequisites installation is not performed any more, saving time.

This trick allows pinpointing specific steps of interest for explicit reuse.
Making it explicit also opens the door to easily distribute responsibilities
to other teams for the different stages.[8]

Robust Pinpointing

The split in the previous example was possible because of the assumption that
pre-requisites change very seldom in a project (with the possible exception of
the initial days). Anyway, it’s possible that the pre-requisites have to
change from time to time, in which case it’s necessary to regenerate the base
images to include them, which might be easily overlooked.

At the expense of an additional layer, though, it’s possible to repeat the
prereqs stroke inside the build and the bundle strokes; these will
mostly resolve into nothing (i.e. no change) unless an addition is put in the
prerequisites, in which case the addition will be honored. The following
dibs.yml implements this approach.

The prereqs program relies upon the DIBS_PREREQS variable, so it has
to be set whenever prereqs will be used.

The prereqs stroke is re-introduced as the first step in both build
and bundle. Most of the times this will be a no-op.

Running the prereqs step can anyway draw time from the build/bundle process
though, so in all cases in which it can be skipped it can be useful to avoid
it. The following example does some refactoring to add buildq (i.e.
the quick version of build), leaving out bundleq (which can undergo a
similar transformation).

build_basics is a new sketch that includes strokes to compile modules
and save artifacts in the cache

the new artifact is used in both the build and buildq sketches,
avoiding repetitions

With this setup:

"normal" work on code can rely upon buildq and skip the prereqs stroke
(which consumes some time)

"safe" work can still rely upon build to ensure that prereqs are
honored. This might come handy when a new prerequisite is added and the
buildq sketch yields an error because of missing dependencies, without the
need to regenerate the full base image (e.g. to test out if the addition to
the prerequisites is sufficient or needs to be changed)

in the medium-long term, though, it’s still better to re-generate the base
image.

Avoiding Repetions: YAML Variables

As in code, repetitions can be dangerous in a dibs.yml file because changes
would have to be applied in multiple places. In the examples above, there are
a few repetitions in the names of images used as base.

YAML allows the definition of anchors and aliases to avoid repetitions
inside the file, like in the following example.

It’s possible to place the YAML "variables" more or less everywhere, although
it is suggested to place them under the variables key.

Avoiding Repetitions: Inheritance

It is also possible to inherit some characteristics from other actions by
using the extends key in the definition of an action. In the following
example, the DIBS_PREREQS envile is defined once (in buildish for
building, in bundlish for bundling) and then used where needed.

These two definitions are abstract and do not specify a type of action
(although only sketches and strokes leverage the envile key)

Using extends allows "importing" all definitions from the referred
element.

Import of traits from ancestors is somehow crude, because a redefinition in
the derived element totally overwrites the ancestor’s data.

Variables Expansion

The variables highest-level key is supposed to be associated to an
array-type value. Each item in this array that is a hash with a single key
function and an array value is subject to expansion. The following is an
example of the join function (which is also the only one available).

When read by dibs, the value associated to anchor whatever is expanded
in-place to something:latest; the application of the operation in-place also
means that all aliases will get this expanded value (like the from statement
in the example).

Stroke Arguments

As anticipated, strokes define programs that will be executed inside a
container. It is possible to pass arguments to these programs, in order to
increase their reusability, via the args key inside a stroke.

In the example above, the second and third argument are objects with a single
key-value pair. Values associated to keys path_cache, path_src, etc. are
expanded as sub-directories of the corresponding zones (cache, src, etc. in
the specific case).

Alien Mode

In the initial example, the dibs.yml file is part of the code itself, but
this need not be. It’s possible to separate concerns of development and
build/bundling using the so-called alien mode.

This mode of operations is somehow similar to a bare git repository, where
there is no sub-directory but the project directory is directly the current
directory. The layout is as follows:

auto
cache
dibs.yml
env
pack
src

The src directory is where the source code is supposed to be placed. It’s
possible to develop code directly there, of course, although it’s probably
better to rely upon the origin directive (or command-line option) and fetch
it remotely.

$ dibs --alien --origin "$ORIGIN"

Going Further

This README file is only meant as an introduction to the possibilities. The
manual contains all details and is the next suggested
reading.