Creating a Release

A release contains one or more pieces of software that work together.
For example, you could create a release of a service with three pieces:
two MySQL nodes and a dashboard app.

There are four fundamental elements in a release:

Jobs describe pieces of the service or application you are releasing

Packages provide source code and dependencies to jobs

Source provides non-binary files to packages

Blobs provide binary files (other than those checked into a source code repository) to packages

The following instructions use an example release that includes two jobs:
a web UI and a background worker.
The two jobs split up the functionality provided by single Ruby app,
ardo_app (you can use simple gist as app).

When deploying your release, BOSH places compiled code and other resources
in the /var/vcap/ directory tree, which BOSH creates on the job VMs.
The four directories you just created, jobs, packages, src, and blobs,
appear on job VMs as /var/vcap/jobs, /var/vcap/packages, /var/vcap/src,
and /var/vcap/blobs, respectively.

Choose whether you want to work one step at a time or one job at a time.
For releases with just a few jobs, going one step at a time is probably easiest.
If you have a larger number of jobs, going one job at a time may be more efficient.

If your release needs templates other than the control script, create them now.

For example if the job can be used to deploy clusters of nodes, especially in
the case of stateful clusters (e.g. a database or distributed data store), you
will want to write a drain script for your job to ensure that the
service is not affected by the rolling provisioning/update operations performed
by BOSH.

On a deployed release, a BOSH Agent runs on each job VM.
BOSH communicates with the Agent, which in turn executes commands in the
control script.
The Agent does this using open source process monitoring software called
Monit.

The file paths that you provide for templates are relative to
the /var/vcap/jobs/<job_name> directory on the VM.
For example, bin/ctl becomes /var/vcap/jobs/<job_name>/bin/ctl on the job VM.
Using bin as the directory where these files go is a convention.

The templates block of the updated spec files for the example jobs look
like this:

Putting each dependency in a separate package provides maximum reusability
along with a clear, modular structure. This is not mandatory; what packages
to create is a matter of preference. You could even opt to put all the
dependencies together in a single package, though that is not recommended.

Note

Use of the pre_packaging file is not recommended, and is not discussed in this tutorial.

Without using pre_packaging for our ardo_app we need to pack gems manually for further usage:

The location where BOSH can find the binaries and other files that the package needs at compile time

Use your dependency graph to determine which dependencies belong in each spec.
Developer preferences and style play a role here.
Consider our example: the spec for Ruby lists rubygems and bundler as dependencies along
with Ruby itself.
Some Ruby developers would do it this way; others would not.

To maximize portability of your release across different versions of stemcells,
never depend on the presence of libraries or other software on stemcells.

To describe binary locations in the files block of the spec:

Find the official site for the binary in question.
For example, Ruby might be at http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p484.tar.gz.

Download the binary from the official location and make sure the file hash matches.

Record the binary name including version number, with a slash and the binary
filename concatenated to it.
It's a good idea to cite the official URL in a comment, in the same line.

BOSH interprets the locations you record in the files section as being
either in the src directory or in the blobs directory.
(BOSH looks in src first.)
When you add the actual blobs to a blobstore (see the next section),
BOSH populates the blobs directory with the correct information.

For packages that depend on their own source code, use the globbing pattern
<package_name>/**/* to deep-traverse the directory in src where
the source code should reside.

Update the spec for each package.
Refer to the example specs below for guidance.

At compile time, BOSH takes the source files referenced in the package specs,
and renders them into the executable binaries and scripts that your deployed
jobs need.

You write packaging scripts to instruct BOSH how to do this.
The instructions may involve some combination of copying, compilation, and
related procedures.
For example:

For a Ruby app like ardo_app, BOSH must copy source files and install Ruby
gems.

For Ruby itself, BOSH must compile source code into a binary.

For a Python app, BOSH must copy source files and install Python eggs.

BOSH relies on you to write packaging scripts that perform the correct operation.

Adhere to these principles when writing packaging scripts:

Use your dependency graph to determine which dependencies belong in each
packaging script.

Begin each script with a set -e -x line.
This aids debugging at compile time by causing the script to exit immediately
if a command exits with a non-zero exit code.

Ensure that any copying, installing or compiling delivers resulting code to
the install target directory (represented as the BOSH_INSTALL_TARGET
environment variable). For make commands, use configure or its equivalent
to accomplish this.

Be aware that BOSH ensures that dependencies cited in the dependencies
block of package spec files are available to the deployed binary.
For example, in the spec file for the Ruby package, we cite libyaml as a
dependency.
This ensures that on the compilation VMs, the packaging script for Ruby has
access to the compiled libyaml package.

If the instructions you provide in the packaging scripts fail to deliver compiled
code to BOSH_INSTALL_TARGET, the job cannot function because the VM has no
way to find the code to run.
This failure scenario can happen if, for example,
you use a make command that delivers compiled code to some standard location
by default.
You can fix the problem by configuring make to compile into
BOSH_INSTALL_TARGET.
See how this is done in the example packaging scripts.

Like control scripts, writing packaging scripts is one of the heavier tasks
entailed in creating a release.
Write your packaging scripts now.
Refer to the examples below for guidance.

When creating a release, you will likely use a source code repository.
But releases often use tar files or other binaries, also known as blobs.
Checking blobs into a repository is problematic if your repository
unsuited to dealing with large binaries (as is true of Git, for example).

BOSH lets you avoid checking blobs into a repository by doing the following:

For dev releases, use local copies of blobs.

For a final release, upload blobs to a blobstore, and direct BOSH to obtain the blobs from there.

In the config directory, you record the information BOSH needs about the
blobstore:

The final.yml file names the blobstore and declares its type, which is either local
or one of several other types that specify blobstore providers.

The private.yml file specifies the blobstore path, along with a secret.

private.yml contains keys for accessing the blobstore and should not be
checked into a repository.
(If you used the --git option when running bosh init-release at the beginning
of this tutorial, private.yml is automatically gitignored.)

The config directory also contains two files whose content is automatically
generated: the blobs.yml file and the dev.yml file.

Adapt the examples below to fit the specifics of your release.
Our example release uses the local type blobstore because otherwise it would
be necessary to explain how to configure a public blobstore such as
Amazon S3, which is too large a topic for this context. More information on full
blobstore configuration can be found here.

The local type blobstore is suitable for learning but the resulting release
cannot be shared.
For that reason, you should configure a non-local, publicly available blobstore
for releases that you intend to share.
Normally, the blobstore you choose when you begin working on a release is used
for all subsequent versions of the release.
Changing the blobstore that a release uses is beyond the scope of this tutorial.

Optional: Include the blobstore_path in the final.yml file. Doing so allows you to gitignore the private.yml file but still allow the release to be downloaded and used on other systems.

Note

The blobstore_secret is required for the local type blobstore. This is true even though the blobstore_secret line is deprecated and its content does not matter. There is never a blobstore_secret line for blobstores of types other than local.

In the package spec file, the files block lists any binaries you downloaded,
along with the URLs from which you downloaded them.
(This assumes that you followed the directions in the Update package specs section.)

Those files are blobs, and now you need the paths to the downloaded blobs on
your local system.

In our example, the spec file for the libyaml_0.1.4 package includes the line:

files:-libyaml_0.1.4/yaml-0.1.4.tar.gz# From http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz

If you downloaded the blob, its local path might be:

~/Downloads/yaml-0.1.4.tar.gz

Go through all your packages and make a list of local paths to the blobs you downloaded.
Now you are ready to inform BOSH about these blobs.

For each blob, run:

bosh add-blob <path_to_blob_on_local_system> <package_name>

e.g.

bosh add-blob ~/Downloads/yaml-0.1.4.tar.gz libyaml_0.1.4

The bosh add-blob command adds a local blob to the collection your release
recognizes as BOSH blobs.

The usage shown above is a blend of requirement and convention.
It works like this:

For the first argument, you provide the path to the blob on your local system

For the second argument, you provide a destination within the blobs directory
in your release

BOSH goes into the blobs directory and creates a subdirectory with
the name of the package that the local blob represents

In the new subdirectory, BOSH creates a symbolic link to a copy of the blob
which BOSH makes in a hidden directory

Using the package name as the second argument of the bosh add-blob command
is recommended because it produces a cleanly-organized blobs directory.

Later, when you upload blobs for a final release, BOSH uses the hidden directory
as a staging area.

Deploying the release requires three or more steps, depending on whether
BOSH is targeting the desired Director, and whether BOSH is already pointing
to a release.

See what director BOSH is targeting:

bosh env

Target a director:

bosh -e <director_alias> log-in

See what releases are available:

bosh releases

If BOSH is already pointing to a release, edit the BOSH deployment manifest.
Otherwise, create a manifest. See BOSH Deployment Manifest for more information.
Simple manifest for ardo_app can be found here (OpenStack) or here (AWS).

Upload the new dev release.

bosh upload-release

Assuming you are in the release directory, no path is needed with the above command.

When you use the bosh create-release --force command to create them, dev
releases depend on locally-stored blobs.
To do a final release, you must upload blobs first.

If files that you need to keep private are uploaded to a public blobstore,
there is no satisfactory way to undo the mistake.
To avoid this situation, complete the following steps immediately before
you upload blobs:

Run bosh blobs to see the list of blobs BOSH is prepared to upload

Proofread the list of blobs displayed by the command

The list should include only the blobs you need for the final release

If the list includes any files that should not be uploaded, find and delete
the symbolic links to them in the blobs directory