Simplified Application Deployment With Cloud Foundry “Manifest”

Posted by Alex Suraci Today, we added a new feature to Cloud Foundry command line tool (‘VMC’) that makes it easier to automate application deployments. The new feature, called “manifests”, describe applications in human-editable manifest documents. Manifest documents can describe anything from a simple “Hello World” app to complex multi-app hierarchies with inter-application dependencies and service binding information. The manifests feature not only adds ease-of-use to VMC, it also ensures consistency and reproducibility of application deployments in Cloud Foundry.

Overview

The manifests feature uses a YAML document, aptly named manifest.yml. You will typically place this manifest document in your app’s root directory, though you can specify a different location by telling VMC which to use with the -m flag. The manifest can be created by hand, automatically created after a vmc push, or explicitly with vmc manifest. With this manifest document, VMC will simply read the input values from the file rather than prompt you for each configuration. Not only can you automate vmc push with manifests, you can also bypass interactive inputs for a large portion of VMC’s commands to make using the command-line tool more efficient and user-friendly. For example, you can leave the app name out when issuing a vmc update command, and VMC will retrieve the app name from an existing manifest document. Here’s the full list of commands that can take advantage of the manifest document:
* vmc push: Now allows you to specify multiple services. Pushes with information from the manifest. If no manifest is found, it will ask if you want to create one after the interaction is finished.
* vmc stats, vmc update, vmc start, vmc stop: If no application name is given, it operates on the application(s) described by the manifest.
* vmc update: Syncs changes from the root of the application if a manifest is present.
* vmc start: Starts the applications in a multi-app deployment in the proper order (taking dependencies into account).
* vmc stop: Stops multi-app deployments by shutting down each app in the reverse of the order in which they were started.
* vmc restart: See vmc stop and vmc start.
* vmc delete: Delete the application Note: For multi-app hierarchies, these will operate only on the sub-app you’re in, rather than always operating on every app in the hierarchy. To operate on every app, invoke the command from the root of the hierarchy.

Getting Started

First, install or update your Cloud Foundry command line tool (‘VMC’) to the latest preview version using the following command:

gem install vmc --pre

You can verify that you got the right version using:

vmc -v

which should show the version to be 0.3.16.beta.1 or higher. The easiest way to get going is to get a manifest document generated from basic application info. If you haven’t pushed your app yet, you can start with

vmc push

as usual, which will now ask if you want to save the configurations as a manifest document:

hello-sinatra(master*) % vmc push
Would you like to deploy from the current directory? [Yn]:
Application Name: hello-sinatra
Application Deployed URL [${name}.${target-base}]: ${name}-suraci.${target-base}
Detected a Sinatra Application, is this correct? [Yn]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]:
How many instances? [1]:
Would you like to bind any services to 'hello-sinatra'? [yN]: y
The following system services are available
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
Please select the one you wish to provision: 2
Specify the name of the service [mysql-2cccd]:
Would you like to bind another service? [yN]:
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [mysql-2cccd]: OK
Binding Service [mysql-2cccd]: OK
Uploading Application:
Checking for available resources: OK
Packing application: OK
Uploading (1K): OK
Push Status: OK
Staging Application 'hello-sinatra': OK
Starting Application 'hello-sinatra': OK

As you can see, just before pushing, we were able to save the deployment configurations as manifest.yml in the same directory.

The manifest document has captured all of the configuration that we entered above for the application push into a description of the application deployment. Once you have a manifest.yml file, you can modify it however you would like, as it’s meant to be human-editable. The structure of the document is freeform, so if you want to define arbitrary values and use them throughout your document, you can. Now if we try pushing again, vmc push will use this to automate everything:

You can also use vmc manifest to create a manifest without pushing. vmc manifest will let you create more complex manifests describing multiple applications in a single hierarchy. Now that you have a manifest document, you don’t really have to do anything else if you don’t want to get fancy. It’ll passively improve VMC’s user interface experience for the commands listed above: vmc push will be interaction-free, making deployment much easier, and many other commands will be efficient to invoke.

Getting Fancy

Child Manifests

A manifest document can inherit properties from a parent manifest like so:

inherit: path/to/parent.yml

This slurps in everything from the parent document ensuring that properties defined in the child manifest are deep-merged with the parent. The symbols are resolved after this merge has taken place, so any properties you set in the child manifest may be used in properties set in the parent. This allows you to provide the basic information, such as service bindings and framework information, in a “base” manifest, which can be “filled in” via a child manifest. For example:

Having various child manifests for different deployment modes (e.g. debug, local, public) that extend base application information provided by a “base” manifest.

Packaging the basic configuration along with your app, which users can extend with their own manifest to override your settings or fill in the blanks for their own deployment.

Symbol Resolution

random-word: A random string of characters. This is useful for ensuring your URLs are unique. Otherwise, symbol resolution simulates lexical scoping, so you can define arbitrary properties, which can be overridden by child manifests or in a nested hash. For example, the following parent:

applications: ./foo: name: bar url: ${name}.${target-base}

…combined with this child manifest:

applications: ./foo: name: baz

…and with a target of api.cloudfoundry.com, will result in a url of baz.cloudfoundry.com when using the child manifest, and bar.cloudfoundry.com when using the parent.

Multi-App Manifests

Manifests also present a way to deploy multiple applications through a single push command. Let’s say you have a modular application comprised of several independent parts, for example, a publisher and a subscriber. You’ll want the subscriber to be started before the publisher, so it doesn’t miss anything that was published. It’s best to have these two applications defined as parts of a whole, where you can specify this dependency. This is done with multi-app manifest documents. Our publisher app will publish messages every second, with the message starting at 0 and increasing for every iteration. The subscriber will simply collect the messages it receives in the order they came in, and display them to the user. To start with, you may want to arrange your applications like so:

./big-app
./big-app/publisher
./big-app/subscriber

This will make using the manifest document more natural. Switch to the big-app directory and use vmc manifest to create your manifest document:

~ % cd ./big-app
big-app % vmc manifest
Configure for which application? [.]: ./publisher
Application Name: publisher
Application Deployed URL [${name}.${target-base}]: publisher-${random-word}.${target-base}
Detected a Sinatra Application, is this correct? [Yn]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]:
How many instances? [1]:
Would you like to bind any services to 'publisher'? [yN]: y
The following system services are available
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
Please select the one you wish to provision: 5
Specify the name of the service [redis-47da2]: redis
Would you like to bind another service? [yN]:
Configure for another application? [yN]: y
Application path?: ./subscriber
Application Name: subscriber
Application Deployed URL [${name}.${target-base}]: subscriber-${random-word}.${target-base}
Detected a Sinatra Application, is this correct? [Yn]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]:
How many instances? [1]:
Would you like to bind any services to 'subscriber'? [yN]: y
The following system services are available
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
Please select the one you wish to provision: 5
Specify the name of the service [redis-a1278]: redis
Would you like to bind another service? [yN]:
Manifest written to manifest.yml.

In this single interactive session we’ve configured a manifest that defines two Sinatra apps, using a single Redis service. We’re using URLs with a bit of randomness (provided by the special random-word symbol) to ensure uniqueness. There’s one thing missing, though. We didn’t specify any dependencies between the apps. If we were to start it now, we could lose some data if the publisher starts before the subscriber:

As you can see, we’ve lost some data here. In the time between the publisher starting and then the subscriber starting, the subscriber has missed four messages. This can be fixed by editing the manifest.yml document to indicate that the publisher depends on the subscriber being started:

Wrapping Up

The manifests feature is intentionally open-ended and flexible, providing the structure for you to define your deployments however you like. Kick the tires a bit and let us know how you think we should enhance this feature. Feel free to direct any suggestions or feedback to the support forums!

Hi,
The manifest feature is amazing. I have one question though. In a typical deployment scenario, we almost always have to seed the database with some initial data that the application can work with. I haven’t seen anyplace where we could specify the seed data to the VMC client in any way. This means that it is quite impossible to automate the whole deployment process using vmc/manifest feature.
I have a scenario where I have around a dozen war files that I need to deploy as part of my application. each war file depends upon a mysql service. And for almost all the services that I provision for my web apps, I have to populate some data in the DB. How do you think I can achieve this using VMC/manifest?