Cloud Foundry Bloghttp://blog.cloudfoundry.com
The latest updates from Cloud FoundryTue, 25 Mar 2014 21:50:45 +0000en-UShourly1http://wordpress.org/?v=3.8.1Cloud Foundry Eclipse 1.6.0: Spring Boot Support and Self-Signed Certificateshttp://blog.cloudfoundry.com/2014/03/25/cloud-foundry-eclipse-1-6-0-spring-boot-support-and-self-signed-certificates/
http://blog.cloudfoundry.com/2014/03/25/cloud-foundry-eclipse-1-6-0-spring-boot-support-and-self-signed-certificates/#commentsTue, 25 Mar 2014 13:00:57 +0000http://blog.cloudfoundry.com/?p=5968Continue reading →]]>Cloud Foundry Eclipse integrates Cloud Foundry server and application management into Eclipse and Spring Tool Suite (STS). You can push applications to a Cloud Foundry server via drag and drop from the Eclipse Project Explorer much in the same way as deploying to a local tomcat server. Application memory and instances can be scaled easily, and services created and bound to an application through drag and drop. A comprehensive description of the plug-in is available at:

Creating a Cloud Foundry Server Instance

Once Cloud Foundry Eclipse 1.6.0 has been installed, open the Servers view in STS/Eclipse, and create a new server instance. This can be done by right-clicking on the view to open the context menu and selecting:

New -> Server

In the first page, select “Cloud Foundry” under “Pivotal”, and click “Next” to open “Cloud Foundry Account”. Click on “Manage Cloud…” to add the URL to the private cloud.

Click “Add” and enter a name and URL for the private cloud.

After clicking “Finish”, a warning may appear indicating that the URL failed to validate. Click “Yes” to keep the URL. This will return to the “Manage Cloud URLs” dialogue.

Click “OK” in the “Manage Cloud URLs” dialogue to return to the credentials page. Enter a username and password and click “Finish”, which will validate the credentials as well as automatically select the first organization and space that it finds as the server’s target space. Alternately, click “Next” to select another organization and space.

While validating the credentials, by clicking on “Finish”, “Next” or “Validate Account”, a confirmation dialogue is displayed showing that the server uses self-signed certificates, and whether to proceed in creating the server instance. If you select “Yes”, the Cloud Foundry Eclipse plug-in will remember the decision for any future server instance creation to this URL, and you will not be prompted again.

Once the server instance is successfully created, it will appear in the Servers view.

Creating and Configuring Spring Boot App

Now we create a Spring Boot application. Right-click on the Project Explorer and select:

Once the project is created and built in the STS workspace, right-click on it and select:

Configure -> Enable as Cloud Foundry App

This step is necessary and allows the project to be deployable to a Cloud Foundry server instance in STS.

NOTE: As of version 1.6.0, this configuration only applies to Java Spring Boot applications, that is, Spring Boot applications that are packaged and deployed as jar files.

Pushing the Spring Boot Application

Once configured, you can simply drag and drop the Spring Boot project onto the private cloud in the Servers view.

This opens the Application Deployment wizard. You can click “Finish” right away, or “Next” to further configure the application. For instance, you can modify the mapped application URL, memory (default is set to 512MB), bind services, or set environment variables.

For our case, we click “Next”, and modify the memory.

Version 1.6.0 now supports open, variable memory configuration. Whereas previous versions of the plug-in restricted the memory choices, you can now enter any MB value over 0 up to your maximum allowed allocation. We change the default 512MB to 437MB.

Clicking “Next” will allow you to bind services, and a further click on “Next” opens the Environment Variables page, which is another new feature in version 1.6.0.

To add a new environment variable, right-click on the table to open the context menu and select “Add”. In our example we enter a RECAPTCHA_PRIVATE_KEY environment variable.

Click “Finish” to push the application to the Cloud Foundry server.

Application Staging and Console Output

Once the application begins deployment, the Eclipse console will automatically open and show progress during different stages of the deployment.

Version 1.6.0 introduces more comprehensive console progress than previous versions of the plug-in, displaying completed stages of the application deployment and start process.

After the Spring Boot application completes staging and has successfully started, the console will display the application’s standard out and error logs.

While the application is running, at any time you can refresh the console output by right-clicking on the application in the Servers view and selecting “Show Console”. This will only show the application’s standard out and error log files.

Editing Environment Variables and Memory

To edit application environment variables and memory, double-click on the application in the Servers view. This will open the Cloud Foundry server editor, and display bound services and instance stats for the application. The “General (Application Restart Required)” section allows environment variables and memory to be adjusted. For these changes to take effect, the application has to be restarted, which can be done directly from the Cloud Foundry server editor by clicking on either “Restart” or “Update and Restart”.

Improved Application Deployment

With version 1.6.0, we have enhanced application deployment from Eclipse-based IDEs. Pushing applications to private servers that use self-signed certificates is now possible. Furthermore you can now easily push Spring Boot projects to Cloud Foundry servers via drag and drop, much like any other web project.

We continue to further enhance the Cloud Foundry experience in Eclipse and STS in each new release of the plug-in.

]]>http://blog.cloudfoundry.com/2014/03/25/cloud-foundry-eclipse-1-6-0-spring-boot-support-and-self-signed-certificates/feed/1Working with the Cloud Controller API in Ruby: Beyond CFoundry.http://blog.cloudfoundry.com/2014/03/24/working-with-the-cloud-controller-api-in-ruby-beyond-cfoundry/
http://blog.cloudfoundry.com/2014/03/24/working-with-the-cloud-controller-api-in-ruby-beyond-cfoundry/#commentsMon, 24 Mar 2014 17:35:41 +0000http://blog.cloudfoundry.com/?p=6412Continue reading →]]>Cloud Controller is the primary API through which third parties interact with Runtime; it encapsulates the myriad internal services that take a user from ‘cf push’ to a running application. Both the Pivotal Developer Console (the web application that ships with Pivotal CF and backs console.run.pivotal.io) and the CloudFoundry CLI interact directly with the Cloud Controller API. The two applications were originally written in Ruby and shared a common Ruby code base for talking to Cloud Controller called CFoundry.

CFoundry provides a friendly interface to the Cloud Controller API. The library encapsulates the logic around endpoint requests, object relationships, and error handling, permitting new users to intuitively navigate the object graph by working with domain objects rather than raw data. The CLI was recently rewritten in Go, and as such, no longer has a dependency on CFoundry. Our team, responsible for Pivotal Developer Console, took this as an opportunity to re-evaluate whether CFoundry was the right choice for our web application.

As Developer Console complexity grew, we found we wanted to be ‘closer to the metal’ so we had the ability to fine-tune the requests we made to the Cloud Controller API. While it is technically possible to make nearly any request that the Cloud Controller API supports via CFoundry, the interface does not encourage it. Further, although Developer Console domain models are largely backed by Cloud Controller data, they also incorporate information from both UAA and a relational database. New developers on our team were frequently confused by the difference between objects returned by the CFoundry gem and those used in the Rails application. Lastly, CFoundry was tightly coupled to the behavior of the domain objects in Developer Console which made it challenging to stub domain object values in integration tests.

We decided to gradually remove the use of CFoundry in our app and replace it with a new pattern the design of which would be guided by our CFoundry experiences. We wanted this new pattern to exhibit a couple of key characteristics:

1) It should be easy to tell when data is loaded. With CFoundry, associated models were loaded as the object graph was traversed and the domain objects in Developer Console failed to encapsulate this behavior. This made it very easy to unwittingly write code that made excessive API requests.

2) It should be easy to tell where the data is coming from. CFoundry uses its domain modeling to infer the API endpoints to which it makes requests. This design makes it extremely easy to add new endpoints, but it makes it very difficult to reason about what particular endpoints will be called in a given situation. Since the Cloud Controller domain changes infrequently, we found that being able to understand what and how endpoints are called was provided more value to our team.

3) It should be easy to know what values each object provides. Domain models should be immutable and always instantiated with a full complement of data. Existing domain models in Developer Console lazily loaded data from Cloud Controller depending on what methods were called on it and what data it already had. This design made it difficult to predict what API calls an object in any particular state might make, which in turn made stubbing web requests for integration tests challenging.

4) It should be easy to provide fake data in tests. Data store access should exist in a layer separate from domain objects to permit simple faking of persisted data for feature specs.

The set of patterns we arrived at were heavily influenced by the idea of Domain Driven Design. In particular, we started by adding Entity, Aggregate, and Repository objects to the Developer Console code base. Our Repository objects encapsulate the retrieval of data from the various services we talk to and emit a fully hydrated and immutable domain object (either an Entity or an Aggregate).1 So far this new pattern has produced a number of positive benefits.

Our team members have found the Repository pattern makes it easier to understand how data is being retrieved. Cloud Controller data is returned as a simple hash by a concise restful client that is responsible for building requests, parsing JSON and error handling. The calls and endpoints necessary to create, retrieve, update and destroy each object are clearly enumerated in that domain object’s Repository. Consolidating all data access that occurs for a particular object gives us a single point where we can choose to optimize or cache the retrieval of data — for example, it is obvious which requests could be made in parallel in order to speed up the overall response.

The Repository pattern also improves our ability to write understandable and maintainable tests because it provides a discrete layer that can be stubbed out during testing. Where before we struggled to create web response stubs that consistently described a particular data set, it is fairly trivial to write a collection of fake Repositories that can maintain state. Further, Entities and Aggregates take only primitives and other Entities at instantiation time, making them easy to create in a test environment. Feature tests will eventually be as simple as telling the repositories about the domain objects they contain and then allowing the rest of the system to operate normally.

Repository objects, in turn, are test driven in an integration environment, meaning we don’t have to guess at what the responses will be, we simply spin up the test and look at the values that are returned by Cloud Controller. This approach also gives us the ability to run our repository integration suite against updated external dependencies to ensure that no regressions have occurred. Our previous approach to integration testing occurred at the GUI layer, necessitating a deploy to the environment in question before we could confirm that our code still worked correctly.

We’re excited that this new way of interacting with Cloud Controller will help us complete new features faster with fewer defects. I’ve posted a gist with some sample objects to help you better visualize our new approach; while this code is certainly not feature complete, we welcome feedback on how this pattern might work (or not) for your own interactions with Cloud Controller.

1 You might be wondering how we create or update these objects; these operations act on separate and similarly immutable objects whose responsibility is specifically for persistence, copying data from the retrieved object as necessary.

]]>http://blog.cloudfoundry.com/2014/03/24/working-with-the-cloud-controller-api-in-ruby-beyond-cfoundry/feed/0Diego Phase 1: Staging in a multiplatform worldhttp://blog.cloudfoundry.com/2014/03/13/diego-phase-1-staging-in-a-multiplatform-world/
http://blog.cloudfoundry.com/2014/03/13/diego-phase-1-staging-in-a-multiplatform-world/#commentsFri, 14 Mar 2014 00:57:49 +0000http://blog.cloudfoundry.com/?p=6332Continue reading →]]>The first phase of Diego’s development has focused on offloading the staging workflow – the task of converting uploaded app bits to compiled droplets ready for running in Cloud Foundry – from the existing DEAs to Diego. From the outset one of Diego’s mandates has been to make it relatively easy to support multiple platforms (e.g. linux + heroku buildpacks/linux + docker/windows) on one Cloud Foundry installation.

This blog post outlines what has emerged out of this first phase of development, and describes Diego’s architecture with an emphasis on how multiplatform support is envisioned to work.

The Pieces

To wrap your head around Diego you need to wrap your head around the components that make Cloud Foundry tick. Here’s a bullet-point list broken out into existing runtime components and new components introduced by Diego.

Runtime Components

Cloud Controller receives user inputs and sends NATS staging messages to Diego. The diego code is under diego_staging_task.rb and differs from the DEA codepath. in particular, we’ve moved the responsibility for computing staging environment variables completely into CC.

Diego Components

ETCD is the high-availability key-value store used to coordinate data across Diego’s components.

Stager listens for staging messages over NATS and constructs and places a staging-specific RunOnce (more on these below) into ETCD.

Executor picks up RunOnces from ETCD and executes them in a garden container (garden is a Go rewrite of warden).

Linux-Smelter transforms a user’s application bits into a compiled droplet. It does this by running Heroku-style buildpacks against the app bits.

FileServer provides blobs for downloading (the smelters live here) and proxies droplet uploads to the CC (this allows us to have a simpler upload interface downstream of the CC).

Other Diego Pieces

In addition to these components (which, basically, map onto separate processes running on seaparate VMs) there are a few additional puzzle pieces/concepts that must be understood when discussing Diego:

RunOnces: The executor is built to be a generic platform-agnostic component that runs arbitrary commands within garden containers. The RunOnce is the bag of data that tells the executor what commands to run. When the executor receives a RunOnce it:

Creates a garden container and applies the memory and disk limits conveyed by the RunOnce

Runs the actions described in the RunOnce. There are four actions:

Download: downloads a blob from a URL and places it in the container

Upload: grabs a file from the container and uploads it to a URL

Run: runs a provided script in the context of provided environment variables

FetchResult: fetches a file from the container and sets its content as the result of the RunOnce

Marks the RunOnce as succeeded/failed and saves it to ETCD

RuntimeSchema is a central repository of models (including the RunOnces) and a persistence layer that abstracts away ETCD. Diego components use the runtime-schema to communicate with each other.

An ordered list of buildpacks (names + download URLs) to run when compiling the application

The location of the user’s app bits (a url to a blobstore)

The environment variables to be applied during the staging process (e.g. VCAP_APPLICATION, VCAP_SERVICES)

An available Diego stager receives the staging NATS message and constructs a RunOnce with the following actions:

A Download action to download the user’s app bits

Download actions for each of the requested buildpacks

A Download action to download the correct smelter (this is selected by stack)

A Run command to run the smelter (along with the environment variables received from CC)

An Upload command to upload the droplet

A FetchResults action to fetch staging metadata from the smelter

The Diego stager then puts the RunOnce in ETCD

A compatible Diego executor (one that matches the desired stack) picks up the RunOnce, spawns a container, and executes its actions. When it does this the executor also streams logs generated by commands run in the container back to the user via Loggregator.

On success, the executor marks the RunOnce as complete and puts it back in ETCD

A Diego stager then pulls out the completed RunOnce and notifies CC that staging is complete.

It’s important to understand that the Diego stager’s role is quite small: it simply interprets the incoming NATS message, produces a valid staging RunOnce, and then conveys the result of executing said RunOnce back to the CC. The pool of Diego executors is doing all the heavy lifting (namely: actually downloading the app bits and producing a droplet).

Multi-platform Entry Points

Diego is built to make supporting multiple-platforms relatively straightforward. The parameter that determines which platform an app is targeted for is the stack:

The user selects a target platform by specifying a stack when pushing an app.

The Diego stager selects a smelter based on the stack and creates a RunOnce with the associated stack.

The Diego executors are configured with a stack parameter. An executor will only pick up a RunOnce if the stack denoted by the Runonce matches the executor’s stack.

Given this, to support a new platform (e.g. Windows) one needs the following pieces:

A smelter designed for the target platform. For linux, Diego has a linux-smelter that runs through Heroku-style buildpacks. For windows, for example, one could construct a smelter that simply validates and repackages a user’s app bits in preparation for running against a .NET stack (in such a case the notion of buildpacks is unnecessary and can be ignord). The smelter’s api is simple: it should be an executable that accepts a certain set of command line arguments and produces a droplet.tgz file.

A platform-specific plugin for the executor. This is compiled into the executor when it is built and solves certian platform-specific issues (e.g. converting environment variables from an array-of-arrays input format to a platform-specific format).

A platform-specific Garden backend plugin. Garden performs all containerization via a backend plugin with a well-defined interface. Garden ships with a linux-backend that constitutes a reference implementation. To target other platforms one simply needs to write an API-compatible backend plugin.

In terms of deployment we envision that most components (the stager, fileserver, etcd, cloud controller, nats) will be deployed to Linux VMs. Only the Diego Executor and Garden need to live on the non-linux target platform. Since these components are written in Go, recompiling and targetting a supported non-linux platform should be relatively straightforward.

Cloud Foundry is the leading open source PaaS offering with a fast growing ecosystem and strong enterprise demand. One of its advantages which attracts many developers is its consistent model for deploying and running applications across multiple clouds such as AWS, VSphere and OpenStack. The approach provides cloud adopters with more choices and flexibilities. Today, NTT Software Innovation Center and School of Software Technology, Zhejiang University, China (ZJU-SST) are glad to announce that we have successfully developed the BOSH CloudStack CPI, which automates the deployment and management process of Cloud Foundry V2 upon CloudStack.

CloudStack is an open source IaaS which is used by a number of service providers to offer public cloud services, and by many companies to provide an on-premises (private) cloud offering, or as part of a hybrid cloud solution. Many enterprises like NTT and BT want to bring Cloud Foundry to CloudStack in an efficient and flexible way in production environment. Therefore, developing BOSH CloudStack CPI for deploying and operating Cloud Foundry upon CloudStack becomes a logical choice.

Technical Details

Since NTT and ZJU-SST developed BOSH CloudStack CPI independently at the beginning, there are many differences in the implementation. Hence, the first step is to merge code repositories of NTT and ZJU-SST into a new repository. We chose to create a new repository in GitHub cloudfoundry-community in order to encourage more developers to join us.

There are some crucial aspects in the process of refactoring CPI(check out the wiki if you are interested in digging more differences between NTT and ZJU-SST implementations):

Stemcell Builder

Create Stemcell

Basic Zone VS Advanced Zone

Fog Support

Stemcell Builder

ZJU-SST used standard Ubuntu10.04 ISO file to build stemcells for both MicroBOSH and BOSH. NTT used Ubuntu10.04 of a backported kernel version due to some compatibility problems in their environment. Unfortunately, aufs, which is essential for warden in Cloud Foundry V2, is missing in the backported kernel. So, we decided to try standard Ubuntu12.04 as the base OS of stemcells for both MicroBOSH and BOSH after brainstorming together. We found that, with a minor patch of cf-release, Ubuntu12.04 is compatible with BOSH and Cloud Foundry. The patch only modifies the deployment process of Cloud Foundry, so it does not impact Cloud Foundry itself. In fact, this issue has been solved since cf-release v160 by updating nginx to 1.4.5.

Create Stemcell

When referring to API call create_stemcell in CPI, ZJU-SST used an extra web server as a entrepot to store the stemcell temporarily while uploading, which follows the OpenStack style, but NTT didn’t use an extra web server and took the volume route same as AWS pattern.

Implementation [A]:

Similar to the OpenStack CPI

Requires no inception server

Requires a web server to save qcow2 files and expose them at HTTP for CloudStack

[Client] –> (SSH upload) –> [WebServer] –> (HTTP) –> [CloudStack]

The API call can’t receive image files directly, it downloads image files from given URLs. The web server is necessary because CloudStack can not directly receive image data from inception server.

Copies (using dd) a raw stemcell image to a volume attached to the inception server

Creates a template from a snapshot of the volume

Both implementations have cons and pros.

[A] user have to setup a web server somewhere, however users can bootstrap MicroBOSH from the outside of CloudStack.

[B] an inception server is always required, however users don’t have to setup a web server.

After a heated discussion in the open source community, we adopted approach B as the default solution for uploading stemcells to bosh director due to approach A is not user-friendly. Meanwhile, we created a new branch for experiment with approach A.

Basic Zone VS Advanced Zone

Before the collaboration between NTT and ZJU-SST, ZJU-SST worked in CloudStack advanced zone while NTT developed in CloudStack basic zone. ZJU-SST preferred advanced zone because according to our test, basic zone was unable to support Cloud Foundry without applying some “tricks” during the network configuration process. On the other hand, NTT did deploy Cloud Foundry in basic zone by using some “tricks”, for instance, deploying a separated BOSH DNS node and so on. However, we all agreed that it’s inconvenient to work in basic zone especially when we need to redeploy some components such as router. Finally, we reached an agreement to support both network types and add an option to switch between them.

Fog Support

Both NTT and ZJU-SST invoked fog gem to send API requests to CloudStack engine. However, APIs of the official fog gem are not rich enough to supporting BOSH Cloudstack CPI. ZJU-SST built a local fog gem which added the missing APIs while NTT made a fog patch with missing APIs in CPI project to work around for the moment. We already have sent a PR to fog and are waiting for it to be merged.

When refactoring work finished, we started a month-long heavy test. Once a bug was found, the bug finder would open an issue and describe detailed informations about it. Then all of the developers will receive the message about this bug via mail list. Any commit to the code repository would be submitted in the form of pull request and the repository owners would review the set of changes, discuss potential modifications, and even push follow-up commits if necessary. Only if the PR passed CI and BAT can it be merged. Simply put, we followed the workflow of other Cloud Foundry repositories in GitHub such as cf-release and bosh. In this way, we controlled the history of the new repository and prevented potentially dangerous code from being added in.

Current Status

Finished development and test work.

Support both basic zone and advanced zone for CloudStack.

Tested on CloudStack 4.0.0 and 4.2.0 with KVM hypervisor.

Successfully deployed Cloudfoundry V2 and had applications running.

Support both Ubuntu10.04 and Ubuntu12.04 stemcells.

Support Ubuntu14.04 stemcell is in the TODO list.

Open sourced and maintained in GitHub.

How to Deploy Cloud Foundry on CloudStack

Steps

Create Inception Server

Bootstrap Micro BOSH

Deploy Cloud Foundry

Setup Full BOSH (Optional)

Create Inception Server

You need a VM on the CloudStack domain where you install a BOSH instance using this CPI. This VM is so-called “inception” server. You will install BOSH CLI and BOSH Deployer gems on this server and run all operations with them.

Why do I need an inception server?

The CloudStack CPI creates stemcells, which are VM templates, by copying pre-composed disk images to data volumes which automatically attached by BOSH Deployer. This procedure is same as that of the AWS CPI and requires that the VM where BOSH Deployer works is running on the same domain where you want to deploy your BOSH instance.

Create Security Groups or Firewall Rules

You also need to create one or more security groups or firewall rules for VMs created by your BOSH instance. We recommend that you create a security group or firewall rule which opens all the TCP and UDP ports for testing. When in production environment, we strongly suggest setting more tight security group or firewall rules.

Boot an Ubuntu Server

We recommend Ubuntu 12.04 64bit or later for your inception server. For those who use Ubuntu 12.10 or later we recommend to select Ubuntu 10.04 or later for the OS type while creating instance from ISO file or registering VM templates.

More Steps

You may find more detailed guide step by step concerning on deploying Cloud Foundry on CloudStack here. In fact, the remaining steps are very straightforward and similar with other deployment methods such as AWS, Vsphere and OpenStack.

Why NTT and ZJU-SST?

This work is powered by NTT and ZJU-SST, we have been working together since last November. Thanks to NTT team member Iwasaki Yudai and ZJU-SST team members Du Jun and Zhang Lei, the main contributors of this project, who had devoted much of their energies to fixing issues and rising to challenges on the CPI work. In addition, we would like to appreciate the selfless help received from Pivotal Cloud team and Nic Williams.

ZJU-SST is the biggest software engineering team of Zhejiang University as well as the leading Cloud Computing research institute in China. ZJU-SST started R&D work on Cloud Foundry and CloudStack about 3 years ago, and more recently launched a comprehensive PaaS platform based on Cloud Foundry V1 serving City Management Department of Hangzhou China. ZJU-SST released BOSH CloudStack CPI for Cloud Foundry V1 last May, and introduced the CPI work at PlatformCF 2013 in Santa Clara, California last September.

NTT, the world’s leading telecom company, has been active in fostering the Cloud Foundry developer and user community in Japan. NTT has been contributing to Cloud Foundry for the last two years and sharing its projects such as Memcached Service, Nise BOSH, a lightweight BOSH emulator and BOSH Autoscaler, with the community. NTT Communications, a subsidiary of NTT Group, has been running a public commercial PaaS Cloudn with Cloud Foundry since March 2013, and a video about their efforts on Cloud Foundry in building a commercial service with it is available at the Cloud Foundry Summit 2013 site.

The decision to work together was motivated in part because ZJU-SST intended to upgrade their previously released CPI to support Cloud Foundry V2 and NTT wanted to improve their independently developed BOSH CloudStack CPI project so that can be compatible with CloudStack advanced zone.

Xiaohu Yang, Executive Vice Dean of School of Software Technology, Zhejiang University, thought highly of this international collaboration. “It will be a win-win cooperation. Open source projects such as Cloud Foundry can serve as an international platform for education and researching”.

Facts and Lessons Learned

This is a successful international collaboration which benefits both NTT and ZJU-SST a lot. With the help from NTT, ZJU-SST is able to release an upgraded CPI which supports Cloud Foundry V2 and CloudStack 4.0/4.2. NTT appreciates ZJU-SST for their huge effort in building a CPI runnable on various CloudStack environments. The most precious assets we get from this cooperation maybe the experience in how to perform international cooperation effectively and how to reach out to the community if help is needed.

Join Us

Questions about the BOSH CloudStack CPI can be directed to BOSH Google Groups. Of course, please open an issue or send a pull request if you want to improve this project.

From the marketplace console (or the cf command line utility), select CloudForge then choose one of the packages available. When you select a package by clicking on “buy this plan”, CloudForge will require you to select a unique name for your instance which will be used to create a new organization name in CloudForge. Each “space” in Cloud Foundry will map to a unique CloudForge organization, and users can create different CloudForge accounts for different Cloud Foundry Applications or Spaces, but this isn’t strictly required. You can always deploy your code from a shared CloudForge repository to different Spaces. As CloudForge is a shared service, it does not need to bind to a specific Cloud Foundry application.

Create a new instance called “PivotalDemo” in your default Space (typically “development”) but do not bind it to a specific application. Note: the account name in CloudForge needs to be unique.

Add the CloudForge service in your Space. When you have successfully created a new CloudForge instance it will be listed in your Space Services. Click on “Manage” to finish configuring your CloudForge account.

The instance name you chose for CloudForge service will be used for your new CloudForge account. By default, any user clicking on “Manage” within Cloud Foundry will be automatically logged into CloudForge via OAuth as the Cloud Foundry Space user. However, you will need to choose a username and a password to access your SVN and Git repositories. These SCM services do not support oAuth and are accessed directly from your SCM client, IDE or your command line. Changing your first and last name are optional.

Regardless of how many developers you have it is always useful to maintain version control, even if you are the only developer. Using a Cloud-based SCM provider like CloudForge means that you always have a full copy of your source code backed up and ready for retrieval from any computer. It also makes it very easy to add collaborators on your projects. In order to add collaborators simply click on Admin->Manage Users.

You can invite as many collaborators as you like (depending upon your plan) to join your CloudForge account regardless of whether they are users on Cloud Foundry (the admin CloudForge account is created when the CloudForge service is selected from the run.pivotal.io marketplace, but subsequent team members do not have to be Cloud Foundry users). Each user will create a new username on CloudForge so you can track their commits, issue updates, and comments separately as you develop your application.

Now that you have invited others to collaborate on your projects it’s time to create your first project. Click on Projects->New Project.

Create a new CloudForge Project to manage your development. After you name your project you can add a SVN or Git repository (or both). If you want to include Bug/Issue Tracking, Agile Planning, Wiki, or Discussions you can also choose to add TeamForge to your project. All of these services will be provisioned within your project and the direct links for each provided on your project landing page.

Once you have coded your application and merged all of your branches into a stable trunk you can easily deploy the app to Cloud Foundry using the CF command utility or directly from the Spring Tool Suite. You are able to deploy to any CloudFoundry target, Space, or Org; your CloudForge project is not bound to a specific instance of Cloud Foundry. Feel free to create as many Projects in CloudForge as you need. Your account can be used to manage multiple applications and projects across several Spaces within Cloud Foundry. If you using the Spring Tool Suite IDE, we recommend installing the CollabNet Desktop as well. This allows you to manage your CloudForge account, including all repositories and all tasks, from within the same environment that manages your code.

]]>http://blog.cloudfoundry.com/2014/03/04/collabnet-cloudforge/feed/0Wrapping libyaml in gohttp://blog.cloudfoundry.com/2014/02/28/wrapping-libyaml-in-go/
http://blog.cloudfoundry.com/2014/02/28/wrapping-libyaml-in-go/#commentsFri, 28 Feb 2014 17:24:23 +0000http://blog.cloudfoundry.com/?p=6191Continue reading →]]>We recently released version 6.0.0 of cf, the command line client for Cloud Foundry. cf was previously written in Ruby, and we have rewritten it in Go. This allowed us to package cf as as a single binary and simplified our deployment strategy enormously.

The YAML problem

Two weeks before our release, we realized that the go YAML library we were using, goyaml, is distributed under the LGPL license. This makes it unusable for cf. We had to quickly find another way to parse YAML in cf.

The definitive YAML implementation is a C library called libyaml. We knew that Go had good support for interfacing with C libraries, so we decided to write our own go bindings to libyaml. This felt a bit risky given that we were releasing in two weeks, so we named our creation ‘gamble’.

Writing the library

Go’s FFI system cgo allows Go programs to use C libraries without adding any ‘wrapper’ code in C. This is a simplified code snippet from gamble:

Here, we enable cgo by importing the "C" package. We can can then reference C types (yaml_parser_t, *uchar), functions (yaml_parser_initialize) and constants (YAML_DOCUMENT_END_EVENT) as if they were members of the "C" package. This makes integrating with C libraries feel amazingly effortless.

Using cgo and libyaml, we were able to implement YAML parsing and marshaling in less than three hundred lines (plus tests):

Building the library

We didn’t want to complicate the build process for applications that use gamble by requiring them to compile libyaml manually. Luckily, when go builds a package, it compiles any C source files in the package’s directory and links them into the binary. This allows us to bundle libyaml source with gamble and avoid any extra build steps. To make this work, we had to include into gamble the macro definitions that would normally be generated by libyaml’s configure script.

Unfortunately, after creating gamble, we learned that cgo did not work with cross-compilation. Though it looks like this has been recently fixed on Go’s master branch, at the time we had to update our CI setup because we could no longer build cf for all of our target platforms from a single machine.

Also, since libyaml uses functions in the C standard library, applications that use gamble depend on libc. With help from the open source community, we have updated our builds to be compatible with most versions of libc that are still in use.

Takeaways

Go’s FFI is a joy to use and will be even more useful when cross-compilation support is officially released. We at Cloud Foundry give our sincere thanks to the go team for their wonderful work.

gamble is distributed under the Apache 2 License. The source is available on github. If you need to parse or serialize YAML from go, please give it a try!

]]>http://blog.cloudfoundry.com/2014/02/28/wrapping-libyaml-in-go/feed/0HM9000: Ready for Launchhttp://blog.cloudfoundry.com/2014/02/22/hm9000-ready-for-launch/
http://blog.cloudfoundry.com/2014/02/22/hm9000-ready-for-launch/#commentsSat, 22 Feb 2014 22:29:04 +0000http://blog.cloudfoundry.com/?p=6184Continue reading →]]>Cloud Foundry (CF) is a platform-as-a-service that, once deployed, makes it easy for developers to deploy, run and scale web applications. Powering this elegant PAAS is a complex distributed system comprised of several interoperating components: the Cloud Controller (CC) accepts user input and directs Droplet Execution Agents (DEAs) to stage and run web applications. Meanwhile, the Router maps inbound traffic to web-app instances, while the Loggregator streams log output back to developers. All these components communicate via NATS, a performant message bus.

It’s possible to boil CF down to a relatively simple mental model: users inform CC that they desire applications and Cloud Foundry ensures that those applications, and only those applications, are actually running on DEAs. In an ideal world, these two states – the desired state and the actual state – match up.

Of course, distributed systems are not ideal worlds and we can – at best – ensure that the desired and actual states come into eventual agreement. Ensuring this eventual consistency is the job of yet another CF component: the Health Manager (HM).

In terms of our simple mental model, HM’s job is easy to express:

collect the desired state of the world (from the CC via HTTP)

collect the actual state (from the DEAs via application heartbeats over NATS)

Despite the relative simplicity of its domain, the Health Manager has been a source of pain and uncertainty in our production environment. In particular, there is only one instance of HM running. Single points of failure are A Bad Thing in distributed systems!

The decision was made to rewrite HM. This was a strategic choice that went beyond HM – it was a learning opportunity. We wanted to learn how to rewrite CF components in place, we wanted to try out new technologies (in particular: Golang and etcd), and we wanted to explore different patterns in distributed computing.

HM9000 is the result of this effort and we’re happy to announce that it is ready for prime-time: For the past three months, HM9000 has been managing the health of Pivotal’s production AWS deployment at run.pivotal.io!

As part of HM9000′s launch we’d like to share some of the lessons we’ve learned along the way. These lessons have informed HM9000′s design and will continue to inform our efforts as we modernize other CF components.

Laying the Groundwork for a Successful Rewrite

Replacing a component within a complex distributed system must be done with care. Our goal was to make HM9000 a drop-in replacement by providing the same external interface as HM. We began the rewrite effort by steeping ourselves in the legacy HM codebase. This allowed us to clearly articulate the responsibilities of HM and generate tracker stories for the rewrite.

Our analysis of HM also guided our design decisions for HM9000. HM is written in Ruby and contains a healthy mix of intrinsic and incidental complexity. Picking apart the incidental complexity and identifying the true – intrinsic complexity – of HM’s problem domain helped us understand what problems were difficult and what pitfalls to avoid. An example might best elucidate how this played out:

While HM’s responsibilities are relatively easy to describe (see the list enumerated above) there are all sorts of insidious race conditions hiding in plain-sight. For example, it takes time for a desired app to actually start running. If HM updates its knowledge of the desired state before the app has started running it will see a missing app and attempt to START it. This is a mistake that can result in duplicates of the app running; so HM must – instead – wait for a period of time before sending the START message. If the app starts running in the intervening time, the pending START message must be invalidated.

These sorts of race conditions are particularly thorny: they are hard to test (as they involve timing and ordering) and they are hard to reason about. Looking at the HM codebase, we also saw that they had the habit – if not correctly abstracted – of leaking all over and complicating the codebase.

Many Small Things

To avoid this sort of cross-cutting complexification we decided to design HM9000 around a simple principle: build lots of simple components that each do One Thing Well.

HM9000 is comprised of 8 distinct components. Each of these components is a separate Go package that runs as a separate process on the HM9000 box. This separation of concerns forces us to have clean interfaces between the components and to think carefully about how information flows from one concern to the next.

For example, one component is responsible for periodically fetching the desired state from the CC. Another component listens for app heartbeats over NATS and maintains the actual state of the world. All the complexity around maintaining and trusting our knowledge of the world resides only in these components. Another component, called the analyzer, is responsible for analyzing the state of the world — this is the component that performs the set-diff and makes decisions. Yet another component, called the sender, is responsible for taking these decisions and sending START and STOP messages.

Each of these components has a clearly defined domain and is separately unit tested. HM9000 has an integration suite responsible for verifying that the components interoperate correctly.

This separation of components also forces us to think in terms of distributed-systems: each mini-component must be robust to the failure of any other mini-component, and the responsibilities of a given mini-component can be moved to a different CF component if need be. For example, the component that fetches desired state could, in principle, move into the CC itself.

Communicating Across Components

But how to communicate and coordinate between components? CF’s answer to this problem is to use a message bus, but there are problems with this approach: what if a message is dropped on the floor? How do you represent the state of the world at a given moment in time with messages?

With HM9000 we decided to explore a different approach: one that might lay the groundwork for a future direction for CF’s inter-component communication problem. We decided, after extensive benchmarking, to use etcd a high-availability hierarchical key-value store written in Go.

The idea is simple, rather than communicate via messages the HM9000 components coordinate on data in etcd. The component that fetches desired state simply updates the desired state in the store. The component that listens for app heartbeats simply updates the actual state in the store. The analyzer performs the set diff by querying the actual state and desired state and placing decisions in the store. The sender sends START and STOP messages by simply acting on these decisions.

To ensure that HM9000′s various components use the same schema, we built a separate ORM-like library on top of the store. This allows the components to speak in terms of semantic models and abstracts away the details of the persistence layer. In a sense, this library forms the API by which components communicate. Having this separation was crucial – it helped us DRY up our code, and gave us one point of entry to change and version the persisted schema.

Time for Data

The power behind this data-centered approach is that it takes a time-domain problem (listening for heartbeats, polling for desired state, reacting to changes) and turns it into a data problem: instead of thinking in terms of responding to an app’s historical timeline it becomes possible to think in terms of operating on data sets of different configurations. Since keeping track of an app’s historical timeline was the root of much of the complexity of the original HM, this data-centered approach proved to be a great simplification for HM9000.

Take our earlier example of a desired app that isn’t running yet: HM9000 has a simple mechanism for handling this problem. If the analyzer sees that an app is desired but not running it places a decision in the store to send a START command. Since the analyzer has no way of knowing if the app will eventually start or not, this decision is marked to be sent after a grace period. That is the extent of the analyzer’s responsibilities.

When the grace period elapses, the sender attempts to send a START command; but it first checks to ensure the command is still valid (that the missing app is still desired and still missing). The sender does not need to know why the START decision was made, only whether or not it is currently valid. That is the extent of the sender’s responsibilities.

This new mental model – of timestamped decisions that are verified by the sender – makes the problem domain easier to reason about, and the codebase cleaner. Moreover, it becomes much easier to unit and integration test the behavior of the system as the correct state can be set-up in the store and then evaluated.

There are several added benefits to a data-centered approach. First, it makes debugging the mini-distributed system that is HM9000 much easier: to understand what the state the world is all you need to do is dump the database and analyze its output. Second, requiring that coordination be done via a data store allows us to build components that have no in-memory knowledge of the world. If a given component fails, another copy of the component can come up and take over its job — everything it needs to do its work is already in the store.

This latter point makes it easy to ensure that HM9000 is not a single point of a failure. We simply spin up two HM9000s (what amounts to 16 mini-components, 8 for each HM9000) across two availability zones. Each component then vies for a lock in etcd and maintains the lock as it does its work. Should the component fail, the lock is released and its doppelgänger can pick up where it left off.

Why etcd?

We evaluated a number of persistent stores (etcd, ZooKeeper, Cassandra). We found etcd’s API to be a good fit for HM9000′s problem space and etcd’s Go driver had fewer issues than the Go drivers for the other databases.

In terms of performance, etcd’s performance characteristics were close to ZooKeeper’s. Early versions of HM9000 were naive about the amount of writes that etcd could handle and we found it necessary to be judicious about reducing the write load. Since all our components communicated with etcd via a single ORM-like library, adding these optimizations after the fact proved to be easy.

We’ve run benchmarks against HM9000 in its current optimized form and find that it can sustain a load of 15,000-30,000 apps. This is sufficient for all Cloud Foundry installations that we know of, and we are ready to shard across multiple etcd clusters if etcd proves to be a bottleneck.

Why Go?

In closing, some final thoughts on the decision to switch to Golang. There is nothing intrinsic to HM9000 that screams Go: there isn’t a lot of concurrency, operating/start-up speed isn’t a huge issue, and there isn’t a huge need for the sort of systems-level manipulation that the Go standard library is so good at.

Nonetheless, we really enjoyed writing HM9000 in Go over Ruby. Ruby lacks a compile-step – which gives the language an enviable flexibility but comes at the cost of limited static analysis. Go’s (blazingly fast) compile-step alone is well worth the switch (large scale refactors are a breeze!) and writing in Go is expressive, productive, and developer friendly. Some might complain about verbosity and the amount of boiler-plate but we believe that Go strikes this balance well. In particular, it’s hard to be as terse and obscure in Go as one can be in Ruby; and it’s much harder to metaprogram your way into trouble with Go ;)

We did, of course, experience a few problems with Go. The dependency-management issue is a well-known one — HM9000 solves this by vendoring dependencies with submodules. And some Go database drivers proved to be inadequate (we investigated using ZooKeeper and Cassandra in addition to etcd and had problems with Go driver support). Also, the lack of generics was challenging at first but we quickly found patterns around that.

Finally, testing in Go was a joy. HM9000 became something of a proving-ground for Ginkgo & Gomega and we’re happy to say that both projects have benefited mutually from the experience.

]]>http://blog.cloudfoundry.com/2014/02/22/hm9000-ready-for-launch/feed/2Cloud Foundry Serving Around the Globehttp://blog.cloudfoundry.com/2014/02/14/cloud-foundry-serving-around-the-globe/
http://blog.cloudfoundry.com/2014/02/14/cloud-foundry-serving-around-the-globe/#commentsFri, 14 Feb 2014 19:59:35 +0000http://blog.cloudfoundry.com/?p=6170Continue reading →]]>The Church of Jesus Christ of Latter-day Saints (LDS Church) is a really large organization with even larger reach – probably much more than most of us realize. Sure, you probably know about the massive genealogy library and databases that they have, but it’s so much more. In my cursory research I found that they have numerous humanitarian programs, benefitting people around the globe with health services, drilling wells for safe water, disaster relief, educational services and more.

Why am I telling you all this? Because, just like in virtually every other industry Software is Eating the World; the mission of the LDS Church is greatly enabled and supported through technological innovation and application. For example, I learned that for a time window around Christmas an LDS Church mobile application broke into the top 10 list of applications downloaded from the Apple Appstore; not top 10 in its category, but top 10 overall!! And just like any other enterprise, the Church wants to get better, faster and more effective at creating the technology that it needs.

I received a wonderful gift; an invitation to participate in the internal launch of a new application development and operations platform aimed at doing just this.

Yep, you guessed it – at the core of this platform is Cloud Foundry. A small but crazy-good team of IT professionals at the Church started in earnest with the open source Cloud Foundry project a bit more than a year ago and have been fine-tuning it for their needs since. Okay, truth be told, what they’ve done is so much more than “fine-tuning”; for those of you that hang out on our mailing lists, you already know well what Mike Youngstrom and Mike Heath have brought to the community. The team has built more than a half dozen service brokers, they’ve customized their DEAs, they’ve built CLI plugins and they’ve constructed a developer console that we would all envy. They’ve contributed countless pull requests and donated whole new frameworks to the open source. Some of that I knew about, much more I didn’t see until this week.

Let me tell you a bit about the event. The two hour event was held in their Salt Lake City offices in a large event space set up with a presentation area as well as more than a dozen booths that would showcase specific parts of the platform. As we were setting up, more than one person made a nervous remark about “if no one comes…” – well, they needn’t have worried, the roughly 200 chairs arranged before the stage were not even close to enough! Let’s put this in perspective. In September when Pivotal held its first Platform event we hoped for 100 people, drawing from the 10s of thousands of technologists in the Silicon Valley. The LDS Church event was INTERNAL with a population to draw from sizing in the mid hundreds; a HUGE percentage of those people made the time for this event.

Shawn Nielsen gave the bulk of the presentation and LIVE (!! – you are my hero Shawn!) demo. He showed the creation of a host of services from the substantial marketplace they’ve created. And then he pushed an app, binding it to not one or two services, but to four. When he then brought the app up in a browser it automatically had single sign on (to rounding applause), application monitoring, security features and several other capabilities, all without the developer needing to do more than bind to those services. The demo rocked!

Following the presentation, attendees were invited to visit the booths to learn more about the various components of and integrations with the platform. Customized services and buildpacks were covered, application lifecycle management integrations described, internal customer use cases presented and so much more. The room was teeming with excitement.

The event was a huge success and I congratulate Shawn Nielsen, Mike Youngstrom, Mike Heath and the rest of the team on what they’ve already accomplished. I’d like to take this moment also to personally and on behalf of Pivotal thank “the Mikes” (as I learned they are referred to within the Church just as they are in our Howard Street offices) for what they are bringing to the Cloud Foundry community. And finally, I have to say that I am completely humbled by what they are doing with Cloud Foundry and reach it has to so many people around the globe.

AppDynamics is an Application Performance Management company that offers solutions to monitor a variety of applications running on public clouds or in private data centers. App Dynamics is excited to support Pivotal’s Cloud Platform by making it easy to monitor Java apps running on Cloud Foundry and Pivotal’s Web Services.

Monitor Apps on Pivotal Web Services

The AppDynamic Java agents are included in the default Java buildpack for Cloud Foundry, so if you have AppDynamics monitoring running, the Cloud Foundry DEA will auto-detect the service and enable the agent in the buildpack. If you start the AppDynamics monitoring for an app already running, just restart the app and the DEA will autodetect the new service.

In the future, Pivotal Web Services will include the AppDynamics SaaS APM services, so you’ll only need to signup for Pivotal Web Services and it will automatically create an AppDynamics account. But for now, don’t forget to signup signup for AppDynamics because you need both accounts!

Pivotal Web Services has both a web based GUI as well as a full featured linux style command line interface (CLI). Once you have a PWS account, you can download a Mac, Windows or Unix CLI from the “Toosl” tab in the PWS dashboard or directly for OSX, Linux, and Windows.

3) Sign-in With Your Pivotal Web Services Credentials

Using the CLI, log into your Pivotal Web Services account. Remember to preface all commands given to Cloud Foundry with cf. Individual Cloud Foundry PaaS clouds are identified by their domain API endpoint. For PWS, the endpoint is api.run.pivotal.io. Cloud Foundry will automatically target your default org (you can change this later) and ask you to select a space (a space is similar to a project or folder where you can keep a collection of apps).

5) Push the app to production with the CLI Upload your app with the cf push command, or (as in the example below) you can specify additional parameters including memory, number of app instances, name and the exact app path.

Pivotal Web Services Console

Production Monitoring with AppDynamics Pro

To get more out of your app monitoring, signup for an AppDynamic Pro account with enhanced features and code level visibility into application performance problems.

Watch the Full Installation Video

If you missed any part of the instructions, you can watch Dustin walk through each step in this screen cast video, or reach him on Twitter @Dustin Whittle.

]]>http://blog.cloudfoundry.com/2014/02/04/monitoring-java-apps-appdynamics/feed/0Remote Dependencies, Convenience, Risk and Other Considerations for Operating Distributed Systemshttp://blog.cloudfoundry.com/2014/02/03/remote-dependencies-convenience-risk-and-other-considerations-for-operating-distributed-systems/
http://blog.cloudfoundry.com/2014/02/03/remote-dependencies-convenience-risk-and-other-considerations-for-operating-distributed-systems/#commentsMon, 03 Feb 2014 08:03:32 +0000http://blog.cloudfoundry.com/?p=6110Continue reading →]]> One deeply held principle by experienced distributed system operators that I have worked with is that you should have no external dependencies to your software other than the ties to minimum requirements of the OS such as common system libraries, utilities, and the kernel of the base OS. This approach should enable recreating a distributed system deployment without any dependencies on the outside world. When something goes wrong, you should have control over your own destiny. Reliance on any external dependency that is managed or hosted by someone else introduces risk that something outside your system can affect your ability to restore and recreate the system any time you need to.

To use a simple metaphor, imagine your system is represented by Jenga blocks and it falls over as Jenga towers inevitably do. However, instead of being able to rebuild your tower you find out that a mandatory required component at the base of your tower is missing or unavailable. No matter what you try, you cannot rebuild the tower exactly how it was before. Your new tower is going to behave differently in unexpected ways and you might topple over because you do not understand all the behaviors when using different building blocks combined in a different way.

Some of the original designers of the software deployment project for Cloud Foundry named BOSH (Mark Lucovsky, Vadim Spiwak, Derek Collison) embraced this principle and tried to create a prescriptive framework that encouraged this approach. They had experience managing large scale distributed systems at Google (the web services APIs). Kent Skaar also did similar for SaaS provider Zendesk. Given a software release that references specific versions of multiple software packages (known as a BOSH release), an instantiation of that release (a BOSH deployment) can be reconstructed at any time with the deployment configuration (a BOSH deployment manifest), the base OS images (the BOSH stemcells) and the software release (the BOSH packages and job templates for applying configuration); at any point in time, properly implemented BOSH releases of large scale distributed systems can be recreated without external dependencies. That means this holds true even when the internet is unavailable.

BOSH does give you the framework hooks to break out of this prescriptive principle and use external dependencies or at least external dependency formats if you choose to for convenience or other reasons. Dr Nic Williams recently implemented tooling to use apt packages instead of compiling from source. another example: some of the Pivotal big data software intentionally targets CentOS/RHEL only and therefore only ships rpm packages rather than compiling Hadoop. A guiding principle is that you should be mindful of the tradeoffs you are making of convenience vs risk and tying your release to only one OS distributor.

Examples of the tradeoffs:

relying on an externally hosted package manager like apt-get could affect the availability or correctness of that dependency when you need it most

relying on debian packages could prevent someone from using your release unmodified with a CentOS image

A recent real-world example demonstrated the risk of an external dependency changing unexpectedly. The coreos/etcd project that Cloud Foundry is using for storing stateful configuration data for the new Cloud Foundry Health Manager codebase had one of the dependencies (goraft/raft) force push to master of their git repository that overwrote some git history required by git to work properly. This situation has limited the flexibility of some users to make code modifications on several previous releases of Cloud Foundry without some tedious intervention.

A common reaction when learning about Cloud Foundry BOSH is to question the prescriptive guidance to compile from source when commonly used distributed package management systems exist in the Linux distributions. My recommendation is to understand the tradeoffs involved and make the best choice for your situation. You should explicitly call out external dependencies if you have them in your system. When your tower inevitably falls over, know how to rebuild it.