Over the last few years Codethink has been helping a series of customers
understand and manage the scale and complexity of their software integration
pipelines. We've seen and tried various approaches to visualise what's
happening, but many projects end up falling back to some combination of:

architecture diagrams

data models

workflow descriptions

organisation charts

A key problem is that most of these approaches tend to describe how we think
(or hope) things are, rather than how they actually work. It's hard for new
people to grok what's happening, and even harder for executives to pinpoint
the root causes of business-level problems. Even worse, as the real world
projects evolve, the pictures don't keep up.

At the Cloud Foundry Summit in 2015, Codethink CEO Paul Sherwood was inspired
by Alex Suraci's talk about Concourse, a new tool for CI which can help to
demystify the process of building, integrating, validating and releasing
complex software.

Given that Codethink works on some truly gigantic whole-stack projects for
enterprise, devices and automotive customers, Paul immediately wondered if
Concourse really is 'the CI that scales with your project'. Could it handle
visualisation and integration of complete systems; from initial toolchain
through BSP, kernel, OS, middleware and applications?

The answer is (more or less) YES!

Here's a minimal Linux stack integration pipeline, showing the flow of a
build from toolchain on the left, through to the output system on the right.
Click the image to see the live CI pipeline at concourse.baserock.org

And the Baserock GENIVI Demo Platform for x86_64:

And now a full integration of OpenStack software:

Important things to note about these pipelines:

they represent actual software, being integrated for real

when a new integration fails, its blindingly obvious where the failure
happens, which is a huge win for interaction with a large ecosystem
contributing to a common platform

the pipeline structure can refresh automatically as the underlying
software architecture changes

Chris Brown, who leads the Concourse team at Pivotal says "It's fantastic to
see these impressive pipelines opened to the public. I was always hoping that
Concourse would be used for this scale of project."

There's still lots of work to do, as Codethink improves and optimises this
approach for commercial use-cases and multi-architecture projects, but early
adopter customers are as excited as we are.

If you want to know more about how this new level of visualisation can help
your team, please get in touch!