Topics

Featured in Development

Understandability is the concept that a system should be presented so that an engineer can easily comprehend it. The more understandable a system is, the easier it will be for engineers to change it in a predictable and safe manner. A system is understandable if it meets the following criteria: complete, concise, clear, and organized.

Featured in Architecture & Design

Sonali Sharma and Shriya Arora describe how Netflix solved a complex join of two high-volume event streams using Flink. They also talk about managing out of order events and processing late arriving data, exploring keyed state for maintaining large state, fault tolerance of a stateful application, strategies for failure recovery, data validation batch vs streaming, and more.

Featured in Culture & Methods

Tim Cochran presents research gathered from ThoughtWorks' varied clients and projects, and shows some of the metrics their teams have identified as guides to creating the platform and the culture for high performing teams.

Reimagining CI/CD Pipelines as Composable Blocks with Bryan Liles

Bryan Liles, senior staff engineer at VMware, presented at DeliveryConf a series of principles for building continuous integration and continuous delivery (CI/CD) pipelines. Liles recommended thinking about CI/CD as patterns instead of implementations, such as Jenkins or Spinnaker. It should be possible to build a platform with composable blocks and replaceable components that are agnostic to a technology stack.

Liles began the talk with a discussion on how many companies traditionally build CI/CD pipelines; the purpose of having these pipelines is to move an application from development into a production environment. In essence, a delivery process gets the application code from source code management, then the pipeline compiles the application and stores the artifacts in a repository as a bundle. Finally, the pipeline ships the bundle to all different environments like dev, test, and prod. Liles said that if a delivery pipeline looks complex, perhaps the technology stack shouldn’t be any more complicated.

According to Liles, engineers should reimagine deployment pipelines as composable blocks and patterns instead of implementations, such as merely focusing on using Jenkins or Spinnaker. For instance, a pipeline could have an integration block for building the application, running tests, and creating artifacts. This block could use Jenkins, but in the future it could use another tool like Circle CI. Another block in the pipeline could be the delivery process that deploys the application, solicits approval, and runs data migrations. Both the integration and delivery blocks have the same base components: a webhook trigger, a workflow, notifications, and approvals. And the integration and delivery blocks are part of a bigger composable block, the deployment pipeline.

Liles said, "we have to think more about composability in our systems"; a deployment pipeline should be a platform-as-a-service (PaaS) that publishes an app, and then ships the app to its destination. Liles proposed to rename the deployment pipeline to a composable PaaS, where engineers can change components effortlessly. An example of this pattern is Kubernetes, which is a platform for building platforms. With Kubernetes, a deployment pipeline could have the same base components as the integration and delivery blocks, but use multiple different tools. For instance, an integration block could use Tekton or Argo for the trigger, workflow, notification, and approval blocks.

Moreover, Liles said that tools like Jenkins, Spinnaker, or ConcourseCI helped to define how to build CI/CD pipelines and how to use them. However, it’s essential to take the lessons learned from these tools and leave them behind, Liles said. Therefore, he recommended thinking in new ways about tools:

We really need to think differently about our tools; the status quo got us to where we are, but it won’t get us to where we need to be. We should be thinking bigger and differently.

Finally, Liles said that tooling is complicated, and the ecosystem should work on building smaller and composable tools. He recommended engineers start to research about how to improve the ecosystem together instead of making what is already here more polished. The building blocks he proposed are only the start, and he said: "the point should be to think more about composability in our systems."