Sunday, July 2, 2017

[Theory] Software craftsmanship Part 3 - Deployment pipeline

In this post I am going to describe common techniques for the deployment process.

Multi-user development leads to a problem of individual errors. Some developers say the phrase "But it works on my machine", although it does not work on another computer.To overcome this problem we can create a mythical independent developer, that will build the project and run unit tests. This mythical developer usually presented as Continuous Integration server, i.e. a robot that runs build and tests on every update of a common code base. In case the build is failed or unit test runner shoes the red flag CI server notifies the team about the problem.

Difference between environments

Any difference between applications leads to a bunch of subtle bugs. There is a need to have one version of the application for the development environment, at the same time we need a small modification of the application for production environment. Let us write this as the contradiction:Contradiction (1)Application should be the same and application should be different.

Solution (1)Divide application on two blocks: one part is immutable and the other is mutable.The application can be divided in two parts: immutable code binaries and mutable configuration. Multiple combinations can produce different versions of the same application: [immutable app binaries] + [development config] = dev app; while the same [immutable app binaries] + [production config] = prod app. This approach allows us to eliminate the bugs that can occur between different binaries for different environments, for example hard coded strings (#ifdef DEBUG). The only difference exists in a configuration file (database connections, log level and so on).
Thus we can archive our application binaries into some immutable package called artefact, e.g. projectA.v.1.zip or serviceB.v3.zip. These artefacts can be placed to a common repository for a later usage.

Testing

There are two main rules for testing:1) Automate tasks as much as possible and to reduce the manual human work. Leave interesting bugs to testers and leave boring job to computers.2) Fail fast. If there is a bug in the application it is better to know about it as earlier as possible. Unit tests are the fastest tests, so we should run them at first. Then go module testing and integrations testing, and lastly the slowest UI tests and end-to-end tests.Unit tests does not require application installation and are very fast, so these tests can be run by CI tool right after the build process.

Integration and UI tests require a distinctive environment, often with its own database. Before showing the work to QA engineers we can run all these tests in a special environment for each commit to our service.

Creating environments

Any person (developer, tester, manager, stakeholder) should be able to pick the exact version of the services (artefacts), apply configuration and add the service to a standalone environment. By default the system should propose the latest artefacts for the new environment.
Each environment should be autonomous and be almost identical to production. This principal can be achieved by usage of Virtual Private Cloud (AWS VPS) or local Docker runner with multiple images. The deployments should be saved the same as production is not destroyed between deployments with its database. If there is a need in a new database, maybe even seeded with tests data, a person can create a new environment. The rest mimics production environment:
The same as blue-green deployment is done in production we can do it in the test environment. After switching to the newly deployed service we will get the next picture:
The switch can be done manually or automatically via some registry service, where every service registers its new address after the deployment.

Configuration

Let me shed light on the configuration and its relationship to the application. For proper versioning and history all configuration files should be placed inside version control system near the service code. Continuous Delivery tool can apply updates to config files with secret codes.
Configuration file might have different properties for external interfaces like payment settings, SMS sending setting, which are not part of the application. Another settings might be specific to the environment, for example, exact email settings for developer or tester.

During deployment a Continuous Delivery tool might remember previous settings and ask about confirmation for applied configuration patching.
The overall deployment pipeline is presented on the next scheme:Happy coding! 😊