Development Operations

A major top-20 regional bank turned to emagine in an effort to drive architectural design and implementation of DevOps practices within a newly created Internal Cloud Team

A major top-20 regional bank turned to emagine in an effort to drive architectural design and implementation of DevOps practices within a newly created Internal Cloud Team. This cross-functional team was responsible for developing new procedures around CI/CD, TDD, source control, infrastructure as code, rapid deployment processes and inclusion of new application/middleware/database operations. As with many large enterprises they were forced to deal with the complexity of a diverse application landscape, well-defined responsibility silos and the learning curve of new technologies and processes specific to the DevOps movement.

Problem Senario

When new servers and networking infrastructure were required, application teams would submit a ticket or make a request to the operations team for the required components. Submission of this request kicked-off the provisioning of the VM's followed by installation of the base operating systems. After the OS was in place a lengthy process, requiring the active involvement of up to six teams was initiated to install all the packages sitting on top of the OS (middleware, database, monitoring, networking, security, messaging, etc.)

The end-to-end execution of this process required up to six weeks – a large number of man hours, contract resources, and consistent management to ensure the delivery process was progressed.

Solution

emagine worked with the client to build an efficient delivery process enabled by Chef, Jenkins, Git/Gerrit and Artifactory – all operating within a larger environment that allowed the client to retain other services that had already been invested in such as VMWare’s vRA, ServiceNow and PAM. In addition to designing and building the technical components of the system the emagine team provided training and best practices guidance to enable the client's resources. This knowledge transfer included, regular group trainings, followed by intense hands-on activity centered around pair-programming and instructor-led development workshops.

As a first step a small team was tasked with building a proof of concept for utilising Chef as the primary configuration management utility. This was done by standing up a private HA Chef server using DRBD on premise and combining the installation of Chef client into the VM provisioning process. Using open source cookbooks and a basic run-list. The POC team was able to quickly deliver a working prototype of a base Linux VM (RHEL) and move on to the implementation of a full solution.

Realising the full benefit of this proof of concept required a complete, customised Chef implementation as well as the introduction of tools and process to support the transition to zero-touch deployment. emagine facilitated this process in the following ways:

Establishing an Artifactory server to manage the Chef artifacts utilised in the provisioning process

Teaching development best practices including unit testing, TDD, version control, and automated testing

Pair-programming with client resources to help them understand how to translate their existing processes into Chef recipes and empowering them to own the process going forward

Impact

The major benefits of this project are realised on two fronts. First and foremost, both the time and resources required to deploy servers were drastically reduced. IT velocity is a major force for success both in the ability for the client to rapidly bring solutions to the marketplace as well as providing fiscal savings to the organisation.

The provisioning of a server now takes 6-11 minutes on average and requires only the passive oversight of 3 departments and no external resources.

The second major area of improvement was the consistency, stability, and reproducibility of the deployment process. Prior to the use of Chef for configuration management the installation and configuration of packages required many manual steps that were executed by system administrators without stringent code tests, specs or reviews. Most of the system administrators had one-off scripts, stored in a shared drive. These used automated pieces of the work and/or procedures which allowed them to performed in a unique way.

Moving to a zero-touch deployment process removed variability and risk from the deployment process. Supplementing this practice with development of best practices, like code review and automated testing, ensures consistently higher quality deployments.

Conclusion

emagine was able to work in partnership with client resources to establish a zero-touch deployment environment built on Chef. The execution of the project included both technical delivery and mentoring on the new technology being introduced.

Successful delivery of this project helped the client realise both operational improvement and expense reduction.