How some organizations are implementing CI/CD

CI/CD implementations and the impetus for them varies among companies, but everyone wants to ensure faster delivery of high-quality software. Following are three examples of companies that have adopted CI/CD in their own way.

Lucidchart improves productivityAbout four years ago, diagramming solution providerLucidchart had 30 developers. The team was “running into issues” because not everyone was aware of the changes going into production.

“We often spent a lot of time tracking down the individual who made code changes that went out with a given release so that we could try to piece together what happened and how best to fix it,” said David Torgerson, director of DevOps at Lucidchart. “Our QA team was having trouble scaling the testing that needed to be done because of all of the different combinations of manual tests that needed to be done.”

The goal was to achieve a regular release cycle, but because they were unable to track down which change caused a problem, releases were “terrifying.” So, they had a meeting following one release to discuss what when wrong. Out of that meeting came the idea of “emergency releases” (aka bug fixes). They started releasing about once every two weeks with emergency releases following during the next couple of days. However, the result was meetings about the emergency releases they had to do because of the problems with the original release.

Worse, because there was no classification for a business-related change, they created a third category of release called “business release,” which led to meetings before release meetings just to determine whether the release was an emergency release or a business release. Eventually, Torgerson and other technical leads realized that the time spent in meetings would be better spent on iterations, so they started moving toward a CI/CD process, albeit slowly because everyone wasn’t ready for the “mindset shift” required.

That first meeting devolved from a discussion to four concurrent arguments. At the second meeting, they agreed that their definition of continuous deployment was “as frequently as possible.”

“If we were looking strictly to be a purist CI/CD shop and appear as a DevOps shop, we probably would do things differently, but our approach has been to improve developer productivity and increase production stability so we’ve opted to piece together best-of-breed [tools] in many cases and we still have some custom-written components that will evolve over time.”

The QA team hated the idea of CI/CD because they didn’t want the manual regression tests they were doing automated. CI/CD seemed like an existential threat. Torgerson said they’re now the biggest CI/CD proponents because they’re having more fun trying to break the product than they ever had doing manual regression tests.

Negotiatus perfects manual processes, then automatesSpend management software providerNegotiatus has grown from 2 to 45 employees since it was founded in 2016. It wanted to improve developer speed by streamlining the onboarding process and improving the reliability of code. They also wanted to improve the speed and accuracy of the deployment process because it had become clear that when the deployment process is manual, it’s easy to forget to run a specific command. What they didn’t want to do was automate processes that weren’t optimized to begin with.

“We try to solve as much as we can manually first, but in an extremely structured way. We basically document a process, run the same exact steps every single time, and ensure people are following them so we can identify any gaps in the process and fill them in,” said Negotiatus CTO Tom Jaklitsch. “The whole thing would run for a month or two and we’d iterate on it.”

Another area targeted for improvement was test coverage because it was less than 30% on a monolithic application at the time. Now, it’s more than 75%. They also made sure all new services were above 95%.

Developers are now required to run their own QA, which has improved code quality and reduced the amount of back-and-forth between developers and QA.

Deployment now happens about 15 times a day, with each deployment taking about 25 minutes. Of the 25 minutes, developers spend one or two minutes at the halfway point making sure everything looks fine. Then, the developer clicks a button and the rest is automated.

“It’s better to be safe than sorry versus automating every step of the process,” said Jaklitsch. “I’d love to get the process down to 10 to 15 minutes, but as of right now, once you kick off and deploy, it kicks off all these automatic processes.”

The deploy ticket queue is populated and managed in Slack with the last person adding their release to the end of the list. That process is being optimized now, Jaklitsch said.

Stealth Communications modernizes its application and processesInternet service provider Stealth Communications was saddled with a legacy business application built in 2009 that was written in C++ and PHP. It had 1.6 million lines of code and took two years to build. The application’s architecture made it extremely difficult to implement CI/CD, so about a year ago, the development team started rewriting it as a microservices application so they could implement CI/CD. More importantly, the new architecture would enable easier upgrades to the UI and easier implementation of new features without massive, infrequent overhauls.

“We want to release features in a real-time fashion without having the user reinstall it or do these upgrade paths that might be intensive or error prone,” said Shrihari Pandit, CEO of Stealth Communications. “Instead of a single stack application that controls everything, we have mini applications that plug in so we can easily take advantage of them. CI/CD is extremely crucial because that’s how you’re going to react to moving markets.”