We in Basefarm manage mission critical applications and today all application owners would like to get more code and innovations out in production faster. Manual deployments of code takes too much time and money, and is also more dependent on people doing deployment correctly. In order to speed up the release cycle an automatic deployment process is needed. Läs mer

How to move mountains – our road to continuous delivery

Developing an agile culture to achieve a faster time to value is more about the people, and how you approach the problem, than what techniques you apply.

Continuous delivery

Continuous delivery is a practice where new features that pass an automated QA regime is automatically installed in a production setting. The practice speeds up delivery of software and avoids costly manual operations, but in turn requires extensive automation and confidence that the automation performs well over time. The practice is being used by all the big web players we use daily, and is now becoming adopted by more and more teams. Sadly, many of these teams will probably fail because they will try to copy the efforts made by another team without adopting it to their own situation.

This article is created to answer frequently asked questions by our customers about how we introduced continuous delivery (and in the beginning, becoming an agile team) while building our core provisioning tool. The situation described by the customers is one where the technology starts to override the “why” of the application. Management and other teams that have previously been burnt by the same issue, have now become skeptical.

We don’t conform to any of the described processes out there. We pick whatever we think will work, and if it works, we keep doing it.

Our Transition

This article is written for those who are interested in our path, and it will be a bit different because it deals with technology only superficially. We think the team aspects behind our transition to continuous delivery is much more interesting than the technologies we used to accomplish it. While technologies have a shelf life of three or four years, our team characteristics should be useful for much longer.

Please don’t consider this article as a prescription on how to adopt continuous delivery. All teams are different, so perhaps another path would be better, or maybe continuous delivery won’t be useful at all to your team. Our team agrees that continuous delivery was right for us, and I think this is where our path really began.

1. Building the right team culture

When we were asked to rewrite the previous version of Basefarm’s provisioning system we wanted to do this using an agile methodology.

For us, this meant building a team culture where we trust each other, with a minimum of office politics, where people can ask for and receive help without being thought less of, where we would rather produce working software instead of having meetings, and where we have all the knowledge needed to build the application inside the team.

As a corollary to this, only those on the team get a say in how the team solves problems. If you are not part of solving the issues that might arise, your opinion will be listened to, but you do not have a right to block the team from trying something new. We think this is an important point to make because humans are often skeptical to change, and granting non-team-members the power to stop experimentation would lead to no progress being made.

Not for everyone

Being an agile team is not for everyone. Over the years, seven people have left our team. Two of these left because they had fundamental issues with what we were doing. These are hardly numbers that are large enough to be statistically significant, but there are people out there who prefer to work in teams that are organized differently. We think it is safe to say that of the people currently on the team, many of them would probably leave a team that was not “agile”. However your team is organized, this is something that needs to be addressed. Our team has an explicit team contract which addresses some of these concerns, and we will hopefully be able to handle any future problems when they appear.

2. Stop wasting time in meetings

The first casualty of the team becoming agile was the “department meeting” because we needed the time taken up by meetings to make changes that could improve our efficiency. In our team, meetings have concrete agendas that are known beforehand, and it is not OK to call meetings with a generic name other than “Standup”, “Grooming/planning” and “Retrospect”, which are our three recurring meeting types. A few people on the team have worked in other teams where there were two-hour meetings every week for keeping the department manager updated. We wanted our team to avoid this style of communication because it steals days of working time every week.

3. Reduce the impact of failure

The next big change we endured was to take a good look at our own team. Developers are often in the habit of pointing to other IT functions and saying we could have done better. An example can be big IT processes where someone at some point make a mistake, and then a process, performed by humans, was put into place to ensure it didn’t happen again. We think this often is the cause for byzantine processes. Every step in the process makes sense individually, but taken as a whole the process becomes big and slow.

When we turned around and looked at ourselves, we found we were stuck in the same trap, but in a different context. The underlying fear was the same as our hosting colleagues had – we figured that production outages were a bad thing, so we had accrued a fair bit of magic over the years to placate the failure gods. Once we allowed ourselves to see this, we also accepted that we would have failures in production. It is inevitable.

By switching the team goal from “no failures” to “as few failures as possible at the greatest speed possible” we have hopefully avoided reintroducing big, complex processes, and we routinely ask new team members how we are doing.

4. Constrain the size of changes

The provisioning system we make is designed in such a way that every change made to the machine room is constrained in scope. This ensures that, if things fail, only the things inside the limited scope fail, not the whole machine room. Sequences of operations is ordered in such a way that steps are performed in the sequence with the least likelihood of failure. When we select systems for our machine rooms, these nonfunctional requirements are reflected in that the systems must provide for updates that are limited. On top of that, there is an extensive automated QA regime so we know when existing behavior changes, but more about that later.

This is a mindset change that sounds very fluffy, but was a very powerful one for us. It allowed us to get rid of a lot of very strict routines which had been put in place mostly to force people to think before making a change, and often doing so by forcing them into meetings. Instead, we could think more about mitigation and recovery, and about creating systems that could be automated to perform the tasks consistently every time.

Our experience is that the number of outages has gone down, even with a trivial process and without being able to point fingers at specific people who started installation of a new version. We mostly blame our automated tests for this success.

5. Use automated testing for safe deployment

We decided to invest in a regression test suite before writing a single line of code. One reason for this was the old system was in a language that doesn’t lend itself to testing as much as our current JDK/Scala combination does, and not having regression tests were proving to be costly. If our provisioning system handles our infrastructure the wrong way our customer solutions might stop working. Our provisioning system is business critical to us.

We also saw that manual testing had a quadratic cost over time, growing with the number of features and the complexity of the system. If down payments are not made to automate your manual tests, the cost is going to become prohibitive very fast.

This is the beauty of automated tests. They exchange the quadratic cost of manual testing with a once-off payment, and a small maintenance cost. The one-off payment is not much bigger that performing a manual test of the same feature once. Our QA regime is the first technical investment we made that enables continuous delivery. If you want hands-off installation of new versions, anyone would prefer to know that they work as well or better than the last one.

6. Keep coordination simple

The next big milestone for us was to move most of the dependencies for our application into the team. In the 90’s it was thought that operations would host an “application server” which would host many applications, which were then delivered by the developers according to a specification.

Our team adopted the big application servers in the start because we wanted to reduce the number of changes we made simultaneously. Working off the existing application server delivery model already in use by Basefarm was a sensible tradeoff for a time, but application servers have a cost associated with them, and the cost comes in terms of coordination with the external operations team. Nowadays, virtual machines cover much of the benefits of application servers, which negates any benefit they had in the past.

The application was rebuilt to embed a small Jetty server inside, causing it to become self-contained. We integrated with a startup script delivered by the operations people, and required an oracle JVM and a small property file to be present on the host running the application.

Operations manual for efficiency

We also delivered a tiny operations manual that told the 24/7 operations team how to restart the application, where logs were located and who to call if a restart didn’t work. From this point on, the development team took on the application server hosting in addition to operating the application itself. The overhead related to the nonfunctional aspects of the application have mostly disappeared as a result. This pattern coincides with what we do for many customers who have in-house developed solutions. At some point, a developer will have to wake up if the problem is big enough.

7. Work yourself away from coordination

Our next big overhead turned out to be patching the database. We delivered a set of patch files that were applied by the DBA. This required coordination between the DBA, the operations team, and the development team.

When this coordination went wrong we got outages. We found ourselves in situations where the application worked perfectly in the test installation, but not in production, and the cause was that the DBA had forgotten to create a table, for whatever reason. No application can be expected to work consistently under this kind of situation.

As a result, we taught our application how to automatically apply the database schema changes using a library called Flyway. This removed all the outages caused by people trying to coordinate a process divided amongst three different functions. We have been working off this model for half a decade, and we have had two outages, which is an order of magnitude better than what we experienced when database updates were manual.

8. Implement Database refactoring for even more efficient deployment

By this time, the application still required one to five minutes of downtime while upgrading it, and the application was seeing enough use that this downtime was becoming a problem. This was by far the biggest problem to overcome for the team. Basefarm is a managed service provider, and the immediate reaction to this downtime was that the upgrades of our application should be moved out of production hours. Our team chose to go another way.

The reason for the downtime was the database. When installing a new version of the application the database got patched, and our trivial implementation of database patching couldn’t guarantee the old code would be happy running off the new database schema. We sat down in the team and applied a set of patterns from a book called “database refactoring” (Ambler), and from that point on we were +1 compatibles on most of our releases, meaning that we had changed our development process so that we didn’t apply changes that the previous version would be unhappy with.

9. Increase communication with the users

Technically, this change was relatively easy. Talking to worried users has never been easy, and we cannot stress enough the importance of spending time with the users of the system so that their level of discomfort stays at a manageable level. We decided early on that we would try to make the development team available to the 550+ Basefarm employees to address issues when they arise because not feeling you can get help is a surefire way of escalating user discomfort.

Providing person-to-person support to 550+ people has worked out, and to this day, the development team is available in a chat channel where anyone can ask questions. We have had to add a handful of specialists to the channel to be able to handle questions that are focused around best practices for using our application, but overall this kind of communication has been instrumental in creating the kind of safety net that has allowed our users to feel comfortable that our regular changes will not interfere with their ability to do their work.

Chat channel for fast communication

The chat channel has also been a benefit to the team because the random interruptions to work we saw before where one of our users would stop by whoever was perceived to be the highest-ranking developer to get help. Interruptions like these break flow and destroy productivity. Having a chat channel lowers the expectations of an immediate response that people in a doorway have, which enables people to poll the channel when they have a bit of free time instead of having to break off what they were doing. In addition, the workload is spread away from senior people who are usually the ones to catch all the questions otherwise.

Don’t underestimate fear

At this point we had a pipeline running our tests which was not connected to the installation system. The final part of the puzzle was to use the existing pipeline for the installation task, too. This was the final step that would allow us a fully automated installation of new versions.

Even with our communication efforts there was a fair bit of fear at this point because our users felt uncertain. They wanted mails each time we upgraded the application in case it broke during the upgrade.

Sending mails is an automatable task, so we complied with this. We were just happy that we didn’t have to upgrade outside of office hours. It turns out that we were asked to remove the automated mails within three months because the upgrades turned out not to fail, which caused the mails to become an irritant. Sometimes, it takes time to build trust in new systems.

The automation of the installation process has not caused any outages to date. We are, again, seeing a much better performance from the automated process than from the manually performed one.

Our starting point

Our team was lucky in that we started out as an empowered, trusted team, mostly left to figure out how we wanted to work on our own. “Succeeding with agile”, Mike Cohn contains a good introduction to what a team needs to be able to succeed at being good at change.

This article started out by trying to focus away from “what” we did, and instead trying to describe “why” we did it. We believe this kind of mistake is very common, and our team has had to fight it constantly as our team evolved. It turns out lean adopters struggled with the same problem, and “Toyota Kata” by Mike Rother contains several war stories from the manufacturing world that have helped us in stopping when we were focusing too much on technology and rituals to the detriment of becoming better at change.

Today, change has become a constant force for our team, and continuous delivery is one aspect that has led us to be able to produce more value in a better way. Tomorrow, we will improve in some other way. Your own path towards becoming better at change will probably be very different, with different forces pulling your team in different directions than ours, but we hope this article can be of some help.

Download our Cloud Guide

What should you focus on in order to take the next step in your company’s cloud journey? With help from over 200 IT professionals and a Cloud Maturity Ladder, this report will help you to focus, prioritize and it will guide you to the next level.

Cookie and Privacy Settings

How we use cookies

We may request cookies to be set on your device. We use cookies to let us know when you visit our websites, how you interact with us, to enrich your user experience, and to customize your relationship with our website.

Click on the different category headings to find out more. You can also change some of your preferences. Note that blocking some types of cookies may impact your experience on our websites and the services we are able to offer.

Essential Website Cookies

These cookies are strictly necessary to provide you with services available through our website and to use some of its features.

Because these cookies are strictly necessary to deliver the website, you cannot refuse them without impacting how our site functions. You can block or delete them by changing your browser settings and force blocking all cookies on this website.

Google Analytics Cookies

These cookies collect information that is used either in aggregate form to help us understand how our website is being used or how effective our marketing campaigns are, or to help us customize our website and application for you in order to enhance your experience.

If you do not want that we track your visist to our site you can disable tracking in your browser here:

Click to enable/disable Google Analytics tracking.

Other external services

We also use different external services like Google Webfonts, Google Maps and external Video providers. Since these providers may collect personal data like your IP address we allow you to block them here. Please be aware that this might heavily reduce the functionality and appearance of our site. Changes will take effect once you reload the page.

Google Webfont Settings:

Click to enable/disable Google Webfonts.

Google Map Settings:

Click to enable/disable Google Maps.

Vimeo and Youtube video embeds:

Click to enable/disable video embeds.

Cookie policy Basefarm

COOKIE POLICY BASEFARM

1. COOKIES 1.1 What are cookies?

By using the website you consent to that Basefarm stores cookies on your computer. Cookies are small text files that are placed on your computer while you are browsing a website.

1.2 Basefarm’s use of cookies

Basefarm uses cookies to facilitate your use of our website. This includes using the information collected by the cookies to confirm your login and to remember personalised details and to facilitate the availability of the services on the website.

Cookies are also used to collect information on how the website is used. In addition, with our cooperation partners we collect anonymous information of which browsers that visit the website to show relevant advertising (interest based advertising).

1.3 Manage your cookies

Most browsers are set up to automatically accept cookies. By changing your browser’s settings you can choose between accepting cookies, receiving information when a cookie is placed, or blocking cookies. The way to manage cookies may differ between browsers, but normally the menu is reached through tools or alternatives. If you decide to block Basefarm’s cookies, this may limit the functionality of the website.

You can find more information about cookies and how to delete or block cookies on the website www.allaboutcookies.org.

1.4 More detailed information

Cookie

Cookie used and domain

Typ of Cookie

What does it do?

How long is the cookie saved?

SESSxxxxxxxxxxxxxxxxxxx

basefarm.com

Session cookie

The cookie is used to keep information (not password) about the site editors (Basefarm marketing department)

1. Introduction Basefarm is committed to protect and respect your privacy. With this privacy policy Basefarm describes how it ensures that your personal data and other data is processed in accordance with applicable data protection laws and cookie legislation.

2. Data controller Basefarm AS, reg. no. 982 211 743, Nydalen Allé 37a, 0484 Oslo, Norway, is the data controller in relation to personal data being processed on the Norwegian and English versions of the website. Basefarm AB, reg. no. 556638-0639, Sveavägen 159, 113 46 Stockholm, Sweden, is the data controller in relation to personal data being processed on the Swedish version of the website. Basefarm BV reg. no. [•], Beechavenue 106, 1119 PP Schiphol-Rijk, Netherlands, is the data controller in relation to personal data being processed on the Dutch version of the website. The aforementioned Basefarm entities are collectively referred to as “Basefarm” in the following. You will find Basefarm’s contact information under section 10.

3. When does basefarm collect personal data? When you or your employer sets up an account or signs up for Basefarm’s newsletter; When you apply for a job at Basefarm or otherwise send a job application to Basefarm; In the event you turn to Basefarm with inquiries or requests via e-mail or telephone; and If you have accepted the use of cookies, Basefarm may also collect your IP address. For more information about Basefarm’s use of cookies, please see section 12.

4. What data may Basefarm collect? The personal data Basefarm may collect includes information about your name and contact details such as address, telephone number and e-mail address, company and any other information you provide. If you apply for a job at Basefarm, Basefarm will process your CV as well as any other information you attach with your application.

5. How does Basefarm process personal data? The personal data collected by Basefarm is used to manage customer relations, assess potential employees and assist customers and website visitors with any requests or inquiries made on the website. The information may also be used for monitoring and development of Basefarm’s business and website, for example by analyzing statistics of website visitors, and to protect Basefarm’s rights. If you apply for a job, Basefarm only uses your personal data for the purposes for which you provided the information. However, Basefarm may save interesting applications even after the recruitment period is over. Such applications may also be transferred to other entities within the Basefarm group.

6. To whom may Basefarm disclose the information? Basefarm will not sell, lease or otherwise transfer any personal data collected to a third party. Basefarm may however transfer the personal data to other companies within the Basefarm group or to business partners if it is necessary to fulfil its obligations towards you. Personal data may be disclosed if it is necessary to: a) Comply with applicable law, regulation or similar or to comply with a legal process, request or order from an executive authority; b) Defend Basefarm’s legal interests; or to c) Detect, prevent, or otherwise avoid fraud, security breaches or technical issues.

7. Links to external websites Basefarm’s website may contain links to third-party websites. Basefarm is not responsible for the processing of your personal data on such websites.

8. Amendments If this policy is amended, Basefarm will publish the amended policy at www.basefarm.com with information about when the amendments will enter into force. If Basefarm carry out any significant changes to the policy, Basefarm may choose to inform by e-mail or by publishing a message on the website.

9. The right to information and recifications You have the right to require information about what personal data Basefarm is processing about you and for what purposes. You are also entitled to have any incomplete or inaccurate data rectified, erased or blocked. Please see the contact information in section 10 should you have any questions about how Basefarm processes your personal data.

11. Security measures Basefarm has taken the organizational and technical security measures required to protect personal data against unauthorized access, modification and deletion.

Preferens

Keep in touch with us - we’re aware that your inbox is a sacred place, and we’ve, built this page to put you in control.

With your email registration you are accepting that Basefarm is storing your personal data information and is using it to administrate your registration. We would like to send you personal emails with company news, content, invitation to events, webinars, reports, offerings, product and service information. Please check the boxes below what kind of personal information you would like to receive from us.

I am hereby giving consent that Basefarm is sending me emails on following topics:

Check to enable permanent hiding of message bar and refuse all cookies if you do not opt in. We need 2 cookies to store this setting. Otherwise you will be prompted again when opening a new browser window or new a tab.