Not a Member or Subscriber?

Can you be remotely agile?

About this Publication

Can you mix distributed open source culture with agile and lean principles? Sonatype is a unique company of open source leaders that produces software development support tools for companies that are striving to manage the risks of using open source software components. We describe how this startup has successfully used Scrum with completely distributed teams spread across multiple time zones. Find out what preconditions exist, principles we’re discovering, and practices we use as we coordinate the work of multiple teams across an open source product line with everyone working from home-based offices.

Experience Report

The pdf download of this experience report is available to Agile Alliance Membersonly. You must be logged in to download the experience report.

Not a member of Agile Alliance? If you wish to become a member, sign up here.

1. INTRODUCTION

Sonatype is a unique company of open source leaders that produce tools for companies that are striving to manage the risks of using open source software. Sonatype is also a completely distributed company. In 2013, they attempted Scrum and had some early successes even though everyone worked out of their home. But they also had struggles applying a pure Scrum approach. I joined in early 2014 to improve their Scrum implementations and work on agile practices across the organization. What I realized was similar to what Jason Little mentions in his recent book, Lean Change Management, “As a Change Agent, it’s never about you or me – it’s about understanding the perspective of the people affected by the change [1]. I needed to keep in mind the culture and the people that grew the organization. Now we have three completely distributed Scrum teams working across multiple products and we are expanding application of agile principles outside of engineering. This paper will focus on some of the emerging principles and practices that have made agile successful in this very unusual environment and what are some of the pre- conditions that made it possible.

2. BACKGROUND

In 2001, Apache Maven was released as an open source build automation tool and by 2006 it became a standard within the open source community. In 2008, members of the Maven project formed a company supporting the open source community and Sonatype was created. Within a couple of years, it became obvious to the founders that there was a larger problem to solve: the open source community was rapidly generating thousands of open source components and with it, new supply chains were emerging that were used to build a wide variety of commercial and government systems.

By 2014, the average software program is estimated to have 60-90% open source code [2][3][4]. However, only some of the organizations producing these software products successfully manage the complex web of security and licensing issues, which could hide within the included open source components. Furthermore, while some automated solutions were available, mostly manual inspection of the source took place to uncover these issues. With security flaws such as Heartbleed and Shellshock gaining public attention in 2014, even the federal government considered mandating, “any supplier of software to the federal government to identify which third-party and open source components are used and verify that they do not include known vulnerabilities” [4]. Seeing these trends as early as 2011, the founders asked themselves: could Sonatype evolve new products for these emerging software supply chains while staying connected to it’s open source roots? The answer lay in discovering a blending of open source and agile approaches [5].

Because many of the original developers from 2008 to 2012 were hired based on their expertise and contributions to open source projects, the original development team was distributed. By January 2015, we had two key products (Nexus and CLM), many new team members, and three engineering teams organized as shown in Table 1. As you can see from this table, we have experimented with small teams and large teams, grouping members in similar time zones and other structures to support our open source culture but adding increasing collaboration through agile software development practices. While some would refer to our teams as dispersed instead of distributed [6], we prefer neither term as the label implies a lack of collaboration. Even this author struggled with the title of this paper as “remote” has a similar implication that collaboration is difficult in this environment.

Table 1: Teams, locations, and timezones for Sonatype product development

From the perspective of the Agile Manifesto and its twelve principles [7], such distributed teams are unlikely to be successful. However, I was finding that some of the teams were successful at inspecting and adapting, becoming more collaborative, and able to deliver on a market cadence.

3. PRECONDITIONS FOR OUR SUCCESS IN COMPLETED DISTRIBUTED AGILE

We are all “equally dispersed” as we all work from our homes. We feel our distributed agile “work style” is not for everyone. It requires a different way of thinking about your work and your work environment. Our success with this approach is based on several “preconditions” (or ways of thinking about the work).

Precondition: Wide T people - Wide T people have deep skills in some unique areas but are willing to train others in their expertise and willing to help team members in other areas (UI, docs, etc). T-shaped people [8] are a “must” in this environment as it adds to the flexibility of how and where team members can work.

One area where this comes into play is in code reviews. Using the “pull request” mechanism in GitHub [9], many of our teams use a working agreement where any new piece of code must receive two “+1” votes from other team members before it is merged into the main branch. To get these positive votes usually requires responding to suggestions to alternative approaches, reminding other team members about effects to other parts of the code base, or applying some new components or approaches the team has decided to experiment with. Usually, more than two team members provide commentary, but it only takes two positive votes for team to agree on moving the code forward or a single negative vote to halt it (even if there are two positives). The side effect is that the team is broadly exposed to experience of different members in different parts of the code base, the domain or new technologies. This enables different team members to work in other parts of the code they may be unfamiliar with and be guided by those more familiar.

Precondition: Willingness to blend work and life - Much like the craftsmen of 200 years ago and those today who work in small communities, where you lived was often where you worked. Work was part of the family and your family was part of your work. There was no concept of separating the two parts of your life. Instead, you learn to blend the two parts of your life instead of forcing a separation.

For instance, it’s not unusual in a typical team meeting (via video) to have a family member walk into view. If the work conversation allows, we introduce the family member to the team member so that we can better understand both work and family contexts. If focus in the work conversation is required, we refrain from this. In some cases, we will be phone and web meeting only (no video) when we know there may be significant background distractions such as kids playing, an ill family member that someone needs to listen for, or other activity in the home. In these cases, the team member usually has headphones on to help them maintain focus during the conversation and mutes their audio except when speaking.

Precondition: Can move between worlds easily - Individuals successful in distributed agile communicate well via threaded conversation tools like GitHub, JIRA, Confluence or email to explain concepts and coordinate work asynchronously. They also can be equally comfortable expressing opinions and insights in live settings via video/Skype, Google Hangout, web/phone meeting, or face-to-face meeting. Furthermore, they can comfortably shift conversations between these asynchronous and synchronous modes.

In a recent example, a new and important customer encountered some performance issues with one of our products. Two team members took on the task of replicating the conditions on their multiple machines, coordinating their efforts via chat and logging results via JIRA comments on the issue. The Product Owner monitored progress and shared summaries with other parts of the business via email. When we got on the phone for a final status call, the call was actually brief as people asked clarifying questions about what they read in various channels and the two team members clarified their findings and provided recommendations to the customer for next steps.

Precondition: Desire for challenge and learning - Because we can hire anywhere in the world, we can attract extremely talented people. This tends to attract other talented individuals that wish to learn from other brilliant people and be challenged by them and to challenge them so as to sharpen their skills further. In a recent virtual lean coffee, a team member expressed how we like to argue over code, user experience or anything (even craft beer). This team member pointed out that the debates are not to always be in conflict but to instead share alternative approaches to any problem. We always learn something about new technologies, best approaches in current context, and individual preferences in these debates. These debates stem from a desire in our company culture to challenge any assumption and test the supporting logic. This desire was pointed out to me as part of our open source heritage soon after I joined Sonatype.

Precondition: Open source mindset - Open source development exemplifies several characteristics that support distributed development: sharing, transparency, craftsmanship, rapid prototyping and a do-it- yourself (DIY) attitude. The DIY mindset might not be obvious at first as many think “open source” is just about using the work of others. However, many “open source” developers are quick to find or develop the tools they need to build much like other DIY projects. They will quickly resolve their own issues instead of waiting on someone else to resolve them.

Another aspect of an “open source mindset” is these individuals have a passion for the work. That is why they have contributed in the past to open source projects outside their normal work. Now, in working for Sonatype, they have an opportunity to explore their passion at work. With that, they tend not to work “9 to 5” but instead work when they can optimally collaborate with teammates to solve problems that motivate them.

Precondition: A coaching environment - In a coaching environment, you have one or more individuals that will introduce new concepts to get the teams to think of ways to excel while maintaining discipline that has helped them arrive at their current level of success. This may be a senior leader or even a member of a team. When I arrived at Sonatype, we had both. As the newly hired agile coach, I became a member of this coaching environment and I used the opportunity to introduce and model other concepts to expand collaboration. First, I introduced the concept of a Product Owner team to collaborate on scope and priorities for our product line. Then, introducing concepts like lean coffee and other techniques allowed more people to communicate in a more facilitative fashion. Now, members of the Product Owner team share in facilitation across our teams.

Precondition: Servant leadership from the top - Probably one of the most critical pre-conditions for the success of such an environment is the leadership. In 2012, Mike Hansen (an individual well-versed in lean principles) took over the engineering group as senior vice president. Mike’s approach to lean leadership is to focus on setting the right environment and then getting out of the way [10]. Mike currently has everyone in engineering reporting directly to him. There is no middle management within engineering. However, there are multiple leaders within the organization: Product Owner, Agile Coach, and technical leads. None of these positions have staff reporting to them. These roles serve the business and the team and can stand up for what they feel is right based on business needs and sustainable pace of the team.

4. EMERGING PRINCIPLES AND PRACTICES FOR DISTRIBUTED AGILE

What is a principle? Why is it important? According to the Merriam-Webster dictionary, a principle is: a comprehensive and fundamental law, doctrine, or assumption
In our work as distributed agile teams, I noticed a few key assumptions or principles starting to emerge that make our distributed environment successful. Practices typically represent a manifestation of a principle and can change over time [11]. But I have noticed some long-standing practices that show evidence of these principles. I describe both in this section.

4.1 Principle: Multiple Open Channels Always

What are some different ways you can stay connected with your distributed team? How can you sense when things are going well or going off the rails? How do you sense an impediment before it happens? To answer these questions, we find that we typically have multiple communication channels active no matter what the activity. Typically it can be a mix of synchronous and asynchronous communication, but there are always multiple channels and those channels can change based on the types of communications needed within the team. This mix of communication channels can take many forms:

Practice: Back Channels – for every synchronous meeting, having a chat channel is critical. Every team has their own chat channel. This chat channel is used as part of every team meeting to aid facilitation, for votes on decisions such as story estimates, and it can be used to help people reconnect with other meeting participants should phone or other communication connections be lost [12].

4.2 Principle: Work With

There is a temptation to work solo when you are working remotely. Yet, distributed agile teams work best when there is an intention to “work with” other members of the team daily [13]. To “work with” someone means you trust that they will help you toward a common goal, they will support you when you struggle with reaching that common goal, and that this help and support is reciprocal. Maintaining trust is the most critical trait on any team and one of the best ways to maintain trust is to make and fulfill small commitments [14]. We have found several ways in our distributed agile environment to enhance these trust-building opportunities through collaboration:

Practice: Planning together - This intention to work together (“work with”) typically starts when we plan our sprints. The team and the Product Owner join together in a web meeting to review the product backlog of work including features and technical debt, estimate the work together via planning poker (voting on effort in the chat channel), and then collaboratively select the highest priority work to pull into the sprint based on past velocity of the team. When we say “collaboratively”, it is typically the team suggesting the work to “pull” into the sprint once they have already discussed the highest priorities and goals for the sprint. In the second part of sprint planning, the team then decides what tasks will complete this work.
Sometimes, the team completely plans all the tasks for the story, but does not assign those tasks. Instead, team members sign up for some of the first tasks leaving the opportunity for others to help if they are available. In some cases, the tasking is nothing more than the team deciding the approach and what the first task will be with the intention that the first team member who “reaches the user story” will then determine what other tasks are needed and review these with the team.

Practice: Commitment and “buddying up” in daily stand-ups - The Scrum practice of the “daily stand- up”, also known as the “daily scrum” [15] is a great opportunity for team members to select a task from the sprint backlog and commit to completing it in the next day to meet overall sprint goals. Often, our team members will even ask other team members which tasks might be best to pick to meet the goals for the current sprint. By end of the stand-up (usually 5-10 minutes for our teams of 4-10 people), everyone has stated their intentions on how they will help the team and report progress the following day. Progress is also indicated as team members move tasks on the shared task board and even in chat at times. The daily stand-up is also an opportunity for team members to look for opportunities to work together. This is more than pairing and more of the concept of a “buddy system”. This is making sure you are “connected” with a person on a specific activity after the stand-up, whether it’s coding, needing someone to review your work or collaborating in a future meeting [16][17].

Practice: Resolving issues together in the daily stand-up - As the teams are distributed, they may not want to disturb others if they are not in the same meeting or working on the same task. For this reason, each day in the “stand-up call” we have a time after the typical scrum ceremony for resolving issues. In some cases during the daily scrum, a team member may mention they have an impediment and ask for a “post scrum” discussion and suggest who they might need to speak with. At the end of scrum, we then go through the topics either in the order they were raised or in the order that may require the most people in the call. This latter approach allows folks to drop off the call as needed when we get to “smaller topics”. This also allows opportunities for collaborative problem solving with team members when they feel they may not otherwise have an opportunity [18].

Practice: Resolving issues together outside the daily stand-up - In some cases, if the impediment is not urgent and scrum may occur in a couple of hours, you will sometimes see a team member suggest a post-scrum discussion in the team’s chat room. There may be a brief clarification of the issue in the chat room or email to help prepare team members for the conversation.

However, when a team member hits an issue that completely blocks their progress, there is no expectation to wait until the next daily scrum. Depending on the time when the issue completely blocks a team member, the team will respond in several different ways. As we are spread across time zones on each team, a more “western” team member may be impeded when most of the team is offline. In these cases, there is usually an asynchronous announcement and discussion via email. If it is resolved through email, the team member that requested help will summarize the suggested resolution via comments on the task board and/or GitHub commits.

When other team members are available, the team will typically start the discussion in the chat room (e.g., “Can anyone @here explain …” or “I wonder if @all of you can help me look at an urgent issue”). The “@here” and “@all” conventions in chat tools like HipChat allows one team member to easily and quickly get the attention of other team members if they are online. The @all option reaches team members logged into the chat room who are currently online or offline and is used typically during core work hours to reach everyone on the team to bring attention to an issue and ask for collaboration partners to resolve the issue.

When the team feels they need to pull in other groups into the discussion (PO team, Ops, Customer Success), they may send out a summary email first to bring the other groups up to speed on the issue. Then, the team will invite these other groups into the team chat room, the post scrum conversation time or set up a special meeting if the issue is extremely urgent.

Practice: Continuous Quality - On our teams, everyone on the team understands that code reviews and creating automated tests are part of the value delivered. Just cranking out code to meet a requirement and a deadline is NOT sufficient and not serving our customers, our company and especially the team. It is expected that other developers on the team would write unit tests and functional tests all automated. In some cases, we even have one team member write the code and another team member write the automated functional tests so that at least two people on the team understand a complex portion of the code. Then others will review this same code through GitHub pull requests. When it comes time for a release, the entire team is involved in testing any new features. There is no “handing off” to a separate QA team or role. Recently, one of our team did bring on a QA person who has now championed new testing approaches for the teams.

Practice: Standards and agreements – Another way the “work with” principle can be illustrated is through the working agreements the team puts in place. In some cases, these are explicit or implicit. Sometimes I will refer to them as the “law of the pack” as members of the group are quick to point out on one of the communications channels when these agreements are violated.

For instance, it is important that team members feel they can hold each other to standards, like a Definition of Done [19] and working agreements. This applies to development team members as well as agreements between PO team and development team. These standards are often reinforced in some of the conversations that take place through GitHub pull requests.

Practice: Available Working Hours – This is an example of an implicit “law of the pack”. Being available to help team members when they need it and having access to the organization (see Work Open) to resolve your issues are important. I’ve provided some examples previously. But when are people available to collaborate?

Most of our team members are online between 11a to 4p ET (GMT -5). For those engineering team members in Europe, they may be online earlier. However, most of them prefer to work in the afternoons and evenings. Also, our US west coast folks are usually coming online by 9 to 10am PT (12-1p ET). What is interesting is that there has never been an explicit agreement on core working hours. They seem to have emerged over time as implicit agreements and can even shift from time-to-time for each team.

Furthermore, another element of the “open space mindset” is to work when you have passion. For us, this can sometimes translate into some team members working for 8 to 12 hours during a single day. At the same time, no one objects when someone on the team says “I’ll be out for [one, two, three]” hours if it’s for a doctor’s appointment, a family event or just to go play volleyball for a couple of hours, because these same team members regularly make and meet commitments. The others trust they will get commitments done because that is their past history. This flexibility, balanced with commitment to getting the work done helps support a sustainable pace [20].

Practice: Team History and Decision recording – While some teams are better at this than others, there is a general agreement to record decisions once they have been debated either through a live meeting or an online channel like chat or email. Decisions made relative to a current sprint are typically captured in the “card” for that work on the task board. Longer standing decisions and agreements are captured in our wiki. Other conversations that team members want to retain are typically captured in our email groups. In each case, the idea is that the history is captured whether it is a decision or an observation or agreement and it should be easily found through an online search.

4.3 Principle: Work Open / Go Anywhere / Leave No Trace

Transparency is critical in a distributed organization. It’s important that anyone in the organization is able to enter any distributed team space to view progress or ask questions of the team without disrupting the work in progress. Furthermore, it means that any team member has the infrastructure available to easily see what any other team member is doing to foster rapid collaboration [21].

Practice: Every team space open - This means that every agile task board is open and visible within the company. Also, every team chat space is accessible to anyone in the company. Even production servers

and administrative controls can be made visible. However, if you are not a member of the team working in that area, you tread lightly. You look, but do not touch and you are welcome to ask questions if current work commitments permit some flexibility. This openness allows anyone in the company to view a team’s work and quickly have questions answered without significant disturbance to the team’s work. In some cases, it may be just the Product Owner or Agile Coach answering these questions for the team, but since we “work open”, they know or can get the answer quickly from the team.

4.4 Principle: Work-Life Blending

Willingness to blend work life and home life not only appears to be a precondition, but it also becomes an important intention in how we work. In working from home, it is better to share your home office context rather than create an artificial separation. Boundaries should still be set to allow focus on work and also to not allow work to overrun personal and family life. However, to separate the two is to lose important context. By letting others know when a family member is sick or you are moving to a new home, it’s more easily understood when you must step away for a period of time and allows other team members to step in to help with the work.

Practice: Signaling Shifts in Focus - When telling professional colleagues about my work-from-home environment, I often get questions such as “What happens when a family member wanders into the room asking questions or wants to show you some ‘interesting’ object” (i.e., it’s important to them but it disrupts your work or your collaboration)? For most of us, it’s a simple triage process: (1) Will “stepping away” impact your work commitment for the day or the current collaboration? (2) Will it impact the work of others?

(3) Will it impact the sprint commitments? In each of these cases, it’s up to the team member to ask the family member to wait or to let team members know they are disrupted, indicate when they will return to the work (and make up the time), and if it impacts the sprint commitments, where they might need a teammate to jump in and help. If it’s a very brief interruption, it’s not unusual to include the family member in the live conversation and actually use it a as a brief opportunity to socialize between family member and colleagues. This would be similar to family members “stopping by the office”.

Practice: Show Up Early to Socialize – One of the first things I noticed on the daily stand-up calls is some team members would show up a few minutes early. This would be an opportunity to socialize and get to know more about those team members. As facilitator, I would sometimes deliberately carry that socializing conversation to the first few minutes of the scrum to help the team socialize. Later on, other team members would sometimes join early to socialize [21].

Practice: Intentionally Social – After our annual face-to-face Engineering gathering in April 2014, we realized we needed get to know each other better. We would have some brief “social conversations” in team channels, but there was little opportunity to socialize across the engineering organization and get to know other co-workers. We had a common “dev” channel on HipChat and anything could go there, but we noticed it wasn’t being used for any social interactions. A “virtual lounge” was suggested as an experiment by one of our Product Owners. Instead of creating a new chat channel, we renamed the “dev” channel to “The Lounge”. In the lounge, we had a working agreement from our face-to-face gathering that we could share anything we wanted.

We had no idea how important “The Lounge” would become. People became more social. We quickly found that having a “Lounge channel” in chat allowed us to share much of our personal context and preferences. This even appeared to increase the sharing of personal context and preferences in the team channels as well. So we now feel the building “virtual social space” for our teams is just as important as building “virtual work space”.

Practice: Travel-Work - Some of our colleagues are exploring how they can work in different locations that inspire them while still working with their teams and delivering on their commitments. One colleague in particular has experimented with working from Alaska to Argentina to visit friends and family, pursue his passion for outdoor sports, and stay connected and contributing to the team [22]. Another colleague extended his stay in Turkey to work from there for a few extra days after a vacation. Yet another colleague arrived a few days in London before a conference to enjoy the culture.

We find each of these approaches to work-life blending can help prolong a sustainable pace to this distributed work style.

4.5 Principle: Work Lean, Not Cheap

Companies can realize significant cost savings when people work remotely. However, this does not mean that we conserve on all expenditures. Remote employees do need a level of support and infrastructure as well that allows some personal flexibility and keeps them connected with the team and organization on a round-the-clock basis [21].

Practice: Home Office setup - Therefore, laptops, monitors, keyboards, SIP phones are all provided from their first day as well as software tools a team member needs. Only office space and broadband connections needs to be provided by the employee which is common in many homes in the Americas and Europe.

5. CONCLUSION

We firmly believe the Agile Manifesto applies to distributed agile teams. We believe all four values and the twelve principles hold [7]. However, we are considering if the principle of “face-to-face communication” may need to be reconsidered for distributed teams that are effectively delivering software. Instead, it may be time to update this principle to “open, authentic communication” regardless of where team members are located. While we have not solved all challenges working in this environment, we continue to inspect and adapt as a distributed agile team and we look forward to sharing more of the story.

6. ACKNOWLEDGEMENTS

First, I would like to thank Mike Hansen for bringing me onboard Sonatype and by creating this unique collaborative environment through his servant leadership over the last three years. I would also like to thank my colleagues Jamie Whitehouse and Jeffry Hesse for reviewing drafts of this paper, helping me clarify details, and (more importantly) for helping me to continue to cultivate this environment. Finally, I would like to thank my Agile2015 shepherd, Johanna Rothman, for her keen sense of the audience and for helping me reshape the structure of this paper to better deliver the story to you, the reader.