May people have concerns about the possibility of using Scrum or other Agile methods on large projects that don’t directly involve software development. Data warehousing projects are commonly brought up as examples where, just maybe, Scrum wouldn’t work.

I have worked as a coach on a couple of such projects. Here is a brief description of how it worked (both the good and the bad) on one such project:

The project was a data warehouse migration from Oracle to Teradata. The organization had about 30 people allocated to the project. Before adopting Scrum, they had done a bunch of up-front analysis work. This analysis work resulted in a dependency map among approximately 25,000 tables, views and ETL scripts. The dependency map was stored in an MS Access DB (!). When I arrived as the coach, there was an expectation that the work would be done according to dependencies and that the “team” would just follow that sequence.

I learned about this all in the first week as I was doing boot-camp style training on Scrum and Agile with the team and helping them to prepare for their first Sprint.

I decided to challenge the assumption about working based on dependencies. I spoke with the Product Owner about the possible ways to order the work based on value. We spoke about a few factors including:

retiring Oracle data warehouse licenses / servers,

retiring disk space / hardware,

and saving CPU time with new hardware

The Product Owner started to work on getting metrics for these three factors. He was able to find that the data was available through some instrumentation that could be implemented quickly so we did this. It took about a week to get initial data from the instrumentation.

In the meantime, the Scrum teams (4 of them) started their Sprints working on the basis of the dependency analysis. I “fought” with them to address the technical challenges of allowing the Product Owner to work on the migration in order based more on value – to break the dependencies with a technical solution. We discussed the underlying technologies for the ETL which included bash scripts, AbInitio and a few other technologies. We also worked on problems related to deploying every Sprint including getting approval from the organization’s architectural review board on a Sprint-by-Sprint basis. We also had the teams moved a few times until an ideal team workspace was found.

After the Product Owner found the data, we sorted (ordered) the MS Access DB by business value. This involved a fairly simple calculation based primarily on disk space and CPU time associated with each item in the DB. This database of 25000 items became the Product Backlog. I started to insist to the teams that they work based on this order, but there was extreme resistance from the technical leads. This led to a few weeks of arguing around whiteboards about the underlying data warehouse ETL technology. Fundamentally, I wanted to the teams to treat the data warehouse tables as the PBIs and have both Oracle and Teradata running simultaneously (in production) with updates every Sprint for migrating data between the two platforms. The Technical team kept insisting this was impossible. I didn’t believe them. Frankly, I rarely believe a technical team when they claim “technical dependencies” as a reason for doing things in a particular order.

Finally, after a total of 4 Sprints of 3 weeks each, we finally had a breakthrough. In a one-on-one meeting, the most senior tech lead admitted to me that what I was arguing was actually possible, but that the technical people didn’t want to do it that way because it would require them to touch many of the ETL scripts multiple times – they wanted to avoid re-work. I was (internally) furious due to the wasted time, but I controlled my feelings and asked if it would be okay if I brought the Product Owner into the discussion. The tech lead allowed it and we had the conversation again with the PO present. The tech lead admitted that breaking the dependencies was possible and explained how it could lead to the teams touching ETL scripts more than once. The PO basically said: “awesome! Next Sprint we’re doing tables ordered by business value.”

A couple Sprints later, the first of 5 Oracle licenses was retired, and the 2-year $20M project was a success, with nearly every Sprint going into production and with Oracle and Teradata running simultaneously until the last Oracle license was retired. Although I don’t remember the financial details anymore, the savings were huge due to the early delivery of value. The apprentice coach there went on to become a well-known coach at this organization and still is a huge Agile advocate 10 years later!

The Product Owner Simulation that I shared last summer has some minor updates based on a stronger emphasis on product vision. In particular, two 5 minute exercises before and after the Product Box exercise help to frame the concept of product vision and make it stronger.

Scrum has really come far!!! Check out “Scrum Your Wedding“. I love the ScrumMaster and Product Owner tips. Here’s a good one:

SCRUM MASTER TIP – The stand-up is overkill in most wedding planning scenarios, but we do think it’s useful to ask the questions at least a few times per sprint, perhaps over email. It’s your job to make sure the questions are asked and answered.

They have taken the core ideas of Scrum and made some intelligent modifications to make it suitable for a fixed deadline event planning scenario. Honestly, I think that the ideas presented here could be a great approach to doing Scrum on other similar fixed deadline projects. Thanks to Hannah Kane and Julia Smith for creating a unique and useful resource!

The Scrum Alliance just announced through a press release the Added Qualifications [PDF] program. From the release:

The Added Qualifications program will begin by first offering courses in Scaling Scrum Fundamentals. Those interested in earning an Added Qualification in Scaling Scrum Fundamentals will need to hold at least one of two foundational certifications, Certified ScrumMaster® or Certified Scrum Product Owner®.

Pair Programming Economics by Olaf Lewitz describes three activities in programming: typing, problem-solving and reading code. How does pair programming help? By making the balance between those three activities better.

I was poling around for a good definition of DevOps and found a thoughtful article written by Ernest Mueller called What is DevOps? Highly recommended reading as it includes lots of insight about the relationship between Agile and DevOps. FWIW, I feel that the concept of the Definition of “Done” is Scrum’s own original take on the same class of ideas: breaking down silos in an organization to get stuff into the marketplace faster and faster. I even talked about operationalizing software development back in 2004 and 2005 as a counterpoint to the project management approach which puts everyone in silos and pushes work through phase gates.

Retrospectives are a key part of continuous improvement in Agile teams. The retrospective techniques that a team uses should be adjusted to the needs of the team. In a Scrum team, for example, the ScrumMaster will often decide on the techniques to use based on the current issues facing the team and then facilitate the retrospective for the team. There are some great resources which give you collections of tried-and-true retrospective techniques including Esther Derby’s book “Agile Retrospectives” and the amazing online tool “Retr-o-mat“. As an active consultant and trainer, I am always looking for new techniques to share with my clients. Sometimes, I even create a new one (or at least new to me). The “What Did You Learn” technique is new: I’ve been using it and testing it for a few years now to refine it.

What Did You Learn?

By itself, this is a powerful question. As part of my work with OpenAgile, I’ve been helping teams and organization to focus on learning as an even broader category than continuous improvement. The Learning Circle and the processes in OpenAgile help with focusing on learning. The question “what did you learn?” is very open ended, and can certainly work as an extremely simple type of retrospective in OpenAgile or in Scrum or other Agile methods. Often people like to have a little more structure and guidance so the “What Did You Learn?” retrospective technique provides four categories of learning for people to think about, share, and discuss within a team.

Setup

Setup for this retrospective is very simple: a flip chart or whiteboard divided into four sections or columns works fine, along with a piece of paper for each person in the retrospective, divided up the same way, and sufficient markers and pens for everyone. Here is a downloadable PDF version of the handout for the “What Did You Learn” retrospective.

The facilitator will also participate at various points if they are a member of the team (e.g. a ScrumMaster). It is easiest to do this with a group in-person, but can also be done reasonably well with video or teleconferencing.

Process

The facilitator introduces the retrospective with a welcome and, if necessary, a recitation of the Retrospective Prime Directive. Then, the process is described to the group. Each of the categories of learning is also explained as follows:

Questions. When you can formulate a question about something, it means that you have learned about a gap in your knowledge. In other words, you have discovered something that you would like to learn.

Information / Data / Facts. These are specific details that relate to some area of knowledge or skill. This category of learning is the simplest and is often what people focus on when asked “what did you learn?” Information tends to be dry and unemotional.

Insights / Concepts / “Aha!” Moments. Often when we have a collection of facts or an experience, we see a pattern or make interesting connections between things. This leads us to the great feeling of an insight. Insights tend to be exciting or scary and have an emotional component.

Action Items. These are decisions about what we would like to do in the future, but they could be extremely short-term or very long-term or anything in between.

There are three main stages in the retrospective as follows:

Individual Reflection. For 10 to 15 minutes, each individual works silently to write down the things that they have learned in the appropriate category on the handout. Everyone should try to get at least a couple things into each of the four categories, but more is welcome.

Sharing with the Group. Systematically going around the group and getting people to read from what they have written. This is another 10 to 15 minutes. This stage should not get bogged down in discussion, but brief clarifying questions should be welcome.

Identifying Important Learning. The group now has open discussion to decide on a small number of things it considers the most important that it has learned. This could be based on popularity, but impact, depth, or uniqueness might also be factors in considering importance. These are the items that get written down on the flip-chart. This is usually the longest part of the retrospective and can take up to 30 minutes.

Applicability

This is an excellent retrospective for a team that is going through a significant transition such as starting a new project, a major change in business direction for a product, or as a wrap up technique for sharing lessons learned with other parts of an organization. It is not a good technique for a brand new team that hasn’t worked together before as there will be little common ground for deciding on the importance of peoples’ various shared learning.

In our professional lives and in doing business, we commonly follow the advice to “dress for success.” We make certain to wear that business suit, or a particular pair of snazzy heels, or a certain color of tie. For better or for worse, we can be judged in the first few seconds of contact with a potential employer or customer by our attire, our hairstyle, our facial expression, our nose ring…

A more subtle way we evaluate a person is through the sound of his or her voice. The voice is a very personal instrument, and it can communicate so much about who you are, your abilities and your intentions.

The voice can tell you whether someone is nervous or at ease. Whether they’re authentic or stringing you a line. Whether they care if they communicate with you or not. When I was a kid, I thought I could detect when someone was lying to me by a certain glitch in the voice, or a tell-tale tone. Often, our brain makes intuitive judgements about what’s being said to us, and is sensitive to vocal rhythm, clarity, tones, and the use of language.

One may think it’s not fair to judge someone by their voice. Let’s face it, a voice – like being short, or having a large nose – is usually unchangeable. But it’s how the voice is used that matters. We all have an inherently full, expressive voice, but things happen to us in life that can negatively influence and/ or harm that voice.

Think of the person who speaks so quietly it’s almost a whisper – you must lean closer to catch what she says. This person may have had some trauma in her life, like being constantly told as a child to ‘be quiet’, to de-voice her. I know people whose greatest fear is public speaking, who quake inwardly and outwardly, even if they have something important to share with others.

Personality is also expressed through the voice. Imagine the annoyingly loud talker sitting nearby in a restaurant. This is certainly someone who wants too much attention and tries to get it by being overbearing. Or the fast-talker, who doesn’t want any other opinions but his own to be expressed, and doesn’t give the listener an opportunity to think or to respond, lest they disagree with him.

Anyone can be trained to use their voice for positive communication. A voice is an instrument that can become effective and optimal with practice.

Here’s a few things to think about in how you use your voice:

Are you clearly enunciating your words so as not to be mis-heard?

Are you directing your voice to the person or people you want to communicate with?

Are you speaking in a rhythm that’s neither too fast nor too slow?

Are you allowing your true feelings or intentions to come through?

Are you being honest?

The voice is just one of the important tools we use to communicate. If your work requires relating to other people in any way, for example, making presentations, or promoting a product, consider how you use your voice and what it may communicate about you!

In IT and high-tech, there is a “natural” prevailing culture that makes this first value incredibly difficult. This difficulty is rooted in traditional “scientific management“, but made even more so by a critical additional factor that is mostly invisible: techies solve problems with tools.

Management wants to define processes with clearly described activities, clear inputs and outputs, and clear sources and recipients of the activity (see the description of SIPOC for an explanation of this thinking). Techies build tools to automate these well-defined processes to improve their efficiency, quality and reliability.

Management creates organizational roles with detailed descriptions, detailed goals and detailed performance measurements (see the description of RACI for an explanation of this thinking). Techies build tools to carefully constrain people to these detailed roles to improve efficiency, quality and reliability.

Management has money. Techies want some of that money. So they build the tools to help management get what they really want: a completely automated organization of computers, machines and robots.

The culture of technology is to solve problems with individuals and interactions by introducing processes and tools. The culture of technology is (almost) inherently anti-Agile.

The culture of technology is to solve problems with individuals and interactions by introducing processes and tools. The culture of technology is (almost) inherently anti-Agile.

Individuals and Interactions

Let’s look at the first part of this value in a bit more depth. When we think about work, most of us work with other people. We bring our unique skills, personality and interests to work, and we work with other people who also bring unique skills, personality and interests. In a high-bureaucracy, high-technology work environment, it is easy to forget about all this uniqueness and instead objectify people. When people sense they are being objectified, mostly they feel bad about it. We want to be acknowledged as thinking, feeling, unique beings with agency. Objectification, no matter the source or the rationale, is depressing and de-humanizing. The Agile Manifesto implicitly recognizes this concept and asks us who follow the Manifesto to try to shift our value-focus.

There are many aspects to this concept of humanizing work. Some things that come to mind immediately include recognizing and encouraging people’s capacity for:

creativity and innovation

learning and problem-solving

caring about others

pride in work

complementarity with others

responsibility

Processes and Tools

This side of the value is also interesting. Processes and tools do not have agency. They do not improve on their own. Instead, processes and tools only either remain the same or degrade. Processes and tools are forces for stasis: they encourage maintenance of the status quo. Only humans introduce new processes and tools.

Technologists live in a philosophical double-standard: we build processes and tools for others to use and which we frequently would not like used on ourselves. (We will discuss the cases where me might both build and benefit from processes and tools in a bit.) This is one of the challenges of the type of work we do in technology, but it also applies to many other types of work. So how do we solve this conundrum? I would assert that the principles of the Agile Manifesto and the various Agile methods and techniques are all answers to this question. They show us possible ways to implement this value (and the others) without getting stuck in processes and tools.

Only humans introduce new processes and tools.

What are Processes Good For, What are Tools Good For?

Some processes are good. Some amount of process is good. How do we determine what is good? Well, it largely depends on context. Some examples:

If a close family member is living in a distant location then the advances in communication tools are extremely helpful: the telegraph, the telephone, the cell phone, email, Skype. These tools create connections where otherwise there would be little or none.

If a great deal of data is created while running a marketing campaign and needs to be stored and manipulated, then computers are amazing tools for this. Computers are much much better than human minds and manual record-keeping for this sort of work.

If you create a fantastic new soup, from scratch, for some special occasion and you want to remember how to make and even share how to make it with others, then you document the process in a recipe.

Context, Emphasis and Crisis

Context here is important. The value of Individuals and Interactions over Processes and Tools is basically a statement that given the right circumstances we can use processes and tools, but that our default approach to work and problem-solving should be to focus on individuals and their interactions. Depending on the state of your work environment this is easier or harder.

For example, a startup company founded by three long-time friends who have not yet employed anyone else is almost certainly going to solve most problems that come up through discussion amongst founders and through the development of their skills and capabilities. As a company gets larger, however, there is pressure to adopt more and more processes and tools. This pressure comes from a deep source: lack of trust. At about 12 people, you reach the limit of the number of people you can have and still have anyone do anything (this limit is referred to obliquely in “The Wisdom of Teams” by Katzenbach and Smith). After 12 people, it becomes harder to avoid role specialization and some basic forms of processes and tools. In other words, bureaucracy starts growing as the organization grows. Even at this size, however, it is still relatively easy to have a very strong emphasis on individuals and interactions. There is another important limit: somewhere around 150 to 200 people, any hope of 100% mutual trust among the members of the organization is lost. This is the point at which processes and tools “naturally” start to truly take over. (This transition can happen even in much smaller organizations if the culture does not emphasize trust-based interactions.)

In small trust-based organizations, crisis is usually addressed by the mechanisms of mutual respect, skill development, informal agreements, and strengthening the interactions between people. In a large organization with low trust, crisis is almost always addressed by the creation of new bureaucracy: sign-offs, audits, traceability, procedures, policies, processes and tools.

The true test of the an organization’s commitment to the first value of the Agile Manifesto is, therefore, how it responds to crisis. When someone makes a mistake, can we help them develop the skill and the support networks to avoid the mistake in the future? Or do we put in place even more restrictive constraints on what that person does and how they do it?

In a large organization with low trust, crisis is almost always addressed by the creation of new bureaucracy.

Beyond IT and High-Tech

For now, all that needs to be said is that this particular value of the Agile Manifesto does not in any way directly refer to software or software development. As such, it is pretty easy to see how it could be applied in many other types of work. However, there are some types of work where processes and tools really do take precedence over individuals and interactions. If we want to apply the concepts of Agile universally (or near-universally), we have to examine some of these exceptions. I will leave that for a future essay.