Kurt Cagle recently wrote an article for Forbes, entitled The End of Agile. Although Forbes’ regular Steve Denning responded effectively a few days later with Why Agile’s Future is Bright, I’d like to chime in with several thoughts beyond what Steve has offered. Here they are:

Look beyond Scrum. Kurt started out with a tongue-in-cheek description of an agile team, but it was really a description of a small team at the very beginnings of learning Scrum. This is the classic “Scrum = Agile” mistake that many people make, and it plays well to people who only have a passing understanding of how agile works in practice. We recommend that you look at teams that are succeeding with agile, rather than teams that are clearly struggling. Better yet, let’s help the teams who are struggling — rather than making fun of them.

Beware of agile purists. Kurt’s observation that agile has become a religion wasn’t far off. It might be more accurate to say that there are Agile purists out there, people who often prove to only have experience with applying agile in straightforward situations — rather than in the enterprise situations that Kurt describes. But there are also pragmatists out there in the Agile space (pragmatism is one of the seven Disciplined Agile principles) who are actively addressing how to apply agile and lean strategies in less-than-ideal situations. We have always recommended that you be open-minded and pragmatic, to do the best you can in the situation that you face and always strive to learn and improve.

Agile works in a very wide range of situations.There is ample research data and case studies showing that agile works in a wide range of situations. For example, in our 2016 Agility at Scale survey we found that organizations were in fact applying agile on large teams, in complex domains, using legacy technologies (including legacy data sources), in regulatory environments, and multi-organization situations including outsourcing. And we have data from other studies going back over a decade that show that Agile strategies have been applied beyond the “small teams with straightforward problems” scenario for quite a while. We recommend that you look around and see how agile is being applied in practice.

Scaling agile requires discipline. The article also states that agile doesn’t scale well to address big problems, yet many organizations are doing just that. To be fair to Kurt, this is a common misunderstanding that is a result of the Scrum = Agile mindset because Scrum purposefully doesn’t address architecture (amongst many topics). But other agile methods do — in particular Agile Modeling and Jim Coplien’s excellent work around Lean Architecture — strategies which are explicitly part of Disciplined Agile. In Disciplined Agile Delivery (DAD), we explicitly include initial architecture modeling early in a project via the Identify Architecture Strategy process goal, we show how to reduce architectural risk via the Prove Architecture Early process goal, and how to evolve architecture throughout Construction via the Produce a Potentially Consumable Solution process goal. DAD teams also have someone in an explicit Architecture Owner role who is responsible for guiding the team through architecture decisions. Furthermore, there is an explicit Enterprise Architecture process blade to support organization-level architecture issues. Finally, DA the idea that Architecture Owners and Enterprise Architects should work closely together. Our recommendation is to explicitly address architecture in your way of working, and this is something that Disciplined Agile provides clear, proven options for.

Data projects can be difficult. The same can be said of non-data projects too. When either domain complexity or technical complexity rise, or both, you need to become more disciplined in your approach to requirements and architecture modeling. As Cagle implies, data projects often suffer from domain complexity, this is particularly true for the enterprise data systems he mentions. Similarly data projects tend to suffer from technical complexity in that they need to integrate data from many disparate sources that often use different technologies, apply different semantics, and are accessed via different strategies. Once again, the context-sensitive choice-driven strategy promoted by Disciplined Agile wins the day over the prescriptive methods and frameworks Cagle appears to be familiar with. To confuse matters further, prescriptive frameworks such as Scrum and SAFe have virtually nothing to say about addressing data issues. Disciplined Agile, on the other hand, encompasses proven strategies from the Agile Data method which are critical to success on data projects. Our recommendation is to embrace the complexity that you face and to recognize that you will need to explicitly tailor your way of working (WoW) to address that complexity.

Agile is applicable to data projects…and may be the superior approach. I’m going to be blunt here – Kurt Cagle was completely and utterly wrong concerning the application of agile strategies to data projects. While traditional data professionals often prefer to do extensive data modeling up front, this strategy has been shown to be ineffective in practice. What does work well is an agile data modeling strategy where high-level modeling is performed early in a project and detailed data modeling performed as needed during construction. You can be very agile when supported by agile database practices such as database refactoring, automated database regression testing, and continuous database integration. And his claim that the focus on enterprise data is relatively new clearly doesn’t reflect the wealth of techniques around enterprise data modeling, master data management, and meta data management that the data community has been following since at least the mid-1990s. And yes, there are agile approaches to all of those practices. As an aside, I started writing this blog in the evenings while attending the World Wide Data Vault Conference in Hannover. Most of the presenters have discussed how they’re taking, or at least supporting agile — and often a Disciplined Agile — approaches on their data projects.

Many other claims the article makes are observably false. Contrary to what Kurt claims, there are many large and complex open source projects, and they clearly benefit from agile and lean strategies. Examples of such projects include Linux (operating system), Hadoop (big data processing), MongoDB (database), Numenta (machine learning), Angular (web application framework), and Docker (software infrastructure) to name but a few. But hey, it’s not as if any of those technologies have become mission critical for your organization. And his discussion of games development? Having actually consulted to several commercial game companies, I can safely say that they were all working in a very agile manner.

You must adapt your approach to address the context that you face. One process size does not fit all. Two of DA’s principles are Context Counts and Choice is Good – different teams in different situations will choose to work in different manners. If you want your teams to be effective then you have to allow them — and better yet, enablethem — to choose their own way of working (WoW).

Adapting your approach requires skill and experience. In some ways, Kurt is right – enterprise-class problems require an enterprise-class agile approach. But he was wrong in assuming that because people were struggling to apply undisciplined agile strategies to these situations that agile didn’t work; the real problem was that they didn’t know how to make it work. The fact is that others have figured these things out, and we’ve captured a lot of these strategies in PMI’s Disciplined Agile (DA) toolkit.

Let’s hope this is the end of undisciplined agile. But as Steve Denning points out, it certainly isn’t the end of agile. I suggest this could be an important beginning for Disciplined Agile approaches.

Although all of the presentations were great, I was particularly enthralled with Bill Inmon’s keynote on Thursday. As you may know Bill is the father of data warehousing and has written 59 (59!) books over his career, clearly putting me to shame. Bill shared some of his experiences in extracting information from text-based sources and described several stories doing so. One story focused on how his team had combined information culled from multiple text-based sources that together indicated that BP had a potential maintenance risk in their Gulf of Mexico operations. Sadly his warning was ignored and several months later BP had a catastrophic oil rig failure on Deepwater Horizon. Another story described how his team processed 5000 online postings from Nike customers and 5000 from Adidas customers. Their analysis indicated that while Adidas was a “normal company,” that on the other hand Nike had quality problems with their shoes. Although Bill contacted Nike to inform them, free of charge, of what he had discovered this advice was also ignored because Nike apparently already had consulting companies providing them with advice. A year later Nike suffered a $2 billion market capitalization loss when Zion Williamson’s sneaker exploded in a basketball game watched by over 100 million people. Another text analysis project led him to discover that airlines are consistently not well liked by their customers, revealing that Bill doesn’t always end up with earth-shattering revelations. Although the stories were interesting, Bill’s description of the techniques he was following and the challenges surrounding text-based data analytics were fascinating.

Data Vault 2 (DV2) is an extension to Inmon’s approach to data warehousing. Dan Lindstedt, the creator of DV2, worked for years with Bill. DV2 brings a lot of very practical strategies to data warehousing. Furthermore, a few years ago DV2 adopted Disciplined Agile Delivery (DAD) for its process which was one of the reasons why I suspect I was invited to speak at the conference.

Kudos to everyone who made the conference a success. I’m looking forward to next year.

At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:

What challenges do you face?

What skills/knowledge does an agile DW/BI coach require?

What strategies should you apply to engage with a DW/BI team?

What are the qualities you should look for in an agile DW/BI coach?

The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.

Issue #1: What Challenges Do You Face Coaching DW/BI Teams?

As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.

In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.

Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?

The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.

Skills/knowledge required of an agile DW/BI coach (click to enlarge).

In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:

As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.

Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?

The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.

Strategies to engage successfully with a DW/BI team (click to enlarge).

The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.

Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?

For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.

To effectively engage with a DW/BI team, you need to gain their trust and respect – to do that you need to show that you understand the challenges that they face and can help them to address them

To do so can be more challenging than with an application team due to the cultural impedance mismatch, the nature of data-oriented technical debt, and the lack of modern development skills within the data community – all of these challenges point to a greater than normal need for coaching on such teams

At the Agile 2017 conference this week Lynn Winterboer was kind enough to invite me to her workshop which explored how to apply agile strategies in the data space. She did a great job of facilitating the group of about 40 people through identifying the challenges currently faced by teams. The main issues that the group explored were:

Product ownership and data

Agile data architecture

Data testing and tooling support

How to include data people, and activities, in development

Lynn will soon be blogging about the results so I’m not going to dive into that here. I suspect that her blog post will be very interesting.

What I’d like to do here is share a few thoughts about what I observed:

The discussion was very healthy. This was a group of very smart people coming from different backgrounds. Everyone was interested in sharing their experiences and working together to solve some tough problems.

Context counts. “Agile and data” is a big topic. A few people were dealing with the issue of how to address data issues on application development projects, some were focused on agile data warehousing/business intelligence, and some were focused on complex data analytics. In our conversations it was very clear that strategies which worked for app development wouldn’t work for analytics, and vice versa. This is why Context Counts is one of Disciplined Agile’s fundamental principles.

The challenges are tough. Everyone was working in organizations that were struggling with these challenges. For each of issue we could have spent much longer exploring the potential solution(s).

Every challenge we discussed are solved issues. I’ve always found it frustrating when people, who are very smart and who have been struggling with a problem for awhile, have clearly never thought to simply google “database testing tools” or “agile architecture” to find out what advice is already out there. When we discussed testing I even asked why people hadn’t done such as search, and then pointed out that there are a lot of tools available and even a few people maintaining lists of such tools (such as 40+ database testing tools). All of these challenges, and more, have solutions described by techniques of the Agile Data and Agile Modeling methods from about 15 years ago. These techniques have of course been adopted, and put into context, by the Disciplined Agile (DA) toolkit and in particular Disciplined Agile Delivery (DAD).

The “factions” behaved differently. Long ago I pointed out that there’s a cultural impedance mismatch between the data and developer communities, and it was pretty easy to observe during this workshop. For example, during the workshop we were asked to identify solutions to the challenges. Lynn wanted us to write this information on flip-chart paper so that she could later scan it and share it with others. For awhile the groups dominated by data people discussed the solutions without writing anything down. Their conversations were good, but they quickly got stuck in analysis paralysis mode because they seemed to be afraid to write down information for fear that they couldn’t easily update it (the challenge with having paper to write on instead of whiteboards). The groups dominated by agile people, the ones focused on development and architecture, started by writing on sticky notes and putting them on the flip chart paper. This fundamental agile modelling strategy enabled them to visualize and share their work transparently, enhancing their conversation and enabling them to move forward easily.

Database (tool) vendors need to up their game. Speaking out tools, database vendors and data warehouse tool vendors really need to up their game. Here’s a few very harsh questions: Does your database tool vendor or ETL tool vendor offer testing tools that enable both black box and white box database testing? Answer: very likely no, because their customers don’t demand that of them. Did your data team even think to ask for such tools? Answer: very likely not. How many database testing books do you think have been written? Answer: very few, go do a search and see for yourself. Why does the data community have such a huge blind spot when it comes to testing? Answer: this is a huge side effect of the cultural impedance mismatch.

I’m very happy to see that Lynn is actively trying to bridge the agile and data communities, helping us to learn from each other. This is something she’s been doing for years and doing it quite well. My experience is that both communities would benefit greatly from more collaboration with each other.

There are several strategies that you can choose to employ with vertically slicing the requirements for a DW/BI solution. These strategies are described in the following table. There are example stories for each strategy as well as some advice for when to apply each strategy.

Table 1. Vertical slicing strategies for a DW/BI solution.

Slicing Strategy

Example Stories

When to Do This

One new data element from a single data source

As a Professor I would like to know the names of my students so that I know who should be there

As a Student I would like to know what courses are taught at the university

Very early days when you are still building out fundamental infrastructure components. Very common for the first iteration or two of Construction. These slices still add real business value, albeit minimal.

One new data element from several sources

As a Professor I would like the student list for a seminar that I teach

As a Student I would like to know what seminars are being taught this semester

Early days during Construction when you are still building out the infrastructure. These slices add some business value, often fleshing a DW data element to include the full range of data values for it.

A change to an existing report

As a Professor I would like to know the standard deviation of marks within a seminar that I teach

As a Student I would like to know how many spots are still available in a seminar

Evolution of existing functionality to support new decision making

A new report

As a Professor I would like to know the distribution curve of student marks in a seminar that I teach so I may adjust accordingly

As a Registrar I would like to know what Seminars are close to being full

Several iterations into Construction when the DW/BI solution has been built up sufficiently.

A new reporting view

As a Registrar I would like to know what the prerequisites are for a seminar so that I can advise students

As a Professor I would like to know the current course load of each student within a seminar that I teach

Several iterations into Construction when the DW/BI solution has been built up sufficiently.

A new DW/DM table

As a Chancellor I would like to track the revenues generated from parking pay meters to identify potential profits to divert to supporting students

As a Professor I would like to recommend suggested readings to help people prepare before taking a seminar

Several iterations into Construction when the DW/BI solution has been built up sufficiently.

There are several interesting things about the stories in the table:

They are written from the point of view of your stakeholders. They aren’t a technical specification. For example, the first story describes how professors want a list of student names but it isn’t saying from what data source(s), what the element names are, … These are design issues, not requirement issues.

They always provide business value. The first story appears to be the beginnings of an attendee list for a seminar. Having something as simple as a list of names does in fact provide a bit of value to professors.

Sometimes that business value isn’t (yet) sufficient. It may take several iterations to implement something that your stakeholders want delivered into production, particularly at first. For example, although a list of student names is the beginnings of a class list it might not be enough functionality to justify putting it into production. Perhaps professors also need to know the program that the student is enrolled in, their current year of study, and basic information about the seminar such as the course name, time, and location of it. The decision as to whether the functionality is sufficient to ship is in the hands of your stakeholder (this is one of the reasons why you want to demo your work on a regular basis).

"In Italy for thirty years under the Borgias they had warfare, terror, murder, bloodshed - but they produced Michelangelo, Leonardo da Vinci, and the Renaissance. In Switzerland they had brotherly love, 500 years of democracy and peace, and what did that produce? The cuckoo clock."