Interfacing between Linear Waterfall and Agile Approaches

This article depicts the best practice approach for integrating Agile approaches and specifically Scrum development with traditional overarching linear approaches, specifically waterfall methodology. The Agile PMO, properly defined, can be positioned to secure Agile-Scrum benefits while maintaining the necessary overarching control.

The challenge

Over the last two decades, various Agile approaches have been introduced and practiced. Of these, in the last 5 to 7 years, Scrum has gained the most popularity resulting from a combination of simplicity, ease of use, and effective public relations. Scrum success in software development organizations has been a powerful driver for roll outs across products, industries and businesses. It was exacerbated by focused marketing efforts on part of Scrum evangelists. Unfortunately, most of these organizations were not structured in a way that supports Agile-Scrum. Even more so, scrum in its raw and pure form is not suitable for the majority of organizations.

The first wave of failed Agile-Scrum implementations brought about an even stricter admonition. The main assertion of Scrum zealots has been that failed implementations are caused by partially embracing the true scrum spirit. The full benefits of the approach can only be attained if the entire organization is reengineered. This fanatical attitude left many project teams across organizations big and small, struggling with their already idealistic implementations. Some have been figuring out on their own, how to combine the contemporary and traditional worlds. Other teams have completely abandoned the Agile-Scrum concepts reverting back to the traditional linear waterfall approach and method. Yet other teams, ridden with guilt, manage Scrum by name only, and hiss vehemently at any project management proponent who is unfortunate enough to advise on re-embracing Agile in a more cognizant approach.

The concepts which are presented and embodied in Agile-Scrum are too good to be ignored; however implementing it outside pure software development requires adaptation.

Complex scenarios for Agile

The main hurdle in achieving the benefits of Agile- Scrum outside software development is integrating it with existing control mechanisms. These control mechanisms are stipulated by various organizational prerequisites and are normally actualized implementing Linear Waterfall methods. Four of these typical organizational prerequisites are depicted below:

Big global corporates – in these hierarchical matrix organizations, top down portfolio control is the rule of the day. The free spirited agile approach has a tough time adjusting to the rigorous controls. Specifically the inherent Agile plan-free concepts, make Agile-Scrum difficult for the organization to swallow.

Highly regulated industries – some industries are required by compliance and governance bodies to have a strict binding control mechanism. These can be for example medical equipment, aircraft, and pharmaceuticals research and product development business units. While individual teams might operate Agile-Scrum, the development process must follow rigid Linear Waterfall methods for traceability and governance.

Complex predefined products –integrated products which include hardware, software, and imbedded are developed as a contract with an end customer based on predefined requirements. In these cases the degree of requirement flexibility is small, though larger than what is anticipated initially. Agile-Scrum concept of a fully flexible backlog suffers considerably in these cases.

Generic IT departments – much of the daily and weekly activities in maintenance driven IT departments is ad hoc. Changes to the daily schedules are numerous and immediate. Constant interferences with the team work are the norm. The concept of time boxing and no interference is difficult to maintain in these situations.

Naturally – many times the four discrete categories detailed above, actually mix; so it is common to find a complex product in a global big corporate which is required to comply with firm regulation.

Based on practical experience, the recommended approach to tackle these scenarios is by empowering the Agile PMO; it acts as an enabler, driver and translator between the emerging Agile-Scrum teams and the Linear Waterfall elements.

Refer to the table below for specific guidelines

The Agile PMO – leading the hybrid organization - guidelines

Scenario

Challenge

Possible solution

Comments

Insights

Big global corporates

Strict controls manifested in Linear Waterfall

The Agile PMO is the buffer between Agile-Scrum teams and the Linear top down control

The Agile PMO is also resourced by administrative staff to ensure compliance with regulations

Product risk is managed on a lifecycle view with members of the Scrum-Agile team;

Backlog populated by Non-functional yet critical requirements and owned by the Agile PMO.

Agile PMO staff maintains traceability of these requirements.

Necessary documentation is part of the backlog

The added administrative effort handled by the PMO is compensated by the increased velocity of the Agile teams.

Administrative PMO staff can also be non-functional product owners to ensure compliance aspects

Scenario

Challenge

Possible solution

Comments

Insights

Complex predefined products

Limited flexibility in product scope tends to deteriorate Agile implementations to Agile by name only; Also, hardware elements of product can’t be performed in an Agile approach

The Agile PMO owns the backlog interfacing with the various components of product development – managing a hybrid Agile-Linear project

This is probably the most difficult and tricky scenario to tackle;

It requires technical as well as leadership propensity and know-how.

Experience shows that by investigating creatively – Agile concepts can be implemented in rigid hardware development environments

Also – rigid product requirements usually allow 20% flexibility

the most value can be reaped in this scenario by developing a customized mixed approach;

Agile stage deliveries can be used to increase flexibility.

Concepts of incremental deliveries may sometimes not be achievable in all product aspects

Scenario

Challenge

Possible solution

Comments

Insights

Generic IT departments

Constant changes to team’s work, inability to see the big picture due to ad-hoc work interfering;

missing a true product owner

The Agile PMO substitutes the product owner role in acting as a buffer to oncoming requests also protecting effort to reasonable levels

Many disheartened IT departments have become bitter when trying to use Agile to their development and ongoing work; the result has been fatigue laden teams, viewing Agile as a vicious manipulation to increase output without genuine management support; more than a single project management approach can be practiced

Noticeably, Kanban works better for these environments;

Time boxing makes sense, however a certain predefined buffer for ad-hoc work should be built into each sprint; Sprint durations should be flexible

Our work with global organizations developing and delivering mobile technology is the basis for two of the above recommendations. We consulted the specific company regarding their product delivery challenges, or should I say problems… their products were late, over budget and delivering reduced scope. Not unlike situations with many product development, IT and software organizations.

The project organization rolled out Scrum method two years prior with limited success. They invested in training and coaching however where at a loss, on connecting Scrum to portfolio management and reporting. This resulted in scrum teams losing alignment with the strategic business objective, a nasty side effect of Scrum implementations. The software Scrum teams were also misaligned with the hardware elements of the product, which were managed as part of an overall program.

We scheduled an initial meeting with approximately 16 stakeholders: product owners and project managers. As we entered the room, it seemed there was an unseen yet tangible wall dividing the room, on one side the Scrum product owners and a few Scrum masters, on the other side project managers and PMO members. During the meeting, as we were gathering information, invisible daggers flew - each group was criticizing the other for the delivery blunders. Members of the Scrum teams accused the ‘traditionals’, as they referred to them, of interfering with their process: changing the requirements, moving people away from the project, enforcing the change control process, and showing up on morning stand ups blaming team members for miss delivering. Members of the project management organization and specifically the PMO said that the Scrum fanatics as they called them, were late with their deliveries, inflexible with changes, and couldn’t provide reasonable progress and status reports.

While we knew that the only way to solve the problems was to integrate the methods, we first had to create an environment of trust and collaboration. We identified a PMO champion that was accepted by both teams as reliable and trustworthy. Together, we developed a change plan, using Kotter’s eight steps approach as a guideline. We searched for the low hanging fruits and found two: increasing transparency of the Scrum weekly achievements and protecting Scrum teams from management interference during sprints. For the second ‘low hanging fruit’ we introduced adaptability to time box durations, suggesting a longer Sprint that would later be useful for integration of Hardware program elements with Scrum. This caused push back from several Scrum zealots, and as I explain elsewhere, you really can’t please all, thus we had to operate with voiced objections throughout. Objections also arose from the ‘traditional’ project stakeholders who didn’t appreciate being told to stop meddling with the projects, not allowing the teams self-manage. By then we had been able to secure support from a kernel group of advocates and exhibit tangible progress. We were diligently working with the PMO champion on translating burn down charts to phase gate view for control purposes. We defined requirement traceability, the backlog of user stories and incorporated the dictionary, translating between sprint planning, execution and the top down Waterfall life cycle. We situated the PMO as a transformation layer and renamed it: the Agile PMO. The defined activities of the Agile PMO alleviated the requisite for strict controls mandated by the global corporate, ensuring the benefits of a Scrum process for project delivery. Our next steps revolved around the hardware elements of the product; we formed a multidisciplinary team of subject matter experts with the goal of enabling Scrum concepts in hardware projects. Most of the work was analyzing the components and process of the hardware project, figuring out process points which could be accelerated. The received benefits were numerous; we defined a Kanban process for hardware scope management and reduced projects duration and costs accordingly. In a year subsequent to launch, we were exhibiting improved results. The Agile PMO was a key differentiator and had a pivotal role in contributing to the success of project and product delivery.

Important best practices to remember that support the concept of an Agile PMO:

Implementing Agile-Scrum as a restricting admonition is exploiting the adaptive nature of Agile;

There is no one right way – no one size fits them all;

There is no silver bullet – you can create what works for you. Kanban is a great tool to use;

Being agile and adaptive also allows being flexible with how one uses the methods, process and Methodology;

Time boxing is great as long as you are flexible to changing the durations of the time box if necessary;

Sometimes the client isn’t directly available, in these cases marketing and product management are a proper alternate;

Arbitrary rules don’t complete projects, people do! Empower your team and yourself to choose the appropriate mix of approaches that enable product delivery.

Summarizing

In my book about the Agile PMO I describe how PMOs can succeed only if they create and enhance value through smart portfolio selection (more about that in a future white paper). With the emerging of Agile approaches and specifically Scrum methods new opportunities have become apparent. Integrating them into an existing control structure – typically presented by a waterfall lifecycle – can be frustrating. We have defined a new key player – the Agile PMO which can be positioned to create a transformation / translation layer between the approached and methods, contributing to higher success levels of these integrations. Michael Nir, President, Sapir Consulting, can be reached at here.

The above whitepaper is excerpted from: The Agile PMO, in print. Join Michael for a 2 day workshop in location globally. The odd couple – Marrying Agile and Waterfall. Learn Breakthrough tools and techniques for integrating Agile in a Waterfall environment; immediately increase value from the AGILE PMO in hybrid setting and Increase Agile / Scrum hybrid implementation success.

About the Author

Michael Nir - President of Sapir Consulting - (M.Sc. Engineering) has been providing operational, organizational and management consulting and training for over 15 years. He is passionate about Gestalt theory and practice, which complements his engineering background and contributes to his understanding of individual and team dynamics in business. Michael authored 8 Bestsellers in the fields of Influencing, Agile, Teams, Leadership and others.
Michael's professional background includes significant expertise in the telecoms, hi-tech, software development, R&D environments and petrochemical & infrastructure industries. He develops creative and innovative solutions in project and product management, process improvement, leadership, and team building programs.

And, reading this: "Limited flexibility in product scope tends to deteriorate Agile implementations to Agile by name only; Also, hardware elements of product can’t be performed in an Agile approach" for me makes sense only if "Agile" is seen in a very restricted way based on the idea, that it HAVE TO be incremental and adaptive with very short iterations in ANY CASE. In my understanding "Agile" means this:

1. Robustness: the ability to maintain effectiveness across a range of tasks, situations, and conditions;2. Resilience: the ability to recover from or adjust to misfortune, damage, or a destabilizing perturbation in the environment;3. Responsiveness: the ability to react to a change in the environment in a timely manner;4. Flexibility: the ability to employ multiple ways to succeed and the capacity to move seamlessly between them;5. Innovation: the ability to do new things and the ability to do old things in new ways; and6. Adaptation: the ability to change work processes and the ability to change the organization.(Based on Power to the Edge: Command and Control in the Information Age / David S. Alberts, Richard E. Hayes. ISBN 1-893723-13-5)

This "Adaptation: the ability to change work processes and the ability to change the organization" and "Flexibility: the ability to employ multiple ways to succeed and the capacity to move seamlessly between them" opens the way of work which I call "Agile Agility" in contrast to a "limited and formalised Agility". This "Agile Agility" encourages to use a way of work which is appropriate also for hardware or for products which can be (and therefore should be) predefined - and at the same time encourages to change fast to a much more adaptive and incremental way of work when it fits better. (Survival of the fittest)

Hans-Peter Korn Thanks for your comment- I agree by and large. What I have noticed is that for scaling Agile to work in big organizations - who are among my clients - it is smarter to connect it to an existing platform - hence the Agile PMO.

.... it is overboarding and complicatin only if you try to use ALL of it in situations which are not approprioate...

Please take in account Bonini's paradox:"As a model of a complex system becomes more complete, it becomes less understandable. Alternatively, as a model grows more realistic, it also becomes just as difficult to understand as the real-world processes it represents" And this quote by Paul Valéry: "Everything simple is false. Everything which is complex is unusable."

So: Scrum in many cases is too simple ... and false. And SAFe in many cases too complex is unusable.

So: Don't follow any of those "schools" as "silver bullets". Better find the appropriate way of "agile" (whatever it might be) for each specific situation.

Funny that I see your answer at this time, as i am facilitating a business modeling workshop in Barcelona. I was just discussing modeling approaches and the continuum of Abstract - Real.Now that i am SAFe accredited I am more entitled :) to the claim that as a model it is just too realOn the psychological level - it is difficult for a model creator to let go of reality and to boil the model down to the essentials it promotes (unless he is Einstein in which case all he needs to do is write E=MC^2)