Article

Improve Your Odds of Project Success

by David Bromlow

August 11, 2009

“Agile” methodologies are a new way of developing the modular object-oriented software used for enterprise SOA. Agile programming breaks a project down into many mini-projects, instead of one huge “Big Bang.” Examine the benefits of this approach.

Enterprise service-oriented architecture (enterprise SOA) enables modular application design and development. The benefits of any SOA in general include breaking complex problems into manageable solutions, standardizing business processes, and grouping common functions together to minimize rework, while encouraging component reuse. Enterprise SOA also uses standards-based Web development technologies, such as Web services, Extensible Markup Language (XML), Java, and Microsoft .NET. It separates and distributes systems processing between presentation, bu
siness logic, and data access layers.

While these characteristics of SOA technologies give us advantages in delivering new systems to meet our changing business needs, it was the object-oriented community that most vividly recognized the limitations of the traditional “waterfall” (sequential) project methodologies.

For example, using a stiff lockdown of requirements early in a project fails to recognize that project teams and their customers typically aren’t aware of what they don’t know. Systems development is a learning process, and as such, project methodology should accommodate this element.

Big Bang deliveries tend to push big risks to the end of the project, compounding losses if the system fails to achieve user acceptance. Waterfall project methodologies are apt to extend the requirements phase of the project in an attempt to get things right the first time. This typically has the effect of delaying difficult tasks and decisions, and of carrying higher levels of risk deeper into the project lifecycle. (The reasons why projects fail have been studied many times; to learn more about them, see “Overcoming Project Failure.”)

Overcoming Project Failure

Numerous studies keep track of failed IT projects, calculate failure rates, analyze causes, and prescribe remedies to improve future success. You may be familiar with the Standish Group’s ongoing CHAOS study or the IEEE Software Hall of Shame; these studies document the greatest project failures each year. (They are defined as reported losses in official corporate financial statements.) The general picture today is that about half of the IT projects started overrun their budgets, substantially miss their go-live dates, fail to meet their critical requirements, or deliver products with significant defects. And many projects have more than one of these problems.

While failure rates are high, the same studies show an improvement in project success rates since the 1990s due to improved project-management strategies. Project failure rates 15 years ago were about twice what they are today. So, the questions are:

Which implemented changes have made the difference?

What can you do to further improve your success rate?

The primary change over time has been the shift toward agile methodologies, whether understood as such or not. The studies point to two main differences between project failure and success:

Project size: Fewer huge projects are attempted today than 15 years ago, or if they are attempted now, they are broken down into smaller, more manageable pieces.

Use of a phased approach: Phased implementations reduce the scope of a project, and an iterative project lifecycle reduces its overall risk.

Through these factors, we see more successes. Complete failures have a tendency to fail earlier, reducing overall losses, and even those projects not considered successful are often closer to their original targets than their predecessors were.

Ironically, most SAP NetWeaver implementers ignore the agile methodologies, which were arguably inspired by the object-oriented software development industry. It is surprising, given the migration toward object-oriented technologies in SAP NetWeaver (enterprise service-oriented architecture [enterprise SOA], Java, ABAP objects, and the modular delivery of standard content in SAP NetWeaver Business Intelligence), how few SAP customers have taken advantage of the benefits of agile methodologies.

In an attempt to resolve these limitations, the object-oriented software community introduced the new “agile” (iterative) methodologies, which are typified by user-centric focus, iterative design and development, phased delivery, and risk mitigation through an early delivery of functional systems with heavy user involvement. (For more information, see “The Agile Advantage in Risk Reduction”.)

The Agile Advantage in Risk Reduction

An agile methodology addresses risks earlier in the project, rather than later. It effectively manages user requirements and provides early value through executable software, instead of postponing the delivery of a working system until the final implementation. An agile methodology flushes out and accommodates change early in the project lifecycle and supplies an early architectural baseline. It also supports component-based systems development, facilitates team collaboration, and, most importantly, ensures quality. Let’s look at each benefit in detail.

Address risks: You can define risk as the difference between worst-case and best-case estimates to complete a project. An iterative approach helps to close the gap between these estimates by addressing the highest risk factors first. Its emphasis on architecture helps to deliver a fully functional working architecture early in the project, mitigating the technology risks of enterprise SOA projects. Waterfall methodologies typically put this delivery off until the development/ realization phase, thereby maintaining a high level of architectural risk deep into the project lifecycle (see “Agile vs. Waterfall Risk Reduction” below).

Note the steep reduction in project risk with the iterative (agile) methodology versus the more traditional (waterfall) methodology.

Manage user requirements: Use cases document a user’s interaction with the system in a time-sequential order, provide a simple step-by-step process for requirements analysis, and allow the project team to easily identify any holes in the requirements. They lend themselves to every step of the development process: requirements, design, development, and testing. Analysts, architects, developers, testers, and users can all compare their deliverables to the original use cases, maintaining a strong user perspective throughout the development process.

Deliver early value: An iterative approach enables a team to deliver functional solutions as soon as possible. Working prototypes demonstrate core functionality at the beginning of the project. An architectural baseline ensures that the technologies put into place will meet the fundamental requirements. In fact, development begins early in an agile project, even as soon as the capture of initial. Thus, if a project is canceled prematurely, customers have working software they can use later as the basis for further development.

Accommodate change early: Project change isn’t bad; it’s the timing and magnitude of the change that causes problems. Agile methodologies manage and promote the discovery of necessary changes early in a project. For a project to embrace positive change, you need to acknowledge that the practice of systems design and development is a learning process. The more you know, the better decisions you will make. Thus, the more you interact with the system and learn about it, the better you will understand what you really want it to do for you. New requirements will emerge as a natural result of systems design, development, testing, and use. An agile methodology anticipates change through prototyping and iterative functional release cycles. On the other hand, some change is unproductive and you want to carefully avoid it. Constant change postpones project completion indefinitely, and late changes to core design features can result in significant rework, cost overruns, and even schedule delays. Different kinds of changes have different levels of impact on the project. Scope reduction changes are low-impact and can be necessary to meet delivery schedules. Design and implementation changes are fairly low-impact due to the component-based design. Architecture changes are also fairly low-impact early in the project, but have a high impact once the construction phase begins. Business-solution changes involving a major reworking of requirements are extremely high-impact if they are made after construction has begun. Most of the time, the issue is not what sort of change you need to make, but when (see “Impact of Change” below).

Time and project phase directly affect the impact of change in a project using an agile methodology.

Early baseline architecture: The primary risk of any system project is the technology it uses. You need to answer the questions: Does the technology exist to meet the customer’s requirements? Can the technology be made to function as desired? Can the technology be implemented cost-effectively? Enterprise SOA projects may have a high architectural risk, so the project manager needs to find out whether the architecture supports all known requirements and the developers can build whatever has been designed. Will the architecture support the performance and scalability requirements? An iterative approach faces these risks by delivering an early, working, architectural baseline. Early testing will prove design decisions before you begin full-scale development. If the system is going to fail, let it fail soon.

Fulfilling component-based design: Agile methodologies provide a systematic approach to defining architecture using new and existing components (non-trivial modules and subsystems that fulfill a clear function). By definition, enterprise SOA is component-based, and such design decisions are made primarily through use cases. By first engaging in systematic modeling on paper, the project team can make changes quickly and inexpensively. Unified Modeling Language (UML), a visual diagramming language, is commonly used to model component design decisions based on requirements scenarios and use cases.

Facilitate team collaboration: Use cases typically form the foundation for collaboration between analysts and users. During design activities, use cases extend the collaboration to include the design team. The developers refer to those same use cases, and the testers incorporate them into test scripts, further extending the collaboration between analysts, designers, developers, testers, and users. Finally, use cases close the collaborative loop during user acceptance. Waterfall methodologies using traditional list-based requirements documentation don’t provide a similar collaborative reference point across all disciplines, team members, and phases of a project.

Ensure quality early: Agile methodologies take quality seriously, realigning the project lifecycle along iterative phases rather than functional phases. The testing discipline and resources are engaged sooner in the project to ensure that requirements are defined in a testable manner, and to test the results of early prototype development. Testing is no longer postponed until the end of the project. This way, you can identify errors and discrepancies between requirements and design early, when reworking them is less costly. Emphasizing architecture and user perspective results in higher levels of quality control.

Modular, object-oriented development demands new design and modeling techniques. It also insists on user-centric requirements definition and new kinds of modeling diagrams for architecture-intensive design.

Furthermore, enterprise SOA development calls for a different approach to quality assurance (QA) and testing, one that traces requirements from their initial definition through systems design and configuration, to testing, and finally through user acceptance. The enterprise SOA development supports iterative development and testing cycles and an incremental delivery of functionality to maintain the involvement of the users.

Anatomy of an Agile Methodology

Some of the most well-known agile methodologies are: context-driven programming, Crystal, extreme programming (XP), lean development, Scrum, and the Unified Process. To better understand the benefits of these agile methodologies, let’s take a look at the most widely used agile methodology, the Unified Process, in more detail.

What is the Unified Process? It is an agile project-management framework that has a proven method for defining project activities, schedules, and milestones. The Unifed Process methodology also provides templates for requirements definition, functional and architectural specifications, as well as project-management deliverables.

The IBM Rational Unified Process (RUP) is a proprietary version of the Unified Process that comes with a full complement of software products to help you to manage a complex technological project. You can also use non-proprietary tools with good results with the Unified Process.

The humpback chart below depicts the Unified Process. It emphasizes the primary difference between its organizational principle of “iterations within phases” and the classical waterfall methodology that cascades down from phase to phase sequentially.

In the “humpback” chart, the phases are column headers across the top: inception, elaboration, construction, and transition. The iterations are column footers: initial, elaboration, construction, and so on, and correspond to their respective project phases.

The “humpback” objects extending across the columns indicate the relative levels of effort that are required for each functional activity, as identified by the labels on each row. When compared with a traditional waterfall methodology (see the traditional waterfall methodology diagram below), you can see that the project is organized around phases and iterations, which equate to points of delivery, rather than to functional activities.

In the traditional waterfall method, project phases are structured around the functional activities of defining requirements, design, development, testing, and implementation.

At the same time, in agile programming, each of the various functional activities are performed across multiple iterations and within each project phase, not restricted to a single project phase.

Traditional waterfall methodology: Each functional activity typically occurs in only one phase, although overlap in functional phases is common. The essential elements are defined during the requirements phase of the project and then frozen. If new elements are needed after initial acceptance, they must be identified through a change-request process and approved for inclusion. In the same manner, design work is based on these initial frozen requirements, development on the frozen design, and testing on completed development. Finally, project go-live depends on successful test completion.

The artificial lockdown of requirements early in the project — as well as the subsequent full-scale effort on design, development, and testing — can result in getting exactly what the customers asked for, but not what they really wanted. Customers don’t always know what they want. Certainly, they aren’t aware of their own blind spots. Throw in limited user involvement at the beginning and end of the project, mixed with a rigorous and intimidating change-control process — even the terminology for the process is intimidating — and it’s easy to see why these projects tend to have more elevated failure rates.

The Unified Process is designed to avoid this problem by providing regular checkpoints with the users — at the end of each iteration — where they interact with the developed system, refine their understanding of what they want, enhance the development team’s understanding so it can deliver what they want, and either approve or pull the plug on the project. The phases of the project define its overall structure, much as a project plan does in a traditional waterfall project. However, each iteration — through requirements, design, development, testing, and implementation — takes on more of a just-in-time (JIT) character as the project team accommodates continuous interaction, feedback, and approval from the users.

Phases and iterations in the Unified Process: Each phase of a Unified Process is analogous to the phases of a waterfall project, but rather than governing the phase’s functional activity, the Unified Process governs the phase’s management emphasis and the corresponding objectives, deliverables, and milestones required to facilitate it. The functional activities that take place are the same for each Unified Process phase, but the emphasis changes over time. (To understand what I mean, see “Unified Process Phases.”)

Each project phase is further broken down into one or more iterations. Each agile iteration then executes all the functional activities that a project normally accomplishes: all the way from business modeling, requirements, analysis and design, and implementation, through testing, deployment, and configuration management, and even to project management and environment management.

Unlike a traditional waterfall project, each agile iteration is a completely self-contained Software Development Lifecycle (SDLC) unto itself. Each iteration includes requirements definition, design, development, testing, and implementation. Each one also results in a system release — whether that is an internal release for prototyping purposes or an external one for the productive use of the user community.

When to Stick with What You Know

Managing a project using any new methodology is no easy task. There is a significant learning curve when you are adopting a new methodology for the first time, and moving to an agile methodology from a waterfall methodology tends to be even more difficult for those project teams who are steeped in the traditions of the waterfall method. The terminology, artifacts, and deliverables of the new method will be unfamiliar. Also, since most customers tend to be suspicious of change, you should expect to make some mistakes and to face some kind of a struggle on your agile, iterative methodology project.

The benefits of making the transition from the waterfall methodology to an agile one are sometimes difficult to find and the costs of any mistakes can be even more difficult to justify. As a guideline, small projects with limited architectural impact are always good candidates for the traditional waterfall methodology (which is, essentially, a single-iteration Unified Process project). The implementation of iterative, agile programming on configured systems that have only limited customizations will amount to fewer benefits than your more customized software development projects will.

The benefits of the SAP NetWeaver platform and enterprise SOA are opening doors for innovation and business agility within the SAP user community. These benefits come with new challenges that the methodologies from which they grew can mitigate. If your company is facing some of these new challenges or is expecting to, this just might be the appropriate time for you to consider adopting the new iterative agile methodologies.

David Bromlow is president of Quivira, an SAP NetWeaver Consulting company based in Overland Park, Kansas, and is a regular contributor to SAP NetWeaver Magazine. He has extensive experience in IT project management, technical leadership, and business dynamics. Focused on SAP NetWeaver technologies, Quivira offers its customers expert technical guidance and services with an emphasis on sound project-management fundamentals. Bromlow’s expertise with agile methodologies is born out of many years of real-world experience implementing both SAP systems and enterprise Web applications. You can reach him at david.bromlow@quiviracorp.com.