Customers of software development services who want to
outsource a software development project face a problem: Traditional methods of
selecting a software developer are expensive, time consuming and optimize the
wrong criteria. They set the stage for delays, cost overruns, and building
software with poor or no return on investment.

Agile methods, including Scrum and XP, have proven successful
at creating great solutions that meet the expectations of their sponsors and
users. Many organizations would like to apply agile approaches, but the
traditional tools for selecting a vendor don’t mix well with agile development.

By conceiving the project from the beginning as an agile project, you can outsource projects effectively and agilely. This paper:

Describes how one team used Scrum to create an agile RFP

Discusses what information should be present in an agile RFP

Proposes how to find a partner to trust through a lean, Agile selection
process

Background

My customer wanted to develop an application to automate
critical work and information flows. These are very complicated, involving
direct customers, end customers, the company itself and its suppliers. The
company had some expensive experiences with waterfall development processes.
Scrum had usually been the solution, so for this project, it seemed logical to
conceive the project to mix well with Scrum. But this presented new problems:
How do you write an agile, Scrum-Compatible RFP? How do you select a company to implement an agile project?

A Google search [1] was remarkably unhelpful with this
problem. The top links all pointed to a paper and presentation from 2003 about
combining Use Cases and User Stories in the RFP process. A request to the
ScrumDevelopment Group produced no responses [2]. So we were on our own.

A Classical RFP is poorly suited to Agile Development

A request for proposal would typically include items like the
template in Box 1. The customer seeks to specify his requirements precisely; the
supplier should say exactly what they are going to do, how long it will take,
and what it will cost. Because requirements, time tables and deliverables are
rigidly specified, this form of contracting does not fit well with agile development processes.

My customer assembled a team of two domain experts and myself
as Scrum coach to put together the RFP. We decided to use Scrum to create the
RFP. During "Sprint Zero", which only lasted a week, we were briefed
by the product owner. Then we created and estimated the product backlog. We
reserved a meeting room for the duration of the project, which gave us a place
for the burn down charts, task board, and other information that gets hung on the wall in a Scrum project.

We considered the RFP itself to be a product, so we used the
agile workshops originally defined by Mike Cohn [3] to figure out who our
readers are and what they want from the document. The users included:

Our own management, who need a clear vision of what the product will be
and what it means for the company

Prospective suppliers, who need to understand the business process which
shall be automated and what the new system should do.

"Everybody" who needs to understand how Scrum would be applied
in this project.

We came up with a table of contents which included important elements of a classical Project Plan [4]

Project Charter

What are overriding the corporate goals -- taken from the corporate
website

Elevator Pitch - what will the product do for its users?

Product Mission - what will the product do for the company?

Current Situation: How is the problem being solved today?

Desired Situation: How should this system solve problem?

Risk Analysis: What has gone wrong in the past and how do we want to
prevent that in the future. What are the big non-functional problems that
have to be solved?

Project Management Procedures, Roles and Responsibilities: A description
of Scrum and its roles.

Budget Recommendation: How much should the project allowed to cost based
on the benefit it will bring to the company?

Each entry in the table of contents became a theme for
grouping the stories. Each story corresponded to some content in the RFP or an
appendix. For example, we had stories like ‘Current Process Diagram’ or ‘Project
Management Procedures’ which corresponded directly to (sub-) chapters in the RFP.

Sprint Planning and Demos

Each sprint lasted 2 weeks. The Sprint Review and Planning
Meetings were not as formally separated as in Scrum by the book. The Product
Owner gave us guidance, first in the Kick-Off meeting, later during the Sprint
Reviews and the team decided how to translate that guidance into content in the
RFP. So the team was truly self organizing and assumed some of the duties of the Product Owner.

Based on our initial estimates, we set the following themes for each Sprint.

Project Vision

Current State

User Interviews

Desired State

Final Document

Retrospective/Next Steps

It didn’t quite work out that way. In the Sprint 1
Retrospective, we realized that we should be delivering the RFP incrementally
and that we should work toward a definition of done. So we decided that a Story
is only finished when the corresponding information has been integrated into the
RFP document and when that chapter has been reviewed within the team for
content, spelling and grammar, and the updated RFP is posted on the Wiki. After
Sprint 3, we could have given the RFP to potential suppliers after any Sprint
(perhaps with a little formatting), and they could have starting the bidding the process.

Sometimes we did break stories down ‘horizontally’, e.g.
each user interview was a story, even though a user interview produced no
content for the RFP. Such stories had to produce a clearly defined deliverable
which was useful for creating the final deliverable. So a user interview story
produced a summary of the interview which the team used a basis to discuss the
feedback.

The First Sprint

The application should automate the information flow between
the company, its customers and its suppliers. The process is very complicated
and had been analyzed previously, so we wanted to build any work which was already available.

The first sprint focused on two issues:

What is the product charter: Who is the product for? What should it do for
them? What should it do for the company?

What previous analysis was available?

The first drafts of the Product Mission and Elevator Pitch
provoked a lot discussion with the product owner and beyond. Small differences
in the wording of either can have tremendous impact on the product, adding or
removing many person-months of work. Lack of clarity increases the risk of
building the wrong product, not including critical functionality, wasting
resources on unneeded functionality or endlessly retargeting the product as
management changes its mind about what the vision is.

Using Scrum for the RFP Project

We used classic Scrum to create the RFP. Stories corresponded
to chapters or subchapters of the RFP. The stories were estimated in Story
Points. At the beginning / end of each sprint we discussed the results with the
product owner and planned the "functionality" for the next sprint.
Each story in the sprint backlog was broken down into tasks and estimated in
hours. We used release and sprint burn down charts to track our progress,
occasionally adjusting the scope to ensure that the project would deliver the RFP on time.

We used ‘tangible tools’ - cards and flip charts - to manage the backlog, tasks and burn down charts.

By the end of Sprint 3, we had a document in close to final
form. There is a special moment of recognition in the Product Owner’s eyes,
when he sees that he will get his product as wants it. People who have worked on
a waterfall basis aren’t used to experiencing that until much later in the
project, if at all. By the last sprint, we were starting to wonder if the
additional work we were investing wasn’t just ‘gold plating’, which made
it easier to declare the RFP ‘Done!’

Contents of an Agile RFP

Agile projects set different priorities than traditional
approaches and this has an impact on the contents of the RFP. The most important
differences between agile and traditional approaches can be summarized as follows:

Agile Project

Traditional

Objective of Project

Meet goals and needs of Organization, Users and Customers by providing them functionality which helps them accomplish their goals.

Realize defined scope within defined time and budget

Scope

Negotiable - realize the minimum set of features required to meet the objectives of users and customers

Precisely defined in advance. Often expressed as wish
list, which gets cut down in the process of negotiations to meet budgetary goals.

Scope Changes

Embraced. The priority of not yet implemented functions can be downgraded in favor of new requests which are judged more important.

Discouraged. Changes are often a source of delay or
cost overruns. Change requests are often used by vendors to restore
profitability to projects where fierce competition resulted in
unprofitable basic prices.

Time

Release quickly to start generating ROI

Release once with all functionality

Quality

Actively defined and agreed upon through definition of ‘done’, which is confirmed at the end of every Sprint

Assured in a separate QA-Phase. This increases the risk of delay and cost overruns, because errors detected late are more expensive to find and fix.

Trust

Pre-Requisite for working agilely, difficult to
establish before work starts.

Ideally planned proactively as a function of the value
which the project should produce.

Risk

ROI is managed continuously by the product owner.

Incremental delivery, regular inspections and prioritization (delivering the most valuable work first) are the primary
tools for mitigating risk.

Risk of cost overrun is a major factor in planning the project. Fixed Price/fixed scope, cost ceilings, penalties for late
delivery are used to minimize financial risk

Because of the differences, our RFP had a similar structure but different contents than a typical, waterfall oriented RFP:

1. Introduction (the Company introduces itself)

2. Goals

Corporate Goals

System Goals (Product Mission)

Project Goals (Elevator Pitch)

3. Budget Considerations

Potential Benefits and Savings

Investment Recommendation

4. Process (Scrum)

Risk Management

Project Organization

Scrum Roles and Responsibilities

Scrum Work Flow

Quality Management

5. Current Situation

6. Desired State

Personae (Roles)

User-Stories

Workflows

7. Selection Process

Trial Run / Competitive Sprints

Introduction and Goals

Agile is goal and business value oriented. So the first
priority of the RFP is to communicate the goals of the company, the project, the
product and its users and other stakeholders. The functionality of the system is
a means to the end. So the RFP started with a presentation of the company, its
goals and the marketing and functional goals of the system.

This section was the subject of substantial discussion with
management. Who are the intended users? How is the market changing and how
should this product be positioned to react to those changes? What should the
product do and what should not do? The results were realistic expectations on
product and a clear vision of what should be built and why.

Budget Considerations

For agile projects, a relationship of trust between customer
and supplier is essential. Trust means that each partner does not need to fear
that his inadequacies will be exploited by his partners. Trust enables conflict
at the level of ideas - What is best for the product, for the customer and for
the customer’s customers? With trust, win-win situations are possible. With
trust, it is possible to give in and compromise. Establishing trust requires
openness and honesty and takes time (together) to build and develop. This
encouraged us to include the budget considerations directly in the RFP and
presently clearly the expected risks of the project, even though it meant
exposing our own warts.

It was difficult to come up with a believable model for the
benefits (costs savings, new revenue, new business models etc.) of the system.
There so are many consultants with spreadsheets of dubious value. So we chose to
simply frame the problem. How many employees will be affected by the system? How
much more business could they process if their productivity were raised by 5%,
10%, etc. Or how much money could be saved?

There was not one answer to this question, but a range of
values for discussion. For each of these ranges, we calculated a value based
investment recommendation, based on the double worst case analysis. [5]

Process (Scrum)

The risk analysis focused primarily on actual problems. We
had done a retrospective on previous projects and had a good idea on what we
wanted to improve or prevent, e.g. SW reliability problems (internal quality),
fitness-for-use problems (external quality), project management and oversight
problems, supplier dependencies, and misalignment of short term vs. long term budget optimizations.

Many of these issues can be addressed effectively by using
Scrum, so we defined Scrum as the process for software development and for
escalation management. We included a description of the Roles, Rituals and
Artifacts. We included some additional metrics on Software Quality, Acceptance
Tests defined and passed, Unit Tests defined and passed, which should be presented at every sprint demo.

Escalations shall be handled by a "Management Oversight
Committee" modeled on Ken Schwaber’s Enterprise Transition Team. Their
job is to resolve impediments that the ScrumMaster can’t.

Originally envisioned as two separate chapters, the Risk
Analysis chapter became the introduction to the process chapter and the
justification for using Scrum.

Some issues aren’t directly addressed by Scrum. How do you
ensure system maintenance over a 10 year period? This is both a financial
question and an engineering question. We simply raised these issues with the suppliers in the RFP.

Current and Desired Situation

Next, an understanding of the context is essential to make
good decisions about how to implement the product. The current situation
described the process as it existed. This made clear how the business functioned
and helped identify optimization potential, both in the execution process and
more importantly, in the customer’s decision process, which could lead to more business for the company.

The section ‘Desired State’ reflected most clearly the
difference inherent in an agile approach. Rather than focusing on the
functionality of the system, this section focused on the users (roles or
personae), their goals and what they want to accomplish with the system.

We spent the most time analyzing the current and desired
situations. As when planning the RFP Project, we started out by brainstorming to
identify the user roles ("Personae") and their function in the
process. Some of the roles were obvious: production workers for our company, our
customer and our suppliers. As the purpose of this system is to reduce operating
costs, these were clearly the people who would most intensively use the system.

An important discovery in the Personae workshop was that the
primary (full time) users of the system are not the purchasers or influencers.
The users’ managers, the suppliers’ and customers’ sales, marketing and
management and even our own top management will be ‘only’ occasional users,
but have much more to say about the success of the system than the primary
users. By giving these users real value, we create Exciter Functions [6] for
those users who are most essential to success of the product.

Further, support users and the system developers also needed
functionality to do their jobs, and creating the right functions (including
delegating support functions back to the user) could lower operating costs substantially.

This quickly produced a draft of the desired situation, not
so much in terms of the detailed functionality, but of the user roles and how
they will change with the new system. It also showed how the system could change
the relationships with customers and suppliers, open new business opportunities or generate new revenue from existing customers.

Specify the Product as User Stories

We ended up writing a complete set of user stories for all
major users of the system. The result was 20 pages long, so we moved them into
an appendix. In the main document, we just included a top level description of
each user role, his/her duties and what that role expects from the system.

Augment the Description with a 4 Track Value Flow Chart

We had set out to describe the processes using Value Steam
Mappings [7]. After all, we wanted to eliminate waste. However, the times
involved in the processes were so variable that it was not possible to come up
with numbers which would be generally accepted. Instead, we created a 4 Track
Value Flow Chart. Each track corresponds to one of the primary actors:

End Customer (Our customer’s customer)

Our Customer

Us

Our Supplier

Sample Three Track Value Flow Chart for a fictional job portal.

The tracks are laid one on top of the other and the
horizontal axis represented time. The flow chart shows activities, processes and
decisions, and who does each. The flow analysis makes it clear where
productivity improvements could be made, when key decisions are made and by
whom, and what interfaces between companies are present and necessary.

Selection Process

How do you know if a team can do agile? How do you know that
they can work with you effectively? Which team is really the better choice? This
is a hard question to answer. The classical approach suggests that the vendor
should produce extensive documentation about the proposed solution (at no cost
to the customer) and limit the risk through fixed price/fixed scope contracts,
cost ceilings, bonus/penalty incentives etc.

These approaches do not work well with agile development
projects, because they create competitive games between vendor and customer,
undermine the trust which is necessary to work together effectively, and are
challenged reacting to variability in scope.

Competitive Sprints, an agile selection process

Some people would like to believe that building complex
software is like going to the grocery store: pick a candy bar off of the shelf,
ask what it costs and decide to buy it. You get no risk and quick gratification.
But building custom software is more like building a race car. A special one-off
product to meet exactly the needs of its sponsor: win races.

As a customer, what are you really buying when you contract
for software development? You may think you are getting a solution. But what you
are really getting is an implementation team. And risk is always part of the
bargain. So what you really want is a team you can trust to build your product
and to minimize the risk of that choice.

Competitive sprints are a lightweight, lean approach to
selecting a software development partner which should dramatically reduce the
risk and cost to all parties.

Selecting a vendor through a bidding process is expensive and risky

The process of writing a huge RFP, evaluating and eventually
accepting complicated bids from largely unknown vendors is wasteful, risky and
expensive. A customer will often invest substantial effort into creating a
Request for Proposal (RFP). I've seen RFPs whose size is measured in binders and
the effort to produce them measured in man-years. A vendor will often invest
comparable effort into winning a big project. On both sides, this effort
produces mostly paper. These artifacts have no value except as means to the goal
of selecting a vendor.

On the vendor side, this effort has to be amortized during
the execution of the project itself. As there is only one winner, the others
make a substantial effort and earn no money. So this risk gets passed on in the
form of higher prices on successful bids.

Competitive bidding can mean offering ruinous prices. When
"winning" means producing at a loss (sometimes to the point where the
supplier goes out of business [8] [Automatic English translation [9]), even the
customer loses. The supplier may have no choice but to play the change-request
game to achieve profitability.

A Lean Approach to reduce cost and risk for all parties

How do you maximize trust and minimize risk and cost? Once
trust has been established, customers are often willing to work with agile
companies on a time and materials basis. Trust greatly reduces administrative
overhead and enables a collaborative approach, so establishing trust should be
the first priority in a project.

Lean thinking encourages us to see the whole, eliminate waste
and defer decisions until enough information is available to make that decision
with reasonable risk. Deferring potentially expensive decisions until their
implications are clear is usually advantageous.

Simply asking for quotes and having a few talks with the
sales & consulting staff is just not enough to establish trust or clarify
all the open questions. Working together with the team and evaluating real
results would be much more effective.

To select a vendor, after creating an agile RFP as described above:

Prequalify potential vendors

Identify potential vendors and send them the RFP.

Ask the vendors questions, in particular about their experience with agile
methods, which will allow you to quickly eliminate uninteresting vendors.

Invite the survivors to bid on the project

Select the final short list of 2 vendors who will be invited to start work
on the project.

Both vendors work on the project in parallel until a clear favorite is
established. Both vendors get paid and both are expected to produce
increments of working software according to the rules of Scrum.

After a few sprints, select one vendor to finish the project.

By deferring the final decision until you have actual
experience with the candidates, you reduce the likelihood of picking a candidate
that "looks good on paper" but cannot really deliver software.

How to prequalify vendors

Identify your candidates and send them the RFP. Ask them
questions which will separate the wheat from the chaff, for example:

Please present the team which will carry out the project. How much
experience do they have with Scrum and XP (Extreme Programming)?

Are you willing to organize the project according to Scrum (as described
in the Process chapter)? What experience do you have with agile projects on
this scale?

Please estimate the user stories in story points. What is your estimate of
overall complexity (size) of the system in Story Points?

What is your expected team velocity in Story Points per Sprint?

Given our target budget, how much of the functionality do you think can be
realized?

Given the ground rules, are you willing to participate in the competition
to select the final vendor?

If the vendors are used to working on an agile basis, they
will have no problems with these questions. If they are not, they will probably
not even be able to respond, especially if deadlines are kept tight.

You will need to meet with prospective teams for a day or so
to answer their questions about the user stories. Afterwards the vendors should
be able to size the system and answer your questions quickly.

If there are more than two vendors still in the running, you
will need to use the answers and the results of the interviews to trim the field
down to two vendors, who then participate in the competition.

Hedge your bets on productivity differences between teams

Productivity differences between individual developers can be
a factor of 10 to 20. Teams converting to Scrum often report a 3 fold increase
in productivity within 6 months. The best Scrum teams have reported improvements
of a factor of 10 compare to industry averages. The difference might be
technical ability, but human chemistry issues are just as important, if not more so.

Even if the difference between to the top two contenders is
only 25%, investing 10 to 20% of the development budget to hedge your bets
reduces your risk substantially and may pay off dramatically.

Let us assume that you plan to spend $2.4 Million over 12
months, or $200'000 / Month to develop the software. If you add one month's
effort to the budget, that would raise the total by 8%. But given the
productivity differences between teams, even investing an additional 25%
probably yields a positive ROI. Furthermore, the cost of delay while you take
three months to pick a partner without producing any usable software should be
much larger than the cost of redundant development for a short period.

How to run the contest

Here are the ground rules. There are two vendors. Both are
going to start your project according to the process you defined in the RFP and
continue for a defined trial period (probably one to three months, but not more
than 25% of the total project duration). After each sprint, both players present
an increment of working functionality and you will decide which partner you want
to continue working with. The winner gets full compensation for the initial
sprints and a contract for the rest of the project. The loser also gets paid
(perhaps only 50 or 75%) and does not get a contract.

Here are the steps in the competition.

Agree on other playing rules: who are the team members? May staff
participate who are not invoiced? Is overtime allowed? Who owns the software
and ideas which are produced by the loser? You may need to ensure that
quality does not get sacrificed for quantity as you will be producing code
which one day will go live.

Agree on the definition of done. This probably should include points which
confirm that the partner is capable of test driven development and
continuous integration. The teams should not incur technical debt. The same
definition should apply to both contestants.

Prioritize the backlog and, working with both vendors, create a set of
fine grained stories which should be implemented during the trial period.

Hold the first sprint planning meeting together with both contestants, so
both teams get the same initial briefing. Both teams get a defined period to
implement their increments of functionality.

If the trial period is longer than one sprint, then continue with the
Scrum process until the trial period is over. Each team should work
independently.

Finish the / each Sprint with a Sprint Review and Retrospective. This
should only include the implementation team and the product owner, as it is
part of the Scrum process. When the last sprint retrospective is complete,
the competition is complete.

If the teams want to give a sales demonstration, this should
happen after the competitive sprints are finished.

At the end of the trial period, you have not just qualitative
and quantitative data on which to base a decision. You have experience working
with your new partner. You have some working software (or not). In short, you
have much more information to base your decision on.

This approach does present some challenges. Managing one team
can be very demanding for the product owner. Managing two teams might be
difficult. This will be even more dramatic if the teams do more than one sprint,
as the sprint and product backlogs will surely diverge quickly.

Who wins?

One the one hand, everybody wins.

The risks and up-front costs of producing and responding to
the Agile RFP are much lower than the traditional approach. By working with the
teams, you build confidence that you can work with them and that they can build
the software you need. You can judge the teams based on working software rather
than attractive presentations or other artifacts.

By starting to work on a solution early, you reduce your time
to market. You will probably get a better solution, because the competition will
spur everyone's creative juices. Even the loser will have some good ideas which
will improve the final result.

On the other hand, you actually have to choose a vendor. On
what criteria should you decide? Just picking the one who develops the most
functionality is risky. It encourages the developers to incur technical debt to
show more exciting functionality. Economic criteria will still play a role.
Whatever you choose, they should ensure that you pick a vendor with whom you
want to work and in whom you have confidence that they can deliver the solution
you required.

Retrospective

Scrum worked well to manage the creation of the RFP. We
defined the scope and met the essential objectives of the project -- even if
some of the stories did not get implemented. The RFP served its purpose by
focusing on the goals of its users. A definition of done was very helpful to
focus on creating the document on a chapter by chapter basis. However the
product owner did not really do incremental "acceptance testing", so
there were a lot a changes and the end of the project.

The customer decided to keep the implementation in house,
rather than contract it out, so the competitive sprint approach was not tested.

Three people worked on the RFP at approximately 50%.
Approximately 4 1/2 Person-Months were invested in the RFP. Was this too much?
This is the big unanswered question. In an agile process, the objective is to
create the specification just in time through direct communication between team
and product owner. This is not possible before a supplier has been selected.

We did learn a lot about the desired application and shifted
the center of gravity of the project in ways we would not have expected had we
not done the user centered design. By writing the stories ourselves, we remained
business focused and not technology focused. We have a complete product backlog,
but have not thought much about release strategies (time to market) because
without an implementation team, we had no way to estimate the stories.

But should we really have written 20 pages of user stories? I
am not sure. I had the feeling we were over-specifying. The more we produce, the
more we will have to communicate to the implementation team. The more the team
gets told what to do, the less they will think about the problem themselves.

Future work

This approach is a step away from a classical submission
process, but is not yet purely agile. Some thoughts for next steps:

Repeat this process in another context. Our process had no aspirations to
creating a generalized process. So your RFP might and probably look
different than this project’s results.

Use competitive sprints in an actual bidding process

Is it possible to involve potential vendors already in the RFP creation
project? At least theoretically, it would be better to have one or more
people from the implementation team involved from the beginning. Whether
consultants from competing companies would have an incentive to work
together is interesting question.