Acceptance Criteria

Did We Build the Right Product? And, Did We Build the Product Right?

Acceptance criteria are important. Unfortunately, we often overlook or undervalue it as an aspect of the iterative planning process. It is super important because projects succeed or fail based on the ability of the team to meet their customers documented and perceived acceptance criteria. When we clearly define the criteria up front, we avoid surprises at the end of a sprint, or release, and ensure a higher level of customer satisfaction. In other words we’re able to answer these two important questions: Did we build the right product? And, Did we build the product right?

What are Acceptance Criteria?

Acceptance Criteria are the conditions that a software product must satisfy to be accepted by a user, customer, or in the case of system level functionality, the consuming system.

Acceptance Criteria are a set of statements, each with a clear pass/fail result, that specify both functional and non-functional requirements, and are applicable at the Epic, Feature, and Story Level. Acceptance criteria constitute our “Definition of Done”, and by done I mean well done.

We’re not talking about horseshoes here, and there is no partial acceptance: either the acceptance criteria is met or it is not.

When to define our Acceptance Criteria?

A trap that I encourage my teams to avoid is writing acceptance criteria after development has started. This leads to merely verifying that the functionality built works rather than verifying that the functionality meets user needs and expectations. If we write and review the criteria before implementation begins, we’re more likely to capture the customer intent rather than the development reality.

What makes good Acceptance Criteria?

Acceptance criteria define when a work item is complete and working as expected. Express criteria clearly, in simple language the customer would use, without ambiguity regarding the expected outcome. This sets our testers up for success, since they will be taking our criteria and translating them into automated test cases to run as part of our continuous integration build.

What. Not how.

Another trap that I coach my teams to avoid is the how trap. Criteria should state intent, but not a solution. (e.g., “User can approve or reject an invoice” rather than “User can click a checkbox to approve an invoice”). The criteria should be independent of the implementation, and discuss WHAT to expect, and not HOW to implement the functionality.

Formats

The Given/When/Then format is helpful way to specify criteria:

Given some precondition When I do some action Then I expect some result

When writing acceptance criteria in this format, it provides a consistent structure. Additionally, it helps testers determine when to begin and end testing for that specific work item.

Sometimes it’s difficult to construct criteria using the given, when, then, format. Particularly when dealing with system level user stories. In those cases, I’ve found that using a verification checklist works well.

Another advantage to verification checklists is that they are also simple to individually mark as complete as we implement functionality.

Leave a comment

14 comments on “Acceptance Criteria”

I see challenges in helping non-software development teams (business teams) to understand the meaning of acceptance criteria. Checklists are useful to illustrate the end result. I sometimes recommend that teams try backing into a user story. So begin with listing the items that are desired first. It can help business teams new to agile understand the purpose of the user story and what the acceptance criteria will look like.

I also think acceptance criteria can be written to early. I’ve had clients try to write all acceptance criteria at the beginning of the project when user stories are first capture. I advise them to have acceptance criteria ready to review at the iteration planning meeting for the iteration the stories are going to be developed

>>> “A user can approve or reject an invoice” rather than “A user can click a checkbox to approve an invoice” <<<

Isn't a checkbox user-facing and therefore something a user may expect?

When I think of "HOW the functionality will be implemented", I think of things which are not user-facing such as "The approved or rejected invoice will be stored in a Postgres table".

That said, deciding on a checkbox is probably too small a detail to include in acceptance criteria unless user testing has already shown this is what users expect. Otherwise, I think small details like this can be left out of acceptance criteria and, after the feature is implemented, the users can give feedback such as "I really wish this were a simple checkbox" then you can change it in the next sprint.

I think the point is that when outlining a feature, it’s not typically necessary to describe the implementation details, but rather the set of criteria that need to be met in order to deliver business value. The same business value of approving or rejecting an invoice can be achieved with a checkbox, or with a dropdown selection, or even with a swipe gesture. Regardless of which method is implemented, the user should still be able to achieve their goal of approving or rejecting an invoice.

Quite often, the implementation details may be implied as part of maintaining a consistent UX across the application, in which case the development team will need to adhere to such details when implementing ANY story. But unless a specific implementation is critical to delivering the business value (i.e. the same value for the end user couldn’t be achieved with a different implementation for the same functionality), it’s often not necessary to mention the “how”.

I agree — in my experience, acceptance criteria are interchangeable with test cases and tend to need to be very specific, down to the user INTERACTION — which does need to list things like “this should be a dropbox”. In an ideal world, I would be able to provide guidance to design that says “user needs to be able to SELECT from a list of options”, and that may or may not translate to a dropdown. In practice, design, engineering, and test almost always demand a lot more detail. Also for analytics, for example, I’ve been asked to specify what event should fire — certainly not user facing but very important detail for getting analytics requirements right (and testable).

I have been referred to this post as providing “the agile view of acceptance criteria” by a colleague whose team is (mis)using acceptance criteria to provide another level of detail of requirements. It seems to me that the post and the discussion is missing a couple of points.
1. Acceptance criteria are generally more specific than requirements, but they are not another level of detail. That might appear contradictory, but what it means is that criteria should refer to an instance (or instances) of the requirement, whereas the requirement itself is more general. Consider a tax preparation program. Probably the most important requirement is that it correctly calculates tax due given incomes and outgoings of various sorts. Clearly, you are not going to be able to test all possible combinations, so your acceptance criteria will specify particular values, or how to generate a valid set of random values. More specific, but not another level of detail.
2. Sometimes the acceptance criteria ARE simply a restatement of the requirements. The example of approving or rejecting an invoice is one such. The requirement might be “as a finance officer I want to be able to accept an invoice”, the acceptance criteria might be “when I perform the accept action, the invoice is accepted (check by examining the record for the invoice)”. No more detail really.
3. On the point of whether the tester needs to know whether to click a checkbox, that is part of the definition of the test case, not of the criterion for acceptance.