Agile, BDD/ATDD, Business Analysis, Software Development, Software Testing, Uncategorized

Scenario-Oriented vs. Rules-Oriented Acceptance Criteria

4 Comments

Acceptance Criteria, Scenarios, Acceptance Tests are, in my experience, often a source of confusion.

Such confusion results in questions like the one asked of Rachel Davies recently, i.e. “When to write story tests” (sometimes also known as “Acceptance Tests” or in BDD parlance “Scenarios”).

In her answer, Rachel highlighted that:

“…acceptance criteria and example scenarios are a bit like the chicken and the egg – it’s not always clear which comes first so iterate!”

In that article, Rachel distinguishes between acceptance criteria and example scenarios by reference to Liz Keogh’s blog post on the subject of “Acceptance Criteria vs. Scenarios”:

“where she explains that acceptance criteria are general rules covering system behaviour from which executable examples (Scenarios) can be derived“

Seeing the acceptance criteria as rules and the scenarios as something else, is one way of looking at it. There’s another way too…

Instead of Acceptance Criteria that are rules and Acceptance Tests that are scenarios… I often find it useful to arrive at Acceptance Criteria that are scenarios, of which the Acceptance Tests are just a more detailed expression.

I.e. Scenario-Oriented Acceptance Criteria.

Expressing the Intent of the Story

Many teams I encounter, especially newer ones, place higher importance on the things we label “acceptance criteria” than on the things we label “acceptance tests” or “scenarios”.

By this I mean that, such a team might ultimately determine whether the intent of the story was fulfilled by evaluating the product against these rules-oriented criteria. Worse still, I’ve observed some teams also have:

  • Overheads of checking that these rule-based criteria are fulfilled as well as the scenario-oriented acceptance tests
  • Teams working in silos where BAs take sole responsibility of criteria and testers take sole responsibility for scenarios
  • A desire to have traceability from the acceptance tests back to the acceptance criteria
  • Rules expressed as rules in two places – the criteria and the code
  • Criteria treated as the specification, rather than using specification by example

If we take the approach of not distinguishing between acceptance criteria and acceptance tests (i.e. see the acceptance tests as the acceptance criteria) then we can encourage teams away from these kinds of problems.

I’m not saying it solves the problem – it’s just easier, in my experience, to move the team towards collaboratively arriving at a shared understanding of the intent of the story this way.

To do this, we need to iteratively evolve our acceptance criteria from rules-oriented to scenario-oriented criteria.

Acceptance Criteria: from rules-oriented to scenario-oriented:

Let’s say we had this user story:

As a snapshot photographer
I want some photo albums to be private
So that I have a backup of my personal photos online

And let’s say the product owner first expresses the Acceptance Criteria as rules:

  • Must have a way to set the privacy of photo albums
  • Private albums must be visible to me
  • Private albums must not be visible to others

We might discuss those to identify scenarios (as vertical slices through the above criteria):

  • Create new private album
  • Make public album private
  • Make private album public

Here, many teams would retain both acceptance criteria and scenarios. That’s ok but I’d only do that if we collectively understood that the information from these two will come together in the acceptance tests… And that whatever we agree in those acceptance tests supersede the previously discussed criteria.

This understanding tends to occur in highly collaborative teams. In newer teams, where disciplines (Business Analysts, Testers, Developers) are working more independently this tends to be harder to achieve.

In those situations (and often even in highly collaborative teams) I’d continue the discussion to iterate over the rules-oriented criteria to arrive at scenario-oriented criteria, carrying over any information captured in the rules, discarding the original rules-based criteria as we go. This might leave me with the following scenarios:

  • Create new private album (visible to me, not to others)
  • Make public album private (visible to me, not to others)
  • Make private album public (visible to me, & others)

Which, with a subtle change to the wording, become the acceptance criteria. I.e. the story is complete when we:

  • Can create new private album (visible to me, not to others)
  • Can make public album private (visible to me, not to others)
  • Can make private album public (visible to me, & others)

Once the story is ‘in-play’ (e.g. during a time-box/iteration/sprint) I’d elaborate these one at a time, implementing just enough to get each one passing as I go. By doing this we might arrive at:

Scenario: Can Create new private album
When I create a new private album called "Weekend in Brighton"
Then I should be able to see the album
And others should not be able to see it

Scenario: Can Make public album private
Given there is a public album called "Friday Night"
When I make that album private
Then I should be able to see the album
And others should not be able to see it

Scenario: Can Make private album public
Given there is a private album called "Friday Night"
When I make that album public
Then I should be able to see the album
And others should be able to see it

By being an elaboration of the scenario-oriented acceptance criteria there’s no implied need to also check the implementation against the original ‘rules-oriented’ criteria.

Agreeing that this is how we’ll work, it encourages more collaborative working – at least between the Business Analysts and Testers.

These new scenario-based criteria become the only means of determining whether the intent of the story has been fulfilled, avoiding much of the confusion that can occur when we have rules-oriented acceptance criteria as well as separate acceptance test scenarios.

In short, more often than not, I find these things much easier when the criteria and the scenarios are essentially one and the same.

Why not  give it a try?

  • Elizabeth Keogh

    Hi Antony, this makes sense. I think the only difference is that I’m calling the examples at the bottom, where you actually have specific albums and people, “scenarios”, where as I’d call the short-hand form for the scenarios you identified “acceptance criteria phrased in scenario form”. It isn’t until we get “visible to me, not to others” that the scenarios start to emerge, and probably if we were talking through them we’d find more (eg: what happens if Jane visits a page where Fred used to have a public album?)

    I like to retain the acceptance criteria in some form, but often only as the title for a scenario or a short description of what we’re trying to achieve in the scenario blurb. I’ve found this useful in the past because we often miss scenarios (despite all the conversation, we don’t know what we don’t know). I’m not completely drawn in by Specification by Example as a result, though I do very much like talking about behaviour and having examples of it. Using the desired behaviour (or “acceptance criteria”) as a title is a really lightweight way of getting the traceability, etc., that you mention.

    My post was mostly aimed at the opposite problem – when people were specifying scenarios too early and in too much detail, resulting in glazed eyes and boredom. I reckon it’s probably OK in that context to capture the acceptance criteria and just talk through scenarios until they’re automated, while also capturing any unusual edge-cases that come up. So in that situation I’d have:

        Make a private album public, 
        Make a new private album, 
        Make a public album private – Given Fred has a public album, when Fred makes the album private and Jane goes to look at it, then Jane should see “This album has been made private”.

    I think we both agree that whatever helps people talk, in their context, is fantastic. It’s been a while since I’ve worked on a project where the testers haven’t *wanted* to be more involved, so thank you for reminding me that this happens.

  • Hi Liz,

    Some of the things you’ve said don’t make sense to me so either I’ve not understood you, you’ve not understood me, or a little bit of both.

    Hopefully we can do something about that… What I think I’m saying, in relation to what you’re saying is:

    * Your “acceptance criteria” are what I call “rules-oriented acceptance criteria”.
    * I find that you can also take a “scenario-oriented” approach to acceptance criteria and I favour that over “rules-oriented acceptance criteria”.
    * Your “scenario titles” are also my “scenario titles” but they can be the same as my “scenario-oriented acceptance criteria” but they can’t be the same as the rules-oriented acceptance criteria.
    * Your “scenarios” are also what I call “scenarios” and I am in general agreement as to what goes into them (although our writing styles may be a little different).

    In more detail…

    Hi Antony, this makes sense. I think the only difference is that I’m calling the examples at the bottom, where you actually have specific albums and people, “scenarios”, where as I’d call the short-hand form for the scenarios you identified “acceptance criteria phrased in scenario form”. It isn’t until we get “visible to me, not to others” that the scenarios start to emerge, and probably if we were talking through them we’d find more (eg: what happens if Jane visits a page where Fred used to have a public album?)

    I can see why you might think that calling the “given-when-then” parts scenarios is different to what I was saying since I’ve re-read the post and it’s not clear that I too also call those scenarios… Although I also acknowledged that some people also refer to them acceptance tests, so that’s not different to what I was trying to say.

    How is calling the “scenario-oriented acceptance criteria” (the phrase I used) different to the phrase you’ve used:  “acceptance criteria phrased in scenario form”?

    A better way of getting my point across might be that I call these “scenario-oriented acceptance criteria”:

    * Can create new private album (visible to me, not to others)
    * Can make public album private (visible to me, not to others)
    * Can make private album public (visible to me, & others)

    And I call things like this scenarios:

    Scenario: Can Create new private album 
    When I create a new private album called “Weekend in Brighton” 
    Then I should be able to see the album And others should not be able to see it

    And, by using scenario oriented acceptance criteria, we can use the criteria as the scenario titles and the scenarios become an elaboration of the criteria, rather than mis-aligned derivatives of the ‘rules-oriented acceptance criteria”.

    I like to retain the acceptance criteria in some form, but often only as the title for a scenario or a short description of what we’re trying to achieve in the scenario blurb. I’ve found this useful in the past because we often miss scenarios (despite all the conversation, we don’t know what we don’t know). 

    If a team found it valuable to retain the ‘rules’ as notes or additional reference material, I’m ok with that… but I’d not want them to be regarded as acceptance criteria. 

    In order to demonstrate that the rules-oriented acceptance criteria in my “private albums” illustration have been met you’d need all three scenarios anyway…  All the rules in my illustration apply in each scenario (i.e. the rules are cross-cutting) and are demonstrated by all three scenarios passing… so I’m not sure what value keeping the rules-oriented acceptance criteria would add.
    Can you demonstrate how the acceptance criteria (as you describe them) or “rules-oriented acceptance criteria” as I describe them, can be used as scenario titles?

    I’m not completely drawn in by Specification by Example as a result…

    I am confused as to how you can not be drawn in by “Specification by Example” since you’re such a strong advocate of Behaviour Driven Development… 

    Unless you have a completely different idea of either “Specification by Example” or “Behaviour Driven Development” to me…

    What I understand from “Specification by Example” is that it’s a general term to describe approaches that use examples, rather than rules, as the basis of a specification. Examples can be any form of stimuli with an associated outcome. I also understand it to imply that we infer the implementation both incrementally and iteratively from the examples and test our understanding by being able to execute the examples against the implementation.

    Test-Driven Development and Behaviour Driven Development are forms of “Specification by Example” since we specify things by means of input-output examples (or stimuli-outcome examples). An input-output example may be very specific or more abstract, depending on what aspects of the input or stimuli is relevant to the scenario. From each example we infer the next evolution of the implemented algorithms to support these examples.

    …though I do very much like talking about behaviour and having examples of it. Using the desired behaviour (or “acceptance criteria”) as a title is a really lightweight way of getting the traceability, etc., that you mention.

    I think what I’m trying to say is that by reframing our rules-oriented acceptance criteria as scenario-oriented we negate the traceability concern. I wouldn’t call it lightweight… maybe zero-weight because we demote the rules to no longer exist or into nothing more than notes – so there’s not need for traceability.

    My post was mostly aimed at the opposite problem – when people were specifying scenarios too early and in too much detail, resulting in glazed eyes and boredom.

    That’s not the opposite problem. It’s a related problem solved in a different way. My solution to that problem is expressed in the blog post as it’s ok to discuss the criteria in terms of rules but (and this next part is where our approaches differ) replace those “rules-oriented acceptance criteria” with “scenario-oriented acceptance criteria” at the earliest opportunity. The key-word there being “replace”. In the context of this problem, I’m only talking about going as far as writing the “scenario-oriented acceptance criteria” not the full scenario.

    As for the examples you gave… the scenarios are exactly the same but for some word-smithing. The final elaborated example you give is a good one… and similar to where I might end up if that was what we wanted to happen. In my example, we had decided in the conversation that the person would get a 404 when attempting to view someone else’ private album and that the scenario was adequate as it was.

    I hope that clarifies what I was trying to communicate a little more.

  • Elizabeth Keogh

    I’m a massive fan of BDD, yes. I like to talk about the behaviour we’re expecting, then look particularly for *unusual* scenarios – edge-cases which aren’t obvious.

    In the cases where the scenarios are obvious from the behaviour we’ve discussed, I’m happy to leave it at that, unless someone wants to automate it. An example might be:

    “Replaced or refunded items should be added back into to stock unless faulty.”

    The four scenarios you could associate with this are obvious.

    However, when I ask, “Can you think of an example in which items aren’t returned to stock even though they’re not faulty?” someone might say, “Oh, do you remember that time when that mother came in to get a refund because the buggy was too small, but it had been recalled anyway? Yes, we need to check for recalls.” And now we have another scenario. *That* one, I’d want to write down – it’s not obvious, and it’s easy to forget. But I’ll write down that specific scenario, with a title of “Check for recalls”. And then someone says, “Oh, we can’t even refund earrings for hygiene reasons, unless they’re faulty!” And now we have another. So I’d write these down, but I wouldn’t bother writing down the obvious ones.

    That’s why I can say that I’m into BDD, but not necessarily driven by specification by example – there *are* other ways to specify, and they’re OK! I find the word “specification” stops people questioning quite so freely as well, so I don’t tend to use it.

    I’m very happy for people to have the conversations around as many scenarios as they want, and pretty much insist on it if it’s something new or unusually complex, but I think people are starting to put *too much* emphasis on capturing every single scenario, when some of them are really obvious. In those situations, I’m happy with rules being captured (as above) instead as proxies for the scenarios they generate. The scenarios will appear when and if we automate. In the meantime, the focus can fall on some of the unusual scenarios instead (which are frequently around new, differentiating and therefore highly valuable functionality). So we don’t “replace” the rules, because often we don’t even bother writing them down!

    The big difference for me is that where a scenario *needs* to be captured, I’ll use specific, realistic, actual scenarios with realistic names and data, rather than capturing scenario-oriented acceptance criteria. I find it pings people’s imagination more, and lets us come up with the other five scenarios that we *also* forgot about.

    I have done the kind of thing you’re talking about before, capturing all the scenarios to use as acceptance criteria. I found it useful in low-discipline environments, where the developers sometimes forget to check what’s working and what isn’t. I’ve got away from that recently by teaching Feature Injection instead, so that everyone has a clear idea of the value of what they’re doing and the risks associated with it, and is motivated to understand their work better.

    I’ve also done it when the scenarios have been discussed during planning meetings, and needed to be captured because the people doing the work weren’t always the same as the ones doing the capturing, or enough time passed for them to forget. Persuading people to step away from fine-grained estimates (also using Feature Injection) and accept some discovery in the process of software development means we no longer have to do that either, and can talk through scenarios as and when it’s needed, thereby keeping the knowledge fresh.

    As a result I suspect that I have a slightly looser process – and I don’t think either of our approaches is necessarily wrong. We may just work in different contexts.

    Have a chat with Dan North about his “Ginger Cake” pattern some time, and please invite me for the conversation if I’m around 🙂

  • I can relate to what you describe here (as in the looser way of working). That’s quite consistent with teams I would be a team-member of…  But then I wouldn’t even be using terms like “acceptance criteria” in those situations. It would be little more than a card, the conversation and lots of collaboration – with whatever notes are necessary to aid our collective memory.

    For those teams that do place a high emphasis on capturing up-front acceptance-criteria and can relate to the problems I outlined, then moving in the direction of scenario-oriented acceptance criteria might prove a useful step towards not needing them at all.

    Thanks for your comments Liz.