Archive for category Software Testing

Old Favourite: Taking Repetition To Task

This originally appeared on my old blog on 16th March 2010…

Others have talked about the virtues of stories as vertical slices of a problem (end-to-end capabilities) rather than horizontal slices (system layers or components). So, if we slice the problem with user stories, how do we slice the user-stories themselves?

If, as I sometimes say, acceptance tests (a.k.a. examples/scenarios/acceptance-criteria) are the knife with which we slice a story into even thinner vertical slices, then I would say my observation of ‘tasks’ is that they are used as the knife used to cut a story into horizontal slices. This feels wrong…

Sometimes I also wonder, hasn’t anyone else noticed that the idea of counting the effort of completed tasks on burn-down/up charts is counter to the value that we measure progress only with working software? Surely it makes more sense to measure progress with passing tests (or “checks” – whichever you prefer).

These are two of the reasons I’ve never felt very comfortable with tasks, because:

  • they’re often applied in such a way that the story is sliced horizontally
  • they encourage measuring progress in a less meaningful way than working software

Tasks are, however, very useful for teams at first. Just like anything else we learn how to do, learning how to do it on paper can often help us then discard the paper and do the workings in our heads. However, what I’ve noticed is that most teams I’ve worked with continue to write and estimate tasks long after the practice is useful or relevant to them.

For example, there comes a time for many teams where tasks become repetitive. “Add x to the Model”, “Change View”… and so on. Is this adding value to the process or are you just doing it because the process says you should do it?

Simply finding that your tasks are repetitive doesn’t mean the team is ready to stop using them. There is another important ingredient, meaningful acceptance criteria (scenarios / acceptance-tests / examples).

I often see stories with acceptance criteria such as:

  • Must have a link to save the profile
  • Must have a drop down to select business sector
  • Business sector must be mandatory

Although these are “acceptance criteria” they aren’t what we mean by acceptance criteria in the context of user stories. Firstly, they are talking about how the user interacts rather than what they need to achieve (I’ve talked about this before). Secondly, they aren’t examples. What we want are the variations that alter the behaviour or response of the product:

  • Should create a new profile
  • Profile cannot be saved with blank “business sector”

As our product fulfils each of these criteria, we are making progress. Jason Gorman illustrates one way of approaching this.

So, if you are using tasks, consider an alternative approach. First, look at your acceptance criteria, make sure they are more like examples and less like instructions. Once that’s achieved, consider slicing each criterion (or scenario) horizontally with the tasks rather than the story. Pretty soon, you’ll find that you don’t need tasks anymore and you can simply measure progress in terms of the new capabilities you add to your product.


, , , , , , , ,

1 Comment

Old Favourite: QA / Testing – what’s the difference?

Software is about the only industry one of the few industries that lumps testing and QA under one banner. It’s one of those things where common misuse of a term results in the community changing it’s meaning… this happens in mainstream language all the time.

Testing something is actually more analogous to quality control or, QC , [although it isn't quality control].

QA is more concerned with the process – collecting information about the performance of the process in order to determine if we are ‘assuring’ (or more realistically increasing the probability of) quality. Statistical information about problems found in the product (during quality control) is just one of many pieces of info useful to someone concerned with QA (which really should be the whole project team)

In short:

QC helps us answer the question ‘does our product work?’

QA helps us answer the question ‘does our process work?’

Unfortunately, in the software industry, all too many teams don’t realise their process doesn’t work until the testers find all the ways in which the product doesn’t work… maybe that’s why software testing has come to be known as QA.

This article originally appeared on my old blog in July 2008. I had already elaborated on this topic in the article “What’s in a word” in Better Software Magazine in March 2008.

, , , ,

2 Comments

You’re almost cuking it…

In “You’re Cuking it Wrong”, Jonas Nicklas, shows several examples of bad scenarios (or acceptance tests whichever term you prefer) and demonstrates better approaches. This is an excellent post on common mistakes made when writing example scenarios with Cucumber.

I think, however, he could have gone further in one case. One of his examples of a bad scenario looks like this:

Scenario: Adding a subpage
Given I am logged in
Given a microsite with a Home page
When I click the Add Subpage button
And I fill in "Gallery" for "Title" within "#document_form_container"
And I press "Ok" within ".ui-dialog-buttonpane"
Then I should see /Gallery/ within "#documents"

(Dude – yep, seen these… I agree… not good). He goes on to suggest it should really look like this:

Scenario: Adding a subpage
Given I am logged in
Given a microsite with a home page
When I press "Add subpage"
And I fill in "Title" with "Gallery"
And I press "Ok"
Then I should see a document called "Gallery"

This is a massive improvement. It keeps the specifics that inform the reader and give them some context (like filling in the “Title” with “Gallery”) and takes them further away from the implementation. Although it gets closer to expressing customer intent, I think it could go further. At the moment, it is describing the ‘what’ and some of the ‘how’. This example makes complete sense if what we are exploring is the design of the UI. I’ve not found these scenarios to be a good place to do that, however.

Instead, in these specifications we want our examples to illustrate the customer intent. The ‘what’ not the ‘how’. There are other places we can capture the ‘how’ – i.e. in the step methods.

Instead, I would write it like this:

Scenario: Adding a subpage
Given a microsite with a home page
Given I am logged in
When I Add a Subpage with a Title of "Gallery"
Then I should see a document called "Gallery"

I’ve removed all the “Tasks” and left only “Activities”. This leaves the user experience completely open. This ensures that when there are UI changes, I only change the code that performs the tasks (clicking, pressing, etc.) and my scenarios evaluate whether the customer intent is still fulfilled – without having to go back and change a lot of files.

Otherwise – great post Jonas :-)

, , ,

7 Comments

Old Favourite: Expected Exceptions

This first appeared on my old blog in November 2008.

I’ve decided that I don’t like typical patterns for testing exceptions. I decided this a while ago as far as “Expected Exception” attributes/annotations are concerned and stuck with the traditional try/catch approach (I’ll explain why in a minute). Now, I’ve decided I don’t like the typical “try/fail or catch” approach and have started using a subtle evolution of it.

First, let me explain why I don’t like Expected Exception attributes/annotations. The final nail in the coffin of this approach was hammered home for me when working with Liz Keogh a while back.

Here is an example in Java of your expected exception pattern (for brevity I won’t include assertions for e.getMessage()):

 

@Test(expected=BeyondMyExpertiseException.class)
public void shouldComplainWhenNotAClass() throws Exception {
	DomainExpert expert = new DomainExpert();
	String thisCheckpoint = nameOfSomeInterface();

	expert.howDoIRestore(thisCheckpoint);
}

 

So, apart from the obvious fact that it is only implicit as to which method threw that exception (because I know that none of the other steps can throw that exception)… and we want our tests to communicate information explicitly, yes? The insight that Liz shared with me is that it changes the flow of information (ok, I’m paraphrasing now) compared to a test that doesn’t expect an exception.

In a ‘positive’ test, the flow of information that is expressed to the reader is What I need->what I do->what I expect. In an expected exception test, this is changed to what I expect->what I need->what I do. The latter just doesn’t flow very well and because it’s different to your positive tests there’s an overhead involved for the reader (me or someone else later on) to process this shift in structure… I’ve found that such tests just don’t jump out at me when I’m scanning the tests…

Since then, despite fashion, I committed to the old-fashioned way of writing exceptions – “try/fail or catch”:

 

@Test
public void shouldComplainWhenNotAClass() throws Exception {
	DomainExpert expert = new DomainExpert();
	String thisCheckpoint = nameOfSomeInterface();

	try {
		expert.howDoIRestore(thisCheckpoint);
		fail("Should have thrown " +
			BeyondMyExpertiseException.class.getSimpleName());
	} catch (BeyondMyExpertiseException e) {
	}
}

 

Ok, I accept, it looks more cluttered by comparison but the flow of information makes more sense and I make it explicit that the expert.howDoIRestore(thisCheckpoint) method call is the one that should have thrown the exception. (Note: The idea here is not to reduce how much you type but to make the test more expressive). The “try/fail or catch approach only works when your code doesn’t throw an exception… If your code throws a different exception, the failure trace just tells you what exception was actually thrown, it doesn’t tell you what exception was expected. So, here is a slightly different way of writing it:

 

@Test
public void shouldComplainWhenNotAClass() throws Exception {
	DomainExpert expert = new DomainExpert();
	String thisCheckpoint = nameOfSomeInterface();
	try {
		expert.howDoIRestore(thisCheckpoint);
		fail();
	} catch (Exception e) {
		assertThat(e,is(instanceOf(
				BeyondMyExpertiseException.class)));
	}
}

 

Notice that I’m only catching Exception now, not BeyondMyExpertiseException. This still feels a little jumbled… Because my assertion is inside the catch block, I have to have the fail() method call just after the call that should throw the exception. Hmmm… don’t like that… Instead, this makes more sense:

 

@Test
public void shouldComplainWhenNotAClass() throws Exception {
	DomainExpert expert = new DomainExpert();
	String thisCheckpoint = nameOfSomeInterface();
	Exception thrownException = null;

	try {
		expert.howDoIRestore(thisCheckpoint);
	} catch (Exception e) {
		thrownException = e;
	}
	assertThat(thrownException,is(instanceOf(
				BeyondMyExpertiseException.class)));
}

 

Giving this failure trace when it fails:

 

java.lang.AssertionError:
Expected: is an instance of
com.testingreflections.atdd.expertise.
    misunderstanding.BeyondMyExpertiseException
        got: < java.lang.UnsupportedOperationException >

 

So, for those who want to type as little as possible, this isn’t for you… But if you want tests that drive out your exception handling to be more expressive, then this is an alternative to the usual “try/fail or catch” approach. I think that perhaps there’s an even better way of doing this… maybe next I’ll see how approximating closures with an anonymous class might help improve the readability of this… Let me know if you know of a better way.

, , , , , , ,

No Comments

Boredom: a Testing Smell?

This first appeared on my old blog in June 2005.

Somebody I know who was doing some (unscripted) testing spoke of being bored the other day… I have always found boredom to be a sign that something is wrong.

I believe, as has been said by Kaner et al, that testing is a brain-engaged activity. If that is the case, why would I ever be bored?

Borrowing the Smell metaphor… I would say that boredom is a Bad Testing Smell. If it isn’t a bad smell, it is a whiff of an underlying bad-smell for sure.

If on the rare occasion I find that I am bored, I’d ask myself:

  1. Am I testing this area more than I need to? (if so, why?)
  2. Am I losing concentration because my brain is tired? Do I need a break?
  3. Is what I am doing so repetitive that perhaps it should be automated?
  4. Is there a better way of testing this feature?

If I answer “yes” to any of these, I know that perhaps I need to do something differently. Whether that is feasible in a given context, might be a different story.

Update 9th July 2010: This applies to scripted testing as much as it does to exploratory testing. Generally, I’ve found that boredom during scripted testing is because we’re asking a human to do something we should be asking a computer to do… The main barrier to this, in my experience, is that it is harder to write the automated tests than to write manual scripted tests… so, I look for ways to make automated tests as easy to write as manual scripted ones. Usually, I can achieve this with a little effort and have found it makes a huge difference.

, , , , ,

No Comments