This blog post was originally a Racket and has been transcribed and rewritten to make sense in blog form.
One of the things that’s been on my mind recently is how testing in an Agile team means setting your own scope. Frequently we’ll have to work out what stuff to test, or what information will be important to the team, and push that up rather than waiting for information about what to do to be handed to us. Going out and finding what to test is very different from the old / traditional ways of testing (where we’d check against requirements handed down to us) and requires a shift in behaviour and mindset.
In the agile projects I’ve recently been working with there’s been a degree of uncertainty in what we’re building or what good looks like. This is because we’ve been working in greenfield development where requirements and acceptance criteria are loosely defined or high level. This has meant that I had to set my own scope of what to test and what good could look so that I could explore to find useful information to the team.
We don’t have a scope but we have ARTIFACTS and code.
So I don’t have a scope, or I’m not being told what to test, what do I do then?
Use the ACs to start: We can start our testing using what we do know, the ACs of the ticket. If there are no ACs then usually we can either guess some or ask the developer what they’d tried to implement. But ACs alone are usually not the whole story as they will only give you the thin layer of information about the quality of what’s been built; instead use this as a jumping off point and build out your testing beyond the happy path and functionality.
Use heuristics: We can use other heuristics (rules of thumb) to guide the scope of our work. Ask yourself “What would I like from this product?” or bring personal experience from other projects to guess what the team might want to know. A heuristic like KISS (Keep It Super Simple) will remind you to not overthink or panic about not having scope, just simply work through what you know about the product or test what you can already see is there.
Talk to people: Do some testing and then play back what you did through a debrief. From talking back to the project team you can refine your future testing based on what information was / is useful to the team or what information results in decisions being made. Talk to project managers or product owners to ask them what the customer needs or is expecting to see from what is under development and then use that to inform your testing.
Get stuck in: Sometimes there just won’t be anything to shape your testing other than the product in front of you. When that happens you’ll have to use your gut instinct and what you already know about the product or project. Don’t wait to be told what to do or for approval, just jump in! Timebox looking at the different features that become apparent and start sharing information about what you’ve found (people will soon tell you if that info is useful or not).
Why setting your own scope is important.
It’s easy to fall into the mindset of checking; that we’ll be told exactly what to test and be able to tick them off one by one (like in old school waterfall projects) because that’s easy. When you’re checking you don’t have to think as much, you can just let someone else’s thinking and product knowledge guide you through what to work on.
But here’s the thing, the people setting the product scope and direction or the ACs aren’t experts in quality, you are. They don’t know what they don’t know about risks, how everything works, how things could fail, non-functional testing or sometimes how integrated systems work. To add value to a project we need to be able to be the ones to help show people that what they didn’t know about might be important.
- Non functional requirements out of scope? Are they really, what about stability of a demo environment under load? What about the saleability of a product being enhanced by its usability?
- Happy paths only? Is that true, what happens if I try something and it crashes the whole system because of poor error handling?
- Not caring about data? What if the data includes embarrassing placeholder text?
These are all things that on the surface the project might not care about, but when we find evidence of a problem are seen as important. That’s because hypothetical risks are easily dismissed whereas an actual issue needs to be dealt with. If we testers can uncover actual issues and put the evidence in front of people it gives people something tangible to manage and can shift their perception of what matters on future projects.
I’ve worked on projects that were adamant that usability and accessibility were not important and were out of scope. But when I used tools to pull out and socialise actual issues related to these areas the project managers saw that extent of this as an actual problem and planned fixes for accessibility immediately.
Yes it’s hard.
Exploration and investigation rather than checking is something that I’ve found myself struggling with as it’s a big mindset switch to make from traditional testing. The reason why it’s harder is because, frankly, it just requires more brain power, creativity and more critical analysis and thinking.
Setting your own scope means that you have to determine what’s important. That’s way more creative than basically reading some documentation for requirements so you can tick them off. It means knowing what the customer might want, knowing what people are working on and holding knowledge about the risks inherent to all of those in your head. You have to know about and think about a much wider array of things: knowing all the types of personas and users that might use the system, knowing about non-functional needs, knowing design and full stack system architectures.
It means having a lot going on in your head. That’s really hard but can add so much value to a team when you can come along pre-demo and say, “Here’s the stuff I’ve looked at based on a scope that I put forward for you and here’s the information that could help you make meaningful decisions”. In this way we take testing away from just being a tick box exercise to something people can engage with and really use.
Working with uncertainty.
Saving time: When you’re waiting for people to tell you what to test it takes time, time that we need for testing. Let’s say there’s a demo on Friday and by Wednesday evening nobody has told you what the scope of the demo is… that means you now only have a day to do any testing and get fixes in! Instead, if we can learn to work with the uncertainty and embrace setting our own scope then we can get started earlier and have more time for testing. We can come in on Monday, say “Okay I know there’s a demo, I’ll start looking at the possible workflows, environment and user behaviours that could happen” and get started. This allows us to start sharing feedback days before the demo and gives people time to react and ourselves time to plan more testing or change our plans based on feedback.
Improving engagement: People find it much easier to say what they like (or don’t like) when you put something in front of them to critique, so reduce your reviewer’s uncertainty by setting your own scope. Put what you plan to test in front of people to comment on, rather than asking them to come up with it all, and you’ll likely get better engagement. Or, as mentioned above, put something real rather than hypothetical in front of people and it’ll likely help them to prioritise making a decision on it.
Owning testing: You’re not going to get in trouble for taking ownership and testing things before you “get told what to do”. By working with uncertainty and pushing forwards with our own scope of testing we’re owning it and showing that we support the team. Plus we’ll find information earlier which shortens loops to feedback (always good).
Callum Akehurst-Ryan's Testing Blog 