Breaking the Test Case Addiction
A Reflection on Expected Behaviour and Misused Test Cases
The Falsehood of “Expected Behaviour”
Following on from my thoughts on “breaking the test case addiction”. There’s an interesting falsehood that what is written in a test case is somehow THE expected behaviour of a system. This comes down to one of the main reasons I believe businesses use and abuse test cases.
There appears to be a pervasive idea that once a test case is written down or documented, it can then be run and verified by another tester and therefore that system is deemed as "behaving correctly”. To put this in another way, the first person who encounters a feature does all of the thinking and the next person doesn’t have to, having the luxury of running tests looking for simple expected values.
This is seriously problematic.
Assumptions and Bias in Test Cases
Firstly, the test case that is written is baked in assumption and bias that we can’t possibly understand or evaluate. For example, we don’t know what questions were asked about the system under test? We don’t know what conversations had happened to get the information for this test case? We don’t know if any conversations happened at all. We don’t know how well the author understands the functionality, acceptance criteria, tooling used, environment configuration, the list goes on. We’re hoping the information in the test case is accurate, but we don’t know.
The Illusion of Confidence
Test cases usually have 2 outcomes; pass or fail. In the case that you ran a test case and it “passed” you may think that your confidence level has increased. Likewise if the test case fails, your confidence would decrease. I’d argue that regardless of the outcome, your confidence shouldn’t be based on someone else’s test idea. It can be used as a baseline, but ultimately, testing should be about the tester using their experience to evaluate software, not blindly following scripts.
Blind Following: A Childhood Analogy
When I was a child, if I ever got in trouble, I’d blame my brother or my cousin as we all did. I used to say that one of them told me to do something, which is why I did it. My mom used to then say to me, “If they told you to jump off a bridge, would you do that as well?”. Following a script blindly is jumping off a bridge and hoping there’s water, not train tracks below. If this sounds familiar and if you find yourself running tests blindly, you need to ask yourself what information you are trying to gain.
A Call for Critical Evaluation
I’m not saying test cases shouldn’t or can’t be run by another tester. I’m just saying “be careful” about what information you’re getting from them. Are you merely absorbing bias looking for someone else’s idea of what the system does or are you performing a critical evaluation of a system for information? If it's the latter, ask yourself a second question. Could I perform a better evaluation without running the same test cases? Is the act of running test cases the most beneficial way to gain confidence in a system or an area of a system under test?