Software testing, a small misunderstanding

Before we go in to myths lets take a look at what software testing is. Software testing is about assessing quality.  Quality is how well a product meets it’s requirements.  The requirements can be regarding security, usability, load and more, but mostly we focus on functionality requirements, we would not be making software if not for the functionality now would we.

Now that’s clear and we will move on to the six myths/misunderstanding, I know there are way more out there but lets start with these

#1 Software testing increases quality

You might be thinking, well of course it does, why else would we be testing. Like I said before software testing is about measuring and assessing quality not increasing it. Let me explain with an example: Let’s say I find a bug. Does that increase quality? No, but it does tell us something about the quality. If somebody would go ahead and fix that bug, that would increase the quality. Like I once read “Testing to increase quality is like standing on a scale to lose weight”.

#2 Testers are the quality police

Some people seem to think that software testers are some kind of quality police, and some testers seem to think they should be in that role. To explain, being the quality police is being the one that stands in front of the production environment and says YOU SHALL NOT PASS, without me giving my approval. Now this might not be an odd role to accept if you work in a department named Quality Assurance, another factor is that testing is often placed last in the software development cycle. Software testers often feel like they are the users last defence against those pesky bugs. But by putting your self in that role you accept responsibility that is not yours. Testers don’t own quality. I admit I’ve been a quality police officer, when ever something went wrong the finger would be pointed my way and I would be asked “didn’t you test that?”. What happened then was that I became more and more reluctant to release and wanted to test more than was necessary. But as we live, we learn, and I learned to stop being that quality police officer and became a quality consultant. When somebody asks can we release? I will always respond yes, but… I will explain what I have tested, what I have found and what I have not found. Then the people involved can make an informed decision. What happened was the ones that had been pushing for release were now better informed and stopped pushing because they were aware of the quality (or lack there of). In the end we had a better quality product because everybody was working for the same goal. So do not be that quality police, be the quality consultant.

#3 Software testing is both expensive and time consuming

I’m going to give you the key to cheep time efficient testing, are you reedy for this? You might recognize the graph below, if not it shows the cost of fixing bugs depending on when they are found.
Cost of fixing a bug
Cost of fixing a bug

As you can see it’s cheapest to fix bugs while still in analysis, this makes sense because we are just dealing with words on paper and mock-ups. It’s most expensive to fix bugs that are found in the wild and this is why: The customer will have to call in and report it, somebody has to take that call, somebody has to reproduce it and that may take more communication with the user with questions such as “What browser were you using”. When that is done some developer has to fix it, he might find it’s coming from a underlying service and sends the bug to the team that owns that service, might even be a third party. Then they take a look and find it’s the code John worked on 2 months back but he is no longer with the company so some other programmer must go through it and figure out how it works. They then finally release the bug fix. You can see that with salary costs alone it is getting quite expensive, not to mention the loss in sales while your product was down, or time taken from new development.

What is interesting about this graph above is that if it is cheapest to find bugs at the beginning, why aren’t we testing there? It’s simple testing requirements and mock-ups. Well there is the problem, testing in not a phase, it’s many actions over the software development cycle. So do not place testing as a phase at the end of development cycle, test early, test often! By doing this you also even out the workload for the software tester that often finds little to do in the beginning of a project, but come close to the due date and he will be working overtime. By testing early and through out the software development cycle you will even the work load and therefore it will take less time then if you do it all at the end. Mind that to do this you have to develop the software in small testable chunks so get your tester involved and ask if the story/requirement you are working is testable, sometimes it might just take 2 extra lines of code to make it’s easily testable. Sometimes you might need to finish another story before this one to make it testable.

If you manage the work correctly, you can test early, find the bugs sooner which makes them cheeped to fix. So you see testing doesn’t have to take a lot of time and also it could save you money in the end.


“Testing can only prove the presence of bugs not their absence.”
— Edsger W. Dijkstra
Need I say more? This slight misunderstanding that if something is tested it is bug free has been around for some time and is not going away, this is why I couldn’t skip mentioning it. The fact is you can never test every thing. For example test input 0 to infinity is just not possible. The things that could go wrong are infinite so you can’t test them all. That’s why we focus our testing on where the bugs are likely to be and that’s what software testers are good at.

#5 It’s bad to find a bug

Bugs… To find a bug is seen as something very negative. Bugs are bad, we don’t want bugs in our software, but we do want to find them, right? Of course we want to find them and fix them but the saying don’t kill the messenger seems to come to mind when bringing this up.

Lets say we are developing some kind of software that is going to be released next week and the tester walks in saying: “Hey I found this bug here, seems I can just go and steal all the users personal information.” The response will often be something like this: “Ohh… wow… so how long will it take to fix? Ohh a month… ahh… but we were going to release next week… why did you have to find that bug?” Now if you think about it the response should have been “WOW thank you for finding that bug, that could have cost us the company”.

Here is a tip! If you want to throw your tester off, thank them for doing the testing and finding the bugs, they will be so surprised they won’t know what to say because they haven’t experienced it before. At least I was thrown off my game when a programmer really thanked me for finding all the bugs in his code that he couldn’t spot him self. Actually that moment is why I felt I needed to talk about this misunderstanding. Finding bugs is in fact a good thing, at least when you find them and not your costumer.

#6 Automatic testing will replace testers

I will tell you right now that test automation will not replace testers anytime soon simply because automatic testing is not testing, like James Bach explains it’s automatic checking. When the tester thought up the test, that was testing, every repetition of that is in fact just checking. James Back gave some good explanations on why it’s not testing. Let’s say you have a test that says “Hello website are you running?” and the website would reply “Yes I am, thanks for asking”. This is useful for checking if the website is running. But if a tester might go test if it was working they might notice that the site has no styles. The thing is automatic checking (that’s what we are calling it from now on) will only do what it is programmed to do, we could add another check for the styles but that would just be repeating what the tester did so no longer testing. The reason we should rename it to automatic checking is because it has nothing to do with testing. You don’t here a programmer say “Hey I’m going to ask my automatic programmer to take my source code and make a program” (another one of James Bach explanations) no they call it compiling because it has nothing to do with programming. So from now on call it automatic checking.

In conclusion…

Software testing does not increase quality on its own, testing measures quality.
Don’t be part of the quality police, be a quality consultant.
Software testing does neither have to be expensive or time consuming. Test early, test often.
You cannot guaranty 100% bug free software because you simply can’t test every scenario.
It’s good to find bugs, remember to thank your tester.
Automated “checking” will not replace the software tester because it’s not testing, it’s checking.