Building The All-Star Lineup of Quality

You do not have to go further than to a random company’s recruitment web page to find out that IT industry of today wants superheroes. A job description states that you have to know your ITIL, COBIT, Six Sigma and whatnot through and through, be able to manage projects, have a truckload of certifications, produce clever code and yes, test too. Latest trends idolize cross-functional teams and work models where anyone or anything can do anyone’s job, preferably within an instant. The fear of losing money drives companies to eliminate anything that even remotely resembles unproductive activity, or personnel for that matter.

But is this combination done too eagerly without contemplating the risks of fitting too much roles and responsibilities inside one set of brains? Could it have dramatic effects on quality? I think so, and after reading this paper hopefully you do too.


The importance of roles or, better yet, positions in a team


Ice-hockey is a big thing in Finland. Long winters, our brawny gene pool and recent success on international arenas are just a number of reasons for this. We have all-star material like Jari Kurri and Teemu Selänne, and especially Finnish goalies are often said to embody the Finnish ice hockey excellence. But for many goalie is the second class role in a hockey team; You become a goalie if you cannot be anything else and all that. In majority of team sports goalie has been the last role to be considered in the lineup.

Positions in Ice HockeyHowever underestimated the goalie role may be, it is crucial for the success of a team. In ice hockey the goalie has the unique task of preventing the puck from going into the goal. Ok, anyone could do that, but in the hands of anyone else this task would get lower priority and the team standards would fall. Because of the role the goalie takes responsibility of the tasks that belong to it and focuses on the means to do them well.

So does the tester. Testing is a task that should belong to anyone involved in software development. With that I agree. But only because of the role or, better yet, position of a tester and the responsibility that comes with it, the means to do testing improve. Of course when doing for example coding, these means improve as a side product, but only from a certain viewpoint. And that viewpoint might not be testing at all.

Let me elaborate.


The risks of eager combination of testing and development


Many companies whose main business is developing IT systems and software try to bring testing closer to development, even as a part of it. For example Agile software development is supported with Lean methodology, which is a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination1. Many of the front-runners of this movement state2 that testing should not be separated from the development and that it should not be done by separate testers. In the most radical examples automation has completely replaced sapient testing…

If testing is done only in development teams and even only by developers, there is some apparent risks:

Firstly engineers often have difficulties to put will into reality. Visionary IT author James Martin stated in his book An Information Systems Manifesto3 that 56% of the problems in software development can be traced back to requirements. Ok, the book was released in 1984 and precise percentages always grow suspicions considering how vast and complex area software development is. The requirements are however not synonym for requirement documentation. Requirements can be anything, ranging from clearly stated explicit ones to highly challenging implicit ones, they can be in all kinds of forms, come from different directions and stakeholders, and ultimately they can be something that just cannot be put to words. Mr. Martin wasn’t that wrong after all?

Lost in ImplementationsSecondly engineers might lose themselves in implementation. When people are too closely involved in development process they can become blind of the problems and risks that may persist in the end result or in the process. They might develop an emotional bond to the product or the way of doing, and eventually inability to say: “Our baby is ugly.” The absence of defocus is also a risk in this situation; The objects under observation are often too small and/or removed from their context that it’s impossible to map the impact they have in big picture. This often leads to indifference on many quality viewpoints such as usability, performance, testability, etc.

Thirdly comes the most fascinating risk of them all; The difference between algorithmic and heuristic thinking. Software development is algorithmic activity by default; It tries to come up with a defined automatic solution to a defined problem. Testing is however heuristic activity; When doing testing the problems and the means to solve them are rarely defined. In testing you apply heuristics, arbitrary choices, educated guesses, rules of thumbs and a number of questions to get answers from an area that is often endless and constantly changing. Testing is questioning. As development is interested in producing successful end result, testing is interested in digging out all the risks that prevent this.

There is definitely more, but hopefully your start to get the picture. From the position of testing someone focuses on these risks and takes responsibility on removing them.


Heuristics – more than just computer science?


Imagine you have a requirement that states “This electronic equipment has to work within the nominal range of 100-250 VAC.” There are different ways to approach this challenge:

  1. “Verify that this electronic equipment works within the nominal range of 100-250 VAC.” This is very elementary verification activity and no actual expertise is required.
  2. “Verify that this electronic equipment works with the nominal values 100, 101, 249 and 250 VAC, but not with nominal values 99 or 251 VAC. Any deviation from this à bug report.” Still very much verification activity, but with some “testing” education behind it. “The tester” shows competence in equivalence partitioning, boundary value analysis and defect management, and has formed a basis for testing, possibly for automation too. I have presented this challenge for tens, perhaps hundreds of IT professionals, even people working in testing positions, and this is the level where most of these professionals stay.
  3. “Test that this electronic equipment works with the nominal values 110 and 220 VAC. Test also that the nominal values <100 and >250 VAC do not cause problems.” Notify that the sentence starts with the word “test”. It is acceptable because there is thinking that might be considered as heuristic; “The tester” knows that 110V and 220V are the most common AC voltages and has created an association from this. He feels also the responsibility to find out that the voltages outside the operating scale do not cause problems, even life-threatening situations.

These three approaches are the most common ones within the testing community that has not fully embraced the heuristic approach. Earlier in this paper I mentioned that testing is questioning. Heuristic tester might have something like this in his/her sleeve:

  • In what environment does this equipment has to operate?
  • To what market is this equipment released (domestic/international)? What kind of currents are there?
  • To what kind of usage it is exposed?
  • Is there specifics for ”has to work”, can it e.g. work partly?
  • Is hazardous operation allowed, and if so, in what terms?
  • What does ”nominal” mean?
  • What does ”VAC” mean?
  • How about DC?
  • Amps?
  • By whom is it used?
  • How long does it have to operate, and in what kind of stress?
  • Are there competing products?
  • Are other systems dependent on it?
  • Is it possible to maintain this equipment?
  • Is there an operation and maintenance guide?
  • Do we have claims that have to be fulfilled?
  • Does it bring added value to this company?
  • Do we have anyone who knows about electronics?

A tester in the makingTesting is considered by many to get its means only from computers science, but that is a very limited viewpoint. As you look at the array of questions above, you might notice that there is so much more to it. For example, when pondering about what kind of market the product is released into, the tester is pulled into the world of anthropology. It is completely different to release something to European market than it is to Middle-Eastern market. A totally different set of rules about culture, religion, history, local ways, etc. That grows a totally different set of questions, and still in a very limited area. “By whom is it used?” ties the product into an end user, who is a thinking and feeling entity. So psychology and, in some extent, philosophy come in play, along with a different set of questions. That’s how heuristics work.

I have often ranted that the more you know about testing the more you understand that it is infinite. It might be daunting for many, but also a welcome challenge for those brave enough to play the position of a tester. There is an endless amount of possibilities, but also an endless amount of tools provided by different fields of science and people mastering them.


Testing vs. verification


At this point someone might think what kind of activity is executing predefined test cases, building automation, etc. That is verification (or checking), an algorithmic activity closely related to testing. Earlier in this paper I mentioned that the means to do testing improve as a side product of coding for example. When looked from the algorithmic vs. heuristic viewpoint, it is not testing that is improved but verification. Software developers are responsible for wonderful things such as TDD, test automation and even test case driven testing, but they are still defined solutions for defined problems, and are in that sense verification.

Verification is no less important than testing, but it is indeed different. Often people working in verification have a strong background in development and via that expertise come up with brilliant means to find that defined solution for that defined problem. Often even testers start to play the position of verification if they find a focus area that suits them. However, if they want to stay in testing too, they need to master the skill of focusing and defocusing. When focusing, a tester drills into one subject and uses a limited set of means to test and verify that specific area in certain context. It is like zooming into a map. And when defocusing a tester starts to wonder about the big picture, determine next focus area, select the correct ones from the vast array of questions and even question the very ways of doing things. “What are we doing here and why we are doing it?” should be asked quite often. Focusing and defocusing can also happen with an instant. For instance in quick iterations and continuous integrations the time between new software versions can be as small as five minutes. That creates a certain demand for rapid testing and verification activities.


How to build The All-Star Lineup of Quality?


Ok, we have done some groundwork for defining the individual qualities of development, testing and verification. It is time for these positions to start playing together. Following process takes place:

  1. Testing starts to ask questions. There can be written requirements, implied needs, use cases, mysterious specs, whatnot, but testing understands that there is always more. Via these questions everyone understands what should be done, when, how well, by whom, to whom, etc. This questioning is constant and continues throughout the project.
  2. Via this understanding development does its work whether it is coding, producing documentation, maintaining hardware, etc.
  3. Testing tests as the development progresses. Testing can target not only code, modules and complete programs/systems, but requirements, documentation, processes, etc. As mentioned before, this is an infinite task. So instead of trying to create an illusion of control and test coverage via test planning and test cases, testing expands the coverage and deepens it as test effort progresses. Instead of reading a map, testing draws a map. Experts in exploratory testing have numerous means to manage this.
  4. Together with testing, verification contemplates which heuristics should be implemented as test cases and preferably automated, creating a test set of algorithms. For example bug fixes are often verified already in code level, sometimes even automated on business process level. Often this approach removes the need for conventional defect management i.e. bug reports are not written, but instead a code is written that prevents this particular bug from happening again. It is understood by everyone that this gives test coverage to only certain areas, in some level of depth and in certain contexts. This set is of course constantly updated and maintained.
  5. Development, testing and verification live this routine until a quality product comes out.

PleasePositions in Quality note that I was not talking about separate entities doing the work, but positions. The goalie has a unique task of preventing the puck from going into the goal. Via this role or, better yet, position the goalie prioritizes this to be the most important task, takes responsibility in becoming very good at this and by this activity improves the standards of the whole team. But even goalie can score a goal. It is a rare happening, but possible nonetheless. It is a team after all. So even though development, testing and verification are positions, they do overlap. Testing can and should use automation, development should test and verify its own code, verification should help in this, etc. And it does not stop to IT department; Business people can join in, even end users, customers and so on. Even CEO might be interested what happens under the hood. So let him/her test!

It is a team after all.

I have now scratched only the surface of this special team building approach. There is so much more to it, things to learn, people to meet. However understanding about the differences and individual strengths of development, testing and verification, the risks of combining them too eagerly and the gradient between algorithmic and heuristic thinking is a very good start when building The All-Star Lineup of Quality.

By Sami Söderblom


  1. Wikipedia (2012). Lean Manufacturing. Referred 29.12.2012. http://en.wikipedia.org/wiki/Lean_manufacturing
  2. Whittaker, James A (2012). Introduction to Google Software Testing. Referred 29.12.2012. http://www.informit.com/articles/article.aspx?p=1854713
  3. Martin, James. An Information Systems Manifesto (Prentice Hall, 1984). ISBN-13 978-0134647692.

Share This:

Leave a Reply