What Is Your Test Quality?

You have consistently written unit tests and you have a line coverage of, let us say, 80% and all of your tests pass. Pretty good, isn’t it? But then you change your code and still all of your tests pass although you have changed code which is covered by your unit tests. In this post, we will take a look at mutation testing which will test the quality of your unit tests.

Introduction

We value our code quality very much. We execute static code analysis, we write unit tests, integration tests, etc. We can set minimum threshold values for certain metrics: we do not allow critical, major analysis issues, all tests must pass, etc. These are all good and valuable items to check in the pipeline to ensure a certain quality level. But what is the value of the metric itself? 

Strengthening Testing through Mutation: A DevOps Engineer’s Experience

I work as a DevOps engineer for a large public-facing application, which has around 90+ microservices (Java-based). Consistently, we are hit by scenarios that we’d discover in the field, which were not caught in any of our testing. Despite improving our test strategies and code coverage assessments, we were unable to assess the "strength" of our test cases.

We looked around for options and solutions that could help us to be more sure about our test cases and the code we develop. As a DevOps engineer, my responsibility was to identify options and trial them to fix this problem and include it as part of our automated build process.

Mutation Testing: Covering Your Code With the Right Test Cases (Part 1)

Mutation Testing Basics

The concept of mutation testing is to modify code in a small way and verify that tests detect that modification. Undetected modification gives a hint to what test is likely missing.

Mutation Procedure Next Steps:

  1. Modify code in small way (creating mutant)
  2. Execute existing test suites (killing mutant)
  3. Verify that at least one existent test failed (mutant killed)
  4. If all test succeded (mutant survived), there is likely a missing test case

But why do we have to consider testing the result of the modified code? Let's take a look at an example: