Unveiling the Power of Mutation Testing: Insights from Newton's Tech Talk

Although mutation testing has existed since 1971, it fell out of favor due to high costs. However, today, it’s a more efficient technique that deserves wider adoption. Simply put, mutation testing measures the effectiveness of your software testing process by identifying faults and defects. In other words, it’s a way to test your tests and ensure your software functions flawlessly. So let’s dive into the details and explore the benefits of mutation testing.

What is a mutation test?

By definition, a “mutation” is:

A permanent change in an organism or the changed organism itself.
Cambridge Dictionary

So by applying this concept to our code, a mutant is a version of the code in which a small change has been made deterministically, and, of course, we expect the test fails against this.

Let’s see an example:

if(someString == null)

It will be changed to:

if(someString != null)

After this small change (a mutation), we should rerun the tests, and if you wrote all the tests correctly, at least one should fail. By doing this, we kill the mutant we just created. But if all the test cases were successful, the mutant survived.

This is a problem! If someone introduces this change unconsciously, the test cases will not catch that, and we will have a bug in our code!

Advantages of Mutation test

A mutation test is a powerful technique for improving the quality and effectiveness of software testing, and some of the main advantages include the following:

  1. Detection of weak spots in the test suite:
    Mutation testing can identify areas of the code not sufficiently covered by the test suite. By introducing many mutations into the code, mutation testing can reveal which areas of the code are most challenging to test and where additional test cases may be needed.

  2. Validation of the test suite:
    Mutation testing ensures that the test suite effectively catches real faults in the code. By comparing the results of running the test suite against the original and mutated codes, mutation testing can objectively measure the test suite’s quality.

  3. Improvement of code quality:
    Mutation testing can help identify weaknesses in the code itself, such as redundant or poorly written code. By highlighting areas of the code prone to faults, mutation testing can encourage developers to improve the quality of the code they write.

  4. Prevention of regression bugs:
    Mutation testing can help ensure that code changes do not introduce new bugs into the system. By running the test suite against the mutated code, mutation testing can reveal if any changes made to the code have unintentionally caused the test suite to fail.

How to perform mutation testing

We just saw what a mutation test is and its benefits to ensure that our test cases have been written correctly and, by doing so, prevent the introduction of new bugs in our code.

An essential tool has been used to ensure that our unit test has covered all our code (or some percentage of them). So we wrote our test cases and put a gate on the pipeline to guarantee the code rate was covered, but will 100% of coverage prevent some changes breaks the code? Let’s write some code:

public class TimeConversion {
    public String convert(String s) {
        int givenHour = Integer.parseInt(s.substring(0, 2));
        var suffix = s.substring(8);
        int convertedHour;
        if (suffix.equalsIgnoreCase("PM") && givenHour < 12) {
            convertedHour = givenHour + 12;

        } else if (suffix.equals("AM") && givenHour >= 12) {
            convertedHour = givenHour - 12;
        } else {
            convertedHour = givenHour;
        }

        return String.format("%02d", convertedHour) + s.substring(2, 8);
    }
}

So, we need a test case for the previous class:

import org.junit.Assert;
import org.junit.Test;

public class TimeConversionTest {

    @Test
    public void testTimeConversion() {
        TimeConversion timeConversion = new TimeConversion();
        Assert.assertEquals("23:00:00", timeConversion.convert("11:00:00PM"));
        Assert.assertEquals("00:00:00", timeConversion.convert("12:00:00AM"));
    }
}

Ok, this is a simple method with a unit test, and as lucky as we are, the test covers all lines. Look:

Image 1 — TimeConversion line coverage

Our test case fully covers our code, but is this trustworthy? So now we know the benefits of the mutation test. Let’s give it a shot and implements it in our code.

Since I’m using Java here, I will introduce you to how to implement a mutation test using an open-source tool called PITest (https://pitest.org/) (which can also be used for Kotlin). First, we need to add the plugin on our pom.xml:

<plugin>
       <groupId>org.pitest</groupId>
       <artifactId>pitest-maven</artifactId>
       <version>1.11.3</version>
       <configuration>
         <targetClasses>
        	<param>com.neewrobert.mutationtest.TimeConversion</param>
         </targetClasses>
         <targetTests>
           <param>com.neewrobert.mutationtest.TimeConversionTest</param>
         </targetTests>
        </configuration>
   </plugin>

You can check how to add the plugin here if you are a Gradle user.

So we added the plugin and told him the target classes we would like to test and where the test case on our project is. It’s simple as that. We don’t need to write any other piece of code. So let’s run the mutation test coverage and see if there is anything wrong with our code:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

And we can generate an HTML report with:

mvn test-compile org.pitest:pitest-maven:report

So now, If we go to target/pit-report , we can see what PitTest found on our code:

Image 2 — pit test full report

So we knew our code was fully covered (it should mean thoroughly tested), but the mutation test says the Mutation Coverage is 89%, and the Test Strength is also 89%. What does this mean?

Mutation coverage

Mutation coverage is the ratio of the number of mutations killed by tests to all number of mutations (regardless of whether it was covered by tests or not)

Test strength

Test strength is the ratio of mutations killed by tests to the number of all mutations covered by tests.

Let’s get deeper into the report:

Image 3 — Full class report. With mutation survived

The report says that 1 of all the mutations introduced in our code survived, meaning the unit test still passed after changing the code. For example, if we change the code from:

if (suffix.equalsIgnoreCase("PM") && givenHour < 12)

To:

if (suffix.equalsIgnoreCase("PM") && givenHour <= 12)

The test case will not fail, but it should! This small tiny change can introduce a new bug in our code that can be hard to identify if we do not test deeply. So there are two different approaches we can follow here:

  1. Refactor the code;

  2. Implements another unit test to cover all the scenarios (not only the lines!).

Our case is a simple method, so adding a new test case should work:

@Test
public void testTimeConversionFail(){
     TimeConversion timeConversion = new TimeConversion();
     Assert.assertNotEquals("24:00:00", timeConversion.convert("12:00:00PM"));
}
Image 4 — Pit test line report. Success
Image 5 — Full class report. Success

By adding a plugin and running a command, we could see that our 100% line coverage does not save us from new bugs in the code.

Real-world scenario

A couple of times ago, I decided to implement and demonstrate the efficacy of the mutation test on a company I was working for. To do so, I went to Sonar and found the top-rated project.

Image 6 — Real-world scenario. Sonar overall report

Then I put the plugin to run the mutation test, and I got the following:

Image 7 — Real-world scenario. Classes detailed report

It was possible to see that the code was not safe and thoroughly tested even with a good coverage report. So imagine someone introducing a new feature or even working on some refactoring. That person may end up adding a new bug, and just by running the existing test case, it will not catch the bug. Then, we worked on it as a team to get a healthy code base. But, of course, the mutation test has some trade-offs, which I will discuss in the next paragraph.

Limitations of mutation testing

One potential limitation of mutation testing is that it can be computationally expensive, especially for large or complex software systems. Running a mutation test can require significant processing power and time, which can be a barrier for some development teams. As a result, some tools offer the possibility to run only for specified classes/files. You can only run it against your core business logic to save you some computing resources.

Another limitation is the potential for false positives. A false positive occurs when the mutation testing tool identifies a fault or defect that is not present in the software. For example, this can happen if the tool makes an incorrect assumption about how the code should behave or if the tool is not configured correctly.

Finally, mutation testing can only test for the specific mutations defined in the test suite. This means it may not catch all potential faults or defects in the software, especially if modifications are not included in the test suite.

Overall, while mutation testing can be a valuable tool for software testing, it’s important to be aware of its limitations and use it in conjunction with other testing methods to ensure comprehensive software testing.

Conclusion

In conclusion, mutation testing is a powerful technique for identifying faults and defects in software, which can help improve software quality and reduce the risk of bugs and errors. However, while mutation testing has been around for several decades, it has only recently become more widely adopted due to technological advancements and increased awareness of its benefits.

Although mutation testing has some limitations, such as potential false positives and the need for specialized tools and expertise, it remains an important tool for software testing. By following best practices and integrating mutation testing into their software development workflow, development teams can ensure that their software is functioning flawlessly and meets the needs of their users.

In today’s rapidly evolving software landscape, it’s more important than ever to use effective and reliable testing techniques like mutation testing to ensure that software is high-quality, secure, and reliable. By leveraging the power of mutation testing, development teams can stay ahead of the curve and deliver software that meets the highest standards of quality and reliability.

Watch Newton's Tech Talk

Thanks for reading.
Now let's get to know each other.

What we do

WAES supports organizations with various solutions at every stage of their digital transformation.

Discover solutions

Work at WAES

Are you ready to start your relocation to the Netherlands? Have a look at our jobs.

Discover jobs

Let's shape our future

Work at WAES

Start a new chapter. Join our team of Modern Day Spartans.

Discover jobs

Work with WAES

Camilo Parra Gonzalez

Camilo Parra Gonzalez

Account Manager