What is Mutation Testing ?

Introduction

Mutation testing is seeding faults into your code and seeing if tests pass or fail. Faults or mutations are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.

Generally, in mutation testing, unit tests are run against automatically modified versions of the application code. When the application code changes, it should produce different results and cause the unit tests to fail.

If a unit test does not fail in this situation, it may indicate an issue with the test suite. As a result, mutation testing is actually able to detect whether each statement is meaningfully tested.

Traditional Test coverage like line, statement, branch measures only which code is executed by the tests. So, it doesn’t check that the tests are actually able to detect faults in the executed code. Furthermore, it is therefore only able to identify code that is definitely not tested.

For example, put all tests with no assertions at all and run coverage tests, you see coverage with no actual tests.

Real-world mutation testing

PIT is a state-of-the-art mutation testing system, providing gold standard test coverage for Java and the JVM. It’s fast, scalable, and integrates with modern test and build tooling. The reports produced by PIT are in an easy-to-read format combining line coverage and mutation coverage information.

Maven Plugin

Getting started with PIT Maven Plugin. Add the plugin to build/plugins in your pom.xml

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>LATEST</version>
    <configuration>
        <targetClasses>
            <param>com.your.package.root.want.to.mutate*
</param>
        </targetClasses>
        <targetTests>
            <param>com.your.package.root*</param>
        </targetTests>
    </configuration>
</plugin>

 

Maven Goals

PIT provides two goals :

1. mutationCoverage goal

The mutationCoverage goal analyses all classes in the codebase that match the target tests and target classes filters.

2. scmMutationCoverage goal

The scmMutationCoverage goal analyses only classes that match the filters. The source file has a given status within the project source control system (by default ADDED or MODIFIED). This provides a quick way to check the coverage of changes prior to checking code in / pushing code to a repository.

For further details, check out the PIT official website for more information:   http://pitest.org/