Mutation Testing with Pitest and Maven

May 10th, 2015 by

Mutation testing makes an interesting addition to the classical test coverage metrics.

They seed mutations (errors) into the code, run the project’s tests afterwards and if the tests fail, the mutation is killed – otherwise it lived and we have a possible indication of an issue with our tests.

In the following short tutorial. I’d like to demonstrate how to setup mutation tests with the PIT/Pitest library and Maven and generate reports.

Detailed mutation report for BookBean

Detailed mutation report for BookBean

 

Adding the Maven Plugin

To integrate Pitest, we’re simply adding the following plug-in to our project’s pom.xml.

<plugin>
	<groupId>org.pitest</groupId>
	<artifactId>pitest-maven</artifactId>
	<version>1.1.5</version>
</plugin>

In addition to write some tests, we’re adding junit and hamcrest-all to our project.

Target Classes and Tests Configuration

We may specify a pattern for classes to be mutated and tests to be used by adding the following lines to the plug-in definition in the pom.xml.

If nothing is specified, the project’s groupId is taken as a default. More detailed information about the other configuration flags can be found in the documentation here.

<configuration>
	<targetClasses>
		<param>com.your.package.root.for.mutation*</param>
	</targetClasses>
	<targetTests>
		<param>com.your.package.root.for.tests*</param>
	</targetTests>
</configuration>

Our Example Project

This is our small project that’s going to be the target for our mutation tests:

Book Entity

Just a simple POJO, immutable, with a toString method implemented

package com.hascode.tutorial.entity;
 
public class Book {
	private final String title;
	private final String id;
 
	public Book(final String title, final String id) {
		this.title = title;
		this.id = id;
	}
 
	// getter ommitted,
 
	@Override
	public String toString() {
		StringBuilder builder = new StringBuilder();
		builder.append("Book [title=").append(title).append(", id=").append(id).append("]");
		return builder.toString();
	}
}

Book Bean

Another POJO, creates new Books, adding an unique identifier and checking some constraints.

package com.hascode.tutorial.service;
 
import java.util.UUID;
 
import com.hascode.tutorial.entity.Book;
 
public class BookBean {
	private static final int MIN_LENGTH = 3;
 
	public Book create(final String title) {
		if (title == null) {
			throw new IllegalArgumentException("title must be set");
		}
		if (title.length() <= MIN_LENGTH) {
			throw new IllegalArgumentException("title must have a minimal length of " + MIN_LENGTH);
		}
		return new Book(title, UUID.randomUUID().toString().toUpperCase());
	}
}

Book Bean Test

A simple test, verifying the book bean’s behaviour for the success path and two possible error paths.

package com.hascode.tutorial.service;
 
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.isEmptyOrNullString;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.notNullValue;
 
import org.junit.Test;
 
import com.hascode.tutorial.entity.Book;
 
public class BookBeanTest {
	private static final String TITLE = "Some book";
 
	BookBean bookBean = new BookBean();
 
	@Test
	public void shouldCreateBook() throws Exception {
		Book book = bookBean.create(TITLE);
		assertThat(book, notNullValue());
		assertThat(book.getTitle(), equalTo(TITLE));
		assertThat(book.getId(), not(isEmptyOrNullString()));
	}
 
	@Test(expected = IllegalArgumentException.class)
	public void shouldFailCreateBookWithNoTitleGiven() throws Exception {
		bookBean.create(null);
	}
 
	@Test(expected = IllegalArgumentException.class)
	public void shouldFailCreateBookWithTooShortTitleGiven() throws Exception {
		bookBean.create("ab");
	}
}

Running Mutations

We’re now ready to run mutation tests. This is done by simply executing the following Maven goal:

$ mvn org.pitest:pitest-maven:mutationCoverage
[..]
[INFO] Found plugin : Default csv report plugin
[INFO] Found plugin : Default xml report plugin
[INFO] Found plugin : Default html report plugin
[INFO] Found plugin : Default limit mutations plugin
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /data/project/pitest-tutorial/target/classes
[INFO] Defaulting to group id (com.hascode.tutorial*)
3:48:50 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
3:48:50 PM PIT >> INFO : Sending 1 test classes to slave
3:48:50 PM PIT >> INFO : Sent tests to slave
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Checking environment
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Found  3 tests
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0
3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : 3 tests received
\3:48:50 PM PIT >> INFO : Calculated coverage in 0 seconds.
3:48:50 PM PIT >> INFO : Created  2 mutation test units
-3:48:51 PM PIT >> INFO : Completed in 1 seconds
================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
--------------------------------------------------------------------------------
> Total  : 1 seconds
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 7 mutations Killed 5 (71%)
>> Ran 7 tests (1 tests per mutation)
================================================================================
- Mutators
================================================================================
> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 4 Killed 3 (75%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 1
--------------------------------------------------------------------------------
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 2 Killed 2 (100%)
> KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
--------------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.044s
[INFO] Finished at: Sun May 10 15:48:51 CEST 2015
[INFO] Final Memory: 12M/205M
[INFO] ------------------------------------------------------------------------

Mutation Test Report

Now we’re ready to have a look at the generated HTML report of the mutation test in target/pit-reports/yyyyMMddhhmm.

Pitest Mutation Report Overview

Pitest Mutation Report Overview

BookBean Mutation Report Overview

BookBean Mutation Report Overview

Detailed mutation report for BookBean

Detailed mutation report for BookBean

Tutorial Sources

Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:

git clone https://bitbucket.org/hascode/pitest-tutorial.git

Resources

Tags: , , , , , , , , , , ,

3 Responses to “Mutation Testing with Pitest and Maven”

  1. Terry Speicher Says:

    I’m trying to get PITest to run on my project in IntelliJ using Maven. I kept getting errors saying that there were no tests run.

    I like your example, so I downloaded it and ran it on my machine. It gave me the same error. I ran the command line that you specified, and here is my output:

    Terrys-MacBook-Pro:hascode-pitest-tutorial-5c6024b22f42 terry$ mvn org.pitest:pitest-maven:mutationCoverage
    [INFO] Scanning for projects…
    [INFO]
    [INFO] ——————————–
    [INFO] Building pitest-tutorial 1.0.0
    [INFO] ——————————–[ jar ]———————————
    [INFO]
    [INFO] — pitest-maven:1.1.5:mutationCoverage (default-cli) @ pitest-tutorial —
    [INFO] Found plugin : Default csv report plugin
    [INFO] Found plugin : Default xml report plugin
    [INFO] Found plugin : Default html report plugin
    [INFO] Found plugin : Default limit mutations plugin
    [INFO] Found shared classpath plugin : Default mutation engine
    [INFO] Adding org.pitest:pitest to SUT classpath
    [INFO] Mutating from /Users/terry/Desktop/hascode-pitest-tutorial-5c6024b22f42/target/classes
    [INFO] Defaulting to group id (com.hascode.tutorial*)
    9:26:02 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
    9:26:02 PM PIT >> INFO : SLAVE : objc[15011]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/bin/java (0x1080ef4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/libinstrument.dylib (0×10817
    9:26:02 PM PIT >> INFO : SLAVE : b4e0). One of the two will be used. Which one is undefined.

    9:26:02 PM PIT >> INFO : Sending 0 test classes to slave
    9:26:02 PM PIT >> INFO : Sent tests to slave
    9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Checking environment

    9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Found 0 tests

    9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

    9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : 0 tests received

    9:26:02 PM PIT >> INFO : Calculated coverage in 0 seconds.
    9:26:02 PM PIT >> INFO : Created 0 mutation test units
    [INFO] ————————————————————————
    [INFO] BUILD FAILURE
    [INFO] ————————————————————————
    [INFO] Total time: 1.139 s
    [INFO] Finished at: 2018-03-28T21:26:02-06:00
    [INFO] ————————————————————————
    [ERROR] Failed to execute goal org.pitest:pitest-maven:1.1.5:mutationCoverage (default-cli) on project pitest-tutorial: Execution default-cli of goal org.pitest:pitest-maven:1.1.5:mutationCoverage failed: No mutations found. This probably means there is an issue with either the supplied classpath or filters.
    [ERROR] See http://pitest.org for more details.
    [ERROR] -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
    Terrys-MacBook-Pro:hascode-pitest-tutorial-5c6024b22f42 terry$

    While it leads me to believe that I have something wrong on my side, I wanted to see if there was something simple that you might know about since it happened when I tried to run your program also.

    Thanks for any help. Even if you are unable to respond or if you do not see anything wrong, I think your article is leading me closer to finding the problem.

    -Terry

  2. Terry Speicher Says:

    So I found the problem. Running the “mvn org.pitest:pitest-maven:mutationCoverage” is not all that needs to be done. The project needs to have the mvn verify goal run to compile the class files. Then running the command line produces the tests.

    Again, this has pointed me in the direction I need to search. Thanks for the great article.

    -Terry

  3. Micha Kops Says:

    Hi Terry,
    I’m glad you found a solution and thanks for sharing it! :)
    Cheers,
    Micha

Search
Categories