Mutation Testing with Pitest and Maven
May 10th, 2015 by Micha KopsMutation testing makes an interesting addition to the classical test coverage metrics.
They seed mutations (errors) into the code, run the project’s tests afterwards and if the tests fail, the mutation is killed – otherwise it lived and we have a possible indication of an issue with our tests.
In the following short tutorial. I’d like to demonstrate how to setup mutation tests with the PIT/Pitest library and Maven and generate reports.
Contents
Adding the Maven Plugin
To integrate Pitest, we’re simply adding the following plug-in to our project’s pom.xml.
<plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <version>1.1.5</version> </plugin>
In addition to write some tests, we’re adding junit and hamcrest-all to our project.
Target Classes and Tests Configuration
We may specify a pattern for classes to be mutated and tests to be used by adding the following lines to the plug-in definition in the pom.xml.
If nothing is specified, the project’s groupId is taken as a default. More detailed information about the other configuration flags can be found in the documentation here.
<configuration> <targetClasses> <param>com.your.package.root.for.mutation*</param> </targetClasses> <targetTests> <param>com.your.package.root.for.tests*</param> </targetTests> </configuration>
Our Example Project
This is our small project that’s going to be the target for our mutation tests:
Book Entity
Just a simple POJO, immutable, with a toString method implemented
package com.hascode.tutorial.entity; public class Book { private final String title; private final String id; public Book(final String title, final String id) { this.title = title; this.id = id; } // getter ommitted, @Override public String toString() { StringBuilder builder = new StringBuilder(); builder.append("Book [title=").append(title).append(", id=").append(id).append("]"); return builder.toString(); } }
Book Bean
Another POJO, creates new Books, adding an unique identifier and checking some constraints.
package com.hascode.tutorial.service; import java.util.UUID; import com.hascode.tutorial.entity.Book; public class BookBean { private static final int MIN_LENGTH = 3; public Book create(final String title) { if (title == null) { throw new IllegalArgumentException("title must be set"); } if (title.length() <= MIN_LENGTH) { throw new IllegalArgumentException("title must have a minimal length of " + MIN_LENGTH); } return new Book(title, UUID.randomUUID().toString().toUpperCase()); } }
Book Bean Test
A simple test, verifying the book bean’s behaviour for the success path and two possible error paths.
package com.hascode.tutorial.service; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.isEmptyOrNullString; import static org.hamcrest.Matchers.not; import static org.hamcrest.Matchers.notNullValue; import org.junit.Test; import com.hascode.tutorial.entity.Book; public class BookBeanTest { private static final String TITLE = "Some book"; BookBean bookBean = new BookBean(); @Test public void shouldCreateBook() throws Exception { Book book = bookBean.create(TITLE); assertThat(book, notNullValue()); assertThat(book.getTitle(), equalTo(TITLE)); assertThat(book.getId(), not(isEmptyOrNullString())); } @Test(expected = IllegalArgumentException.class) public void shouldFailCreateBookWithNoTitleGiven() throws Exception { bookBean.create(null); } @Test(expected = IllegalArgumentException.class) public void shouldFailCreateBookWithTooShortTitleGiven() throws Exception { bookBean.create("ab"); } }
Running Mutations
We’re now ready to run mutation tests. This is done by simply executing the following Maven goal:
$ mvn org.pitest:pitest-maven:mutationCoverage [..] [INFO] Found plugin : Default csv report plugin [INFO] Found plugin : Default xml report plugin [INFO] Found plugin : Default html report plugin [INFO] Found plugin : Default limit mutations plugin [INFO] Found shared classpath plugin : Default mutation engine [INFO] Adding org.pitest:pitest to SUT classpath [INFO] Mutating from /data/project/pitest-tutorial/target/classes [INFO] Defaulting to group id (com.hascode.tutorial*) 3:48:50 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue. 3:48:50 PM PIT >> INFO : Sending 1 test classes to slave 3:48:50 PM PIT >> INFO : Sent tests to slave 3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Checking environment 3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Found 3 tests 3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0 3:48:50 PM PIT >> INFO : SLAVE : 3:48:50 PM PIT >> INFO : 3 tests received \3:48:50 PM PIT >> INFO : Calculated coverage in 0 seconds. 3:48:50 PM PIT >> INFO : Created 2 mutation test units -3:48:51 PM PIT >> INFO : Completed in 1 seconds ================================================================================ - Timings ================================================================================ > scan classpath : < 1 second > coverage and dependency analysis : < 1 second > build mutation tests : < 1 second > run mutation analysis : < 1 second -------------------------------------------------------------------------------- > Total : 1 seconds -------------------------------------------------------------------------------- ================================================================================ - Statistics ================================================================================ >> Generated 7 mutations Killed 5 (71%) >> Ran 7 tests (1 tests per mutation) ================================================================================ - Mutators ================================================================================ > org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -------------------------------------------------------------------------------- > org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator >> Generated 4 Killed 3 (75%) > KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 1 -------------------------------------------------------------------------------- > org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator >> Generated 2 Killed 2 (100%) > KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 -------------------------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.044s [INFO] Finished at: Sun May 10 15:48:51 CEST 2015 [INFO] Final Memory: 12M/205M [INFO] ------------------------------------------------------------------------
Mutation Test Report
Now we’re ready to have a look at the generated HTML report of the mutation test in target/pit-reports/yyyyMMddhhmm.
Tutorial Sources
Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:
git clone https://bitbucket.org/hascode/pitest-tutorial.git
Resources
Tags: ci, junit, maven, metrics, mutation, pit, pitest, report, reporting, tdd, test, testing
March 29th, 2018 at 5:33 am
I’m trying to get PITest to run on my project in IntelliJ using Maven. I kept getting errors saying that there were no tests run.
I like your example, so I downloaded it and ran it on my machine. It gave me the same error. I ran the command line that you specified, and here is my output:
Terrys-MacBook-Pro:hascode-pitest-tutorial-5c6024b22f42 terry$ mvn org.pitest:pitest-maven:mutationCoverage
[INFO] Scanning for projects…
[INFO]
[INFO] ——————————–
[INFO] Building pitest-tutorial 1.0.0
[INFO] ——————————–[ jar ]———————————
[INFO]
[INFO] — pitest-maven:1.1.5:mutationCoverage (default-cli) @ pitest-tutorial —
[INFO] Found plugin : Default csv report plugin
[INFO] Found plugin : Default xml report plugin
[INFO] Found plugin : Default html report plugin
[INFO] Found plugin : Default limit mutations plugin
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /Users/terry/Desktop/hascode-pitest-tutorial-5c6024b22f42/target/classes
[INFO] Defaulting to group id (com.hascode.tutorial*)
9:26:02 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
9:26:02 PM PIT >> INFO : SLAVE : objc[15011]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/bin/java (0x1080ef4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/libinstrument.dylib (0×10817
9:26:02 PM PIT >> INFO : SLAVE : b4e0). One of the two will be used. Which one is undefined.
9:26:02 PM PIT >> INFO : Sending 0 test classes to slave
9:26:02 PM PIT >> INFO : Sent tests to slave
9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Checking environment
9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Found 0 tests
9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0
9:26:02 PM PIT >> INFO : SLAVE : 9:26:02 PM PIT >> INFO : 0 tests received
9:26:02 PM PIT >> INFO : Calculated coverage in 0 seconds.
9:26:02 PM PIT >> INFO : Created 0 mutation test units
[INFO] ————————————————————————
[INFO] BUILD FAILURE
[INFO] ————————————————————————
[INFO] Total time: 1.139 s
[INFO] Finished at: 2018-03-28T21:26:02-06:00
[INFO] ————————————————————————
[ERROR] Failed to execute goal org.pitest:pitest-maven:1.1.5:mutationCoverage (default-cli) on project pitest-tutorial: Execution default-cli of goal org.pitest:pitest-maven:1.1.5:mutationCoverage failed: No mutations found. This probably means there is an issue with either the supplied classpath or filters.
[ERROR] See http://pitest.org for more details.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
Terrys-MacBook-Pro:hascode-pitest-tutorial-5c6024b22f42 terry$
While it leads me to believe that I have something wrong on my side, I wanted to see if there was something simple that you might know about since it happened when I tried to run your program also.
Thanks for any help. Even if you are unable to respond or if you do not see anything wrong, I think your article is leading me closer to finding the problem.
-Terry
March 29th, 2018 at 5:41 am
So I found the problem. Running the “mvn org.pitest:pitest-maven:mutationCoverage” is not all that needs to be done. The project needs to have the mvn verify goal run to compile the class files. Then running the command line produces the tests.
Again, this has pointed me in the direction I need to search. Thanks for the great article.
-Terry
March 29th, 2018 at 8:06 am
Hi Terry,
I’m glad you found a solution and thanks for sharing it! :)
Cheers,
Micha