Microbenchmarks with JMH / Java Microbenchmark Harness
October 2nd, 2017 by Micha KopsWriting microbenchmarks for parts of our applications is not always easy – especially when the internals of the virtual machine, the just-in-time-compiler and such things are coming into effect.
Java Microbenchmark Harness is a tool that takes care of creating JVM warmup-cycles, handling benchmark-input-parameters and running benchmarks as isolated processes etc.
Now following a few short examples for writing microbenchmarks with JMH.
Contents
Project Setup
Using Maven we simply need to add the following two dependencies to our project’s pom.xml:
<dependencies> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-core</artifactId> <version>1.19</version> </dependency> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-generator-annprocess</artifactId> <version>1.19</version> </dependency> </dependencies>
In addition, we’re adding the Maven Shade Plugin to assemble a fat-jar for running the benchmarks:
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.2</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <finalName>benchmark</finalName> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>org.openjdk.jmh.Main</mainClass> </transformer> </transformers> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> </configuration> </execution> </executions> </plugin>
Setup using Maven Archetype
More easier is to use the provided Maven archetype org.openjdk.jmh:jmh-java-benchmark-archetype to generate a new benchmark project e.g.:
mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=com.hascode.tutorial \ -DartifactId=jmh-benchmark-sample \ -Dversion=1.0.0
Simple Benchmark
We’re now ready to implement our first, simple benchmark using org.openjdk.jmh.Main as starting point and runner and by annotating the method to benchmark with @Benchmark – that’s all!
package com.hascode.tutorial; import java.io.IOException; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.runner.RunnerException; public class DefaultsBenchmarkExample { public static void main(String[] args) throws IOException, RunnerException { org.openjdk.jmh.Main.main(args); } @Benchmark public void sampleMethod(){ } }
We may run our benchmarks now using a fat-jar e.g. like this:
mvn clean package && java -cp target/benchmark.jar com.hascode.tutorial.DefaultsBenchmarkExample
# JMH version: 1.19 # VM version: JDK 1.8.0_131, VM 25.131-b11 # VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java # VM options: <none> # Warmup: 20 iterations, 1 s each # Measurement: 20 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: com.hascode.tutorial.DefaultsBenchmarkExample.sampleMethod # Run progress: 0.00% complete, ETA 00:13:45 # Fork: 1 of 10 # Warmup Iteration 1: 3605181491.535 ops/s # Warmup Iteration 2: 3600786119.281 ops/s # Warmup Iteration 3: 3614718349.890 ops/s # Warmup Iteration 4: 3123247490.136 ops/s [..] Iteration 1: 3556337136.253 ops/s Iteration 2: 3280579831.750 ops/s Iteration 3: 3528262154.060 ops/s Iteration 4: 3492611702.649 ops/s Iteration 5: 3527094300.955 ops/s [..] # Run progress: 4.85% complete, ETA 00:13:12 # Fork: 2 of 10 # Warmup Iteration 1: 3549568604.608 ops/s # Warmup Iteration 2: 3589118701.881 ops/s # Warmup Iteration 3: 3553118270.819 ops/s [..] Result "com.hascode.tutorial.DefaultsBenchmarkExample.sampleMethod": 3545540088.367 ±(99.9%) 17575281.787 ops/s [Average] (min, avg, max) = (3051953885.239, 3545540088.367, 3652281748.873), stdev = 74414843.592 CI (99.9%): [3527964806.580, 3563115370.154] (assumes normal distribution)
or using an IDE:
Tuning the Warmup Configuration
In the following example, we’ll be adding some configuration for the vm warmup using the @Warmup annotation.
package com.hascode.tutorial; import java.io.IOException; import java.util.concurrent.TimeUnit; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.Fork; import org.openjdk.jmh.annotations.Warmup; import org.openjdk.jmh.runner.RunnerException; public class WarmupConfigExample { public static void main(String[] args) throws IOException, RunnerException { org.openjdk.jmh.Main.main(args); } @Benchmark @Fork(0) @Warmup(iterations = 10, time = 500, timeUnit = TimeUnit.MILLISECONDS) public void sampleMethod() { } }
# JMH version: 1.19 # VM version: JDK 1.8.0_131, VM 25.131-b11 # VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java # VM options: # Warmup: 10 iterations, 500 ms each # Measurement: 20 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: com.hascode.tutorial.WarmupConfigExample.sampleMethod # Run progress: 0.00% complete, ETA 00:00:25 # Fork: N/A, test runs in the host VM # *** WARNING: Non-forked runs may silently omit JVM options, mess up profilers, disable compiler hints, etc. *** # *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. *** # Warmup Iteration 1: 3533060659.401 ops/s # Warmup Iteration 2: 3513785656.121 ops/s # Warmup Iteration 3: 3595708958.880 ops/s # Warmup Iteration 4: 3576232220.453 ops/s # Warmup Iteration 5: 3482621862.885 ops/s # Warmup Iteration 6: 3521885445.695 ops/s # Warmup Iteration 7: 3545500280.775 ops/s # Warmup Iteration 8: 3523578528.180 ops/s # Warmup Iteration 9: 3547621249.120 ops/s # Warmup Iteration 10: 3487009994.200 ops/s Iteration 1: 3571613038.862 ops/s Iteration 2: 3594909751.769 ops/s Iteration 3: 3551277284.450 ops/s Iteration 4: 3538796644.638 ops/s Iteration 5: 3530181250.100 ops/s Iteration 6: 3500270929.480 ops/s Iteration 7: 3539906819.276 ops/s Iteration 8: 3489324994.493 ops/s Iteration 9: 3510483375.086 ops/s Iteration 10: 3533637346.608 ops/s Iteration 11: 3542051580.501 ops/s Iteration 12: 3535201317.693 ops/s Iteration 13: 3543712258.627 ops/s Iteration 14: 3489945232.067 ops/s Iteration 15: 3484418015.987 ops/s Iteration 16: 3502514821.205 ops/s Iteration 17: 3535290894.166 ops/s Iteration 18: 3476540124.706 ops/s Iteration 19: 3489130100.256 ops/s Iteration 20: 3496757588.873 ops/s Result "com.hascode.tutorial.WarmupConfigExample.sampleMethod": 3522798168.442 ±(99.9%) 27407380.668 ops/s [Average] (min, avg, max) = (3476540124.706, 3522798168.442, 3594909751.769), stdev = 31562380.336 CI (99.9%): [3495390787.774, 3550205549.110] (assumes normal distribution) # Run complete. Total time: 00:00:25 Benchmark Mode Cnt Score Error Units WarmupConfigExample.sampleMethod thrpt 20 3522798168.442 ± 27407380.668 ops/s
Parameterized Benchmarks
In the next example, we’ll be implementing a parameterized benchmark:
package com.hascode.tutorial; import java.io.IOException; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.BenchmarkMode; import org.openjdk.jmh.annotations.Fork; import org.openjdk.jmh.annotations.Level; import org.openjdk.jmh.annotations.Mode; import org.openjdk.jmh.annotations.Param; import org.openjdk.jmh.annotations.Scope; import org.openjdk.jmh.annotations.Setup; import org.openjdk.jmh.annotations.State; import org.openjdk.jmh.runner.RunnerException; public class ParametersExample { public static void main(String[] args) throws IOException, RunnerException { org.openjdk.jmh.Main.main(args); } @Fork(value = 1, warmups = 1) @Benchmark @BenchmarkMode(Mode.Throughput) public void runSample(BenchmarkParams params) { for (int i = 0; i <= params.loops; i++) { params.sb.append("num ").append(i).append("\n"); } System.out.println(params.sb.toString()); } @State(Scope.Benchmark) public static class BenchmarkParams { @Param({"1", "20", "40", "100", "1000"}) public int loops; public StringBuffer sb; @Setup(Level.Invocation) public void setup() { sb = new StringBuffer(); } } }
Output:
# JMH version: 1.19 # VM version: JDK 1.8.0_131, VM 25.131-b11 # VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java # VM options: <none> # Warmup: 20 iterations, 1 s each # Measurement: 20 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: com.hascode.tutorial.ParametersExample.runSample # Parameters: (loops = 1) # Run progress: 0.00% complete, ETA 00:06:40 # Warmup Fork: 1 of 1 # Warmup Iteration 1: num 0 num 1 num 0 num 1 num 0 num 1 num 0 num 1 num 0 num 1 num 0 num 1 num 0 num 1 num 0 num 1 [..] num 994 num 995 num 996 num 997 num 998 num 999 num 1000 400.640 ops/s Result "com.hascode.tutorial.ParametersExample.runSample": 376.221 ±(99.9%) 29.341 ops/s [Average] (min, avg, max) = (248.222, 376.221, 400.640), stdev = 33.790 CI (99.9%): [346.880, 405.563] (assumes normal distribution) # Run complete. Total time: 00:06:51 Benchmark (loops) Mode Cnt Score Error Units ParametersExample.runSample 1 thrpt 20 150800.311 ± 6103.086 ops/s ParametersExample.runSample 20 thrpt 20 18131.695 ± 307.077 ops/s ParametersExample.runSample 40 thrpt 20 9302.495 ± 472.481 ops/s ParametersExample.runSample 100 thrpt 20 3837.838 ± 233.328 ops/s ParametersExample.runSample 1000 thrpt 20 376.221 ± 29.341 ops/s
Programmatic Configuration
In addition we may skip using annotations and use the integrated OptionsBuilder to configure our benchmarks:
package com.hascode.tutorial; import org.openjdk.jmh.runner.Runner; import org.openjdk.jmh.runner.RunnerException; import org.openjdk.jmh.runner.options.Options; import org.openjdk.jmh.runner.options.OptionsBuilder; public class ProgrammaticalConfigExample { public static void main(String[] args) throws RunnerException { Options opts = new OptionsBuilder() .include(".*") .warmupIterations(10) .measurementIterations(20) .jvmArgs("-Xms4g", "-Xmx4g") .shouldDoGC(true) .forks(1) .build(); new Runner(opts).run(); } }
Output:
# JMH version: 1.19 # VM version: JDK 1.8.0_131, VM 25.131-b11 # VM invoker: /usr/lib/jvm/jdk1.8.0_131/jre/bin/java # VM options: -Xms4g -Xmx4g # Warmup: 10 iterations, 1 s each # Measurement: 20 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: com.hascode.tutorial.ProgrammaticalConfigExample.sample # Run progress: 0.00% complete, ETA 00:00:30 # Fork: 1 of 1 # Warmup Iteration 1: ≈ 10⁻¹⁰ s/op # Warmup Iteration 2: ≈ 10⁻¹⁰ s/op # Warmup Iteration 3: ≈ 10⁻¹⁰ s/op # Warmup Iteration 4: ≈ 10⁻¹⁰ s/op # Warmup Iteration 5: ≈ 10⁻¹⁰ s/op # Warmup Iteration 6: ≈ 10⁻¹⁰ s/op # Warmup Iteration 7: ≈ 10⁻¹⁰ s/op # Warmup Iteration 8: ≈ 10⁻¹⁰ s/op # Warmup Iteration 9: ≈ 10⁻¹⁰ s/op # Warmup Iteration 10: ≈ 10⁻¹⁰ s/op Iteration 1: ≈ 10⁻¹⁰ s/op Iteration 2: ≈ 10⁻¹⁰ s/op Iteration 3: ≈ 10⁻¹⁰ s/op Iteration 4: ≈ 10⁻¹⁰ s/op Iteration 5: ≈ 10⁻¹⁰ s/op Iteration 6: ≈ 10⁻¹⁰ s/op Iteration 7: ≈ 10⁻¹⁰ s/op Iteration 8: ≈ 10⁻¹⁰ s/op Iteration 9: ≈ 10⁻¹⁰ s/op Iteration 10: ≈ 10⁻¹⁰ s/op Iteration 11: ≈ 10⁻¹⁰ s/op Iteration 12: ≈ 10⁻¹⁰ s/op Iteration 13: ≈ 10⁻¹⁰ s/op Iteration 14: ≈ 10⁻¹⁰ s/op Iteration 15: ≈ 10⁻¹⁰ s/op Iteration 16: ≈ 10⁻¹⁰ s/op Iteration 17: ≈ 10⁻¹⁰ s/op Iteration 18: ≈ 10⁻¹⁰ s/op Iteration 19: ≈ 10⁻¹⁰ s/op Iteration 20: ≈ 10⁻¹⁰ s/op Result "com.hascode.tutorial.ProgrammaticalConfigExample.sample": ≈ 10⁻¹⁰ s/op # Run complete. Total time: 00:00:49 Benchmark Mode Cnt Score Error Units ProgrammaticalConfigExample.sample avgt 20 ≈ 10⁻¹⁰ s/op
OpenJDK JMH Examples
There are plenty of good examples to be found in the OpenJDK Mercurial repository here: http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/
JMH IntelliJ Plugin
There is a plugin for IntelliJ that helps generating benchmarks (assuming that both dependencies for jmh-core and annotations are on the classpath:
Tutorial Sources
Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:
git clone https://bitbucket.org/hascode/jmh-benchmark-sample.git
Resources
Troubleshooting
- “Error: A JNI error has occurred, please check your installation and try again Exception in thread “main” java.lang.NoClassDefFoundError: org/openjdk/jmh/runner/RunnerException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.openjdk.jmh.runner.RunnerException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) … 7 more” Use the Maven Shade Plugin to assemble a fat Jar, that then may be run using java -jar fatjar.jar package.MainClass
Tags: benchmark, harness, jdk, jmh, jvm, microbenchmark, profiling, tools