Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.
January 30th, 2019 by Micha KopsA lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.
Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.
Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.
In the following short tutorial I am going to demonstrate how to start Apache Kafka as well as a classical Postgresql database from a JUnit 5 integration test.
Contents
Prerequisites
We need two things for the following tutorial: Docker installed (see the exact requirements) and Maven.
Since we’re using JUnit5, we need to add the JUnit dependencies (I’ve used the JUnit BOM for these dependencies here) and also the testcontainers-junit adapter library.
Using Maven our project’s pom.xml should include the following basic dependencies:
<dependencyManagement> <dependencies> <dependency> <groupId>org.junit</groupId> <artifactId>junit-bom</artifactId> <version>5.3.2</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <!-- JUnit 5 --> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter-api</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter-engine</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.junit.platform</groupId> <artifactId>junit-platform-launcher</artifactId> <scope>test</scope> </dependency> <!-- Testcontainers --> <dependency> <groupId>org.testcontainers</groupId> <artifactId>junit-jupiter</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> </dependencies>
Examples
Kafka
Our first example targets my favorite stream processing platform, Apache Kafka….
Dependencies
We need to add two dependencies .. one for the testcontainers-kafka support and the kafka client library:
<!-- Kafka Container --> <dependency> <groupId>org.testcontainers</groupId> <artifactId>kafka</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> <!-- Kafka Client/Producer --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.9.0.1</version> </dependency>
Integration Test
This is our Kafka integration test. We’re producing and consuming some records send over the containerized Kafka instance.
This is not a real valid test but demonstrates that our container is running and available.
package it; import static org.junit.jupiter.api.Assertions.assertTrue; import java.util.Arrays; import java.util.Properties; import java.util.concurrent.CountDownLatch; import java.util.concurrent.atomic.AtomicInteger; import java.util.stream.IntStream; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerRecord; import org.junit.Assert; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.testcontainers.containers.KafkaContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; @Testcontainers public class ContainerizedKafkaIT { public static final String MY_TOPIC = "my-topic"; public static final int NUMBER_OF_MESSAGES = 100; @Container public KafkaContainer kafkaContainer = new KafkaContainer(); @Test @DisplayName("kafka server should be running") void shouldBeRunningKafka() throws Exception { assertTrue(kafkaContainer.isRunning()); } @Test @DisplayName("should send and receive records over kafka") void shouldSendAndReceiveMessages() throws Exception { var servers = kafkaContainer.getBootstrapServers(); System.out.printf("servers: %s%n", servers); var props = new Properties(); props.put("bootstrap.servers", servers); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("group.id", "group-1"); props.put("auto.offset.reset", "earliest"); var counter = new AtomicInteger(0); CountDownLatch waitABit = new CountDownLatch(1); new Thread(() -> { KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(props); kafkaConsumer.subscribe(Arrays.asList(MY_TOPIC)); while (counter.get()<NUMBER_OF_MESSAGES) { ConsumerRecords<String, String> records = kafkaConsumer.poll(200); records.forEach(record -> { System.out .printf("%d # offset: %d, value = %s%n", counter.incrementAndGet(), record.offset(), record.value()); }); } waitABit.countDown(); }).start(); try ( Producer<String, String> producer = new KafkaProducer<>(props)) { IntStream.range(0, NUMBER_OF_MESSAGES).forEach(i -> { var msg = String.format("my-message-%d", i); producer.send(new ProducerRecord<>(MY_TOPIC, msg)); System.out.println("Sent:" + msg); }); } waitABit.await(); Assert.assertEquals(NUMBER_OF_MESSAGES, counter.get()); } }
The test should produce an output similar to this one but the tests needs some serious improvements, so results may differ…
ℹ︎ Checking the system... ✔ Docker version should be at least 1.6.0 ✔ Docker environment should have more than 2GB free disk space servers: PLAINTEXT://localhost:32902 Sent:my-message-0 Sent:my-message-1 Sent:my-message-2 Sent:my-message-3 Sent:my-message-4 Sent:my-message-5 Sent:my-message-6 Sent:my-message-7 Sent:my-message-8 Sent:my-message-9 Sent:my-message-10 Sent:my-message-11 Sent:my-message-12 Sent:my-message-13 Sent:my-message-14 Sent:my-message-15 Sent:my-message-16 Sent:my-message-17 Sent:my-message-18 Sent:my-message-19 Sent:my-message-20 Sent:my-message-21 Sent:my-message-22 Sent:my-message-23 Sent:my-message-24 Sent:my-message-25 Sent:my-message-26 Sent:my-message-27 [..] Sent:my-message-98 Sent:my-message-99 1 # offset: 20, value = my-message-20 2 # offset: 21, value = my-message-21 3 # offset: 22, value = my-message-22 4 # offset: 23, value = my-message-23 5 # offset: 24, value = my-message-24 6 # offset: 25, value = my-message-25 7 # offset: 26, value = my-message-26 8 # offset: 27, value = my-message-27 9 # offset: 28, value = my-message-28 [..] 79 # offset: 98, value = my-message-98 80 # offset: 99, value = my-message-99
Postgresql
To provide another example, we will now write an integration test for a classical RDBMS, Postgresql...
Dependencies
We need to add the dependencies for the testcontainers-postgresql adapter and the JDBC driver needed:
<!-- Postgres Container --> <dependency> <groupId>org.testcontainers</groupId> <artifactId>postgresql</artifactId> <version>${testcontainers.version}</version> <scope>test</scope> </dependency> <!-- Postgres Driver for JDBC Connection --> <dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <version>42.2.5</version> </dependency>
Integration Test
Again we’re writing two fast integration tests to verify that the database container is started and another one writing and reading from the started Postgresql database.
package it; import static org.junit.jupiter.api.Assertions.assertTrue; import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.testcontainers.containers.PostgreSQLContainer; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; @Testcontainers public class ContainerizedPostgresIT { @Container private PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer() .withDatabaseName("mydb") .withUsername("user") .withPassword("secret"); @Test @DisplayName("postgres should be running") void shouldBeRunningPostgres() throws Exception { System.out.printf("postgres db running, db-name: '%s', user: '%s', jdbc-url: '%s'%n ", postgresqlContainer.getDatabaseName(), postgresqlContainer.getUsername(), postgresqlContainer.getJdbcUrl()); assertTrue(postgresqlContainer.isRunning()); } @Test @DisplayName("should write and read from database") void shouldReadFromDatabase() throws Exception { Class.forName("org.postgresql.Driver"); var connection = DriverManager .getConnection(postgresqlContainer.getJdbcUrl(), postgresqlContainer.getUsername(), postgresqlContainer.getPassword()); connection.prepareStatement("CREATE DATABASE article_db").execute(); connection.prepareStatement("CREATE table articles (title VARCHAR , url VARCHAR)").execute(); connection.prepareStatement("INSERT INTO articles VALUES('Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java','https://www.hascode.com/2018/11/implementing-reactive-client-server-communication-over-tcp-or-websockets-with-rsocket-and-java/')").execute(); connection.prepareStatement("INSERT INTO articles VALUES('Setting up Kafka Brokers for Testing with Kafka-Unit','https://www.hascode.com/2018/03/setting-up-kafka-brokers-for-testing-with-kafka-unit/')").execute(); connection.prepareStatement("INSERT INTO articles VALUES('Managing Architecture Decision Records with ADR-Tools','https://www.hascode.com/2018/05/managing-architecture-decision-records-with-adr-tools/')").execute(); var result = connection .prepareStatement("SELECT title,url FROM articles ORDER BY title ASC").executeQuery(); result.next(); assertTrue(result.getString("title").equals( "Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java")); result.next(); assertTrue(result.getString("title").equals( "Managing Architecture Decision Records with ADR-Tools")); result.next(); assertTrue(result.isLast() && result.getRow() == 3); } }
The test should produce an output similar to this one:
ℹ︎ Checking the system... ✔ Docker version should be at least 1.6.0 ✔ Docker environment should have more than 2GB free disk space postgres db running, db-name: 'mydb', user: 'user', jdbc-url: 'jdbc:postgresql://localhost:32915/mydb'
Tutorial Sources
Please feel free to download the tutorial sources from my Bitbucket repository, fork it there or clone it using Git:
git clone https://bitbucket.org/hascode/testcontainers-tutorial.git
Resources
- Testcontainers Website
- JUnit 5 Website
- Testcontainers Documentation – Database containers
- Testcontainers Documentation – Kafka
Article Updates
- 2020-02-08: Updated the Kafka example to start reading data from the beginning (thanks @Jan Vermeir for the hint)
Other Testing Tutorials of Mine
Please feel free to have a look at other testing tutorial of mine (an excerpt):
- Testing Java Applications for Resilience by Simulating Network Problems with Toxiproxy, JUnit and the Docker Maven Plugin
- Generating JUnit Tests with Java, EvoSuite and Maven
- Layout Testing with Galen, JUnit and Maven
- Mocking HTTP Interaction with Java, JUnit and MockServer
- Testing Asynchronous Applications with Java and Awaitility
- Mutation Testing with Pitest and Maven
- BDD Testing with Cucumber, Java and JUnit
- Running categorized Tests using JUnit, Maven and Annotated-Test Suites
- Mocking, Stubbing and Test Spying using the Mockito Framework and PowerMock
And more…
Tags: bom, container, database, db, docker, jdbc, junit, junit5, jupiter, kafka, maven, postgres, tdd, testcontainers, testing
January 29th, 2020 at 10:28 am
Hi Micha, I’ve tried running the code in your example in ContainerizedKafkaIT.java. It shows messages being sent to the producer (I get a list of log messages like
‘servers: PLAINTEXT://localhost:32868
Sent:my-message-0′)
but no messages are read by the consumer.
I’m using this Java version:
java -version
openjdk version “11.0.5″ 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.5+10, mixed mode)
What could be wrong?
regards, Jan
January 29th, 2020 at 12:53 pm
Answering my own question: you should add
props.put(“auto.offset.reset”, “earliest”);
to the consumer properties to make it start reading from the beginning. If you don’t the consumer might start too late.