A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.
Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.
Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.
In the following short tutorial I am going to demonstrate how to start Apache Kafka as well as a classical Postgresql database from a JUnit 5 integration test.
Prerequisites
We need two things for the following tutorial: Docker installed (see the exact requirements) and Maven.
Since we’re using JUnit5, we need to add the JUnit dependencies (I’ve used the JUnit BOM for these dependencies here) and also the testcontainers-junit adapter library.
Using Maven our project’s pom.xml should include the following basic dependencies:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.junit</groupId>
<artifactId>junit-bom</artifactId>
<version>5.3.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<!-- JUnit 5 -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<scope>test</scope>
</dependency>
<!-- Testcontainers -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
Examples
Kafka
Our first example targets my favorite stream processing platform, Apache Kafka….
Dependencies
We need to add two dependencies .. one for the testcontainers-kafka support and the kafka client library:
<!-- Kafka Container -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>kafka</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<!-- Kafka Client/Producer -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>
</dependency>
Integration Test
This is our Kafka integration test. We’re producing and consuming some records send over the containerized Kafka instance.
This is not a real valid test but demonstrates that our container is running and available.
package it;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.Arrays;
import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.junit.Assert;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
@Testcontainers
public class ContainerizedKafkaIT {
public static final String MY_TOPIC = "my-topic";
public static final int NUMBER_OF_MESSAGES = 100;
@Container
public KafkaContainer kafkaContainer = new KafkaContainer();
@Test
@DisplayName("kafka server should be running")
void shouldBeRunningKafka() throws Exception {
assertTrue(kafkaContainer.isRunning());
}
@Test
@DisplayName("should send and receive records over kafka")
void shouldSendAndReceiveMessages() throws Exception {
var servers = kafkaContainer.getBootstrapServers();
System.out.printf("servers: %s%n", servers);
var props = new Properties();
props.put("bootstrap.servers", servers);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("group.id", "group-1");
props.put("auto.offset.reset", "earliest");
var counter = new AtomicInteger(0);
CountDownLatch waitABit = new CountDownLatch(1);
new Thread(() -> {
KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(props);
kafkaConsumer.subscribe(Arrays.asList(MY_TOPIC));
while (counter.get()<NUMBER_OF_MESSAGES) {
ConsumerRecords<String, String> records = kafkaConsumer.poll(200);
records.forEach(record -> {
System.out
.printf("%d # offset: %d, value = %s%n", counter.incrementAndGet(), record.offset(),
record.value());
});
}
waitABit.countDown();
}).start();
try (
Producer<String, String> producer = new KafkaProducer<>(props)) {
IntStream.range(0, NUMBER_OF_MESSAGES).forEach(i -> {
var msg = String.format("my-message-%d", i);
producer.send(new ProducerRecord<>(MY_TOPIC, msg));
System.out.println("Sent:" + msg);
});
}
waitABit.await();
Assert.assertEquals(NUMBER_OF_MESSAGES, counter.get());
}
}
The test should produce an output similar to this one but the tests needs some serious improvements, so results may differ…
ℹ︎ Checking the system...
✔ Docker version should be at least 1.6.0
✔ Docker environment should have more than 2GB free disk space
servers: PLAINTEXT://localhost:32902
Sent:my-message-0
Sent:my-message-1
Sent:my-message-2
Sent:my-message-3
Sent:my-message-4
Sent:my-message-5
Sent:my-message-6
Sent:my-message-7
Sent:my-message-8
Sent:my-message-9
Sent:my-message-10
Sent:my-message-11
Sent:my-message-12
Sent:my-message-13
Sent:my-message-14
Sent:my-message-15
Sent:my-message-16
Sent:my-message-17
Sent:my-message-18
Sent:my-message-19
Sent:my-message-20
Sent:my-message-21
Sent:my-message-22
Sent:my-message-23
Sent:my-message-24
Sent:my-message-25
Sent:my-message-26
Sent:my-message-27
[..]
Sent:my-message-98
Sent:my-message-99
1# offset: 20, value = my-message-20
2# offset: 21, value = my-message-21
3# offset: 22, value = my-message-22
4# offset: 23, value = my-message-23
5# offset: 24, value = my-message-24
6# offset: 25, value = my-message-25
7# offset: 26, value = my-message-26
8# offset: 27, value = my-message-27
9# offset: 28, value = my-message-28
[..]
79# offset: 98, value = my-message-98
80# offset: 99, value = my-message-99
Postgresql
To provide another example, we will now write an integration test for a classical RDBMS, Postgresql...
Dependencies
We need to add the dependencies for the testcontainers-postgresql adapter and the JDBC driver needed:
<!-- Postgres Container -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<!-- Postgres Driver for JDBC Connection -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.5</version>
</dependency>
Integration Test
Again we’re writing two fast integration tests to verify that the database container is started and another one writing and reading from the started Postgresql database.
package it;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
@Testcontainers
public class ContainerizedPostgresIT {
@Container
private PostgreSQLContainer postgresqlContainer = new PostgreSQLContainer()
.withDatabaseName("mydb")
.withUsername("user")
.withPassword("secret");
@Test
@DisplayName("postgres should be running")
void shouldBeRunningPostgres() throws Exception {
System.out.printf("postgres db running, db-name: '%s', user: '%s', jdbc-url: '%s'%n ",
postgresqlContainer.getDatabaseName(),
postgresqlContainer.getUsername(),
postgresqlContainer.getJdbcUrl());
assertTrue(postgresqlContainer.isRunning());
}
@Test
@DisplayName("should write and read from database")
void shouldReadFromDatabase() throws Exception {
Class.forName("org.postgresql.Driver");
var connection = DriverManager
.getConnection(postgresqlContainer.getJdbcUrl(), postgresqlContainer.getUsername(),
postgresqlContainer.getPassword());
connection.prepareStatement("CREATE DATABASE article_db").execute();
connection.prepareStatement("CREATE table articles (title VARCHAR , url VARCHAR)").execute();
connection.prepareStatement("INSERT INTO articles VALUES('Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java','https://www.hascode.com/2018/11/implementing-reactive-client-server-communication-over-tcp-or-websockets-with-rsocket-and-java/')").execute();
connection.prepareStatement("INSERT INTO articles VALUES('Setting up Kafka Brokers for Testing with Kafka-Unit','https://www.hascode.com/2018/03/setting-up-kafka-brokers-for-testing-with-kafka-unit/')").execute();
connection.prepareStatement("INSERT INTO articles VALUES('Managing Architecture Decision Records with ADR-Tools','https://www.hascode.com/2018/05/managing-architecture-decision-records-with-adr-tools/')").execute();
var result = connection
.prepareStatement("SELECT title,url FROM articles ORDER BY title ASC").executeQuery();
result.next();
assertTrue(result.getString("title").equals( "Implementing Reactive Client-Server Communication over TCP or Websockets with RSocket and Java"));
result.next();
assertTrue(result.getString("title").equals( "Managing Architecture Decision Records with ADR-Tools"));
result.next();
assertTrue(result.isLast() && result.getRow() == 3);
}
}
The test should produce an output similar to this one:
ℹ︎ Checking the system...
✔ Docker version should be at least 1.6.0
✔ Docker environment should have more than 2GB free disk space
postgres db running, db-name: 'mydb', user: 'user', jdbc-url: 'jdbc:postgresql://localhost:32915/mydb'
Tutorial Sources
Please feel free to download the tutorial sources from my GitHub repository, fork it there or clone it using Git:
git clone https://github.com/hascode/testcontainers-tutorial.git
Resources
Article Updates
-
2020-02-08: Updated the Kafka example to start reading data from the beginning (thanks @Jan Vermeir for the hint)
Other Testing Tutorials of Mine
Please feel free to have a look at other testing tutorial of mine (an excerpt):
And more…