Embedded Kafka for Spring Boot Testing without using Docker

Wed, 25 May 2022 00:00:00 +0200

Sometimes it is nice to set up an embedded Kafka broker for testing without the need to have Docker installed (e.g. for using testcontainers-lib).

The following snippet shows, how to set up an embedded Kafka instance for testing for a Spring Boot project.

Setup

Using Maven, this is our Spring Boot project with dependencies needed:

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>(1)
        <version>2.6.4</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <groupId>com.hascode.tutorial</groupId>
    <artifactId>kafka-testing</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>17</java.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
        <kafka.version>3.1.0</kafka.version>(2)
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>(3)
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>(4)
            <scope>test</scope>
        </dependency>

        <dependency>(5)
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <scope>test</scope>
        </dependency>
        [..]
</project>

JUnit5 Java Maven Snippet

Fri, 14 May 2021 00:00:00 +0200

Goals

Use JUnit Maven BOM for version alignment
Add minimal dependencies for JUnit5 Java Projekt

Setup

Excerpt from the Maven project’s pom.xml:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.junit</groupId>
      <artifactId>junit-bom</artifactId>
      <version>5.7.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

JUnit5 does not work with older versions of the Maven Compiler Plugin and the Surefire Plugin used for test execution. Setting their versions in the pom.xml is done like this:

Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.

Wed, 30 Jan 2019 00:00:00 +0100

A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.

Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.

Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.

Testing OpenAPI Swagger Schema Compliance with Java, JUnit and assertj-swagger

Fri, 31 Aug 2018 00:00:00 +0200

The OpenAPI and Swagger API description format are becoming important standards to specify API contracts for RESTful web services and the Microservices trend pushes the need for such contracts even further.

Therefore arises the need for software architects, testers and developers to write tests to verify if an exposed API follows such a specified contract.

In the following tutorial I will demonstrate a setup with Java, Maven, JUnit and the designated contract-testing-library, assertj-swagger that verifies the validity of such a contract exposed by a Spring Boot application against a local stored definition.

Testing Java Applications for Resilience by Simulating Network Problems with Toxiproxy, JUnit and the Docker Maven Plugin

Sun, 29 Jul 2018 00:00:00 +0200

When implementing distributed systems, client-server architectures and simple applications with network related functionalities, everything is fine when we’re in the development or in the testing stage because the network is reliable and the communicating systems are not as stressed as they are in production.

But to sleep well we want to validate how resilient we have implemented our systems, how they behave when the network fails, the latency rises, the bandwidth is limited, connections time out and so on.

Setting up Kafka Brokers for Testing with Kafka-Unit

Wed, 28 Mar 2018 00:00:00 +0200

When writing test for applications that interact with Kafka brokers we often need to setup a decent environment including an instance of Kafka and ZooKeeper.

Though Kafka ships with some classes for testing, setting up a simple testing environment is quite easier with the kafka-unit library that offers JUnit test rule support or a fast programmatic setup within no time.

In the following short example, I’d like to show how create a simple setup using Maven, Surefire and kafka-unit.

Implementing, Testing and Running Procedures for Neo4j

Tue, 27 Feb 2018 00:00:00 +0100

A lot of features are already included in the Neo4j graph database system but sometimes we want to extends its capabilities and implement functions and procedures by ourselves that we may reuse.

In the following tutorial I will demonstrate how to implement a procedure for Neo4j, how to write and run tests using JUnit and an embedded graph database and last but not least how to setup Neo4j with Docker and our stored procedure installed in no time.

Software Architecture Exploration and Validation with jqAssistant, Neo4j and Cypher

Sun, 31 Dec 2017 00:00:00 +0100

I have written about other software system analyzing and validation tools before but today I would like to introduce a new tool named jqAssistant that supports software architects, developers and analysts in a variety of tasks like analyzing given structures, validating architectural or quality constraints and generating reports.

Therefore jqAssistant analyzes given projects or artifacts and stores the gathered information – that is enriched by a variety of existing plugin-ins – in a Neo4j graph database.

Using JUnit 5 Parameterized Tests, Argument Sources and Converters

Sat, 19 Aug 2017 00:00:00 +0200

With JUnit 5 the possibilities to write parameterized tests have changed and improved a lot.

The following short overview covers all new types of possible parameter sources for JUnit 5 tests as well as the new conversion API for test arguments.

In addition we’re showing how parameterized tests were written in JUnit 4.

Figure 1. Running JUnit5 in IntelliJ

About

We will be covering all available types of parameter sources in the following sections – all that you need as a prerequisite is Java ™, Maven and a few minutes of your time.

Assuring Architectural Rules with ArchUnit

Mon, 03 Jul 2017 00:00:00 +0200

Maintaining architecture rules and constraints for a specific software project or an application is not easy as textual documentation is easily forgotten after a while and hard to verify.

ArchUnit is a testing library that allows developers and software architects to write down such rules as executable tests that may be run by the development teams and the integration servers.

In the following article I will demonstrate the basic features of this library by applying rules and constraints to an existing application.

Template Driven Test POJO Generation with Fixture Factory and Java

Tue, 20 Jun 2017 00:00:00 +0200

In our tests we often need to create a bunch of test-objects that are populated with random-data. This data needs to follow specific rules as identifiers need to be unique or must be incremented, string-properties must follow special conventions and so on. In the following short tutorial I will demonstrate how to generate such test data using the Fixture Factory library.

Figure 1. Fixture Factory and JUnit

Writing BDD-Style Webservice Tests with Karate and Java

Thu, 06 Apr 2017 00:00:00 +0200

There is a new testing framework out there called Karate that is build on top of the popular Cucumber framework.

Karate makes it easy to script interactions with out web-services under test and verify the results. In addition it offers us a lot of useful features like parallelization, script/feature re-use, data-tables, JsonPath and XPath support, gherkin syntax, switchable staging-configurations and many others.

In the following tutorial we’ll be writing different scenarios and features for a real-world RESTful web-service to demonstrate some of its features.

Handling System Properties, Environment Variables, STDOUT/STDERR in JUnit Tests with System Rules

Mon, 19 Dec 2016 00:00:00 +0100

When important data is written to STDIN/STDOUT and an application relies on specific system properties or environment variables, writing tests is getting more complicated.

System Rules is a collection of JUnit rules that helps us writing Java tests for everything that deals with java.lang.System.

In the following short examples I’d like to demonstrate how to deal with system properties, environment variables, STDOUT and STDERR and capturing both for testing e.g. for some golden master refactoring.

Elasticsearch Integration Testing with Java

Tue, 23 Aug 2016 00:00:00 +0200

When building up search engines, indexing tons of data into a schema-less, distributed data store, Elasticsearch has always been a favourite tool of mine.

In addition to its core features, it also offers tools and documentation for us developers when we need to write integration tests for our Elasticsearch powered Java applications.

In the following tutorial I’d like to demonstrate how to implement a small sample application using Elasticsearch under the hood and how to write integration-tests with these tools for this application afterwards.

LDAP Testing with Java: ApacheDS vs Embedded-LDAP-JUnit

Mon, 04 Jul 2016 00:00:00 +0200

When writing applications that interchange information with LDAP directory services there is always the need to write integration tests for these components and services.

Therefore we need a the possibility to start-up an embedded LDAP server, fill it with test-data and control its life-cycle during the test-phases.

In the following tutorial I’d like to demonstrate two candidates that fulfil this purpose, the ApacheDS test integrations and a small library named embedded-ldap-junit.

Layout Testing with Galen, JUnit and Maven

Mon, 16 May 2016 00:00:00 +0200

Writing tests not only to verify the behaviour of a web site but also the correctness of its layout especially for responsive websites is not always easy.

Luckily the Galen Framework eases the task of writing layout tests for us, offering a specialized domain-specific-language to write layout-specifications, it integrates well with Selenium Grid, Sauce Labs or BrowserStack, it offers an easy way to deal with different browser sizes and responsive designs and it generates nice, detailed test reports.

Generating JUnit Tests with Java, EvoSuite and Maven

Sun, 28 Feb 2016 00:00:00 +0100

Generating test suites for existing code allows us to verify the behaviour of an application before we’re making changes to its code base or for regression testing.

In the following short tutorial I’d like to demonstrate how to derive test suites from an existing Java application using EvoSuite and the EvoSuite Maven plug-in.

EvoSuite offers some nice features like running in a sandbox to avoid dangerous operations, virtual file-system and network and optimizing of different coverage criteria.

Mocking HTTP Interaction with Java, JUnit and MockServer

Tue, 05 Jan 2016 00:00:00 +0100

When writing tests for our software components sometimes we need to mock external services based on the HTTP protocol, might it be a RESTful web-service, an XML-RPC call or a simple GET request to some web-server.

In the following short tutorial I’d like to demonstrate how to create a mock HTTP server for testing and how to bootstrap and bind it to the life-cycle of a classical build-management tool like Maven.

Testing Asynchronous Applications with Java and Awaitility

Sun, 23 Aug 2015 00:00:00 +0200

Writing tests for asynchronous applications has never been much fun as we’re always struggling with the problem how to determine state changes, handle process terminations, dealing with timeouts or failures and stuff like this.

Awaitility eases this process for us offering a nice DSL, rich support for languages like Scala or Groovy and an easy-to-use syntax that’s even more fun when using it with Java 8′s lambda expressions.

In the following short introduction I’d like to demonstrate writing some tests different scenarios.

Mutation Testing with Pitest and Maven

Sun, 10 May 2015 00:00:00 +0200

Mutation testing makes an interesting addition to the classical test coverage metrics.

They seed mutations (errors) into the code, run the project’s tests afterwards and if the tests fail, the mutation is killed – otherwise it lived and we have a possible indication of an issue with our tests.

In the following short tutorial. I’d like to demonstrate how to setup mutation tests with the PIT/Pitest library and Maven and generate reports.

Marrying Java EE and BDD with Cucumber, Arquillian and Cukespace

Wed, 07 Jan 2015 00:00:00 +0100

Having written about the basics of using Cucumber in a Java project in my last blog article, I now would like to demonstrate how to use a similar setup in a Java EE web project with Arquillian and the Cukespace library.

In the following tutorial, we’re going to write a full Java EE web application and add BDD-style tests to the project so that we’re able to test our business layer on the one hand and the user interface on the other hand using Arquillian Drone and Selenium.

Java EE: Setting up and Testing Form-Based JDBC Authentication with Arquillian and Maven

Sun, 21 Dec 2014 00:00:00 +0100

Especially when it comes to testing, setting up a decent environment for a secured Java EE web application isn’t always an easy thing to do.

In the following tutorial I’d like to demonstrate how to create a secured web application using form-based authentication and a JDBC realm to fetch users and roles and how to run the application in an embedded container for testing and development.

Additionally I’d like to show how to write and run integration tests to verify the security setup using a setup of Maven, Embedded GlassFish, Arquillian, jUnit and rest-assured.

Micro Benchmarking your Tests using JUnit and JUnitBenchmarks

Sun, 10 Mar 2013 00:00:00 +0100

I recently stumbled upon a nice framework that allows to convert simple JUnit tests into micro benchmarks named JUnitBenchmarks.

It allows to set basic benchmark options and and to generate charts by adding some simple annotations and a test rule to your tests.

One might argue if it is wise to mix the aspects, testing and benchmarking and I’d agree for sure – nevertheless I think this framework can be handy sometimes so let’s create some benchmarks using JUnit and JUnitBenchmarks..

Running categorized Tests using JUnit, Maven and Annotated-Test Suites

Thu, 06 Dec 2012 00:00:00 +0100

Sometimes we need to classify the tests in a project and a possible solution to achieve this goal is to assign different categories to the tests.

Often we’re doing this to separate the execution of fast-running and long-running tests or to run a specific set of tests that is only applicable in special situations.

To run a specific set of categorized tests there are different options and in the following tutorial we’ll be covering two of them: by configuring the Maven Surefire Plug-in or by using a JUnit Test Suite and the JUnit annotations.

New features in JUnit 4.11

Sun, 18 Nov 2012 00:00:00 +0100

JUnit is one of the most popular testing frameworks out there. Version 4.11 has just been released and offers some nice improvements that you shouldn’t miss.

Dependencies

In older versions of JUnit there were two dependencies .. junit:junit contained an old version of hamcrest and could cause some nasty trouble .. junit:junit-dep just referenced hamcrest the maven way.

Now with version 4.11 there is just junit:junit with clean references to hamcrest and junit:junit-dep is relocated to junit:junit.

Make your Tests more readable with custom Hamcrest Matchers

Sun, 28 Oct 2012 00:00:00 +0200

Everyday we’re writing tests for our software and sometimes we’re in a situation where we’re testing a specific type or object very often.

Luckily Hamcrest allows us to create custom matchers by subclassing from a given variety of available matchers.

Adding jUnit and Hamcrest

First add the dependencies for JUniti and Hamcrest to your project’s pom.xml or alternative build system.

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.10</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.hamcrest</groupId>
        <artifactId>hamcrest-all</artifactId>
        <version>1.3</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Selenium WebDriver, Selenium Server and PageObjects by Example

Tue, 06 Mar 2012 00:00:00 +0100

A lot has changed since Selenium RC and WebDriver has given us a new syntax to write tests for our web pages. PageObjects add an abstraction to the pages under test and finally we’re able to programmatically start Selenium server instances and use them to run the tests.

In the following tutorial, we’re writing some tests using PageObjects, WebDriver, Selenium Server and finally we’re even taking some screenshots of our tested web pages..

Ordering your JUnit Rules using a RuleChain

Tue, 21 Feb 2012 00:00:00 +0100

JUnit Rules are a handy solution if one needs to alter test methods or wants to share common functionality between several test cases. JUnit 4.10 introduced a new class to order several rules according to our needs using a so called rule-chain.

In the following example, we’re going to create a simple custom rule and afterwards bind several instances of it in a specified order to a test method.

Adding JUnit

Just one Maven dependency needed here – JUnit 4.10

REST-assured vs Jersey-Test-Framework: Testing your RESTful Web-Services

Mon, 05 Sep 2011 00:00:00 +0200

Today we’re going to take a look at two specific frameworks that enables you to efficiently test your REST-ful services: On the one side there is the framework REST-assured that offers a nice DSL-like syntax to create well readable tests – on the other side there is the Jersey-Test-Framework that offers a nice execution environment and is built upon the JAX-RS reference implementation, Jersey.

In the following tutorial we’re going to create a simple REST service first and then implement integration tests for this service using both frameworks.

Mocking, Stubbing and Test Spying using the Mockito Framework and PowerMock

Sun, 27 Mar 2011 00:00:00 +0100

Today we’re going to take a look at the Mockito framework that not only does sound like my favourite summer cocktail but also offers nice testing, mocking/stubbing, test-spying features and mock injections.

After that we’re going to take a look on how to mock static or final classes by extending Mockito’s capabilities with PowerMock.

Prerequisites

We don’t need much for the following samples .. Java of course, Maven dependency management and that’s all ..

Enterprise Java Bean EJB 3.1 Testing using Maven and embedded Glassfish

Sat, 01 Jan 2011 00:00:00 +0100

Are you playing around with the shiny new 3.1 EJB API?

Using Maven for your Java projects?

Need an easy way to write and execute tests for your EJBs that depends on an Java Application Server?

No problem using Maven Archetypes, the Maven EJB Plugin and the GlassFish embedded Application Container..

Prerequisites

For the following tutorial we’re going to need an installation of Maven and of course – the Java Development Kit!

Object-relational Mapping using Java Persistence API JPA 2

Mon, 11 Oct 2010 00:00:00 +0200

Today we’re going to take a look at the world of object-relational Mapping and how it is done using the Java Persistence API by creating some basic examples, mapping some relations and querying objects using JPQL or the Criteria API..

Prerequisites

Java 6 JDK
Maven >= 2
If you’d like to take a look behind the scenes e.g. how entities are mapped in your database you could install a RDBMS of your choice .. or just use Derby/JavaDB that is bundled with the JDK 6

Eclipse Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Favorites

Spare my time when using static imports ..

Window > Preferences > Java > Editor > Content Assist > Favorites:

com.google.common.collect.Lists
com.jayway.restassured.matcher.RestAssuredMatchers
com.jayway.restassured.RestAssured
io.restassured.matcher.RestAssuredMatchers
io.restassured.RestAssured
org.hamcrest.MatcherAssert
org.hamcrest.Matchers
org.mockito.Mockito
javaslang.API
javaslang.Predicates

Template to insert a static logger instance

Go Windows > Preferences > Java > Editor > Templates > New …

Enter logger as name and as template:

${:import(org.slf4j.Logger,org.slf4j.LoggerFactory)}
private static final Logger LOG = LoggerFactory.getLogger(${enclosing_type}.class);

Afterwards you’re able to type logger in your code and ctrl+space gives the option to insert the logger

IntelliJ Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Live Templates

JUnit 5 Test

@org.junit.jupiter.api.Test

@org.junit.jupiter.api.DisplayName("$NAME$")

void $METHOD$() throws Exception {

    $END$

}

Figure 1. IntelliJ Live Template Editor

Edit Variables:

NAME
METHOD: default-value camelCase(NAME)

Figure 2. Editing template variables

SLF4J Logger Template

private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger( $CLASS$.class );

Edit variables:

CLASS: expression: className()

Favorite Shortcuts

Select whole block in Editor

Cmd+Tab+Down

Select similar blocks in editor

Ctrl+G

Copy the absolute Path of the current File into Clipboard

Cmd+Shift+C

Java Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Remote Debug a Pod’s Java Process

Simple steps for remote debugging a Java process running on a k8 pod:

Edit deployment and add the following parameters to the Java start line: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=127.0.0.1:5005

Also add the following port mapping at the section container → ports in the deployment:
```
- containerPort: 5005
  protocol: TCP
```
Safe, wait for the new pods and then add a port forward for port 5005 for this pod:
```
kubectl port-forward podname 5005
```

Spring Boot Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Define and Configure Log Groups

This allows to configure a group of loggers at the same time

Define a log group named myaspect with two packages
application.properties
```
logging.group.myaspect=com.hascode.package1,com.hascode.package2
```
Configure the log group and set all loggers to level TRACE
application.properties
```
logging.level.myaspect=TRACE
```

This is also possible as parameter on startup

java -Dlogging.level.myaspect=TRACE myapp.jar

Use JUnit 5 with Spring Boot

Use newer versions of Surefire and Failsafe plugins:

<properties>
[..]
  <maven-failsafe-plugin.version>2.22.0</maven-failsafe-plugin.version>
  <maven-surefire-plugin.version>2.22.0</maven-surefire-plugin.version>
</properties>

Junit on Micha Kops' Tech Notes

Orchestrator Agent with Parallel Sub-Agents in Kiro CLI

Overview

Architecture

Step 1: Define Specialized Agents

Embedded Kafka for Spring Boot Testing without using Docker

Setup

JUnit5 Java Maven Snippet

Setup

Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.

Testing OpenAPI Swagger Schema Compliance with Java, JUnit and assertj-swagger

Testing Java Applications for Resilience by Simulating Network Problems with Toxiproxy, JUnit and the Docker Maven Plugin

Setting up Kafka Brokers for Testing with Kafka-Unit

Implementing, Testing and Running Procedures for Neo4j

Software Architecture Exploration and Validation with jqAssistant, Neo4j and Cypher

Using JUnit 5 Parameterized Tests, Argument Sources and Converters

About

Assuring Architectural Rules with ArchUnit

Template Driven Test POJO Generation with Fixture Factory and Java

Writing BDD-Style Webservice Tests with Karate and Java

Handling System Properties, Environment Variables, STDOUT/STDERR in JUnit Tests with System Rules

Elasticsearch Integration Testing with Java

LDAP Testing with Java: ApacheDS vs Embedded-LDAP-JUnit

Layout Testing with Galen, JUnit and Maven

Generating JUnit Tests with Java, EvoSuite and Maven

Mocking HTTP Interaction with Java, JUnit and MockServer

Testing Asynchronous Applications with Java and Awaitility

Mutation Testing with Pitest and Maven

Marrying Java EE and BDD with Cucumber, Arquillian and Cukespace

Java EE: Setting up and Testing Form-Based JDBC Authentication with Arquillian and Maven

Micro Benchmarking your Tests using JUnit and JUnitBenchmarks

Running categorized Tests using JUnit, Maven and Annotated-Test Suites

New features in JUnit 4.11

Dependencies

Make your Tests more readable with custom Hamcrest Matchers

Adding jUnit and Hamcrest

Selenium WebDriver, Selenium Server and PageObjects by Example

Ordering your JUnit Rules using a RuleChain

Adding JUnit

REST-assured vs Jersey-Test-Framework: Testing your RESTful Web-Services

Mocking, Stubbing and Test Spying using the Mockito Framework and PowerMock

Prerequisites

Enterprise Java Bean EJB 3.1 Testing using Maven and embedded Glassfish

Prerequisites

Object-relational Mapping using Java Persistence API JPA 2

Prerequisites

Eclipse Snippets

Favorites

Template to insert a static logger instance

IntelliJ Snippets

Live Templates

JUnit 5 Test

SLF4J Logger Template

Favorite Shortcuts

Select whole block in Editor

Select similar blocks in editor

Copy the absolute Path of the current File into Clipboard

Java Snippets

Remote Debug a Pod’s Java Process

Spring Boot Snippets

Define and Configure Log Groups

Use JUnit 5 with Spring Boot