Schema on Micha Kops' Tech Notes

Avro Schema Compatibility and the Confluent Schema Registry

Sun, 08 Jun 2025 00:00:00 +0200

What is Schema Compatibility?

Schema compatibility ensures that data serialized by a producer using one schema can be deserialized by a consumer using another. The Confluent Schema Registry provides several compatibility modes to enforce these rules when updating schemas.

There are two key directions of compatibility:

Backward Compatibility (Upward Compatibility): Old consumers can read data produced with a newer schema.
Forward Compatibility (Downward Compatibility): New consumers can read data produced with an older schema.

Quick Kafdrop Setup with Helm Charts

Thu, 10 Aug 2023 00:00:00 +0200

Figure 1. Kafdrop Topic Viewer

In the ever-expanding world of data streaming and event-driven architecture, Apache Kafka has emerged as a cornerstone for reliable and scalable data processing. However, managing and monitoring Kafka clusters can often present its own set of challenges. This is where Kafdrop, a web-based Kafka consumer group and topic viewer, comes to the rescue. With its intuitive interface and insightful visualizations, Kafdrop offers developers and operators an efficient way to gain valuable insights into Kafka clusters.

Converting XML Schema (XSD) to Protocol Buffers (Protobuf)

Wed, 02 Feb 2022 00:00:00 +0100

Sometimes one needs to derive a Google Protocol Buffers schema from an XML schema .. e.g. from an Enterprise Architect Export.

Tool used here: schema2proto (GitLab project)

Steps

Download schema2proto-lib from the global Maven repository: https://search.maven.org/search?q=schema2proto
Run Schema2Proto against a give XSD schema file and with a given output directory:

java -jar schema2proto-lib-1.53.jar Schema2Proto --outputDirectory=src/main/protobuf input.xsd

A yaml config file may be given instead of cli parameters:

java -jar schema2proto-lib-1.53.jar Schema2Proto ----configFile=config.yaml input.xsd

Testing OpenAPI Swagger Schema Compliance with Java, JUnit and assertj-swagger

Fri, 31 Aug 2018 00:00:00 +0200

The OpenAPI and Swagger API description format are becoming important standards to specify API contracts for RESTful web services and the Microservices trend pushes the need for such contracts even further.

Therefore arises the need for software architects, testers and developers to write tests to verify if an exposed API follows such a specified contract.

In the following tutorial I will demonstrate a setup with Java, Maven, JUnit and the designated contract-testing-library, assertj-swagger that verifies the validity of such a contract exposed by a Spring Boot application against a local stored definition.

Integrating Swagger into a Spring Boot RESTful Webservice with Springfox

Wed, 01 Jul 2015 00:00:00 +0200

Spring Boot allows us to create RESTful web-services with ease, Swagger specifies a format to describe the capabilities and operations of these services and with Swagger UI it is possible to explore our REST API with a nice graphical user interface in our browser.

Springfox is a project that aims at creating automated JSON API documentation for API’s built with Spring and is used in the following tutorial to integrate Swagger into a sample application.

A short Overview of Neo4j Indexing Strategies

Sun, 25 Jan 2015 00:00:00 +0100

When it comes to indexing in a Neo4j graph database, different options exist for a developer to create and maintain the index.

In the following short examples I’d like to demonstrate different possibilities for index management.

Figure 1. Simple Domain Model

Dependencies

Only one dependency is needed to run the following examples and start an embedded neo4j server – I’m using Gradle here to manage my dependencies but Maven, Ivy, SBT should work without a problem, too.

Using Apache Avro with Java and Maven

Sat, 08 Mar 2014 00:00:00 +0100

Apache Avro is a serialization framework similar to Google’s Protocol Buffers or Apache Thrift and offering features like rich data structures, a compact binary format, simple integration with dynamic languages and more.

In the following short five minute tutorial, we’re going to specify a schema to serialize books in a JSON format, we’re using the Avro Maven plugin to generate the stub classes and finally we’re serializing the data into a single file.

Contract-First Web-Services using JAX-WS, JAX-B, Maven and Eclipse

Tue, 23 Aug 2011 00:00:00 +0200

Using the contract-first approach to define a web service offers some advantages in contrast to the code-first approach.

In the following tutorial we’re going to take a look at some details of this approach and we’re going to implement a real SOAP service using JAX-WS, Maven and the Eclipse IDE.

Finally we’re going to run our service implementation on an embedded Jetty instance and we’re going to take a look at soapUI and how to test our service using this neat tool.

Kafka Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Start an Image with kcat / kafka-cat for Debugging

kubectl -n NAMESPACE run "$(whoami)-debug" -it --rm \
    --image=confluentinc/cp-kafkacat:6.1.9 \
    --restart=Never \
    -- bash

Dockerfile for Kafka Analysis Container with different Tools

With jq, kafka console tools, schema registry tools and kafkacat installed ….

Dockerfile

FROM confluentinc/cp-kafka:6.2.1 as cp-kafka
FROM confluentinc/cp-schema-registry:6.2.1 as cp-schema-registry

FROM debian:10-slim
ARG DEBIAN_FRONTEND=noninteractive

# Install necessary tools
RUN apt-get update && apt-get install -y \
    curl \
    jq \
    yq \
    && rm -rf /var/lib/apt/lists/*

# Install kafkacat binary
RUN apt-get update && apt-get install -y kafkacat && rm -rf /var/lib/apt/lists/*

# Copy Kafka binaries
COPY --from=cp-kafka /usr/bin/kafka-* /usr/bin/
COPY --from=cp-schema-registry /usr/bin/schema-registry* /usr/bin/

# Copy entrypoint script
COPY entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh

ENTRYPOINT ["/usr/bin/entrypoint.sh"]

Postgres Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Administration & Configuration

Connections

Show Max Connections (value and source)

SELECT setting, source, sourcefile, sourceline
FROM pg_settings
WHERE name = 'max_connections';

Set Max Connections

ALTER system SET max_connections = 250;

Kill Connections for a Database

SELECT pg_terminate_backend(pid)
FROM pg_catalog.pg_stat_activity
-- we don't want to kill our own connection
WHERE pid != pg_backend_pid()
-- we don't want to kill connections to other databases
AND datname = 'MYDATABASE'

Show Statement Timeout Settings for All Users

SELECT r.rolname,
       (SELECT unnest(s.setconfig) FROM pg_db_role_setting s WHERE s.setrole = r.oid AND s.setconfig::text LIKE '%statement_timeout%' LIMIT 1) AS statement_timeout
FROM pg_roles r
WHERE r.rolcanlogin
ORDER BY r.rolname;