Kafka Java Quickstart with Docker

Goals Setup Kafka and Zookeeper with Docker and docker-compose Create a message consumer and producer in Java Kafka Setup We’re using docker-compose to set up our message broker, zookeper and other stuff using confluent-platform. This is our docker-compose.yaml config file from Confluent’s following GitHub repository. docker-compose.yaml --- version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:7.0.1 hostname: zookeeper container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-kafka:7.0.1 hostname: broker container_name: broker depends_on: - zookeeper ports: - "29092:29092" - "9092:9092" - "9101:9101" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_JMX_PORT: 9101 KAFKA_JMX_HOSTNAME: localhost schema-registry: image: confluentinc/cp-schema-registry:7.0.1 hostname: schema-registry container_name: schema-registry depends_on: - broker ports: - "8081:8081" environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092' SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 connect: image: cnfldemos/kafka-connect-datagen:0.5.0-6.2.0 hostname: connect container_name: connect depends_on: - broker - schema-registry ports: - "8083:8083" environment: CONNECT_BOOTSTRAP_SERVERS: 'broker:29092' CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000 CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081 CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components" CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR ksqldb-server: image: confluentinc/cp-ksqldb-server:7.0.1 hostname: ksqldb-server container_name: ksqldb-server depends_on: - broker - connect ports: - "8088:8088" environment: KSQL_CONFIG_DIR: "/etc/ksql" KSQL_BOOTSTRAP_SERVERS: "broker:29092" KSQL_HOST_NAME: ksqldb-server KSQL_LISTENERS: "http://0.0.0.0:8088" KSQL_CACHE_MAX_BYTES_BUFFERING: 0 KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081" KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor" KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor" KSQL_KSQL_CONNECT_URL: "http://connect:8083" KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1 KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true' KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true' ksqldb-cli: image: confluentinc/cp-ksqldb-cli:7.0.1 container_name: ksqldb-cli depends_on: - broker - connect - ksqldb-server entrypoint: /bin/sh tty: true ksql-datagen: image: confluentinc/ksqldb-examples:7.0.1 hostname: ksql-datagen container_name: ksql-datagen depends_on: - ksqldb-server - broker - schema-registry - connect command: "bash -c 'echo Waiting for Kafka to be ready... && \ cub kafka-ready -b broker:29092 1 40 && \ echo Waiting for Confluent Schema Registry to be ready... && \ cub sr-ready schema-registry 8081 40 && \ echo Waiting a few seconds for topic creation to finish... && \ sleep 11 && \ tail -f /dev/null'" environment: KSQL_CONFIG_DIR: "/etc/ksql" STREAMS_BOOTSTRAP_SERVERS: broker:29092 STREAMS_SCHEMA_REGISTRY_HOST: schema-registry STREAMS_SCHEMA_REGISTRY_PORT: 8081 rest-proxy: image: confluentinc/cp-kafka-rest:7.0.1 depends_on: - broker - schema-registry ports: - 8082:8082 hostname: rest-proxy container_name: rest-proxy environment: KAFKA_REST_HOST_NAME: rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092' KAFKA_REST_LISTENERS: "http://0.0.0.0:8082" KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081' ...

January 29, 2022 · 8 min · 1500 words · Micha Kops

Logback and Spring Boot - Change Log Level to custom format

Create a custom converter This class converts the well known log levels to a custom format CustomLogLevelConverter.java package com.hascode; public class CustomLogLevelConverter extends ClassicConverter { @Override public String convert(ILoggingEvent event) { switch (event.getLevel().toInt()) { case Level.ERROR_INT: return "ERROR!!!"; case Level.WARN_INT: return "WARN!!"; case Level.INFO_INT: return "INFO!"; case Level.TRACE_INT: return "DEBUG"; default: return event.getLevel().toString(); } } } Register the converter The following Logback config includes some defaults and registers our custom converter. ...

August 19, 2021 · 1 min · 165 words · Micha Kops

Factory Reset for Google Pixel-C Android Tablet

Steps Reboot the system by pressing "Power" and "Volume down" simultaneously Boot menu should appear, use "Volume up/down" to select the menu item "Android Recovery", "Power" to confirm System reboots, a screen appears with a message "No command" - this is no error though it looks like one Press "Power" and "Volume up" together and a menu "Android Recovery" appears Use "Volume up/down" to select the menu item "Wipe data/factory reset", "Power" to confirm User data on the device is deleted and the original meu is shown Select the menu item "Reboot system now" and confirm by pressing the "Power" Button The device reboots and you may configure the Android system …​ ...

June 13, 2021 · 1 min · 112 words · Micha Kops

Writing a React Component Test with Jest and Testing Library

Goals render a React component in a test mock HTTP/REST calls to the backend verify results achieve the above using typescript, jest and testing-library Application under Test This is our application’s shortened package.json, generated by create-react-app, adding dependencies for …​ jest testing-library typescript react dom implementations (jest-dom/react-dom) axios (for the HTTP/REST call) package.json { [..] "dependencies": { "@testing-library/jest-dom": "^5.11.4", "@testing-library/react": "^11.1.0", "@testing-library/user-event": "^12.1.10", "@types/jest": "^26.0.15", "@types/node": "^12.0.0", "@types/react": "^17.0.0", "@types/react-dom": "^17.0.0", "react": "^17.0.2", "react-dom": "^17.0.2", "react-scripts": "4.0.3", "typescript": "^4.1.2", "web-vitals": "^1.0.1", "axios": "^0.19.0" }, [..] "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, [..] } ...

June 4, 2021 · 3 min · 559 words · Micha Kops

Configure HP Printer for Linux

Goals Installation We just need to install hplibs thats all .. sudo apt install hplip hplip-gui xsane hp-setup (1) hp-plugin 1 Fastest way is to enter the printer’s IP address, for network scans it might be necessary to disable/modify the firewall. Resources HP Linux Imaging and Printing Detailed article from Linuxmint forums

May 14, 2021 · 1 min · 52 words · Micha Kops

GitHub Release Pipeline for Java

Goals Set up Maven build pipeline for a Java 11 app Release Maven artifact on GitHub using GitHub actions Setup Maven Assuming that we have a project named sample-app released for my hascode GitHub account: We’re adding some release information to our project’s pom.xml: pom.xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.hascode</groupId> <artifactId>sample-app</artifactId> <version>1.0.0-SNAPSHOT</version>bookmark-manager <name>sample-app</name> <description>hasCode.com Bookmark Manager</description> <scm> <developerConnection>scm:git:https://github.com/hascode/sample-app.git </developerConnection> </scm> <distributionManagement> <repository> <id>github</id> <name>GitHub</name> <url>https://maven.pkg.github.com/hascode/sample-app</url> </repository> </distributionManagement> <properties> <java.version>11</java.version> <project.scm.id>github</project.scm.id> </properties> [..] </project> ...

May 14, 2021 · 2 min · 398 words · Micha Kops

Install Docker on Linux

Goals Installing a specific Docker version on (Debian-based) Linux Freeze the version to avoid automatic updates Installation curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - (1) sudo add-apt-repository \ (2) "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update (3) sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu (4) sudo apt-mark hold docker-ce (5) 1 Add the Docker GPG key 2 Add the Docker repository 3 Update the index 4 Install docker 5 Freeze the version to avoid unwanted automatic updates ...

May 14, 2021 · 1 min · 86 words · Micha Kops

Install Kubernetes Components - Kubeadm, Kubectl, Kubelet

Goals Install kubeadm, kubectl and kubelet on Debian-based Linux Freeze their versions to avoid automatic updates Installation curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - (1) cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list (2) deb https://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update (3) sudo apt-get install -y kubelet=1.15.7-00 kubeadm=1.15.7-00 kubectl=1.15.7-00 (4) sudo apt-mark hold kubelet kubeadm kubectl (5) 1 Add the GPG key 2 Add the kubernetes repo to the sources list 3 Update the index 4 Install kubelet, kubeadm and kubectl It’s important to use the same version for kubelet, kubeadm and kubectl. ...

May 14, 2021 · 1 min · 117 words · Micha Kops

JUnit5 Java Maven Snippet

Goals Use JUnit Maven BOM for version alignment Add minimal dependencies for JUnit5 Java Projekt Setup Excerpt from the Maven project’s pom.xml: <dependencyManagement> <dependencies> <dependency> <groupId>org.junit</groupId> <artifactId>junit-bom</artifactId> <version>5.7.1</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependencies> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <scope>test</scope> </dependency> </dependencies> JUnit5 does not work with older versions of the Maven Compiler Plugin and the Surefire Plugin used for test execution. Setting their versions in the pom.xml is done like this: ...

May 14, 2021 · 1 min · 85 words · Micha Kops

MySQL and phpMyAdmin Setup with Docker-Compose

Goals Setup mySQL with phpMyAdmin connected using docker-compose (for development purpose) Prerequisites docker-compose installed Setup This is our docker-compose.yml: version: '3.2' services: db: image: mysql:8.0 container_name: mysql-container restart: always ports: - '6603:3306' environment: MYSQL_ROOT_PASSWORD: 12345678 app: depends_on: - db image: phpmyadmin/phpmyadmin container_name: phpmyadmin restart: always ports: - '8080:80' environment: PMA_HOST: db Running docker-compose up Starting mysql-container ... done Starting phpmyadmin ... done Attaching to mysql-container, phpmyadmin [..] We may now login using the following ultra-secure credentials ;) User: root, Password: 12345678 ...

May 14, 2021 · 1 min · 81 words · Micha Kops