Container on Micha Kops' Tech Notes

Building OCI Images with Buildah

Sun, 13 Oct 2024 00:00:00 +0200

Figure 1. Cover Image: Building OCI images with Buildah

In the world of containers, Buildah stands out as a powerful yet lightweight tool for building OCI (Open Container Initiative)-compliant container images. Unlike traditional tools like Docker, Buildah takes a daemonless approach, making it secure, flexible, and ideal for modern development workflows.

Key Features

Daemonless Architecture: Buildah doesn’t rely on a background service (like Docker’s daemon). Instead, it directly manipulates images, ensuring a smaller footprint and lower resource usage.
Rootless and Secure: You can run Buildah without root privileges, making it safer, especially in multi-user environments or CI/CD pipelines.
Dockerfile-Free Flexibility: While Buildah supports Dockerfiles, it also enables image creation without one. Developers can use shell commands or scripts, offering full control over the image-building process.
OCI Compliance and Versatility: Buildah creates both OCI and Docker-formatted images, ensuring compatibility with tools like Podman, Docker, and Kubernetes.
Lightweight and Scriptable: Its minimal design makes Buildah perfect for automation and scripting, particularly in resource-constrained environments.

C4 Modeling with PlantUML and AsciiDoc

Tue, 01 Nov 2022 00:00:00 +0100

C4 models allow us to visualize software architecture by decomposition in containers and components.

Viewpoints are organized in hierarchical levels:

Context Diagrams (Level 1)
Container Diagrams (Level 2)
Component Diagrams (Level 3)
Code Diagrams (Level 4)

C4-PlantUML offers a variety of macros and stereotypes that make modeling fun.

An example in PlantUML:

sample.puml

@startuml
!include <c4/C4_Context.puml>
!include <c4/C4_Container.puml>

left to right direction

Person(user, "User")

System_Ext(auth, "AuthService", "Provides authentication and authorization via OIDC")

System_Boundary(zone1, "Some system boundary") {


    System(lb, "Load Balancer")

    System_Boundary(az, "App Cluster") {
        System(app, "App Servers") {
            Container(app1, "App1", "Docker", "Does stuff")
            Container(app2, "App1", "Docker", "Does stuff")

            ContainerDb(dbSess, "Session DB", "Redis")
            ContainerDb(db1, "RBMS 1", "AWS RDS Postgres")
            ContainerDb(db2, "RBMS 2", "AWS RDS Postgres")

            ' both app servers sync sessions via redis
            Rel(app1, dbSess, "Uses", "Sync Session")
            Rel(app2, dbSess, "Uses", "Sync Session")

            ' both app servers persist data in RDBMS
            Rel(app1, db1, "Uses", "Persist/query relational data")
            Rel(app2, db2, "Uses", "Persist/query relational data")
        }
    }
}

Rel(user, lb, "call")
Rel(lb, app1, "delegate")
Rel(lb, app2, "delegate")
Rel(app1, auth, "Verify", "User auth")
Rel(app2, auth, "Verify", "User auth")

SHOW_FLOATING_LEGEND()

@enduml

Fast Link Checks using Filiph Linkchecker and Docker

Mon, 02 May 2022 00:00:00 +0200

This Linkchecker claims to be way faster than blc and wummel/linkchecker.

Using Docker, we may validate our site’s links in no time and without complex setup.

docker run --rm tennox/linkcheck hascode.io

Resources

https://github.com/filiph/linkcheck

Setting up multiple Postgresql Instances with docker-compose

Tue, 08 Feb 2022 00:00:00 +0100

A simple setup when two Postgres databases prefilled with schema/data needed.

docker-compose.yml

version: '3.6'
services:
    postgres1:
        image: postgres
        restart: always
        environment:
            - DATABASE_HOST=127.0.0.1
            - POSTGRES_USER=root
            - POSTGRES_PASSWORD=root
            - POSTGRES_DB=root
        ports:
            - "15432:15432"
        volumes:
            - ./postgres1-init.sql:/docker-entrypoint-initdb.d/docker_postgres_init.sql

    postgres2:
        image: postgres
        restart: always
        environment:
            - DATABASE_HOST=127.0.0.1
            - POSTGRES_USER=root
            - POSTGRES_PASSWORD=root
            - POSTGRES_DB=root

        ports:
            - "25432:25432"
        volumes:
            - ./postgres2-init.sql:/docker-entrypoint-initdb.d/docker_postgres_init.sql

And our sample init scripts:

CREATE USER tester WITH PASSWORD 'tester' CREATEDB;
CREATE DATABASE testdb
    WITH
    OWNER = tester
    ENCODING = 'UTF8'
    LC_COLLATE = 'en_US.utf8'
    LC_CTYPE = 'en_US.utf8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

Kafka Java Quickstart with Docker

Sat, 29 Jan 2022 00:00:00 +0100

Goals

Setup Kafka and Zookeeper with Docker and docker-compose
Create a message consumer and producer in Java

Kafka Setup

We’re using docker-compose to set up our message broker, zookeper and other stuff using confluent-platform.

This is our docker-compose.yaml config file from Confluent’s following GitHub repository.

docker-compose.yaml

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

  schema-registry:
    image: confluentinc/cp-schema-registry:7.0.1
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

  connect:
    image: cnfldemos/kafka-connect-datagen:0.5.0-6.2.0
    hostname: connect
    container_name: connect
    depends_on:
      - broker
      - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
      CONNECT_REST_ADVERTISED_HOST_NAME: connect
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
      CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR

  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:7.0.1
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
      - connect
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "broker:29092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect:8083"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  ksqldb-cli:
    image: confluentinc/cp-ksqldb-cli:7.0.1
    container_name: ksqldb-cli
    depends_on:
      - broker
      - connect
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

  ksql-datagen:
    image: confluentinc/ksqldb-examples:7.0.1
    hostname: ksql-datagen
    container_name: ksql-datagen
    depends_on:
      - ksqldb-server
      - broker
      - schema-registry
      - connect
    command: "bash -c 'echo Waiting for Kafka to be ready... && \
                       cub kafka-ready -b broker:29092 1 40 && \
                       echo Waiting for Confluent Schema Registry to be ready... && \
                       cub sr-ready schema-registry 8081 40 && \
                       echo Waiting a few seconds for topic creation to finish... && \
                       sleep 11 && \
                       tail -f /dev/null'"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      STREAMS_BOOTSTRAP_SERVERS: broker:29092
      STREAMS_SCHEMA_REGISTRY_HOST: schema-registry
      STREAMS_SCHEMA_REGISTRY_PORT: 8081

  rest-proxy:
    image: confluentinc/cp-kafka-rest:7.0.1
    depends_on:
      - broker
      - schema-registry
    ports:
      - 8082:8082
    hostname: rest-proxy
    container_name: rest-proxy
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'

Install Docker on Linux

Fri, 14 May 2021 00:00:00 +0200

Goals

Installing a specific Docker version on (Debian-based) Linux
Freeze the version to avoid automatic updates

Installation

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - (1)

sudo add-apt-repository \ (2)
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get update (3)

sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu (4)

sudo apt-mark hold docker-ce (5)

1	Add the Docker GPG key
2	Add the Docker repository
3	Update the index
4	Install docker
5	Freeze the version to avoid unwanted automatic updates

MySQL and phpMyAdmin Setup with Docker-Compose

Fri, 14 May 2021 00:00:00 +0200

Goals

Setup mySQL with phpMyAdmin connected using docker-compose (for development purpose)

Prerequisites

docker-compose installed

Setup

This is our docker-compose.yml:

version: '3.2'

services:
   db:
      image: mysql:8.0
      container_name: mysql-container
      restart: always
      ports:
       - '6603:3306'
      environment:
        MYSQL_ROOT_PASSWORD: 12345678

   app:
      depends_on:
       - db
      image: phpmyadmin/phpmyadmin
      container_name: phpmyadmin
      restart: always
      ports:
       - '8080:80'
      environment:
        PMA_HOST: db

Running

docker-compose up
Starting mysql-container ... done
Starting phpmyadmin      ... done
Attaching to mysql-container, phpmyadmin
[..]

We may now login using the following ultra-secure credentials ;) User: root, Password: 12345678

Setting up a Kubernetes Master Node

Fri, 14 May 2021 00:00:00 +0200

Goals

Setup a kubernetes master node on a Linux machine

Setup

Initialize the cluster on the master node

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

This might take a few minutes … afterward we set up our local kubeconfig:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Verify the Cluster Setup

Shows that the cluster is responding and kubectl working:

kubectl version

WordPress Docker Setup

Fri, 14 May 2021 00:00:00 +0200

Goals

Run WordPress via Docker / Docker-Compose
Increase the Upload Filesize Limit

Create Docker Compose Configuration

Create a docker-compose.yml:

version: '3.1'

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress:/var/www/html

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

volumes:
  wordpress:
  db:

Run Docker Compose / Start Containers

docker-compose up
WARNING: Found orphan containers (wordpress-docker_phpmyadmin_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting wordpress-docker_db_1        ... done
Starting wordpress-docker_wordpress_1 ... done
Attaching to wordpress-docker_db_1, wordpress-docker_wordpress_1
[..]
db_1         | 2021-04-03T18:58:17.247963Z 0 [Note] mysqld: ready for connections.
db_1         | Version: '5.7.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

Annotation based Kubernetes and Openshift Manifests for Java Applications with ap4k

Thu, 28 Feb 2019 00:00:00 +0100

Writing our manifest files for Kubernetes / Openshift often forces us to edit xml, json and yml files by hand.

A new library, ap4k allows to specify metadata for these manifest files directly in our Java code using annotations.

In the following short example I am going to demonstrate how to generate manifest files using Maven and ap4k.

Figure 1. ap4k Tutorial

Dependencies

Using Maven we just need to add the following one dependency to our project’s pom.xml

Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.

Wed, 30 Jan 2019 00:00:00 +0100

A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.

Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.

Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.

Testing Java Applications for Resilience by Simulating Network Problems with Toxiproxy, JUnit and the Docker Maven Plugin

Sun, 29 Jul 2018 00:00:00 +0200

When implementing distributed systems, client-server architectures and simple applications with network related functionalities, everything is fine when we’re in the development or in the testing stage because the network is reliable and the communicating systems are not as stressed as they are in production.

But to sleep well we want to validate how resilient we have implemented our systems, how they behave when the network fails, the latency rises, the bandwidth is limited, connections time out and so on.

Docker Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Restrict Network

Can be useful when using a third-party image that we do not trust

Run with no network

docker run --network none <image>

Run with private isolated network

At least containers attached to this network can talk with another

docker network create --internal my_isolated_network
docker run --network my_isolated_network <image>

Block using firewall

e.g. using iptables or ipfw

# Get container's IP
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container_name>

# Block all outbound connections from that IP
sudo iptables -I DOCKER-USER -s <container_ip> -j DROP

Helm Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Common operations

Add Helm Repository

helm repo add NAME URL

e.g. for the Bitnami repository:

helm repo add bitnami https://charts.bitnami.com/bitnami                                                                                                                                                         1 ↵
"bitnami" has been added to your repositories

List Repositories

helm repo list                                                                                                                                                                                                 130 ↵
NAME    URL
bitnami https://charts.bitnami.com/bitnami

Searching in a Helm Repository

helm search repo wordpress
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/wordpress       15.2.30         6.1.1           WordPress is the world's most popular blogging ...
bitnami/wordpress-intel 2.1.31          6.1.1           DEPRECATED WordPress for Intel is the most popu...

Kafka Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Start an Image with kcat / kafka-cat for Debugging

kubectl -n NAMESPACE run "$(whoami)-debug" -it --rm \
    --image=confluentinc/cp-kafkacat:6.1.9 \
    --restart=Never \
    -- bash

Dockerfile for Kafka Analysis Container with different Tools

With jq, kafka console tools, schema registry tools and kafkacat installed ….

Dockerfile

FROM confluentinc/cp-kafka:6.2.1 as cp-kafka
FROM confluentinc/cp-schema-registry:6.2.1 as cp-schema-registry

FROM debian:10-slim
ARG DEBIAN_FRONTEND=noninteractive

# Install necessary tools
RUN apt-get update && apt-get install -y \
    curl \
    jq \
    yq \
    && rm -rf /var/lib/apt/lists/*

# Install kafkacat binary
RUN apt-get update && apt-get install -y kafkacat && rm -rf /var/lib/apt/lists/*

# Copy Kafka binaries
COPY --from=cp-kafka /usr/bin/kafka-* /usr/bin/
COPY --from=cp-schema-registry /usr/bin/schema-registry* /usr/bin/

# Copy entrypoint script
COPY entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh

ENTRYPOINT ["/usr/bin/entrypoint.sh"]

Postgres Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Administration & Configuration

Connections

Show Max Connections (value and source)

SELECT setting, source, sourcefile, sourceline
FROM pg_settings
WHERE name = 'max_connections';

Set Max Connections

ALTER system SET max_connections = 250;

Kill Connections for a Database

SELECT pg_terminate_backend(pid)
FROM pg_catalog.pg_stat_activity
-- we don't want to kill our own connection
WHERE pid != pg_backend_pid()
-- we don't want to kill connections to other databases
AND datname = 'MYDATABASE'

Show Statement Timeout Settings for All Users

SELECT r.rolname,
       (SELECT unnest(s.setconfig) FROM pg_db_role_setting s WHERE s.setrole = r.oid AND s.setconfig::text LIKE '%statement_timeout%' LIMIT 1) AS statement_timeout
FROM pg_roles r
WHERE r.rolcanlogin
ORDER BY r.rolname;