Database on Micha Kops' Tech Notes

Postgres with docker-compose or Docker and pg_stat_statements enabled

Wed, 29 Mar 2023 00:00:00 +0200

pg_stat_statements is useful to gather performance information about queries so lets add it to our dockerized postgres database.

Using docker-compose

Using docker-compose we just need to add the following docker-compose.yaml:

docker-compose.yml

version: '3.5'

services:
  postgres:
    container_name: postgres_container
    image: postgres
    ports:
      - "5432:5432"
    volumes:
      - /var/lib/postgresql/data
    # used for query profiling, deactivate for enhanced performance
    command: postgres -c shared_preload_libraries=pg_stat_statements -c pg_stat_statements.track=all -c max_connections=200
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: thepassword

We simply start our Postgres database with docker-compose:

MySQL and phpMyAdmin Setup with Docker-Compose

Fri, 14 May 2021 00:00:00 +0200

Goals

Setup mySQL with phpMyAdmin connected using docker-compose (for development purpose)

Prerequisites

docker-compose installed

Setup

This is our docker-compose.yml:

version: '3.2'

services:
   db:
      image: mysql:8.0
      container_name: mysql-container
      restart: always
      ports:
       - '6603:3306'
      environment:
        MYSQL_ROOT_PASSWORD: 12345678

   app:
      depends_on:
       - db
      image: phpmyadmin/phpmyadmin
      container_name: phpmyadmin
      restart: always
      ports:
       - '8080:80'
      environment:
        PMA_HOST: db

Running

docker-compose up
Starting mysql-container ... done
Starting phpmyadmin      ... done
Attaching to mysql-container, phpmyadmin
[..]

We may now login using the following ultra-secure credentials ;) User: root, Password: 12345678

WordPress Docker Setup

Fri, 14 May 2021 00:00:00 +0200

Goals

Run WordPress via Docker / Docker-Compose
Increase the Upload Filesize Limit

Create Docker Compose Configuration

Create a docker-compose.yml:

version: '3.1'

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress:/var/www/html

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

volumes:
  wordpress:
  db:

Run Docker Compose / Start Containers

docker-compose up
WARNING: Found orphan containers (wordpress-docker_phpmyadmin_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting wordpress-docker_db_1        ... done
Starting wordpress-docker_wordpress_1 ... done
Attaching to wordpress-docker_db_1, wordpress-docker_wordpress_1
[..]
db_1         | 2021-04-03T18:58:17.247963Z 0 [Note] mysqld: ready for connections.
db_1         | Version: '5.7.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.

Wed, 30 Jan 2019 00:00:00 +0100

A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.

Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.

Testcontainers is a testing library that offers lightweight throwaway instances of anything able to run in a Docker container, with bindings to configure the specific containers and also provides wrappers to manage our own custom containers.

Using jetstreamDB as in-memory Database for Java

Sat, 30 Jun 2018 00:00:00 +0200

JetstreamDB is a in-memory database engine for Java that claims to be built for ultra-high speed and the ability of managing complex data structures by storing real Java objects instead of serializing data structures to other database specific formats.

In the following short example I would like to demonstrate how to create and read items from such a database by building a small article management sample app.

Creating and Providing HipChat Integrations with Atlassian Connect, Nodejs and Express

Tue, 18 Aug 2015 00:00:00 +0200

HipChat is Atlassian’s alternative to Slack and its solution to team collaboration chats. Atlassian Connect offers developer tools to bootstrap applications, connect to Atlassian’s cloud products with easy and in combination with HipChat’s REST APIs allows us to write integrations for such a chat server in no time.

In the following tutorial I’d like to show how to write an integration within a few steps using Atlassian Connect, Node.js and Express and how to connect the integration to a HipChat server.

A short Overview of Neo4j Indexing Strategies

Sun, 25 Jan 2015 00:00:00 +0100

When it comes to indexing in a Neo4j graph database, different options exist for a developer to create and maintain the index.

In the following short examples I’d like to demonstrate different possibilities for index management.

Figure 1. Simple Domain Model

Dependencies

Only one dependency is needed to run the following examples and start an embedded neo4j server – I’m using Gradle here to manage my dependencies but Maven, Ivy, SBT should work without a problem, too.

Java Persistence API: Controlling the Second-Level-Cache

Mon, 21 Apr 2014 00:00:00 +0200

Using the Java Persistence API and a decent persistence provider allows us to configure and fine-tune when and how the second level cache is used in our application.

In the following short examples, we’re going to demonstrate those features written as JUnit test cases and running on a H2 in-memory database.

Figure 1. Persistence Unit Configuration

Setup

First of all we need some basic setup to run the following examples .. we need to select a JPA persistence provider and database, create a persistence-unit configuration and an environment to run tests on an in-memory database.

Creating elegant, typesafe Queries for JPA, mongoDB Morphia and Lucene using Querydsl

Thu, 13 Feb 2014 00:00:00 +0100

Querydsl is a framework that allows us to create elegant, type-safe queries for a variety of different data-sources like Java Persistence API (JPA) entities, Java Data Objects (JDO), mongoDB with Morphia, SQL, Hibernate Search up to Lucene.

In the following tutorial we’re implementing example queries for different environments – Java Persistence API compared with a JPQL and a criteria API query, mongoDB with Morphia and last but not least for Lucene.

Docker Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Restrict Network

Can be useful when using a third-party image that we do not trust

Run with no network

docker run --network none <image>

Run with private isolated network

At least containers attached to this network can talk with another

docker network create --internal my_isolated_network
docker run --network my_isolated_network <image>

Block using firewall

e.g. using iptables or ipfw

# Get container's IP
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container_name>

# Block all outbound connections from that IP
sudo iptables -I DOCKER-USER -s <container_ip> -j DROP

Postgres Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Administration & Configuration

Connections

Show Max Connections (value and source)

SELECT setting, source, sourcefile, sourceline
FROM pg_settings
WHERE name = 'max_connections';

Set Max Connections

ALTER system SET max_connections = 250;

Kill Connections for a Database

SELECT pg_terminate_backend(pid)
FROM pg_catalog.pg_stat_activity
-- we don't want to kill our own connection
WHERE pid != pg_backend_pid()
-- we don't want to kill connections to other databases
AND datname = 'MYDATABASE'

Show Statement Timeout Settings for All Users

SELECT r.rolname,
       (SELECT unnest(s.setconfig) FROM pg_db_role_setting s WHERE s.setrole = r.oid AND s.setconfig::text LIKE '%statement_timeout%' LIMIT 1) AS statement_timeout
FROM pg_roles r
WHERE r.rolcanlogin
ORDER BY r.rolname;

SQL Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Count unique / all values

SELECT COUNT(DISTINCT <FIELDNAME>), COUNT(ALL <FIELDNAME>) FROM <TABLE>;

Partially anonymize e-mail addresses

UPDATE <TABLE_NAME>
SET <EMAIL_FIELD>=
INSERT(
<EMAIL_FIELD>,
POSITION('@' IN <EMAIL_FIELD>),
100,
CONCAT(FLOOR(1 + (RAND() * 100)),'@hascode.com'))
WHERE
POSITION('@' IN <EMAIL_FIELD>)>0;

Find duplicate entries

SELECT COUNT(*), <FIELDNAME> FROM <TABLENAME>
GROUP BY <FIELDNAME>
HAVING COUNT(*)>1;

MySQL Fix Zero Dates for Data Imports

Error: ERROR 1067 (42000) at line 1234: Invalid default value for ‘datefield’

In newer MySQL Versions zero in date or zero dates are forbidden .. check this with: