Snippets on Micha Kops' Tech Notes

AsciiDoc Snippets

Fri, 10 May 2019 00:00:00 +0200

AsciiDoc Syntax and Features

Using AsciiMath

Include stem into document header

:stem:

Then use inline, e.g.:

stem:[sqrt(4) = 2]

or as block

[stem]
++++
sum_(i=1)^n i^3=((n(n+1))/2)^2
++++

AsciiMath Syntax

Collapsible Blocks/Sections

.Click to show the content
[%collapsible]
====
Long content here...
====

Source: Asciidoctor Docs

Tools and Ecosystem

Docker / Podman Container

Including stuff like:

Asciidoctor
Asciidoctor Diagram with ERD and Graphviz integration (supports plantuml and graphiz diagrams)
Asciidoctor PDF
Asciidoctor EPUB3
Asciidoctor FB2
Asciidoctor Mathematical
Asciidoctor reveal.js
AsciiMath
Source highlighting using Rouge, CodeRay or Pygments
Asciidoctor Confluence
Asciidoctor Bibtex
Asciidoctor Kroki
Asciidoctor Reducer

AWS Snippets

Thu, 01 Mar 2018 00:00:00 +0100

AWS Command Line Interface

Installation

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.0.30.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

AWS Documentation

RDS

Export Database Configuration

Export instance configuration

aws rds describe-db-instances --db-instance-identifier arn:aws:rds:eu-central-1:123456789:db:hascode-prd-db --no-paginate

Export parameter group configuration

aws rds describe-db-parameters --db-parameter-group-name PARAM_GROUP_NAME >> param_group_conf.json

Export option group configuration

aws rds describe-option-groups --option-group-name OPT_GROUP_NAME >> option_group_conf.json

Generate Signed URLs with Linux Tools

e.g. for accessing a website behind a CloudFront distribution using a canned policy.

Write the policy file

How to build a quick Lucene Search

Thu, 25 Mar 2010 00:00:00 +0100

Helo – today I wanted to post a small tutorial for a small index and search operation using the Lucene indexer and Maven for the project setup.

Setup

Create an empty Maven sample project using the Eclipse Maven Plugin or use the following console command:
```
mvn archetype:create -DgroupId=com.hascode.demo.search -DartifactId=lucene-sample
```

Here is my pom.xml there are some dependencies for Lucene defined:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.hascode.demo.search</groupId>
  <artifactId>lucene-sample</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>My Lucene Search Sample</name>
  <description>Lucene Search Sample</description>
  <dependencies>
    <dependency>
      <groupId>org.apache.lucene</groupId>
      <artifactId>lucene-core</artifactId>
      <version>2.4.1</version>
    </dependency>
    <dependency>
      <groupId>lucene</groupId>
      <artifactId>lucene</artifactId>
      <version>1.4.3</version>
    </dependency>
  </dependencies>
</project>

AMQP and RabbitMQ Snippets

Mon, 01 Mar 2010 00:00:00 +0100

rabbitmqctl

Create admin user with full host permissions

# create new user named 'theadmin'
rabbitmqctl add_user theadmin thepassword

# make 'theadmin' admin
rabbitmqctl set_user_tags theadmin administrator

# give 'theadmin' permissions for all hosts
rabbitmqctl set_permissions -p / theadmin ".*" ".*" ".*"

AMQP Exchange Types

1. Fanout Exchange

Description: A fanout exchange routes messages to all of the queues that are bound to it. It doesn’t take the routing key into consideration. Instead, it simply broadcasts the message to all bound queues.

GitHub Snippets

Mon, 01 Mar 2010 00:00:00 +0100

GitHub Actions

Step to check variables and redistribute as env

The following step does ..

check of given variables are set, if not, exit with an error that is visible in the action’s log
provide the given input as environment variable in GITHUB_ENV

- name: configuration
      env:
        VAR1: ${{ needs.configure.outputs.something }}
        VAR2: ${{ vars.SOMETHING_OTHER }}/
        VAR3: "something_other_other"
      run: |
        for var in VAR1 VAR2 VAR3; do [ -n "${!var}" ] || { echo "$var is missing"; exit 1; }; echo "$var=${!var}" >> "$GITHUB_ENV"; done

GitLab Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Generate AsciiDoc Documentation and Publish it with GitLab Pages

We setup a repository and add a directory named docs there .. this is the home of our AsciiDoc files.

We’re using asciidoctor/docker-asciidoctor as Docker image for tool provisioning

This is the .gitlab-ci.yml, we’re running the stage only when something in the docs directory has changed.

stages:
  - "Build docs"

# The name of the job activates the GitLab pages publication
pages:
  image: asciidoctor/docker-asciidoctor
  stage: "Build docs"
  tags:
    - build
  script:
    - sh ./gen_docs.sh
    - mv output public
  only:
    refs:
      - master
    changes:
      - /docs/*
  artifacts:
    paths:
      - public
    expose_as: 'Documentation Archive'

Helm Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Common operations

Add Helm Repository

helm repo add NAME URL

e.g. for the Bitnami repository:

helm repo add bitnami https://charts.bitnami.com/bitnami                                                                                                                                                         1 ↵
"bitnami" has been added to your repositories

List Repositories

helm repo list                                                                                                                                                                                                 130 ↵
NAME    URL
bitnami https://charts.bitnami.com/bitnami

Searching in a Helm Repository

helm search repo wordpress
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/wordpress       15.2.30         6.1.1           WordPress is the world's most popular blogging ...
bitnami/wordpress-intel 2.1.31          6.1.1           DEPRECATED WordPress for Intel is the most popu...

IntelliJ Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Live Templates

JUnit 5 Test

@org.junit.jupiter.api.Test

@org.junit.jupiter.api.DisplayName("$NAME$")

void $METHOD$() throws Exception {

    $END$

}

Figure 1. IntelliJ Live Template Editor

Edit Variables:

NAME
METHOD: default-value camelCase(NAME)

Figure 2. Editing template variables

SLF4J Logger Template

private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger( $CLASS$.class );

Edit variables:

CLASS: expression: className()

Favorite Shortcuts

Select whole block in Editor

Cmd+Tab+Down

Select similar blocks in editor

Ctrl+G

Copy the absolute Path of the current File into Clipboard

Cmd+Shift+C

Java Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Remote Debug a Pod’s Java Process

Simple steps for remote debugging a Java process running on a k8 pod:

Edit deployment and add the following parameters to the Java start line: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=127.0.0.1:5005

Also add the following port mapping at the section container → ports in the deployment:
```
- containerPort: 5005
  protocol: TCP
```
Safe, wait for the new pods and then add a port forward for port 5005 for this pod:
```
kubectl port-forward podname 5005
```

Kafka Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Start an Image with kcat / kafka-cat for Debugging

kubectl -n NAMESPACE run "$(whoami)-debug" -it --rm \
    --image=confluentinc/cp-kafkacat:6.1.9 \
    --restart=Never \
    -- bash

Dockerfile for Kafka Analysis Container with different Tools

With jq, kafka console tools, schema registry tools and kafkacat installed ….

Dockerfile

FROM confluentinc/cp-kafka:6.2.1 as cp-kafka
FROM confluentinc/cp-schema-registry:6.2.1 as cp-schema-registry

FROM debian:10-slim
ARG DEBIAN_FRONTEND=noninteractive

# Install necessary tools
RUN apt-get update && apt-get install -y \
    curl \
    jq \
    yq \
    && rm -rf /var/lib/apt/lists/*

# Install kafkacat binary
RUN apt-get update && apt-get install -y kafkacat && rm -rf /var/lib/apt/lists/*

# Copy Kafka binaries
COPY --from=cp-kafka /usr/bin/kafka-* /usr/bin/
COPY --from=cp-schema-registry /usr/bin/schema-registry* /usr/bin/

# Copy entrypoint script
COPY entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh

ENTRYPOINT ["/usr/bin/entrypoint.sh"]

Linux Snippets

Mon, 01 Mar 2010 00:00:00 +0100

These are not only linux snippets but also bash snippets and snippets using tools that run under Linux, *nix or sometimes even MacOSX, I should reorder this article someday ;)

Settings for more reliable bash scripts

set -euo pipefail

this gives us …

-e: exit script if a single command fails
-u: exit script if an unset variable is used
-o pipefail: return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status

Maven Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Extract Coordinates from the POM

Helpful for build and integration environments, pipelines etc.

Exctract the Project version from the pom.xml

mvn org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate -Dexpression=project.version -q -DforceStdout

Override a Property via local configuration

e.g. to override the property xxx from your project’s pom.xml

Create a directory .mvn in your project directory root
Create a file named maven.config in this directory
Insert -Dxxx=newvalue into the file to override this property

Don’t forget to add .mvn to your .gitignore!

Misc Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Data Normalizing

Formula

newVal = (oldVal-min) / (max-min)

Ugly Scala Example

package com.hascode
import scala.collection.mutable.LinkedList

object NormalizerExample extends App {
    val dataSet = LinkedList(1., 6.5, 3., 6.2, 20., 31.2, 50.2, 12., 0.24, 1.224, 2.2, 3.)
    for ((num, index) <- dataSet.zipWithIndex) {
        dataSet(index) = (num - dataSet.min) / (dataSet.max - dataSet.min)
    }
    println("Normalized: " + dataSet)
}

Normalized: LinkedList(0.01521216973578863, 0.12921819759798853, 0.05947594797769014, 0.12324029048767723, 0.3982240175619966, 0.6213992163469515, 1.0, 1.0, 0.07531115879828326, 0.40498283261802576, 0.7319742489270387, 1.0)

Neo4j Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Aggregate existing Labels

MATCH (n) RETURN DISTINCT labels(n)

Or unwinding label pairs

MATCH (n)
WITH DISTINCT labels(n) AS labels
UNWIND labels AS label
RETURN DISTINCT label
ORDER BY label

PHP Snippets

Mon, 01 Mar 2010 00:00:00 +0100

WordPress anonymize IP in comments

Add as last line in theme’s functions.php:

function wpb_remove_commentsip($comment_author_ip) {
  return '127.0.0.1';
}
add_filter('pre_comment_user_ip','wpb_remove_commentsip');

WordPress remove version info

Add as last line in theme’s functions.php:

function remove_version_info() {
  return '';
}
add_filter('the_generator', 'remove_version_info');

List of all WordPress Hooks

Python Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Virtual Environments

Create and activate a venv

python -m venv /path/to/new-env (1)
source /path/to/new-env/bin/activate (2)

1	Creates a new virtual environment
2	Activate the new virtual environment

Deactivate venv

deactivate

Save used dependencies to a file

pip freeze > dependencies.txt

Install dependencies from file

pip install -r dependencies.txt

Storing and fetching Credentials from the System Keyring

I am using jaraco/keyring here:

pip install keyring

import keyring

keyring.set_password("system", "db.sample.password", "xoxo")

print(keyring.get_password("system", "db.sample.password"))

Scala Snippets

Mon, 01 Mar 2010 00:00:00 +0100

SBT – Eclipse Plugin

Add to your ~/.sbt/plugins/plugins.sbt

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.1.1")

Spring Boot Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Define and Configure Log Groups

This allows to configure a group of loggers at the same time

Define a log group named myaspect with two packages
application.properties
```
logging.group.myaspect=com.hascode.package1,com.hascode.package2
```
Configure the log group and set all loggers to level TRACE
application.properties
```
logging.level.myaspect=TRACE
```

This is also possible as parameter on startup

java -Dlogging.level.myaspect=TRACE myapp.jar

Use JUnit 5 with Spring Boot

Use newer versions of Surefire and Failsafe plugins:

<properties>
[..]
  <maven-failsafe-plugin.version>2.22.0</maven-failsafe-plugin.version>
  <maven-surefire-plugin.version>2.22.0</maven-surefire-plugin.version>
</properties>

SQL Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Count unique / all values

SELECT COUNT(DISTINCT <FIELDNAME>), COUNT(ALL <FIELDNAME>) FROM <TABLE>;

Partially anonymize e-mail addresses

UPDATE <TABLE_NAME>
SET <EMAIL_FIELD>=
INSERT(
<EMAIL_FIELD>,
POSITION('@' IN <EMAIL_FIELD>),
100,
CONCAT(FLOOR(1 + (RAND() * 100)),'@hascode.com'))
WHERE
POSITION('@' IN <EMAIL_FIELD>)>0;

Find duplicate entries

SELECT COUNT(*), <FIELDNAME> FROM <TABLENAME>
GROUP BY <FIELDNAME>
HAVING COUNT(*)>1;

MySQL Fix Zero Dates for Data Imports

Error: ERROR 1067 (42000) at line 1234: Invalid default value for ‘datefield’

In newer MySQL Versions zero in date or zero dates are forbidden .. check this with:

XML Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Ignore Namespaces in XPath Query

e.g. Query for all xxx nodes ignoring their namespace:

xmllint --xpath '//*[local-name()="xxx"]' input.xml

An example parsing URLs from a sitemap XML.

The URLs are located in //url/loc where all nodes are bound to the namespace http://www.sitemaps.org/schemas/sitemap/0.9.

The following query ignores the namespace

xmllint --xpath '//*[local-name()="url"]/*[local-name()="loc"]' sitemap.xml

Pretty Print XML in the Console using xmllint

echo '<blogs><blog url="https://www.hascode.com/">hasCode.com</blog></blogs>' | xmllint --format -
<?xml version="1.0"?>
<blogs>
    <blog url="https://www.hascode.com/">hasCode.com</blog>
</blogs>