These are not only linux snippets but also bash snippets and snippets using tools that run under Linux, *nix or sometimes even MacOSX, I should reorder this article someday ;)

Settings for more reliable bash scripts

set -euo pipefail

this gives us …​

  • -e: exit script if a single command fails

  • -u: exit script if an unset variable is used

  • -o pipefail: return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status

Bash Variable Expansion to check if a variable is set

echo ${XOXO:?} (1)
zsh: XOXO: parameter not set

XOXO='hey there';echo ${XOXO:?} (2)
hey there
1 the variable is not set, an error is display, exitCode 1
2 the varianle is set and therefore display, exitCode 0

Print last command’s exit code

echo $?

Trim leading whitespaces

using awk

echo ' whitespace there is ' | awk '{$1=$1;print}'
whitespace there is

Repeat command until sucessfully completed

while! [command]; do sleep 1; done

Remove Color Codes from Output

using col
somecmd | col
there are fragments of the color codes left
using sed
somecmd | sed 's/\x1b\[[0-9;]*m//g'

Bash Templates / Templating

envsubst replaces variables with values from env.

e.g.:

export FOO=xoxo && cat << EOF | envsubst
Hello, this is ${FOO} from a bash template
EOF
Hello, this is xoxo from a bash template

Passwort encrypt a file with OpenSSL and PBKDF2

Encrypting the file
openssl enc -aes-256-cbc -salt -pbkdf2 -in file.tbz -out file.tbz.enc
enter AES-256-CBC encryption password:
Verifying - enter AES-256-CBC encryption password:
Decrypting the file
openssl enc -d -aes-256-cbc -pbkdf2 -in file.tbz.enc -out file.tbz
enter AES-256-CBC decryption password:

Check TLS/SSL Certificates for Flaws with testssl

install with brew
brew install testssl
run testssl
testssl.sh [options] <URI>
example
testssl.sh hascode.io
[..]
 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Testing cipher categories

 NULL ciphers (no encryption)                  not offered (OK)
 Anonymous NULL Ciphers (no authentication)    not offered (OK)
 Export ciphers (w/o ADH+NULL)                 not offered (OK)
 LOW: 64 Bit + DES, RC[2,4] (w/o export)       not offered (OK)
 Triple DES Ciphers / IDEA                     not offered
 Obsolete CBC ciphers (AES, ARIA etc.)         not offered
 Strong encryption (AEAD ciphers)              offered (OK)
[..]

Resources

Create a temp directory

mktemp -d

CPU Stress Test

for i in $(seq $(getconf _NPROCESSORS_ONLN)); do yes > /dev/null & done

Source: Linuxconfig.org

Dates

Create ISO-8601 String for current Date

date +"%Y-%m-%dT%H:%M:%S%:z"

Read Unix Timestamp

date -d @1234556789

Create *nix timestamp

From Given ISO-8601 String

date --utc --date "2011-11-11 11:11:11" +%s

With a relative dimension e.g. two days ago

date --date='4 days ago' +"%s"

Bash Delete First Line

Use sed '1d' to remove the first line

e.g.

cat << EOF | sed '1d'
first
second
third
EOF

second
third

Insert something after n’th character

echo 'Hello .' | sed 's/.\{5\}/& world/'

Bash convert upper to lowercase

tr '[:upper:]' '[:lower:]'

e.g.

echo "Hello WorlD" | tr '[:upper:]' '[:lower:]'                130 ↵
hello world

Shell Brace Expansion

$ echo xx{1..20}
xx1 xx2 xx3 xx4 xx5 xx6 xx7 xx8 xx9 xx10 xx11 xx12 xx13 xx14 xx15 xx16 xx17 xx18 xx19 xx20

or create five directories:

$ mkdir mydirs-{1..5}

Convert screencast video

mencoder -idx input.ogv -ovc lavc -oac mp3lame -o output

Cut ogv videos without reencoding with ffmpeg

ffmpeg -i in.ogv -vcodec copy -acodec copy -ss 00:00:00 -t 00:10:00 out.ogv

Create thumbnails for every image in directory

for i in *.png; do convert -thumbnail 160 $i thumb-$i; done;

Gather RAM Information

sudo dmidecode --type 17

monitor udev events from usb port etc..

udevadm monitor

Extract Content from HTML by CSS Selector

Using htmlq a tool similar to jq for JSON or yq for YAML

Extracting the page content from a blog article of mine on www.hascode.com
curl -s https://www.hascode.com/2019/01/using-throwaway-containers-for-integration-testing-with-java-junit-5-and-testcontainers/ | htmlq --pretty 'div.post'

<div class="post" id="post-1529">
  <h2>Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.
  </h2>
  <small>
    January 30th, 2019 by <a href="/about/" rel="author">Micha Kops</a></small>
  <div class="entry">
    <p>A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.
    </p>
    <p>
      Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.
    </p>
[..]

Analyze Apache Logs with goaccess

sudo apt install goaccess

goaccess access.log -c

Pretty Print JSON (Python installed)

echo '{"tutorials": 100, "url": "https://www.hascode.com/"}' | python -m json.tool

Result in the following output:

{
"tutorials": 100,
"url": "https://www.hascode.com/"
}

Pretty Print and colorize JSON (Python installed)

echo '{"tutorials": 100, "url": "https://www.hascode.com/"}' | python -m json.tool | pygmentize -l javascript

bash/ use cursor up/down keys for command history up/down

bindkey "^[[A" history-search-backward
bindkey "^[[B" history-search-forward

Grep and Pipe – Keep color codes

grep --color=always | less

Execute last command as root

sudo !!

Copy to Clipboard from Console

Use xclip 
sudo apt-get install xclip

Examples:

uptime | xclip

Clicking the third mouse button allows to paste the result in any x-application

TrueCrypt GUIless/console mount encrypted volume

sudo truecrypt -t -k "" --protect-hidden=no /path/to/encfile /path/to/mountpoint

Enable MTP Support

sudo apt-get install mtp-tools mtpfs

Bash check if variable is set

Detects also empty strings 
if [ -z ${VARIABLE+x} ]

Example: Output variable RELEASE_VERSION if set, if not set or empty string, exit with error

"if [ -z ${RELEASE_VERSION+x} ]; then echo 'RELEASE_VERSION NOT SET' && exit 1; fi && echo \"Release: ${RELEASE_VERSION}\""

Bash check if directory exists

Watch out not to forget important whitespaces in the if-statement 
if [ ! -d $dir ]
then
    echo "Directory $dir does not exist"
else
    echo "Directory $dir exists"
fi

Sort Keywords in AsciiDoc Files in a given Directory

#!/bin/bash

# Directory to search for .adoc files
DIRECTORY="path/to/directory"

# Loop through all files with the .adoc extension in the directory and all subdirectories
find "$DIRECTORY" -name "*.adoc" | while read file
do
    # Extract keywords from the article
    keywords=$(grep -i "^:keywords:" "$file" | awk -F: '{print $3}')

    # Sort the keywords alphabetically
    sorted_keywords=$(echo "$keywords" | tr ',' '\n' | sort | tr '\n' ',' | sed 's/,$//')

    # Update the file with the sorted keywords
    sed -i "s/^:keywords:.*/:keywords: $sorted_keywords/" "$file"
done

Connect the device and detect vendor- / product-id

mtp-detect | grep idVendor
mtp-detect | grep idProduct

Add to /etc/udev/rules.d/55-android.rules:

SUBSYSTEM=="usb", ATTR{idVendor}=="VENDORID", ATTR{idProduct}=="PRODUCTID", MODE="0666"
sudo service udev restart

Connect:

mtpfs -o allow_other /media/GalaxyNexus

Disconnect:

fusermount -u /media/GalaxyNexus

Convert all mp3′s in a directory to aac

for f in *.mp3; do ffmpeg -i "$f" -acodec aac -strict experimental -ab 128k "${f%.mp3}.aac"; done

Secure Socket Shell (SSH)

Generate SSH Keys

ssh-keygen -t rsa -C "my@email.com"

Force SSH Login with Password only

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no

Mount SSH filesystem with user mapping and symlinks

sudo sshfs user@host.com:/path/to/directory /path/to/local-directory -o uid=1000,allow_other,follow_symlinks,transform_symlinks,workaround=rename

SSH Tunnel Local Forward

Forward traffic from localhost on port 44443 over the jump-server with ip address 1.2.3.4 on port 666 to the host some-host and port 443

ssh -4 -v -L localhost:44443:some-host:443 user@1.2.3.4 -p666

Find out which application runs on a given port and protocol

sudo fuser -n tcp 8080

List your hardware

sudo lshw

Directory tree comparison

diff -r dir1 dir2

Recording audio from the command line to a mp3

arecord -f cd -t raw | lame -x -r – outfile.mp3

List files and dirs by size

du -cks * |sort -gr

Faster compress files and directories

Use pigz instead of gzip
install pigz
sudo apt install pigz // linux
// or
brew install pigz // mac
compress a file
pigz filename
compress a directory
tar --use-compress-program="pigz -k " -cf dir.tar.gz dir

Backup directory using rsync with progress bar

rsync -h --progress --stats -r -tgo -l -p -D --update --delete-after /path/to/dir-to-be-saved /path/to/targetdir/

Debian/Ubuntu save a list of all installed packages / reinstall with list

dpkg --get-selections > installed-packages
dpkg --set-selections < installed-packages

Show installed files for a package

dpkg-query -L PACKAGE_NAME

Benchmarking the Network Speed between to Workstations

sudo apt-get install iperf

On the target workstation start iperf in server mode:

iperf -s

On the other workstation start the benchmark

iperf -c SERVER_IP_ADDRESS

Tell SVN not to store your password in .svn ..

svn command --no-auth-cache

Execute function on shell script exit

#!/bin/sh
# exit function
onExit() {
# do something here .. cleanup etc ....
}
# bind exit function
trap onExit ERR EXIT

# here's the normal code to be executed

Join multiple PDF Files with GhostScript

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=output_file.pdf file1.pdf file2.pdf ...

Curl send header

curl -H "headername:value" url

e.g. curl -H "REMOTE_USER:foo" http://developers.android.com

Curl send POST request

curl -X POST url

e.g. curl -X POST http://developers.android.com

Curl get all response headers

curl -i url

e.g. curl -i http://developers.android.com

Curl send JSON

curl -H "Content-Type: application/json" -XPOST -d '{"id":null, "title":"New book", "published":"2009-11-15T14:12:12"}' http://localhost:8080/javaee7-wildfly-liquibase-tutorial-1.0.0/rs/book

Run Apache Benchmark Tool (10 Requests, 5 Concurrent)

ab -n 10 -c 5 http://site.com

Screen detach session/ list / reattach

Detach

Ctrl+A+D

List:

$ screen -ls
There is a screen on:
2859.pts-0.host    (11/07/11 20:18:00)    (Detached)
1 Socket in /var/run/screen/S-user.

Reattach:

screen -r 2859.pts-0.host

Installer Script to Download and add a Binary to the Path

#!/bin/bash

# Define the binary name and URL
BINARY_NAME="mybin"
BINARY_URL="https://hascode.io/mybin"

# Define the install location
INSTALL_LOCATION="$HOME/bin"

# Define the shell profile file
if [ -n "$BASH_VERSION" ]; then
    SHELL_PROFILE_FILE="$HOME/.bashrc"
elif [ -n "$_VERSION" ]; then
    SHELL_PROFILE_FILE="$HOME/.rc"
else
    echo "Error: Unsupported shell."
    exit 1
fi

# Ask the user if they want to proceed with installation
read -p "Do you want to install $BINARY_NAME? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then

    # Download the binary
    echo "Downloading $BINARY_NAME from $BINARY_URL"
    curl -sSL "$BINARY_URL" -o "$INSTALL_LOCATION/$BINARY_NAME"

    # Add the binary to the path
    echo "Adding $BINARY_NAME to your PATH"
    echo "export PATH=\"$INSTALL_LOCATION:\$PATH\"" >> "$SHELL_PROFILE_FILE"

    # Print completion message
    echo "Installation of $BINARY_NAME is complete! Please open a new terminal window or run 'source $SHELL_PROFILE_FILE' to use it."
else
    echo "Installation cancelled by user."
fi

OpenSSL Check Certificate

in this example, the response shows, that a certificate is missing…​

openssl s_client -host HOSTNAME -port PORT
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 337 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

OpenSSL download certificate from host/port

openssl s_client -connect HOST:PORT

Download and cut out certificate part only …​

openssl s_client -connect HOST:PORT | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Join multiple videos into a single one

mencoder -oac copy -ovc copy -o outfile infile1 infile2 ... infileN

or if ls -l filename* matches the correct order ..

mencoder -oac copy -ovc copy -o outfile filename*

netcat test connection

e.g. if a kafka broker is alive

nc -vz localhost 9092

netcat emulate http server

sudo nc -l 80

Emulate a http server using python

python -m SimpleHTTPServer // old
python -m http.server // newer

Emulate a mail server using python

python -m smtpd -n -c DebuggingServer localhost:1025

Colored and comfortable alternative to top

if not installed: sudo apt-get install htop

htop

Colorize Logs

Three different tools to solve this: multitail, grc, ccze

Installation:

sudo apt-get install multitail grc ccze

multitail

multitail logfile

grc

grc tail -f logfile

ccze

tail -f logfile | ccze -m ansi // or html or curses

Changing the Locale

  • for current session only: LANG=en_US.UTF-8

  • permanently for current user: add LANG=en_US.UTF-8 to ~/.bashrc

  • default system locale: use LANG=en_US.UTF-8 in /etc/default/locale

Sending Wake on Lan (WoL) Packes

Using Linux Mint/Ubuntu…​

sudo apt install etherwake
sudo etherwake MAC-ADDRESS

journalctl prevent line truncate

journalctl -x -u UNITNAME | less

Remove first n lines from text file

with sed

sed -i -e 1,3d thefile.txt

with tail

tail -n +4 thefile.txt

HTTP Request via Telnet

telnet HOST/IP PORT
GET / HTTP/1.0
Domain: hascode.com
\n
\n

Example:

 telnet www.test.de 80
 Trying 20.82.65.183...
 Connected to nxsw-ingress.westeurope.cloudapp.azure.com.
 Escape character is '^]'.
 GET / HTTP/1.0
 Domain: www.test.de
 HTTP/1.1 301 Moved Permanently
 Date: Thu, 11 Nov 2021 09:45:20 GMT
 Content-Length: 0
 Connection: close
 Location: https://www.test.de/
 X-Cache-Status: MISS
 Connection closed by foreign host.

Script to convert a Sitemap XML to Redirection Rules in a .htaccess

Changing the domain

#!/bin/bash

# set the old and new domain names
old_domain="www.hascode.com"
new_domain="hascode.io"

# read the sitemap file and extract the URLs
urls=($(grep -o '<loc>.*<\/loc>' sitemap.xml | sed 's/<loc>\(.*\)<\/loc>/\1/'))

echo $urls

# loop over the URLs and create redirect rules
for url in "${urls[@]}"; do
    # replace the old domain with the new one in the URL
    redirect_url=${url/$old_domain/$new_domain}

    # write the redirect rule to the .htaccess file
    echo "Redirect 301 ${url} ${redirect_url}" >> .htaccess
done

Script to test converted URLs from a Sitemap XML for http connectivity

#!/bin/bash

# Specify the base domain to be replaced
OLD_DOMAIN="www.hascode.com"
NEW_DOMAIN="hascode.io"

# Parse sitemap.xml and extract URLs
URLS=($(grep -oP "(?<=<loc>)[^<]+" sitemap.xml))

# Loop through URLs and test accessibility
for url in "${URLS[@]}"
do
  # Replace old domain with new domain
  new_url="${url//$OLD_DOMAIN/$NEW_DOMAIN}"

  # Test if URL is accessible and returns a valid status code
  if curl -s --head "$new_url" | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null; then
    echo "$new_url is accessible"
  else
    echo "$new_url is not accessible"
  fi
done

Read Data from a structured file in a Loop into Variables

Given this text file:

textfile.log
05:00 Wake up
05:30 Start work
11:45 Break and eat

This shell script reads in two variables and prints them - in a loop.

readlog.sh
#!/bin/bash

while read TIME EVENT
do
        echo "At $TIME happens: $EVENT"
done < textfile.log

Running the script produces the following output:

$ sh readlog.sh
At 05:00 happens: Wake up
At 05:30 happens: Start work
At 11:45 happens: Break and eat

Bash Script output execution time

echo "Doing something..."
trap times EXIT

Scan for Credentials / Password in Project Files

Using git-secrets

Install git-secrets
brew install git-secrets
Scanning
git-secrets --scan

Using trufflehog

Installing trufflehog
brew install trufflehog
Scanning
trufflehog filesystem /path/to/sth

Switch back and forth between two directories

Use pushd and popd .. e.g.:

Using pushd and popd
pwd (1)
/tmp
pushd ~/.ssh (2)
~/.ssh /tmp
pwd (3)
/Users/me/.ssh
popd (4)
/tmp
pwd (5)
/tmp
1 We’re in directory /tmp
2 We’re switching to ~/.ssh
3 We verify that we’re in ~/.ssh now
4 We switch back to our original directory
5 We verify that we’re in /tmp again

Better diff tools

difftastic

Website: https://difftastic.wilfred.me.uk/

Installation

install difftastic
brew install difftastic

Use with git

~/.gitconfig
[diff]
        tool = difftastic

[difftool]
        prompt = false

[difftool "difftastic"]
        cmd = difft "$LOCAL" "$REMOTE"

[pager]
        difftool = true

dyff

Website: https://github.com/homeport/dyff

Installation

brew install homeport/tap/dyff

Use for kubectl diff

export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"

kubectl diff [...]

Search with fzf

Installation

install fzf
brew install fzf

Change the default search command

export FZF_DEFAULT_COMMAND="fd --type f"

Shutdown on USB activity

Website: https://github.com/hephaest0s/usbkill

Dealing with CVS files with xsv

Website:

Installation

using brew
brew install xsv
using cargo
cargo install xsv

Commands

Source: XSV Documentation on GitHub

cat

Concatenate CSV files by row or by column.

count

Count the rows in a CSV file. (Instantaneous with an index.)

fixlengths

Force a CSV file to have same-length records by either padding or truncating them.

flatten

A flattened view of CSV records. Useful for viewing one record at a time. e.g., xsv slice -i 5 data.csv | xsv flatten.

fmt

Reformat CSV data with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.)

frequency

Build frequency tables of each column in CSV data. (Uses parallelism to go faster if an index is present.)

headers

Show the headers of CSV data. Or show the intersection of all headers between many CSV files.

index

Create an index for a CSV file. This is very quick and provides constant time indexing into the CSV file.

input

Read CSV data with exotic quoting/escaping rules.

join

Inner, outer and cross joins. Uses a simple hash index to make it fast.

partition

Partition CSV data based on a column value.

sample

Randomly draw rows from CSV data using reservoir sampling (i.e., use memory proportional to the size of the sample).

reverse

Reverse order of rows in CSV data.

search

Run a regex over CSV data. Applies the regex to each field individually and shows only matching rows.

select

Select or re-order columns from CSV data.

slice

Slice rows from any part of a CSV file. When an index is present, this only has to parse the rows in the slice (instead of all rows leading up to the start of the slice).

sort

Sort CSV data.

split

Split one CSV file into many CSV files of N chunks.

stats

Show basic types and statistics of each column in the CSV file. (i.e., mean, standard deviation, median, range, etc.)

table

Show aligned output of any CSV data using elastic tabstops.

bat: cat with color and whitespace character display

Install:

brew install bat
Example: Display yaml file and show special characters
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: values.yaml
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #·Default·values·for·foo.␊
   2   │ #·This·is·a·YAML-formatted·file.␊
   3   │ #·Declare·variables·to·be·passed·into·your·templates.␊
   4   │ ␊
   5   │ replicaCount:·1␊
   6   │ ␊
   7   │ image:␊
   8   │ ··repository:·nginx␊
   9   │ ··pullPolicy:·IfNotPresent␊
  10   │ ··#·Overrides·the·image·tag·whose·default·is·the·chart·appVersion.␊
  11   │ ··tag:·""␊
  12   │ ␊
  13   │ imagePullSecrets:·[]␊
  14   │ nameOverride:·""␊
  15   │ fullnameOverride:·""␊
  16   │ ␊
  17   │ serviceAccount:␊
  18   │ ··#·Specifies·whether·a·service·account·should·be·created␊
  19   │ ··create:·true␊
  20   │ ··#·Annotations·to·add·to·the·service·account␊
  21   │ ··annotations:·{}␊
  22   │ ··#·The·name·of·the·service·account·to·use.␊
  23   │ ··#·If·not·set·and·create·is·true,·a·name·is·generated·using·the·fullname·template␊
  24   │ ··name:·""␊
[..]

Using watch to react to changes

Useful parameters
-n

Interval between updates in seconds

-t

Hide the watch command header

-d

Highlight differences

-g

Exit when a change happens

-b

Play a sound on change

Example: Output the date every 5 seconds and highlight the changes:

watch -t -n 5 -d date

Compare two OpenAPI Specifications with openapi-diff

Mounting our specs into the Docker container

docker run -v $(pwd)/:/specs:ro openapitools/openapi-diff:latest /specs/myspec-1.0.0.yaml /specs/myspec-1.1.0.yaml

zsh autosuggestions

  1. Clone the autosuggestions plugin into the plugin dir

    git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

  2. Edit the ~/.zshrc

    plugins=(zsh-autosuggestions)