These are not only linux snippets but also bash snippets and snippets using tools that run under Linux, *nix or sometimes even MacOSX, I should reorder this article someday ;) |
Settings for more reliable bash scripts
set -euo pipefail
this gives us …
-
-e: exit script if a single command fails
-
-u: exit script if an unset variable is used
-
-o pipefail: return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status
Bash Variable Expansion to check if a variable is set
echo ${XOXO:?} (1)
zsh: XOXO: parameter not set
XOXO='hey there';echo ${XOXO:?} (2)
hey there
1 | the variable is not set, an error is display, exitCode 1 |
2 | the varianle is set and therefore display, exitCode 0 |
Print last command’s exit code
echo $?
Trim leading whitespaces
using awk
echo ' whitespace there is ' | awk '{$1=$1;print}'
whitespace there is
Repeat command until sucessfully completed
while! [command]; do sleep 1; done
Remove Color Codes from Output
somecmd | col
there are fragments of the color codes left |
somecmd | sed 's/\x1b\[[0-9;]*m//g'
Bash Templates / Templating
envsubst
replaces variables with values from env
.
e.g.:
export FOO=xoxo && cat << EOF | envsubst
Hello, this is ${FOO} from a bash template
EOF
Hello, this is xoxo from a bash template
Passwort encrypt a file with OpenSSL and PBKDF2
openssl enc -aes-256-cbc -salt -pbkdf2 -in file.tbz -out file.tbz.enc
enter AES-256-CBC encryption password:
Verifying - enter AES-256-CBC encryption password:
openssl enc -d -aes-256-cbc -pbkdf2 -in file.tbz.enc -out file.tbz
enter AES-256-CBC decryption password:
Check TLS/SSL Certificates for Flaws with testssl
brew install testssl
testssl.sh [options] <URI>
testssl.sh hascode.io
[..]
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 not offered
TLS 1.1 not offered
TLS 1.2 offered (OK)
TLS 1.3 offered (OK): final
NPN/SPDY not offered
ALPN/HTTP2 h2, http/1.1 (offered)
Testing cipher categories
NULL ciphers (no encryption) not offered (OK)
Anonymous NULL Ciphers (no authentication) not offered (OK)
Export ciphers (w/o ADH+NULL) not offered (OK)
LOW: 64 Bit + DES, RC[2,4] (w/o export) not offered (OK)
Triple DES Ciphers / IDEA not offered
Obsolete CBC ciphers (AES, ARIA etc.) not offered
Strong encryption (AEAD ciphers) offered (OK)
[..]
Resources
Create a temp directory
mktemp -d
CPU Stress Test
for i in $(seq $(getconf _NPROCESSORS_ONLN)); do yes > /dev/null & done
Dates
Create ISO-8601 String for current Date
date +"%Y-%m-%dT%H:%M:%S%:z"
Read Unix Timestamp
date -d @1234556789
Create *nix timestamp
From Given ISO-8601 String
date --utc --date "2011-11-11 11:11:11" +%s
With a relative dimension e.g. two days ago
date --date='4 days ago' +"%s"
Bash Delete First Line
Use sed '1d'
to remove the first line
e.g.
cat << EOF | sed '1d'
first
second
third
EOF
second
third
Insert something after n’th character
echo 'Hello .' | sed 's/.\{5\}/& world/'
Bash convert upper to lowercase
tr '[:upper:]' '[:lower:]'
e.g.
echo "Hello WorlD" | tr '[:upper:]' '[:lower:]' 130 ↵
hello world
Shell Brace Expansion
$ echo xx{1..20}
xx1 xx2 xx3 xx4 xx5 xx6 xx7 xx8 xx9 xx10 xx11 xx12 xx13 xx14 xx15 xx16 xx17 xx18 xx19 xx20
or create five directories:
$ mkdir mydirs-{1..5}
Convert screencast video
mencoder -idx input.ogv -ovc lavc -oac mp3lame -o output
Cut ogv videos without reencoding with ffmpeg
ffmpeg -i in.ogv -vcodec copy -acodec copy -ss 00:00:00 -t 00:10:00 out.ogv
Create thumbnails for every image in directory
for i in *.png; do convert -thumbnail 160 $i thumb-$i; done;
Gather RAM Information
sudo dmidecode --type 17
monitor udev events from usb port etc..
udevadm monitor
Extract Content from HTML by CSS Selector
Using htmlq a tool similar to jq for JSON or yq for YAML
curl -s https://www.hascode.com/2019/01/using-throwaway-containers-for-integration-testing-with-java-junit-5-and-testcontainers/ | htmlq --pretty 'div.post'
<div class="post" id="post-1529">
<h2>Using Throwaway Containers for Integration Testing with Java, JUnit 5 and Testcontainers.
</h2>
<small>
January 30th, 2019 by <a href="/about/" rel="author">Micha Kops</a></small>
<div class="entry">
<p>A lot of boilerplate code is written when developers need to test their applications with different connected systems like databases, stream platforms and other collaborators.
</p>
<p>
Docker allows to handle those dependencies but there is still some glue code required to bind the container’s lifecycle and the configuration to the concrete integration test.
</p>
[..]
Analyze Apache Logs with goaccess
sudo apt install goaccess
goaccess access.log -c
Pretty Print JSON (Python installed)
echo '{"tutorials": 100, "url": "https://www.hascode.com/"}' | python -m json.tool
Result in the following output:
{
"tutorials": 100,
"url": "https://www.hascode.com/"
}
Pretty Print and colorize JSON (Python installed)
echo '{"tutorials": 100, "url": "https://www.hascode.com/"}' | python -m json.tool | pygmentize -l javascript
bash/ use cursor up/down keys for command history up/down
bindkey "^[[A" history-search-backward
bindkey "^[[B" history-search-forward
Grep and Pipe – Keep color codes
grep --color=always | less
Execute last command as root
sudo !!
Copy to Clipboard from Console
Use xclip |
sudo apt-get install xclip
Examples:
uptime | xclip
Clicking the third mouse button allows to paste the result in any x-application
TrueCrypt GUIless/console mount encrypted volume
sudo truecrypt -t -k "" --protect-hidden=no /path/to/encfile /path/to/mountpoint
Enable MTP Support
sudo apt-get install mtp-tools mtpfs
Bash check if variable is set
Detects also empty strings |
if [ -z ${VARIABLE+x} ]
Example: Output variable RELEASE_VERSION if set, if not set or empty string, exit with error
"if [ -z ${RELEASE_VERSION+x} ]; then echo 'RELEASE_VERSION NOT SET' && exit 1; fi && echo \"Release: ${RELEASE_VERSION}\""
Bash check if directory exists
Watch out not to forget important whitespaces in the if-statement |
if [ ! -d $dir ]
then
echo "Directory $dir does not exist"
else
echo "Directory $dir exists"
fi
Sort Keywords in AsciiDoc Files in a given Directory
#!/bin/bash
# Directory to search for .adoc files
DIRECTORY="path/to/directory"
# Loop through all files with the .adoc extension in the directory and all subdirectories
find "$DIRECTORY" -name "*.adoc" | while read file
do
# Extract keywords from the article
keywords=$(grep -i "^:keywords:" "$file" | awk -F: '{print $3}')
# Sort the keywords alphabetically
sorted_keywords=$(echo "$keywords" | tr ',' '\n' | sort | tr '\n' ',' | sed 's/,$//')
# Update the file with the sorted keywords
sed -i "s/^:keywords:.*/:keywords: $sorted_keywords/" "$file"
done
Connect the device and detect vendor- / product-id
mtp-detect | grep idVendor
mtp-detect | grep idProduct
Add to /etc/udev/rules.d/55-android.rules
:
SUBSYSTEM=="usb", ATTR{idVendor}=="VENDORID", ATTR{idProduct}=="PRODUCTID", MODE="0666"
sudo service udev restart
Connect:
mtpfs -o allow_other /media/GalaxyNexus
Disconnect:
fusermount -u /media/GalaxyNexus
Convert all mp3′s in a directory to aac
for f in *.mp3; do ffmpeg -i "$f" -acodec aac -strict experimental -ab 128k "${f%.mp3}.aac"; done
Secure Socket Shell (SSH)
Generate SSH Keys
ssh-keygen -t rsa -C "my@email.com"
Force SSH Login with Password only
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no
Mount SSH filesystem with user mapping and symlinks
sudo sshfs user@host.com:/path/to/directory /path/to/local-directory -o uid=1000,allow_other,follow_symlinks,transform_symlinks,workaround=rename
SSH Tunnel Local Forward
Forward traffic from localhost on port 44443
over the jump-server with ip address 1.2.3.4
on port 666
to the host some-host
and port 443
ssh -4 -v -L localhost:44443:some-host:443 user@1.2.3.4 -p666
Find out which application runs on a given port and protocol
sudo fuser -n tcp 8080
List your hardware
sudo lshw
Directory tree comparison
diff -r dir1 dir2
Recording audio from the command line to a mp3
arecord -f cd -t raw | lame -x -r – outfile.mp3
List files and dirs by size
du -cks * |sort -gr
Faster compress files and directories
Use pigz instead of gzip
|
sudo apt install pigz // linux
// or
brew install pigz // mac
pigz filename
tar --use-compress-program="pigz -k " -cf dir.tar.gz dir
Backup directory using rsync with progress bar
rsync -h --progress --stats -r -tgo -l -p -D --update --delete-after /path/to/dir-to-be-saved /path/to/targetdir/
Debian/Ubuntu save a list of all installed packages / reinstall with list
dpkg --get-selections > installed-packages
dpkg --set-selections < installed-packages
Show installed files for a package
dpkg-query -L PACKAGE_NAME
Benchmarking the Network Speed between to Workstations
sudo apt-get install iperf
On the target workstation start iperf in server mode:
iperf -s
On the other workstation start the benchmark
iperf -c SERVER_IP_ADDRESS
Tell SVN not to store your password in .svn ..
svn command --no-auth-cache
Execute function on shell script exit
#!/bin/sh
# exit function
onExit() {
# do something here .. cleanup etc ....
}
# bind exit function
trap onExit ERR EXIT
# here's the normal code to be executed
Join multiple PDF Files with GhostScript
gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=output_file.pdf file1.pdf file2.pdf ...
Curl send header
curl -H "headername:value" url
e.g. curl -H "REMOTE_USER:foo" http://developers.android.com
Curl send POST request
curl -X POST url
e.g. curl -X POST http://developers.android.com
Curl get all response headers
curl -i url
e.g. curl -i http://developers.android.com
Curl send JSON
curl -H "Content-Type: application/json" -XPOST -d '{"id":null, "title":"New book", "published":"2009-11-15T14:12:12"}' http://localhost:8080/javaee7-wildfly-liquibase-tutorial-1.0.0/rs/book
Run Apache Benchmark Tool (10 Requests, 5 Concurrent)
ab -n 10 -c 5 http://site.com
Screen detach session/ list / reattach
Detach
Ctrl+A+D
List:
$ screen -ls
There is a screen on:
2859.pts-0.host (11/07/11 20:18:00) (Detached)
1 Socket in /var/run/screen/S-user.
Reattach:
screen -r 2859.pts-0.host
Installer Script to Download and add a Binary to the Path
#!/bin/bash
# Define the binary name and URL
BINARY_NAME="mybin"
BINARY_URL="https://hascode.io/mybin"
# Define the install location
INSTALL_LOCATION="$HOME/bin"
# Define the shell profile file
if [ -n "$BASH_VERSION" ]; then
SHELL_PROFILE_FILE="$HOME/.bashrc"
elif [ -n "$_VERSION" ]; then
SHELL_PROFILE_FILE="$HOME/.rc"
else
echo "Error: Unsupported shell."
exit 1
fi
# Ask the user if they want to proceed with installation
read -p "Do you want to install $BINARY_NAME? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
# Download the binary
echo "Downloading $BINARY_NAME from $BINARY_URL"
curl -sSL "$BINARY_URL" -o "$INSTALL_LOCATION/$BINARY_NAME"
# Add the binary to the path
echo "Adding $BINARY_NAME to your PATH"
echo "export PATH=\"$INSTALL_LOCATION:\$PATH\"" >> "$SHELL_PROFILE_FILE"
# Print completion message
echo "Installation of $BINARY_NAME is complete! Please open a new terminal window or run 'source $SHELL_PROFILE_FILE' to use it."
else
echo "Installation cancelled by user."
fi
OpenSSL Check Certificate
in this example, the response shows, that a certificate is missing…
openssl s_client -host HOSTNAME -port PORT
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 337 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
OpenSSL download certificate from host/port
openssl s_client -connect HOST:PORT
Download and cut out certificate part only …
openssl s_client -connect HOST:PORT | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
Join multiple videos into a single one
mencoder -oac copy -ovc copy -o outfile infile1 infile2 ... infileN
or if ls -l filename* matches the correct order ..
mencoder -oac copy -ovc copy -o outfile filename*
netcat test connection
e.g. if a kafka broker is alive
nc -vz localhost 9092
netcat emulate http server
sudo nc -l 80
Emulate a http server using python
python -m SimpleHTTPServer // old
python -m http.server // newer
Emulate a mail server using python
python -m smtpd -n -c DebuggingServer localhost:1025
Colored and comfortable alternative to top
if not installed: sudo apt-get install htop
htop
Colorize Logs
Three different tools to solve this: multitail, grc, ccze
Installation:
sudo apt-get install multitail grc ccze
multitail
multitail logfile
grc
grc tail -f logfile
ccze
tail -f logfile | ccze -m ansi // or html or curses
Changing the Locale
-
for current session only:
LANG=en_US.UTF-8
-
permanently for current user: add
LANG=en_US.UTF-8
to~/.bashrc
-
default system locale: use
LANG=en_US.UTF-8
in/etc/default/locale
Sending Wake on Lan (WoL) Packes
Using Linux Mint/Ubuntu…
sudo apt install etherwake
sudo etherwake MAC-ADDRESS
journalctl prevent line truncate
journalctl -x -u UNITNAME | less
Remove first n lines from text file
with sed
sed -i -e 1,3d thefile.txt
with tail
tail -n +4 thefile.txt
HTTP Request via Telnet
telnet HOST/IP PORT
GET / HTTP/1.0
Domain: hascode.com
\n
\n
Example:
telnet www.test.de 80
Trying 20.82.65.183...
Connected to nxsw-ingress.westeurope.cloudapp.azure.com.
Escape character is '^]'.
GET / HTTP/1.0
Domain: www.test.de
HTTP/1.1 301 Moved Permanently
Date: Thu, 11 Nov 2021 09:45:20 GMT
Content-Length: 0
Connection: close
Location: https://www.test.de/
X-Cache-Status: MISS
Connection closed by foreign host.
Script to convert a Sitemap XML to Redirection Rules in a .htaccess
Changing the domain
#!/bin/bash
# set the old and new domain names
old_domain="www.hascode.com"
new_domain="hascode.io"
# read the sitemap file and extract the URLs
urls=($(grep -o '<loc>.*<\/loc>' sitemap.xml | sed 's/<loc>\(.*\)<\/loc>/\1/'))
echo $urls
# loop over the URLs and create redirect rules
for url in "${urls[@]}"; do
# replace the old domain with the new one in the URL
redirect_url=${url/$old_domain/$new_domain}
# write the redirect rule to the .htaccess file
echo "Redirect 301 ${url} ${redirect_url}" >> .htaccess
done
Script to test converted URLs from a Sitemap XML for http connectivity
#!/bin/bash
# Specify the base domain to be replaced
OLD_DOMAIN="www.hascode.com"
NEW_DOMAIN="hascode.io"
# Parse sitemap.xml and extract URLs
URLS=($(grep -oP "(?<=<loc>)[^<]+" sitemap.xml))
# Loop through URLs and test accessibility
for url in "${URLS[@]}"
do
# Replace old domain with new domain
new_url="${url//$OLD_DOMAIN/$NEW_DOMAIN}"
# Test if URL is accessible and returns a valid status code
if curl -s --head "$new_url" | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null; then
echo "$new_url is accessible"
else
echo "$new_url is not accessible"
fi
done
Read Data from a structured file in a Loop into Variables
Given this text file:
05:00 Wake up
05:30 Start work
11:45 Break and eat
This shell script reads in two variables and prints them - in a loop.
#!/bin/bash
while read TIME EVENT
do
echo "At $TIME happens: $EVENT"
done < textfile.log
Running the script produces the following output:
$ sh readlog.sh
At 05:00 happens: Wake up
At 05:30 happens: Start work
At 11:45 happens: Break and eat
Bash Script output execution time
echo "Doing something..."
trap times EXIT
Scan for Credentials / Password in Project Files
Using git-secrets
brew install git-secrets
git-secrets --scan
Using trufflehog
brew install trufflehog
trufflehog filesystem /path/to/sth
Switch back and forth between two directories
Use pushd
and popd
.. e.g.:
pwd (1)
/tmp
pushd ~/.ssh (2)
~/.ssh /tmp
pwd (3)
/Users/me/.ssh
popd (4)
/tmp
pwd (5)
/tmp
1 | We’re in directory /tmp |
2 | We’re switching to ~/.ssh |
3 | We verify that we’re in ~/.ssh now |
4 | We switch back to our original directory |
5 | We verify that we’re in /tmp again |
Better diff tools
difftastic
Website: https://difftastic.wilfred.me.uk/
Installation
brew install difftastic
Use with git
[diff]
tool = difftastic
[difftool]
prompt = false
[difftool "difftastic"]
cmd = difft "$LOCAL" "$REMOTE"
[pager]
difftool = true
dyff
Website: https://github.com/homeport/dyff
Installation
brew install homeport/tap/dyff
Use for kubectl diff
export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"
kubectl diff [...]
Search with fzf
Installation
brew install fzf
Change the default search command
export FZF_DEFAULT_COMMAND="fd --type f"
Shutdown on USB activity
Dealing with CVS files with xsv
Website:
Installation
brew install xsv
cargo install xsv
Commands
- cat
-
Concatenate CSV files by row or by column.
- count
-
Count the rows in a CSV file. (Instantaneous with an index.)
- fixlengths
-
Force a CSV file to have same-length records by either padding or truncating them.
- flatten
-
A flattened view of CSV records. Useful for viewing one record at a time. e.g., xsv slice -i 5 data.csv | xsv flatten.
- fmt
-
Reformat CSV data with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.)
- frequency
-
Build frequency tables of each column in CSV data. (Uses parallelism to go faster if an index is present.)
- headers
-
Show the headers of CSV data. Or show the intersection of all headers between many CSV files.
- index
-
Create an index for a CSV file. This is very quick and provides constant time indexing into the CSV file.
- input
-
Read CSV data with exotic quoting/escaping rules.
- join
-
Inner, outer and cross joins. Uses a simple hash index to make it fast.
- partition
-
Partition CSV data based on a column value.
- sample
-
Randomly draw rows from CSV data using reservoir sampling (i.e., use memory proportional to the size of the sample).
- reverse
-
Reverse order of rows in CSV data.
- search
-
Run a regex over CSV data. Applies the regex to each field individually and shows only matching rows.
- select
-
Select or re-order columns from CSV data.
- slice
-
Slice rows from any part of a CSV file. When an index is present, this only has to parse the rows in the slice (instead of all rows leading up to the start of the slice).
- sort
-
Sort CSV data.
- split
-
Split one CSV file into many CSV files of N chunks.
- stats
-
Show basic types and statistics of each column in the CSV file. (i.e., mean, standard deviation, median, range, etc.)
- table
-
Show aligned output of any CSV data using elastic tabstops.
bat: cat with color and whitespace character display
Install:
brew install bat
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: values.yaml
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ #·Default·values·for·foo.␊
2 │ #·This·is·a·YAML-formatted·file.␊
3 │ #·Declare·variables·to·be·passed·into·your·templates.␊
4 │ ␊
5 │ replicaCount:·1␊
6 │ ␊
7 │ image:␊
8 │ ··repository:·nginx␊
9 │ ··pullPolicy:·IfNotPresent␊
10 │ ··#·Overrides·the·image·tag·whose·default·is·the·chart·appVersion.␊
11 │ ··tag:·""␊
12 │ ␊
13 │ imagePullSecrets:·[]␊
14 │ nameOverride:·""␊
15 │ fullnameOverride:·""␊
16 │ ␊
17 │ serviceAccount:␊
18 │ ··#·Specifies·whether·a·service·account·should·be·created␊
19 │ ··create:·true␊
20 │ ··#·Annotations·to·add·to·the·service·account␊
21 │ ··annotations:·{}␊
22 │ ··#·The·name·of·the·service·account·to·use.␊
23 │ ··#·If·not·set·and·create·is·true,·a·name·is·generated·using·the·fullname·template␊
24 │ ··name:·""␊
[..]
Using watch to react to changes
Example: Output the date every 5 seconds and highlight the changes:
watch -t -n 5 -d date
Compare two OpenAPI Specifications with openapi-diff
Mounting our specs into the Docker container
docker run -v $(pwd)/:/specs:ro openapitools/openapi-diff:latest /specs/myspec-1.0.0.yaml /specs/myspec-1.1.0.yaml
zsh autosuggestions
-
Clone the autosuggestions plugin into the plugin dir
git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
-
Edit the
~/.zshrc
plugins=(zsh-autosuggestions)