thomas-leister.de

Migrate Mastodon Media Storage from Minio S3 to SeaweedFS S3

thomas.leister@mailbox.org (Thomas Leister) — Fri, 23 Jan 2026 15:03:21 +0100

Until now, I have enjoyed using the Minio S3 Server together with my Mastodon instance metalhead.club to store media files. However, since last summer, the Minio project has unfortunately taken a turn that has prompted me to look for alternatives: First, the internal file browser was removed from the administration interface. Shortly thereafter, it was announced that Minio would no longer be maintained as an open source project. In addition, the distribution of Minio binaries was discontinued. This meant that there were no more updates. Minio has effectively been dead ever since at least as far as the community version is concerned.

However, I found a great alternative in SeaweedFS, which I recently switched my Mastodon instance to. Here, I describe the problems I encountered and how I solved them.

Install SeaweedFS

Installation is very simple: SeaweedFS consists of a single binary, which is downloaded and stored in /usr/local/bin:

curl -L https://github.com/seaweedfs/seaweedfs/releases/download/4.07/linux_amd64_full.tar.gz | tar xvz -C /usr/local/bin/

Next, a new SeaweedFS system user is created:

useradd --system seaweedfs

and a corresponding Systemd service is created:

/etc/systemd/system/seaweedfs.service:

[Unit]
Description=SeaweedFS Server
After=network.target
[Service]
Type=simple
User=seaweedfs
Group=seaweedfs
ExecStart=/usr/local/bin/weed server -s3 -ip 127.0.0.1 -ip.bind=127.0.0.1 -volume.max=0 -dir /mnt/s3storage1/seaweedfs
WorkingDirectory=/usr/local/bin
SyslogIdentifier=seaweedfs
[Install]
WantedBy=multi-user.target

/usr/local/bin/weed server -s3 starts the SeaweedFS binary weed and with it the components master server, volume server, filer, and the s3 frontend.
-ip 127.0.0.1 -ip.bind=127.0.0.1: Internal communication interfaces and the S3 API should remain internal to localhost, as we are not running SeaweedFS in a cluster and the S3 interface should only be accessible to the internal Nginx proxy.
-volume.max=0 ensures that the number of storage volumes is not limited. Otherwise, you could set the maximum amount of storage SeaweedFS is allowed to use (default: 1 volume = 30 GB). In my case, I want to use all available storage.
-dir /mnt/s3storage1/seaweedfs: This is where the volumes are stored in the form of files.

Activate and start the service:

systemctl enable --now seaweedfs.service

SeaweedFS should start after a few seconds. See also:

systemctl status seaweedfs

Create a bucket

A new S3 bucket can now be created and configured using the “weed shell.” In my case, the new bucket is called “metalheadclub-media.”

Open shell:

weed shell

Create bucket:

s3.bucket.create -name metalheadclub-media

Set permissions (dynamically, instead of a json config file):

s3.configure -access_key=ACCESS_KEY -secret_key=SECRET_KEY -buckets=metalheadclub-media -user=mastodon -actions=Read,Write,List -apply

The following is displayed for confirmation:

{
"identities": [
{
"name": "mastodon",
"credentials": [
{
"accessKey": "ACCESS_KEY",
"secretKey": "SECRET_KEY",
"status": ""
}
],
"actions": [
"Read:metalheadclub-media",
"Write:metalheadclub-media",
"List:metalheadclub-media"
],
"account": null,
"disabled": false,
"serviceAccountIds": [],
"policyNames": []
}
],
"accounts": [],
"serviceAccounts": []
}

Anonymous (public) users only have read access to the media files:

s3.configure -buckets=metalheadclub-media -user=anonymous -actions=Read -apply

Output:

{
"identities": [
{
"name": "mastodon",
"credentials": [
{
"accessKey": "ACCESS_KEY",
"secretKey": "SECRET_KEY",
"status": ""
}
],
"actions": [
"Read:metalheadclub-media",
"Write:metalheadclub-media",
"List:metalheadclub-media"
],
"account": null,
"disabled": false,
"serviceAccountIds": [],
"policyNames": []
},
{
"name": "anonymous",
"credentials": [],
"actions": [
"Read:metalheadclub-media"
],
"account": null,
"disabled": false,
"serviceAccountIds": [],
"policyNames": []
}
],
"accounts": [],
"serviceAccounts": []
}

You can then exit the shell by typing “quit.”

Migrating data from Minio to SeaweedFS

The SeaweedFS server is now active, the bucket has been created and assigned the necessary permissions. Time to migrate the data from Minio to SeaweedFS!

There were two things to keep in mind for the migration:

The migration should be as “silent” as possible and have no impact on the users of my Mastodon instance.
Since I wanted to use the same virtual machine with the same storage for SeaweedFS and storage was limited, the data should be moved from Minio to SeaweedFS, not copied!

As with previous migrations, the strategy was as follows:

Put the new backend into operation
Configure the frontend proxy with both backends => Frontend first queries the new backend, falling back to the old backend in case of error
Reconfigure the Mastodon instance: Stores new media in the new backend, while old media remains accessible from the old backend
Media is slowly moved from the old storage to the new storage in the background
… until the old storage is finally empty and can be removed
Done!

Two S3 backends for a smooth transition

I have visualized the configuration for data migration here:

The graphic shows the dependencies between the Nginx proxies and the S3 servers. It starts with a media.metalhead.club server, which forms the front end. Behind it, one path leads directly to Minio, while another path leads to the SeaweedFS backend via another proxy, s3.650thz.de.

The upper row (media.metalhead.club, Minio server) already existed – the SeaweedFS backend and the s3.650thz.de server are new additions. The media.metalhead.club Nginx proxy was to play a special role again (similar to the migration from Scaleway S3 to Minio) and switch flexibly between the two S3 backends during the migration of the data. The new SeaweedFS backend should primarily respond to requests. In the event that the new backend was unable to deliver the data, a fallback was implemented to ensure that the request could then be answered by the old Minio backend.

The key lies in these lines:

# During data migration: Forward request to @minio if @seaweedfs returns 404 or 403 (access denied)
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 403 = @minio;

Virtual Host style addressing for SeaweedFS S3

An important difference in the handling of the SeaweedFS backend compared to Minio is that SeaweedFS does not support virtual host addressing of S3 buckets! With Minio, a bucket can be addressed in two ways:

bucket.s3server.tld or
s3server.tld/bucket

The former is therefore not possible with SeaweedFS. Instead, the SeaweedFS Wiki suggests rewriting addresses in the latter form:

...
server_name ~^(?:(?<bucket>[^.]+)\.)?s3\.yourdomain\.com;
...
# If bucket subdomain is not empty,
# rewrite request to backend.
if ($bucket != “”) {
 rewrite (.*) /$bucket$1 last;
}
...

However, I was unable to implement this type of configuration in my media.metalhead.club VirtualHost because it led to an error:

[error] 1388933#1388933: *5700433 rewrite or internal redirection cycle while redirect to named location "@seaweedfs", client: xxx, server: media.metalhead.club, request: "GET /cache/custom_emojis/images/xxx.png HTTP/2.0", host: "media.metalhead.club"

So I quickly moved this mechanism to another vHost called “s3.650thz.de”:

upstream seaweedfs {
 hash $arg_uploadId consistent;
 server 127.0.0.1:8333 fail_timeout=0;
 keepalive 20;
}

server {
 listen 80;
 listen [::]:80;
 listen 443 ssl;
 listen [::]:443 ssl;

 # Enable http2
 http2 on;

 server_name ~^(?<bucket>[^.]+)\.s3\.650thz\.de s3.650thz.de;

 ssl_certificate /etc/acme.sh/s3.650thz.de/fullchain.pem;
 ssl_certificate_key /etc/acme.sh/s3.650thz.de/privkey.pem;

 ignore_invalid_headers off;
 client_max_body_size 0;
 proxy_buffering off;
 proxy_request_buffering off;

 location / {
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 proxy_http_version 1.1;
 proxy_set_header Connection "";
 chunked_transfer_encoding off;

 if ($bucket != "") {
 rewrite ^/(.*)$ /$bucket/$1 break;
 }

 proxy_pass http://seaweedfs; # This uses the upstream directive definition to load balance
 }
}

Ultimately, the vHost only serves to convert “virtual host style addressing” into conventional “path style addressing.”

(The “vHost style addressing” in the media.metalhead.club vHost requires the proxy_pass directives, as paths such as proxy_pass http://seaweedfs/bucket cannot be specified here.)

My media.metalhead.club for the migration phase now looks like this:

upstream minio {
 server 127.0.0.1:9000;
 keepalive 2;
}

server {
 listen 80;
 listen [::]:80;
 listen 443 ssl;
 listen [::]:443 ssl;

 server_name media.metalhead.club;

 # Enable http2
 http2 on;

 # SSL
 ssl_certificate /etc/acme.sh/media.metalhead.club/fullchain.pem;
 ssl_certificate_key /etc/acme.sh/media.metalhead.club/privkey.pem;

 client_max_body_size 100M;

 root /var/www/media.metalhead.club;

 location / {
 access_log off;
 try_files $uri @seaweedfs;
 }

 # Display index.html if user is browsing https://media.metalhead.club
 location = / {
 alias /var/www/media.metalhead.club/;
 }


 #
 # Own SeaweedFS S3 server (new)
 #

 location @seaweedfs {
 # Set headers
 proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
 proxy_set_header Connection '';
 proxy_set_header Authorization '';

 # Hide headers
 proxy_hide_header Set-Cookie;
 proxy_hide_header 'Access-Control-Allow-Origin';
 proxy_hide_header 'Access-Control-Allow-Methods';
 proxy_hide_header 'Access-Control-Allow-Headers';
 # Hide amazon S3 headers, which are sent by Minio for compatibility
 proxy_hide_header x-amz-id-2;
 proxy_hide_header x-amz-request-id;
 proxy_hide_header x-amz-meta-server-side-encryption;
 proxy_hide_header x-amz-server-side-encryption;
 proxy_hide_header x-amz-bucket-region;
 proxy_hide_header x-amzn-requestid;
 proxy_hide_header x-amz-meta-mtime;

 # ignore headers
 proxy_ignore_headers Set-Cookie;

 # Pass headers
 proxy_pass_header Date;
 proxy_pass_header Last-Modified;
 proxy_pass_header ETag;
 proxy_pass_header Cache-Control;
 proxy_pass_header Expires;

 # Connection to backend server
 proxy_pass http://[::1];
 proxy_connect_timeout 1s;
 proxy_send_timeout 30s;
 proxy_read_timeout 30s;

 # Buffer settings
 proxy_request_buffering off;
 proxy_buffering on;
 proxy_buffers 16 64k;
 proxy_busy_buffers_size 128k;
 proxy_buffer_size 32k;

 # Migration Workaround: Forward request to @minio if @seaweedfs returns 404 or 403 (access denied)
 proxy_intercept_errors on;
 recursive_error_pages on;
 error_page 404 403 = @minio;

 add_header 'Access-Control-Allow-Origin' '*';
 add_header X-Cache-Status $upstream_cache_status;
 add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
 add_header X-Served-By "s3.650thz.de SeaweedFS";
 }

 #
 # Own Minio S3 server (old)
 #
 location @minio {
 limit_except GET {
 deny all;
 }

 # Set headers
 proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
 proxy_set_header Connection '';
 proxy_set_header Authorization '';

 # Hide headers
 proxy_hide_header Set-Cookie;
 proxy_hide_header 'Access-Control-Allow-Origin';
 proxy_hide_header 'Access-Control-Allow-Methods';
 proxy_hide_header 'Access-Control-Allow-Headers';
 # Hide amazon S3 headers, which are sent by Minio for compatibility
 proxy_hide_header x-amz-id-2;
 proxy_hide_header x-amz-request-id;
 proxy_hide_header x-amz-meta-server-side-encryption;
 proxy_hide_header x-amz-server-side-encryption;
 proxy_hide_header x-amz-bucket-region;
 proxy_hide_header x-amzn-requestid;
 proxy_hide_header x-amz-meta-mtime;

 # ignore headers
 proxy_ignore_headers Set-Cookie;

 # Pass headers
 proxy_pass_header Date;
 proxy_pass_header Last-Modified;
 proxy_pass_header ETag;
 proxy_pass_header Cache-Control;
 proxy_pass_header Expires;

 # Connection to backend server
 proxy_pass http://minio;
 proxy_connect_timeout 1s;
 proxy_send_timeout 30s;
 proxy_read_timeout 30s;

 # Buffer settings
 proxy_request_buffering off;
 proxy_buffering on;
 proxy_buffers 16 64k;
 proxy_busy_buffers_size 128k;
 proxy_buffer_size 32k;

 add_header 'Access-Control-Allow-Origin' '*';
 add_header X-Cache-Status $upstream_cache_status;
 add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
 add_header X-Served-By "s3.650thz.de Minio";
 }
}

Using add_header X-Served-By “s3.650thz.de ...”, I also have the HTTP header indicate whether a file was delivered by Minio or SeaweedFS. This allows me to check later whether the data migration worked as hoped.

Configuring the Mastodon instance

In order for Mastodon to store the new media files in SeaweedFS, the correct parameters had to be set in the .env.production configuration:

S3_ENABLED=true
S3_BUCKET=metalheadclub-media
AWS_ACCESS_KEY_ID=ACCESS_KEY
AWS_SECRET_ACCESS_KEY=SECRET_KEY
S3_ALIAS_HOST=media.metalhead.club
S3_REGION=hyper2
S3_PROTOCOL=https
S3_HOSTNAME=media.metalhead.club
S3_ENDPOINT=https://s3.650thz.de
S3_SIGNATURE_VERSION=v4

After restarting all Mastodon services, the change was active:

cd /etc/systemd/system
systemctl restart mastodon*

I could easily check whether my change was successful: I waited for new posts with images to appear in the global Mastodon timeline and opened the image in a new tab. The Firefox developer tools then allow you to view the HTTP headers for the file. As hoped, the X-Served-By header was set to “SeaweedFS.” :) …

and the old graphics could still be accessed. They were delivered with X-Served-By “Minio.”

Data transfer

Great! Now it was time to start the actual data migration. Away from Minio – over to SeaweedFS.

Actually, such a migration is not rocket science. I configured rclone with both S3 buckets:

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = hyper2
endpoint = http://127.0.0.1:9000
location_constraint = hyper2
acl = public-read
[seaweedfs]
type = s3
provider = SeaweedFS
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
endpoint = http://127.0.0.1:8333
acl = public-read

I tested access to both buckets with a simple file listing:

rclone lsd seaweedfs:metalheadclub-media
rclone lsd minio:metalheadclub-media

I then began testing by first copying only smaller directories such as site_uploads from Minio to SeaweedFS (instead of moving them!):

rclone sync minio:metalheadclub-media/site_uploads/ seaweedfs:metalheadclub-media/site_uploads/

Once again, I used browser tools to test how the affected graphics (e.g., banner graphics at https://metalhead.club/about) were delivered. While they were initially delivered by Minio, they were delivered by SeaweedFS after the rclone sync.

After this test was also successful, I migrated the Mastodon media data piece by piece to SeaweedFS (this time by moving it with move instead of copying it with sync!).

List of directories to be moved:

root@s3 /root # rclone lsd minio:metalheadclub-media/
0 2026-01-16 14:01:17 -1 accounts
0 2026-01-16 14:01:17 -1 backups
0 2026-01-16 14:01:17 -1 cache
0 2026-01-16 14:01:17 -1 custom_emojis
0 2026-01-16 14:01:17 -1 imports
0 2026-01-16 14:01:17 -1 media_attachments
0 2026-01-16 14:01:17 -1 site_uploads

My commands for moving:

rclone move --delete-empty-src-dirs minio:metalheadclub-media/accounts seaweedfs:metalheadclub-media/accounts
rclone move --delete-empty-src-dirs minio:metalheadclub-media/backups seaweedfs:metalheadclub-media/backups
rclone move --delete-empty-src-dirs minio:metalheadclub-media/custom_emojis seaweedfs:metalheadclub-media/custom_emojis
rclone move --delete-empty-src-dirs minio:metalheadclub-media/imports seaweedfs:metalheadclub-media/imports
rclone move --delete-empty-src-dirs minio:metalheadclub-media/media_attachments seaweedfs:metalheadclub-media/media_attachments
rclone move --delete-empty-src-dirs minio:metalheadclub-media/site_uploads seaweedfs:metalheadclub-media/site_uploads

If you pay close attention, you will notice that I have omitted one directory: the media cache directory /cache! This is by far the largest and receives special treatment …

Trouble with the large /cache directory

This directory contains all files that do not originate from your own Mastodon server. In other words, cached remote media data and preview images. Depending on the cache retention time, the directory can become very large. In the case of metalhead.club, I was dealing with about 700 GB and approximately 3 million small files. In fact, it’s not the total size that’s the problem, but the number of files.

Of course, I also tried to move this directory to the new storage:

rclone move --delete-empty-src-dirs minio:metalheadclub-media/cache seaweedfs:metalheadclub-media/cache

… but after a few minutes, it failed due to insufficient RAM. I added more RAM (about 16 GB), but again, the rclone tool was so RAM-hungry that it was terminated by the OOM killer.

All right. If I couldn’t move all the data from the cache directory, I could just leave it in the old Minio bucket until it had accumulated! This is because the /cache directory only contains files that are not very important for operation and are normally cleaned up by Mastodon after a specified period of time anyway (at metalhead.club: cache retention 15 days).

“Just leave it and wait” sounds like an appealing strategy, but it’s not that simple. This is because the switch of the Mastodon software to the new bucket means that Mastodon’s internal deletion routines no longer work. The bucket remains as full as it was last used. And new files are pushing their way onto the system. The result: memory consumption increases.

If you have a lot of free memory, you may actually be able to simply wait until 15 days have passed and then delete the entire bucket. However, I only had about 170 GB of free storage. This would have been used up well before 15 days by new media added to the new bucket. So waiting was not an option. The old media had to be deleted bit by bit.

First, I tried using rclone commands. This command deletes all files older than 15 days:

rclone -v --min-age 15d delete --rmdirs minio:metalheadclub-media/cache

… - actually. Once again, the RAM requirements were too high (in some cases, 15 GB consumption by rclone alone!).

With a few optimizations, you can significantly reduce consumption:

GOGC=20 rclone --list-cutoff 100000 --min-age 15d delete --rmdirs minio:metalheadclub-media/cache

See also: https://rclone.org/faq/# rclone-is-using-too-much-memory-or-appears-to-have-a-memory-leak

Just when I thought I had solved the problem, another error popped up:

ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get “http://127.0.0.1:9000/metalheadclub-media?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=cache%2F”: net/http: timeout awaiting response headers

Apparently, listing the files took so long that the S3 server itself timed out (after about 10 minutes). How should I solve this? There were simply too many files in the bucket.

I also had no luck with Minio’s own tool mc:

./mc rm --recursive --force --older-than 15d minio/metalheadclub-media/cache
mc: <ERROR> Failed to remove `minio/metalheadclub-media/cache` recursively. Get "http://localhost:9000/metalheadclub-media/? delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=cache": read tcp [::1]:53662->[::1]:9000: i/o timeout

How could I get rid of 3 million small files? As it turned out, it was actually very simple…

Deleting files using S3 Lifecycle Rules…

because I had completely forgotten about a practical S3 mechanism: Lifecycle Rules. These rules allow you to specify what should happen to files during or after their lifecycle. One use case is automatic deletion. Exactly what I needed! And the best part is that this deletion takes place internally in the backend and does not have to be controlled by external tools. This allowed me to cleverly circumvent my timeout problem with file listing.

So I used the mc tool to expire all files in /cache after 15 days:

mc ilm add minio/metalheadclub-media/cache --expire-days 15

Done!

The current status can be viewed with

mc admin scanner status minio/

To make the Minio internal file scanner work faster and thus delete files more quickly when they have exceeded their expiration date, the process can be accelerated a little further:

mc admin config set minio/ scanner speed=fastest

Conclusion

After 15 days, all old files had been deleted and I was able to delete the Minio bucket, remove Minio, and also remove the old bucket from my Nginx configurations. The lines

# Migration Workaround ...
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 403 = @minio;

and

location @minio {
 ...
}

were removed and Nginx was reloaded. Metalhead.club was now completely migrated to SeaweedFS.

By the way, thanks to Stefano Marinelli (@stefano@bsd.cafe) for the SeaweedFS article under FreeBSD. This ultimately convinced me to give SeaweedFS a try and tackle the migration.

Migrating old Dovecot configuration to Dovecot 2.4 (Debian Trixie)

thomas.leister@mailbox.org (Thomas Leister) — Sat, 01 Nov 2025 10:10:33 +0100

The new Debian version “Trixie” has been available for upgrading from Debian “Bookworm” for several weeks now – and as a result, some of my readers have decided to update their mail server setup from my “Mail server with Dovecot, Postfix, MySQL, and Rspamd on Debian 10 Buster [v1.0] (German)”. The setup described for Debian Buster works just as well for Debian Bookworm – but Debian Trixie brings a change:

The latest Debian version makes a leap to Dovecot 2.4 with the included Dovecot version. This means that old Dovecot configurations are no longer compatible!

The configuration syntax has changed significantly in some respects, and old configurations from previous Dovecot versions can no longer be read. This article discusses the changes in the new version 2.4 and explains the migration using the example of the mail server instructions mentioned above. All changes take place in the file /etc/dovecot/dovecot.conf. No further changes (e.g., to the database schema or similar) are necessary.

You can find a complete configuration file at the end of this post.

Overall, the following has changed:

The way configuration parameters can be nested
Names of individual parameters
Variable names and functions

All changes are described in the Dovecot documentation. However, if you followed my instructions when setting up your mail server, you can simply continue reading and follow the steps described to achieve a working configuration.

Step 1: Add config version numbers

Dovecot 2.4 introduces versioning of the configuration syntax. Therefore, the following lines must be added at the beginning of the file:

# Dovecot config and storage versions
dovecot_config_version = 2.4.0
dovecot_storage_version = 2.4.0

It seems that the omissions of the past have now been rectified ;-). Future changes to the syntax can then be intercepted by Dovecot itself or warnings can be issued in the event of changes.

SSL settings

The SSL settings are renamed or changed as follows:

ssl_cert => ssl_server_cert_file (and omit the angle bracket at the beginning)
ssl_key => ssl_server_key_file (and omit the angle bracket at the beginning)
ssl_dh => ssl_server_dh_file (and omit the angle bracket at the beginning)
ssl_prefer_server_ciphers => ssl_server_prefer_ciphers. Values should not be “yes” or “no”, but ‘client’ or “server”. Default: Client. Setting can be removed.
ssl_min_protocol: can be omitted, default is already TLSv1.2
disable_plaintext_auth=yes => auth_allow_cleartext=no. Is default - can be omitted.

Overall, this results in:

ssl = required
ssl_server_cert_file = /etc/acme.sh/mydomain.tld/fullchain.pem
ssl_server_key_file = /etc/acme.sh/mydomain.tld/privkey.pem
ssl_server_dh_file = /etc/dovecot/dh4096.pem
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384: ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

PassDB / UserDB and auth_username_format

The two PassDB and UserDB sections become:

passdb sql {
query = SELECT username AS user, domain, password FROM accounts WHERE username = ‘%{user | username | lower}’ AND domain = ‘%{user | domain | lower}’ and enabled = true;
default_password_scheme = SHA512-CRYPT
}
userdb sql {
query = SELECT concat(quota, ‘M’) AS quota_storage_size FROM accounts WHERE username = ‘%{user | username | lower}’ AND domain = ‘%{user | domain | lower}’ AND sendonly = false;
iterate_query = SELECT username, domain FROM accounts where sendonly = false;
}

Pay special attention to the customized variables, e.g., %{user | username | lower }. Behind concat(quota, ‘M’) AS quota_storage_size there is also a small bug fix that I want to include here. ;-)

In addition, the variable is also customized for auth_username_format:

auth_mechanisms = plain login
auth_username_format = %{user | lower }

In order for the UserDB and PassDB sections just defined to work at all, the access data for the MySQL database is defined in a new SQL section:

sql_driver = mysql
mysql /var/run/mysqld/mysqld.sock {
user = vmail
password = mypassword
dbname = vmail
}

mypassword must of course be customized!

The file /etc/dovecot/dovecot-sql.conf can be deleted completely—it is no longer needed.

Mail location

The definition of the mail storage location mail_location is split into several parameters and replaced as follows:

mail_driver = maildir
mail_path = ~/mail
mailbox_list_layout = fs

In addition, the mail_home parameter is given new variable names:

mail_home = /var/vmail/mailboxes/%{user | domain }/%{user | username }

Protocols

The protocols setting and the two sections protocol imap and protocol lmtp become this block:

protocols {
lmtp = yes
imap = yes
sieve = yes
}
protocol imap {
mail_plugins {
imap_quota = yes
imap_sieve = yes
}
mail_max_userip_connections = 50
imap_idle_notify_interval = 29 mins
}
protocol lmtp {
mail_plugins {
sieve = yes
notify = yes
push_notification = yes
}
postmaster_address = postmaster@mydomain.tld
}

Plugins Section

The plugins segment

plugins {
...
}

no longer exists. Instead, the content becomes global.

Sieve Plugin Settings

sieve_plugins, sieve_before, and sieve become:

sieve_plugins {
sieve_imapsieve = yes
sieve_extprograms = yes
}
sieve_script spam-global {
type = before
path = /var/vmail/sieve/global/spam-global.sieve
}
sieve_script personal {
type = personal
path = /var/vmail/sieve/%{user | domain }/%{user | username }/scripts
active_path = /var/vmail/sieve/%{user | domain }/%{user | username }/active-script.sieve
}

All imapsieve* parameters become:

# From Mailbox to Spam
mailbox Spam {
sieve_script spam {
type = before
cause = copy
path = /var/vmail/sieve/global/learn-spam.sieve
}
}
# From Spam to another folder (learn HAM)
imapsieve_from Spam {
sieve_script ham {
type = before
cause = copy
path = /var/vmail/sieve/global/learn-ham.sieve
}
}

sieve_global_extensions becomes:

sieve_global_extensions {
vnd.dovecot.pipe = yes
}

Quota

quota and quota_exceeded_message become:

quota “User quota” {
driver = count
}
quota_exceeded_message = User %{user} has exceeded the storage volume. / User %{user} has exhausted allowed storage space.

The old “fs” driver for quota should no longer be used. Instead, count is now used. quota_exceeded_message also gets new variable names.

The quota plugin is activated in the new mail_plugins blog:

mail_plugins {
quota = yes
}

imap_quota has also been activated in the previously mentioned protocol imap block.

Changing access pemmissions

Since the MySQL access data is now stored directly in the main configuration file and is no longer stored in a separate file, access rights to dovecot.conf are further restricted. No one except root should be able to read the file:

chmod 600 /etc/dovecot/dovecot.conf

Complete sample configuration

For easier comparison, here is the complete file /etc/dovecot/dovecot.conf again:

# Dovecot config and storage versions
dovecot_config_version = 2.4.0
dovecot_storage_version = 2.4.0
###
### Protocol settings
#############################
protocols {
lmtp = yes
imap = yes
sieve = yes
}
protocol imap {
mail_plugins {
imap_quota = yes
imap_sieve = yes
}
mail_max_userip_connections = 50
imap_idle_notify_interval = 29 mins
}
protocol lmtp {
mail_plugins {
sieve = yes
notify = yes
push_notification = yes
}
postmaster_address = postmaster@mydomain.tld
}
##
## TLS Config
## Quelle: https://ssl-config.mozilla.org/#server=dovecot&version=2.3.9&config=intermediate&openssl=1.1.1d&guideline=5.4
##
ssl = required
ssl_server_cert_file = /etc/acme.sh/mydomain.tld/fullchain.pem
ssl_server_key_file = /etc/acme.sh/mydomain.tld/privkey.pem
ssl_server_dh_file = /etc/dovecot/dh4096.pem
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
###
### Dovecot services
################################
service imap-login {
inet_listener imap {
port = 143
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
mode = 0660
group = postfix
user = postfix
}
}
service auth {
### Auth socket für Postfix
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
### Auth socket für LMTP-Dienst
unix_listener auth-userdb {
mode = 0660
user = vmail
group = vmail
}
}
###
### SQL settings
###
sql_driver = mysql
mysql /var/run/mysqld/mysqld.sock {
user = vmail
password = mypassword
dbname = vmail
}
###
### Client authentication
#############################
auth_mechanisms = plain login
auth_username_format = %{user | lower }
passdb sql {
query = SELECT username AS user, domain, password FROM accounts WHERE username = '%{user | username | lower }' AND domain = '%{user | domain | lower}' and enabled = true;
}
userdb sql {
query = SELECT concat(quota, 'M') AS quota_storage_size FROM accounts WHERE username = '%{user | username | lower }' AND domain = '%{user | domain | lower}' AND sendonly = false;
iterate_query = SELECT username, domain FROM accounts where sendonly = false;
}
##
### Address tagging
###
recipient_delimiter = +
###
### Mail location
#######################
mail_uid = vmail
mail_gid = vmail
mail_privileged_group = vmail
mail_home = /var/vmail/mailboxes/%{user | domain }/%{user | username }
mail_driver = maildir
mail_path = ~/mail
mailbox_list_layout = fs
###
### Mailbox configuration
########################################
namespace inbox {
inbox = yes
mailbox Spam {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
}
###
### Mail plugins
############################
mail_plugins {
quota = yes
}
sieve_plugins {
sieve_imapsieve = yes
sieve_extprograms = yes
}
sieve_script spam-global {
type = before
path = /var/vmail/sieve/global/spam-global.sieve
}
sieve_script personal {
type = personal
path = /var/vmail/sieve/%{user | domain }/%{user | username }/scripts
active_path = /var/vmail/sieve/%{user | domain }/%{user | username }/active-script.sieve
}
# From Mailbox to Spam
mailbox Spam {
sieve_script spam {
type = before
cause = copy
path = /var/vmail/sieve/global/learn-spam.sieve
}
}
# From Spam to another folder (learn HAM)
imapsieve_from Spam {
sieve_script ham {
type = before
cause = copy
path = /var/vmail/sieve/global/learn-ham.sieve
}
}
# Sieve extensions only allowed in global context
sieve_global_extensions {
vnd.dovecot.pipe = yes
}
sieve_pipe_bin_dir = /usr/bin
###
### IMAP Quota
###########################
quota_exceeded_message = Benutzer %{user} hat das Speichervolumen ueberschritten. / User %{user} has exhausted allowed storage space.
quota "User quota" {
driver = count
}

Updates

2026-02-06: Added default_password_scheme to passdb section. Added “Changing access permissions” section.
2025-11-26: Removed user = vmail from service lmtp, to avoid lmtp(1301): Error: conn unix:/run/dovecot/anvil: net_connect_unix(/run/dovecot/anvil) failed: Permission denied error

Copying or moving Incus containers between hosts

thomas.leister@mailbox.org (Thomas Leister) — Sun, 19 Oct 2025 14:27:54 +0200

Below, I will briefly explain how to transfer Incus containers between two Incus hosts that are not connected to a shared cluster. In my case, both hosts are equipped with ZFS-based container storage. See also: “Running Incus with ZFS in a VM and moving containers to a new storage pool”

Open API

If a container is to be moved between two hosts, a transfer path must first be created between the two Incus hosts. To do this, the API is made available on the network interface on the source host:

incus config set core.https_address [2001:db8::1]:8443

The IPv6 address in square brackets must, of course, be replaced with the IP address of your own interface. For an IPv4 address, the brackets are omitted.

Next, a trust token is generated on the source server:

incus config trust add sourcehost

The source server is added to the target server:

incus remote add sourcehost 2001:db8::1

The trust token previously issued by the source server must be entered at the end. As soon as

Client certificate now trusted by server: sourcehost

appears in the console, the first transfer of the container from the source server to the target server can take place.

Transferring data

The container’s root file system can be transferred in two ways:

a) In one go, with downtime lasting as long as the data transfer. Recommended for small containers.
b) In several steps via a second, incremental sync. With a few seconds to minutes of downtime. Recommended especially for larger containers with 100+ GB. Only works with file systems such as ZFS that support snapshots.

a) Simple transfer for smaller containers

# (on the source server)
incus stop mycontainer
# (on the target server)
incus copy sourcehost:mycontainer mycontainer --stateless

Or b): Incremental transfer

With incremental transfer, the current status of the container is first transferred in the background. The parts that were changed during the transfer (because the container was still in operation) are delivered in a second step. However, this second sync runs much faster because it only transfers a delta and not all data again.

Create snapshot on the source server:

incus snapshot create mycontainer

(The container remains in operation and is not stopped yet)

Start initial transfer to the target server:

incus copy sourcehost:mycontainer mycontainer --stateless

After the first transfer, the source container is terminated and a second, final sync is triggered:

# (on the source server)
incus stop mycontainer
incus snapshot create mycontainer
# (on the target server)
incus copy sourcehost:mycontainer mycontainer --refresh

Finally, the container is started on the target server:

incus config unset mycontainer volatile.apply_template
incus start mycontainer

Close API

Finally, the API on the source server can be closed again if it is not to be used for other purposes:

incus config unset core.https_address

Running Incus with ZFS in a VM and moving containers to a new storage pool

thomas.leister@mailbox.org (Thomas Leister) — Sun, 19 Oct 2025 12:12:10 +0200

For many years, I have been relying on the virtualization and container management tool “Incus” (formerly “LXD”) to host my services. Incus runs in a virtual machine and helps me to create separation at the application level. For example, there is an Incus container for trashserver.net, another for metalhead.club, etc. The root file systems of the individual containers are located in a ZFS file system. This allows me to create space-saving snapshots of my containers before critical maintenance actions, e.g., before updates or operating system upgrades.

Since I recently upgraded the underlying storage, I would like to briefly introduce my storage setup and document for myself (but also for you ;-) ) what I paid attention to and how I moved my containers to the new storage.

At the hypervisor level (i.e., on the physical server), storage for my containers is provided in the form of LVM volumes. This allows me to offer the large storage space in many individual volumes for other VMs as well and to dimension it according to resource requirements.

An LVM volume (block device) is then passed to the VM via libvirt/QEMU and the virtio-blk driver. I use the following parameters for storage in libvirt:

Driver: virt-blk
Cache mode: none (cache is already handled by zfs)
io = native (usually better than io=threads)
discard = unmap (for Trim support)
serial = megastor

I would like to highlight the “serial” setting, which allows you to specify a serial number/ID/unique name for the transferred block device. If you connect multiple storage devices to your VM, each storage device can be identified without any doubt. For example, a device named “megastor” will appear in the VM as “/dev/disk/by-id/virtio-megastor”. This is particularly helpful if…

… you are no longer sure what is actually between /dev/vdb and /dev/vdd and which storage device is which?
… or you want to remove /dev/vdc, but you are afraid that the next time you reboot, /dev/vdd will move up and take its place. Then the chaos would be complete.

This is because device names for storage devices are not “stable” under Linux! Only partition UUIDs are stable (or alternatively disk IDs!).

In the VM, the block device is used to create a simple ZFS pool, e.g. via:

zpool create -o ashift=12 megastor /dev/disk/by-id/virtio-megastor

The ZFS pool is also registered in Incus so that it can be used with the containers:

incus storage create megastor zfs source=megastor --description="megastor storage for containers"

Trim Support

Some time ago, only the virtio-scsi driver supported trim to mark and release free storage space on the SSD. Regular “trimming” leads to significantly better performance. However, since some QEMU versions, the leaner “virtio-blk” driver also supports trimming.

Incidentally, ZFS regularly executes a trim cron job (/etc/cron.d/zfsutils-linux) itself via the zfsutils-linux package, so it is not necessary to execute fstrim yourself.

However, it is important to note that, as mentioned above, discard=unmap is defined for the QEMU storage device. And in my case, a “discard” parameter must also be entered in /etc/crypttab at the hypervisor level due to LUKS encryption. Otherwise, the trimming will not be passed through to our storage layer. Example:

luks-d429b69b-1c1a-234a-8bcf -64232e83f156 UUID=d429b69b-1c1a-234a-8bcf-64232e83f156 /var/lib/megastor.key discard

In addition, LVM should also be configured appropriately: vim /etc/lvm/lvm.conf:

issue_discards = 1

Moving existing Incus VMs to new storage

As mentioned at the beginning, I installed new SSDs in the system and set up new ZFS pools accordingly, so now the Incus containers should also be moved to the new “megastor” pool. There are several methods for doing this…

The easy way with a few minutes of downtime

This is how it works with Incus:

incus stop mycontainer
incus move mycontainer --storage megastor

Before the transfer, the container is stopped, which takes a certain amount of time depending on the size of the container and the performance of the storage, thus causing downtime. However, this method is less complex. It is worthwhile for smaller containers with a size of around 50–100 GB.

With incremental transfer – but short downtime

If you want to avoid downtime as much as possible, you can proceed differently:

Create a snapshot of the source container during operation
Copy the snapshot to the destination in the background
Stop the source container (start of downtime)
Transfer the difference since the last transfer (usually only a few MB to GB)
Remove the old container and rename the new container.
Start the new container (end of downtime).

Downtime is reduced to a minimum. I have already described the basic procedure in my article “Move a Mastodon instance with less than 3 minutes downtime (LXD/ZFS-based)” and have worked around LXD, so to speak. However, the same also works with Incus on-board tools and can be applied to the transfer of a container from one storage to another (not just to the transfer between two hosts!).

# 1. Create a snapshot during operation
incus snapshot create mycontainer
# 2. Transfer the snapshot to the new storage “megastor”
incus copy mycontainer new-mycontainer --storage megastor
# 3. Stop the container and create another snapshot
incus stop mycontainer
incus snapshot create mycontainer
# 4. Transfer changes since the start of the first transfer
incus copy mycontainer new-mycontainer --storage megastor --refresh
# 5. Delete old container, rename new container
incus delete mycontainer
incus move new-mycontainer mycontainer
# 6. Start new container
incus config unset mycontainer volatile.apply_template
incus start mycontainer

The incus config unset... ensures that the new container retains the old MAC addresses, IPs, etc., and is not assigned new addresses.

If you want, you can also insert additional “snapshot - copy” sequences between steps 2 and 3:

incus snapshot create mycontainer
incus copy mycontainer new-mycontainer --storage megastor --refresh

This is particularly useful if a lot of time has passed since the first transfer. If a lot has changed in the container during this time, the final sync will also take significantly longer and result in longer downtime. This can be counteracted by approaching it in several “snapshot - copy” steps. It is important not to forget the --refresh flag after the first copy command.

Transfer Mastodon S3 storage from self-hosted Minio to Hetzner S3

thomas.leister@mailbox.org (Thomas Leister) — Wed, 24 Sep 2025 17:32:04 +0200

On behalf of a customer, I recently had the task of transferring an S3 bucket for a Mastodon instance from a self-hosted Minio instance to Hetzner S3. I would like to explain the procedure and the configurations I used here.

The client’s Mastodon instance had previously been supplied with media and media cache by a Minio instance on the same server. However, the media cache continued to grow as the Mastodon instance became increasingly networked. Even by reducing the cache retention period to a few days, the costs for S3 storage were too high. This was because it was located on a powerful but relatively expensive external volume of a Hetzner cloud server.

At the end of 2024, Hetzner introduced Hetzner S3 storage to its product portfolio, opening up an attractive alternative to the Minio instance. An alternative that stands out above all for its low cost. The cost for a 300 GB volume was just under €16/month. Not very much for a business – but certainly a lot for a small, donation-funded Mastodon instance. In comparison, the price for 1000 GB (!) of Hetzner S3 storage is just under €6/month. So the decision to switch was quickly made.

Step 1: Parallel operation of two buckets

To make the transition to the new storage as smooth as possible for users and to keep downtime to a minimum, I decided to run both buckets in parallel during the data migration.

Old bucket “minio”: Delivers the media files stored to date (read-only)
New bucket “hetzner-s3”: Receives new media files, stores them, and delivers them

The upstream Nginx proxy was configured so that the new bucket was queried first when a file request was made. If the requested file could not be found there, the old S3 bucket was queried.

Proxy configuration before:

upstream minio {
 server 127.0.0.1:9000;
}

server {
 [...]

 location / {
 proxy_set_header Host 'mastodon-media.s3.domain.tld';
 proxy_set_header Connection '';
 proxy_set_header Authorization '';
 proxy_hide_header Set-Cookie;
 proxy_hide_header 'Access-Control-Allow-Origin';
 proxy_hide_header 'Access-Control-Allow-Methods';
 proxy_hide_header 'Access-Control-Allow-Headers';
 proxy_hide_header x-amz-id-2;
 proxy_hide_header x-amz-request-id;
 proxy_hide_header x-amz-meta-server-side-encryption;
 proxy_hide_header x-amz-server-side-encryption;
 proxy_hide_header x-amz-bucket-region;
 proxy_hide_header x-amzn-requestid;
 proxy_ignore_headers Set-Cookie;

 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 proxy_connect_timeout 300;
 proxy_http_version 1.1;
 proxy_set_header Connection "";
 chunked_transfer_encoding off;

 add_header 'Access-Control-Allow-Origin' '*';
 add_header X-Cache-Status $upstream_cache_status;
 add_header X-Content-Type-Options nosniff;
 add_header Content-Security-Policy "default-src 'none'; form-action 'none'";

 proxy_pass https://minio;
 }
}

Proxy configuration afterwards - with parallel operation of both buckets (fallback to old bucket):

upstream hetzner {
 server mastodon-media.nbg1.your-objectstorage.com;
}

upstream minio {
 server 127.0.0.1:9000;
}

server {
 [...]

 resolver 9.9.9.9;

 location / {
 proxy_set_header Host 'mastodon-media.nbg1.your-objectstorage.com';
 proxy_set_header Connection '';
 proxy_set_header Authorization '';
 proxy_hide_header Set-Cookie;
 proxy_hide_header 'Access-Control-Allow-Origin';
 proxy_hide_header 'Access-Control-Allow-Methods';
 proxy_hide_header 'Access-Control-Allow-Headers';
 proxy_hide_header x-amz-id-2;
 proxy_hide_header x-amz-request-id;
 proxy_hide_header x-amz-meta-server-side-encryption;
 proxy_hide_header x-amz-server-side-encryption;
 proxy_hide_header x-amz-bucket-region;
 proxy_hide_header x-amzn-requestid;
 proxy_ignore_headers Set-Cookie;

 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 proxy_connect_timeout 300;
 # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
 proxy_http_version 1.1;
 proxy_set_header Connection "";
 chunked_transfer_encoding off;

 add_header 'Access-Control-Allow-Origin' '*';
 add_header X-Cache-Status $upstream_cache_status;
 add_header X-Content-Type-Options nosniff;
 add_header Content-Security-Policy "default-src 'none'; form-action 'none'";

 proxy_pass https://hetzner; # This uses the upstream directive definition to load balance

 # Fallback to oldbucket
 proxy_intercept_errors on;
 error_page 404 403 = @oldbucket;
 }

 location @oldbucket {
 proxy_set_header Host 'mastodon-media.s3.domain.tld';
 proxy_set_header Connection '';
 proxy_set_header Authorization '';
 proxy_hide_header Set-Cookie;
 proxy_hide_header 'Access-Control-Allow-Origin';
 proxy_hide_header 'Access-Control-Allow-Methods';
 proxy_hide_header 'Access-Control-Allow-Headers';
 proxy_hide_header x-amz-id-2;
 proxy_hide_header x-amz-request-id;
 proxy_hide_header x-amz-meta-server-side-encryption;
 proxy_hide_header x-amz-server-side-encryption;
 proxy_hide_header x-amz-bucket-region;
 proxy_hide_header x-amzn-requestid;
 proxy_ignore_headers Set-Cookie;

 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 proxy_connect_timeout 300;
 proxy_http_version 1.1;
 proxy_set_header Connection "";
 chunked_transfer_encoding off;

 add_header Cache-Control public;
 add_header 'Access-Control-Allow-Origin' '*';
 add_header X-Cache-Status $upstream_cache_status;
 add_header X-Content-Type-Options nosniff;
 add_header Content-Security-Policy "default-src 'none'; form-action 'none'";

 proxy_pass http://minio;
 }
}

The new “location” block “@oldbucket” and the reference to it if files cannot be found in the new bucket are decisive here:

# Fallback to oldbucket
proxy_intercept_errors on;
error_page 404 403 = @oldbucket;

It is important at this point not only to catch 404 errors, but also 403 errors (“forbidden”). Hetzner S3 Storage apparently responds with this error code when a resource is requested whose parent directories do not (yet) exist.

Step 2: Saving new media in the new bucket

Up to this point, users of the Mastodon instance should hardly have noticed anything. The proxy change takes effect in a fraction of a second and is completely transparent to the user. However, there will be a brief downtime of several seconds as the Mastodon configuration is adjusted and all Mastodon services must be restarted.

The following settings were used as an example for the new bucket in Mastodon’s .env.production:

S3_ENABLED=true
S3_BUCKET=mastodon-media
AWS_ACCESS_KEY_ID=<myaccessid>
AWS_SECRET_ACCESS_KEY=<myaccesskey>
S3_ALIAS_HOST=media.domain.tld
S3_REGION=nbg1
S3_PROTOCOL=https
S3_HOSTNAME=media.domain.tld
S3_ENDPOINT=https://nbg1.your-objectstorage.com
S3_SIGNATURE_VERSION=v4

The changes will take effect once the Mastodon services have been restarted. New media files will now be stored exclusively in the new S3 bucket:

cd /etc/systemd/system
systemctl restart mastodon-*

After a few seconds, Mastodon is ready to go again.

Step 3: Migrate old data to new bucket

The critical part is now complete. Now you can relax and transfer the old data to the new bucket. To do this, I use the “rclone” tool, which makes it very easy to synchronize two S3 buckets with each other. To do this, configurations for the two S3 providers are first created under ~/.config/rclone/rclone.conf. Here is another example:

[hetzner-s3]
type = s3
provider = Other
access_key_id = <accessid>
secret_access_key = <accesskey>
endpoint = nbg1.your-objectstorage.com
acl = public-read
region = nbg1

[minio-s3]
type = s3
provider = Minio
access_key_id = <accessid>
secret_access_key = <accesskey>
endpoint = s3.domain.tld
acl = public-read
region = server1

The endpoint, region, access ID, and access key must, of course, be adapted to the respective bucket. Another important setting is acl = public-read, which ensures that transferred files have the correct permissions. If this setting is not set correctly, the files may not be publicly readable and the media storage cannot be used in the sense of Mastodon.

Now follows a series of rclone commands that move the media files from the old minio-s3 to the new hetzner-s3 storage. Please note that the copy subcommand of rclone is used here. This does not synchronize deletions, but only transports non-existent files from A to B. Unlike sync! This would delete newly arrived files in the new bucket. So caution is advised. copy is the right command for our purpose.

The most important files first:

rclone copy --progress --transfers=8 minio-s3:mastodon-media/custom_emojis/ hetzner-s3:mastodon-media/custom_emojis/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/accounts/ hetzner-s3:mastodon-media/accounts/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/site_uploads/ hetzner-s3:mastodon-media/site_uploads/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/media_attachments/ hetzner-s3:mastodon-media/media_attachments/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/imports/ hetzner-s3:mastodon-media/imports/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/accounts/ hetzner-s3:mastodon-media/cache/accounts/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/custom_emojis/ hetzner-s3:mastodon-media/cache/custom_emojis/

This is followed by the less important media files. These are only cached media and preview images for websites. If you want to save time and traffic, you can also omit these (or reduce the retention time for these media in Mastodon to a few days in advance—this will significantly reduce the amount of data to be transferred!).

rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/preview_cards/ hetzner-s3:mastodon-media/cache/preview_cards/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/media_attachments/ hetzner-s3:mastodon-media/cache/media_attachments/

This process can take up to half a day, depending on the performance of the servers involved and the size of the bucket.

Step 4: Disconnect the old bucket

Once all data has been transferred, it is advisable to keep the old bucket for a while in case any data has not been transferred correctly. However, you can disconnect the old bucket from the proxy on a trial basis and check whether the Mastodon instance runs correctly without the old bucket. To do this, remove the @oldbucket location block and comment out (or remove) the following lines:

#upstream minio {
# server 127.0.0.1:9000;
#}

[...]

# Fallback to oldbucket
#proxy_intercept_errors on;
#error_page 404 403 = @oldbucket;

After running systemctl reload nginx, only the new bucket will be used and the website’s functionality can be checked. Don’t forget: For serious testing, clear your browser cache!

If everything is still running smoothly after a few days, the old bucket can be deleted. Done!

Automatic dark mode for Drawio diagrams using CSS filters

thomas.leister@mailbox.org (Thomas Leister) — Fri, 01 Aug 2025 06:05:59 +0200

In my latest blog post “Nginx HTTP/3 proxy server displays content from the wrong virtual host,” I included a diagram from Diagrams.net / Drawio, which is an SVG file.

When you export your diagram from Draw.io, you can choose whether you want to apply a light or dark color theme to the exported graphic. I usually choose the light one here so that the diagram fits well with the light default theme of thomas-leister.de. But how do I deal with this in the dark version of my blog? Users automatically see the dark theme if they have set their operating system accordingly. I wondered if there was an easy way to convert the light SVG graphics to dark versions if necessary. After all, CSS now supports filter, right?

Fortunately, I was able to find a simple solution to the problem fairly quickly: “One line - Dark Mode using CSS”. And this is the trick I found:

filter: invert(1) hue-rotate(180deg);

This CSS line first inverts all colors (i.e., from light to dark and vice versa). Black is converted to white (and vice versa), but colors are also inverted. This is not intentional. Therefore, colored content is converted back using a hue-rotate.

I implemented the following in my blog’s CSS stylesheet:

@media (prefers-color-scheme: dark) {
 /*
 * Invert colors for SVG diagrams
 */
 figure.diagram img {
 filter: invert(1) hue-rotate(180deg);
 }
}

Only graphics located in an <figure> HTML element of the class diagram are affected by the conversion – and only if the user has selected a dark theme in their operating system settings.

If I now want to embed an SVG drawing in a blog entry, I do so using the following shortcode:

{{< figure src="images/my-drawing.drawio.svg" caption="My description" class="diagram" >}}

And this is what the result can look like (top in dark mode – bottom in light mode):

Nginx HTTP/3 proxy server displays content from the wrong virtual host

thomas.leister@mailbox.org (Thomas Leister) — Wed, 30 Jul 2025 17:38:27 +0200

At the end of May 2025, I published the metalhead.club song via a Faircamp website at music.metalhead.club. However, not all users were able to follow my links to the Faircamp site without problems. In a few cases, users reported at least one of the following errors to me:

When users accessed the album page: Error 404 - not found
When users accessed the main page: The watch.metalhead.club globe was displayed

After a few attempts, I was able to reproduce the error sporadically myself. In the access logs of the Nginx proxy, I noticed that all erroneous requests were made with HTTP/3.

Since the proportion of HTTP/3-compatible web browsers is steadily increasing, I had decided a few months earlier to use a very recent version of Nginx with HTTP/3 enabled on the server. However, I had not enabled HTTP/3 for music.metalhead.club. This raises the question: …

Why is HTTP/3 used?

For my Mastodon instance at metalhead.club, HTTP/3 was enabled as the only virtual host on the Nginx server, primarily to reduce connection setup time for users located further away and thus shorten loading times. For this virtual host, the Alt-Svc header was also set to “h3” (i.e., HTTP/3). However, this setting did not apply to other virtual hosts such as music.metalhead.club – so why did various web browsers still send HTTP/3-based requests to the non-HTTP/3 virtual host music.metalhead.club?

To explain this phenomenon, it is important to know that HTTP/3, like its predecessor HTTP/2, has a feature called “connection coalescing.” If …

IP address
port, and
SSL certificate

are the same, “old” QUIC connections are reused (see also RFC 7540 “Connection reuse” – also known as “Connection Coalescing”). So even though a different virtual host on the server may be addressed later, an existing QUIC connection to the server is used if the three parameters match. Whether this “other” virtual host supports HTTP/3 at all is no longer checked. In my case, the following apparently happened:

A user first visits metalhead.club. The browser recognizes HTTP/3 support via the “Alt-Svc” header
Further connections to metalhead.club are now established using HTTP/3 – a QUIC connection is set up for this purpose
The user then navigates to music.metalhead.club. This other virtual host uses the same IP/port/SSL certificate combination (because of the wildcard certificate). The browser recognizes this and uses the existing QUIC connection

As a result, the web browser now also communicates with the web server using HTTP/3 when accessing music.metalhead.club.

… but music.metalhead.club does not speak QUIC!

However, there is no QUIC listener in the Nginx configuration for music.metalhead.club. And this is where the second pitfall comes into play: If a virtual host that does not have a QUIC listener is requested via a QUIC connection, Nginx searches for the “default” QUIC virtual host. The “default” vHost is the one that has been explicitly defined as such – or the one that (as in my case) has been implicitly defined by Nginx through the configuration order. In my case, there was only one vHost that speaks QUIC: the vHost for metalhead.club.

Requests to music.metalhead.club were ultimately processed by the vHost for metalhead.club because there was no other vHost with QUIC enabled.

Since this vHost is only a proxy that points to another Nginx instance (“Nginx app server”), the request is forwarded again. The app server Nginx instance in my setup finally receives the request for music.metalhead.club – but cannot do anything with it. After all, it does not have a vHost that could be responsible for music.metalhead.club (see diagram). So once again, a “default” vHost is selected. And since watch.metalhead.club is the first vHost that appears in the configuration of the second Nginx, the request is forwarded to watch.metalhead.club and answered there: The user sees the globe on the home page… or a 404 error page, because the requested album overview does not exist on watch.metalhead.club, of course.

Red: The incorrect route between the two Nginx instances. Green: The intended route. (1) Due to the missing QUIC listener, the metalhead.club vHost is addressed. (2) This in turn now points to the wrong Nginx app server.

The solution

Mystery solved! But how can the problem be fixed? The core problem is that web browsers assume that QUIC connections can also be reused for other virtual hosts if the IP address, port, and certificate are the same. Therefore, we should create the appropriate conditions for this on the server side. The solution to the problem is therefore to enable QUIC for all virtual hosts that share an IP address/port/certificate combination. In my case, I also enabled QUIC for music.metalhead.club (and watch.metalhead.club) on the proxy Nginx. This solved the problem.

Speeding up global DNS resolution by avoiding CNAMES

thomas.leister@mailbox.org (Thomas Leister) — Thu, 19 Jun 2025 10:27:52 +0200

As shown in my article “A small CDN for my Mastodon instance metalhead.club,” I like to use CNAMEs to organize my DNS entries. I usually create a DNS record for each host that maps its hostname to the IP. Then I use one or more CNAME entries to link certain (sub)domains to these CNAMEs depending on the service and purpose. This helps with overview and can make organization easier—especially if IP addresses for hosts need to be changed. Thanks to the chaining of entries, only the last mapping from server hostname to IP needs to be adjusted if the IP address of the target host changes.

Using the example from the article mentioned above, a CNAME chain might look like this:

[thomas@thomas-nb]~% dig media.metalhead.club
;; ANSWER SECTION:
media.metalhead.club. 1800 IN CNAME metalheadclub-media.cdn.650thz.de.
metalheadclub-media.cdn.650thz.de. 21600 IN CNAME s3.650thz.de.
s3.650thz.de. 3600 IN A 5.1.72.141

So first media.metalhead.club is resolved, then metalheadclub-media.cdn.650thz.de, and finally s3.650thz.de. Only then is the IP address for the target host determined. I like this in the sense of order, because the structure makes sense to me in my head. However, as I discovered when analyzing my small CDN for metalhead.club, this is not particularly conducive to performance. This is because the use of CNAMES also has a significant disadvantage: the time required for DNS resolution increases with each CNAME in the chain up to the IP address.

For example, resolving the second line from the “Answer Section” alone means: first resolve .de (query DNS root server), then query .de nameserver for 650thz.de, then query 650thz.de nameserver for cdn, and finally query the cdn nameserver for metalhead.club-media. Only then is it clear: s3.650thz.de must be resolved. The game starts again until the IP address is finally determined.

It should be clear that this chain of two CNAME entries takes time to be resolved by the user’s DNS resolver. For regional users who are within range of these name servers, the effort is not so significant, as resolving each link in the chain usually only takes a few tenths of a millisecond.

However, the situation is different for users who (as in the case of metalhead.club) have to access name servers in Europe from the US or Australia, for example. This is because:

Root server: Global
.de name server: Global
650thz.de: Regional, EU
cdn.650thz.de: Regional, EU

My chain involves four zones on four (virtual) name servers. Each one must be queried individually, and only two of them are globally positioned, meaning they can be accessed worldwide with minimal latency. All regional servers are only accessible to those outside the EU with significantly increased runtimes. The fact that two (from their perspective “slow”) regional DNS name servers have to be queried in the case of a US user means that the increased latencies add up very quickly.

From the perspective of a US user or a user from Asia, it is therefore particularly important that resolution chains are kept as short as possible. This becomes particularly clear when using a tool such as globalping.io and looking at the global resolution times in DNS mode:

While EU users get media.metalhead.club (media-old.metalhead.club in the screenshot) resolved in about 0.004–0.070 seconds, someone in Japan has to wait a lot longer: about 1.5 seconds! Only then can their browser start connecting to the final destination server.

Of course, this only applies to the first DNS query after the TTL has expired. Subsequent queries are usually answered by the DNS resolver from the cache, so they are answered orders of magnitude faster. However, after the TTL expires, the first request takes another 1.5 seconds! However, given the small number of users in some regions, it cannot be assumed that they use a common DNS resolver. Therefore, it is important to me that uncached requests are also answered quickly.

So I decided to do without CNAME chains altogether. And this is how it worked:

In the metalhead.club zone, I delegated the subdomain media.metalhead.club directly to Scaleway’s GeoIP nameserver.
The Scaleway nameserver then resolves directly to an IP address that matches the user’s region.

This means that CNAMEs are no longer involved in the resolution: the entire resolution is now based solely on zone delegation.

The whole process could be improved further by taking the following measures:

No separation of “normal” DNS nameserver (Core-Networks.de) and GeoIP nameserver (Scaleway): This would eliminate the need for the “media” zone delegation.
Global accessibility of all nameservers (however, this is a significant cost factor and also requires a new DNS provider).

But even without these two improvements, I was able to significantly reduce resolution time simply by dispensing with CNAMES, as this screenshot shows:

That’s much better: after optimization by dispensing with CNAMES, the resolution of media.metalhead.club now takes only around 0.7 seconds instead of 1.9 seconds!

By the way: for technical reasons, the main domain metalhead.club resolves to IP addresses anyway. No optimization was necessary here.

Creating my own small CDN for my Mastodon instance metalhead.club

thomas.leister@mailbox.org (Thomas Leister) — Thu, 19 Jun 2025 08:11:04 +0200

Since the big Twitter wave that flooded the Mastodon network and, more broadly, the Fediverse in the fall and winter of 2022, international users have been playing a bigger role for metalhead.club. The service is hosted entirely in Germany, and that was still the case until recently. However, with the increasing number of international members come new challenges: for example, the rapid delivery of content.

As long as users are mainly located in Germany and Europe, latency times to the “Full Metal Server” in Frankfurt are low. However, the situation is different for users from Canada, the US, and Australia, for example, of whom there are a significant number on metalhead.club. For these users, using metalhead.club was sometimes a bit of a test of patience, as videos and larger images in particular appeared on the website with a slight delay. I can only simulate the situation in the browser, but even a ping of more than 200 ms spoils the fun of scrolling through the timeline in some places.

Recently, a US user brought a specific problem to my attention when playing a video within the US. He could only stream the iFixit video with constant interruptions. This prompted me to finally consider a CDN – a content delivery network.

The purpose of such a network is to bring the content available to a user of a platform or website closer to the user, so to speak. This is not just meant metaphorically, but is actually a physical matter: even if we assume that internet nodes, signal repeaters, and other equipment do not introduce additional latency into the system on the route from here to the US, it is well known that light in undersea cables is not infinitely fast. This fact alone means that web content must actually be offered (physically) closer to the customer. A CDN therefore consists of many data center locations or servers that are installed close to users – ideally in metropolitan areas, where you can also benefit from a well-developed network infrastructure. Depending on the use case, this can be in certain regions of the world or even across the globe. The content is mirrored on all (or some) of the servers, depending on requirements. When requesting content, a US user is not redirected to an EU server, but automatically retrieves the content from a server in their vicinity (a US data center). This process usually remains transparent to the user. There are two established methods for determining the location from which content is accessed (and which server is responsible):

Anycast-based CDNs
“GeoIP”-based CDNs

Anycast-based CDNs

The first CDN type is the “gold standard” and enables very elegant (and accurate) routing: In principle, the same public IP address is assigned to several servers scattered around the world. The routing protocol BGP decides which server is actually addressed when a request is made. This protocol essentially determines the paths that an IP packet must take to reach its destination. In most cases, there are several ways to reach the destination. The routing system knows which one involves the lowest costs (or the lowest latency!). In this respect, the work is handed over to the routing system and it is relied upon to find the fastest route through the Internet. That is what it is there for – that is what it was developed for. The route does not always have to be the shortest geographically. If the latency on route A to server A is less than route B to server B, the IP packets are automatically routed to server A. This server then responds. The other servers that can be reached under the same IP address are not aware of the request. Incidentally, such an anycast is also behind all globally available DNS resolvers such as Google DNS, Quad9, or Cloudflare DNS. The IP address is always the same, but in the background, you are redirected to the nearest server without noticing.

GeoIP-based CDNs

The term GeoIP refers to a large database – a GeoIP database, such as those offered by companies like MaxMind (and used, for example, in my project watch.metalhead.club). The database is fed with publicly available routing information and stores which IP address ranges (and thus ASNs) are assigned to which companies. Once the ownership of an IP address is clear, conclusions can be drawn about the geographical area of use via the assigned company. IPv6 addresses that come from the 2003::/19 network (i.e., 2003:: to 2003:1fff:ffff:ffff:ffff:ffff:ffff:ffff) belong to Deutsche Telekom, for example. This also makes it clear where a user with such an IP address comes from when requesting content: from Germany. GeoIP databases store these relationships so that they can be queried: “Where does a user with the IP address xxx come from?” The database then provides a more or less accurate answer.

The accuracy of GeoIP-based CDNs

My experience with the free offerings of such databases has shown me that this works quite well at the country level, but it is usually not possible to locate users down to the city level. And perhaps that is just as well. There are better and worse databases. The most accurate databases are usually behind a costly subscription model. This is because GeoIP databases need to be maintained. IP addresses and subnets are traded and sold, especially in times of IPv4 address shortages, and sold to other providers. An address range that just belonged to an African mobile phone provider may belong to a small Asian company tomorrow. If the databases are not regularly maintained, the information becomes outdated and worthless.

And this is precisely where the major disadvantage of GeoIP lies. While Anycast-based CDNs rely on routing systems already knowing the shortest routes (in terms of latency), GeoIP relies on additional collected information. The assignment of IP address <=> owner is not difficult to determine and is public. The difficulty is much more likely to lie in correctly determining the type of use and region of use for an owner. After all, it could be a small business, but it could also be an ISP that only distributes the addresses to its customers. Only the owner or ISP knows how such distribution takes place and how large the geographical area of use is. Additional tracking information can be used to make this assignment more accurate. For example, thanks to smartphones, an online advertising company could combine GPS location and IP address to determine a user’s location more accurately. If this GPS information is then sold to the GeoIP provider, the latter can determine the location of the IP address much more accurately. But that’s another story…

How does a user get forwarded to the right server if Anycast is not used? It’s simple: via the Internet’s directory service – the DNS.

If you want to build a GeoIP-based CDN, you need a DNS server that returns the appropriate IP address for the user’s region. For each request to the DNS service, the IP address is determined and the appropriate target host is identified in the background based on a GeoIP database. I deliberately write “the IP address” because which IP address is examined depends …

Resolver IP or EDNS Client Subnet (ECS)?

One might assume that the address of the requesting user would be examined, for example, the public IP address of the smartphone or laptop, but this is not the case. This is because the name server responsible for my domain, e.g., metalhead.club, almost never sees this client. It is the DNS resolvers that request the correct IP for a service for the client – not the clients themselves. So all the GeoIP name server can evaluate is the IP address of the resolver.

In many cases, this means that the user themselves has not been located, but their DNS resolver can be identified. In most cases, for private customers, this is the standard DNS resolver of the customer’s ISP, for example Deutsche Telekom. In most cases, the location of the DNS resolver therefore roughly corresponds to the location of the customer – at least when viewed at country level. So if my name server sees an IP that can be assigned to a Deutsche Telekom resolver, I can usually assume that the user making the request is also from Germany.

However, there are also special cases: namely, globally available and ISP-independent, open DNS resolvers. For example:

Cloudflare DNS
Google DNS
Quad9
DNS4EU

These often do not allow conclusions to be drawn about a specific country. In order to still be able to roughly locate the customer actually making the request behind the DNS request, there is an EDNS extension called “ECS” (EDNS Client Subnet) that allows additional information to be sent to my name server: namely, the subnet of the original user. The resolver naturally knows this client IP and forwards it to my name server in anonymized form (only the subnet instead of the exact IP address). The name server can now use the subnet forwarded via ECS instead of the resolver IP for its GeoIP database and obtains a more precise result for the geographical origin of the request.

ECS is not supported by all public resolvers – DNS4EU, for example, does not yet support ECS – even though the feature is on the operator’s to-do list:

DNS4EU currently does not support EDNS, but we want to add support for this feature. Unfortunately, we do not have a timeline on when this should be done. Please check the project page https://www.joindns4.eu/learn for updates.

So what should you use? A CDN provider or self-hosting?

Anycast systems are expensive. Very expensive. And impossible for someone like me to implement. Because to do so, you would need to have your own IP address range for which you can define BGP routes yourself. So the issue was quickly settled for me. Anycast? I don’t operate it myself. But there is another option: After all, you can rent space in existing Anycast systems or simply rent CDNs. Renting Anycast IPs is also costly and is not offered in this form by the hosts I use or want to use.

However, freely rentable CDNs are a dime a dozen: BunnyCDN, Akamai, Cloudflare, KeyCDN, … these are just a few popular examples. These CDNs are often sold in combination with DNS services, S3 storage, or DDoS protection. There are also some offers that are affordable for small platform operators on the internet. I was particularly interested in BunnyCDN. For around €20-25 per month, I could have the approximately 2.5 TB of metalhead.club media played worldwide every month. However, there are two issues that currently concern me:

I would have to relinquish TLS termination and have the media hosted externally. However, my users rely on me to keep all data in my own hands.
The green electricity issue: It has become easier to find server hosts that operate exclusively with green electricity. Unfortunately, the situation is not so good with CDN providers: Only Cloudflare seems to use 100% green electricity.

This means that renting space in an existing CDN is also out of the question (for now?).

Building your own Geo-IP-based CDN

So the only option left is to build it yourself!

My expectations are relatively low. I am, of course, aware that a self-built GeoIP-based CDN will quickly reach its limits in terms of efficiency, accuracy, and performance. This is especially true if, like me, you plan to invest very little money in it. My primary goal is to improve the user experience for some of my metalhead.club members. I won’t achieve the perfect solution this way.

Since the supply situation within Europe is fine and latency measurements using online tools (more on that later!) have delivered satisfactory results, I have focused on two locations in particular: the American continent and Asia. With a total of three media servers for media.metalhead.club, this would roughly cover the entire globe.

Using the “Ping Simulation” tool, I looked at what latencies I can expect globally if I distribute the servers as I had imagined. Namely, as the server host Hetzner allows me to do with its cloud. I already have a Hetzner account and am familiar with their products – plus, they meet my requirements (German company, GDPR-compliant, 100% green electricity). And when it comes to locations, Hetzner offers exactly what I want. In addition to European data centers, there are also two in the US and one in Singapore.

The simulation showed that I can reach large parts of the world with around 100 ms or less. Of course, this always depends on the local provider, but the overall picture was fairly consistent and only in a few locations were the packet times significantly higher than 150 ms. So I quickly booked a cloud server in Ashburn (VA, USA) and one in Singapore with Hetzner.

Implementation

As far as vServers are concerned, I opted for the smallest and cheapest models offered by Hetzner. I wanted to start small and try it out first. As it turns out, the CPX11 cloud servers have been more than sufficient so far. They only need to cache and deliver static files – nothing more.

Nginx caching proxy

Only one Nginx instance runs on the servers, which

… receives requests to media.metalhead.club
checks whether the requested content is already available locally
(if not, it is requested from the original S3 bucket and cached)
returns it to the client

My configuration for Nginx:

proxy_cache_path /tmp/nginx-cache-metalheadclub-media levels=1:2 keys_zone=s3_cache:10m max_size=30g
inactive=48h use_temp_path=off;
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name media.metalhead.club;
include snippets/tls-common.conf;
ssl_certificate /etc/acme.sh/media.metalhead.club/fullchain.pem;
ssl_certificate_key /etc/acme.sh/media.metalhead.club/privkey.pem;
client_max_body_size 100M;
# Register backend URLs
set $minio_backend 'https://metalheadclub-media.s3.650thz.de';
root /var/www/media.metalhead.club;
location / {
access_log off;
try_files $uri @minio;
}
#
# Own Minio S3 server (new)
#
location @minio {
limit_except GET {
deny all;
}
resolver 9.9.9.9;
proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_pass $minio_backend$uri;
# Caching to avoid S3 access
proxy_cache s3_cache;
proxy_cache_valid 200 304 48h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_lock on;
proxy_cache_revalidate on;
expires 1y;
add_header Cache-Control public;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
add_header X-Content-Type-Options nosniff;
add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
add_header X-Served-By "cdn-us.650thz.de";
}
}

By the way: Adding the X-Served-By header makes debugging and checking the function easier. For every image loaded from media.metalhead.club, it is very easy to determine which of the three hosts actually transferred the file. The header can be viewed in the developer tools of any web browser, for example.

GeoIP-based DNS zones

Now all that’s missing is the GeoIP component so that metalhead.club members are redirected to the appropriate server. At this point, I could have configured my own GeoIP/GeoDNS-enabled name server (as I have already done in a customer project), but I didn’t want to put too much effort into my experiment. So I looked around a bit and found a very interesting offer at Scaleway. There, you can rent name servers that not only support the GeoIP feature but can also be used with external domains (i.e., domains that are not registered with Scaleway). Depending on usage, such a service costs only a few cents to a low single-digit euro amount in my case. I’ll be able to say exactly how much once my CDN has been running for a while ;-)

However, I didn’t want to hand over my 650thz.de root zone to Scaleway. This should continue to be hosted by Core-Networks.de. Instead, I created my own zone “cdn.650thz.de” and set the Scaleway name servers as authoritative name servers. However, I first had to prove ownership of cdn.650thz.de to Scaleway. So I haven’t set the NS entries yet, but first created a verification entry in the metalhead.club zone:

_scaleway-challenge IN TXT 3600 "verifizierungs-string"

After about 30 minutes, it was confirmed that I had control over the domain, and I was able to start entering the Scaleway name servers for cdn.650thz.de in the 650thz.de zone at Core-Networks.de:

cdn 86400 NS ns0.dom.scw.cloud. cdn 86400 NS ns1.dom.scw.cloud.

(I deleted the _scaleway-challenge entry again)

By the way: I use the 650thz.de domain for all infrastructure in the background, as metalhead.club belongs to the 650thz.de project. Don’t let that confuse you.

In the cdn.650thz.de zone, I finally added the GeoIP entry for metalheadclub-media.cdn.650thz.de:

The default entry points to s3.650thz.de – the previous S3 media server in Frankfurt. However, if a client location in North or South America is detected, the CDN server cdn-us.650thz.de is used instead. If a location in Asia or the Pacific region is detected, the CDN server cdn-ap.650thz.de is referenced.

Incidentally, the period at the end of each FQDN in the input mask is essential! If no period is set, an entry in the same zone is referenced – which cannot work.

In retrospect, chaining DNS records using CNAME has proven to be unfavorable for users who are far away. More on this under “Further improvements: DNS optimization”

Switching to CDN operation

In order to respond to all requests to media.metalhead.club with the appropriate media server, one more change was necessary: The entry for media.metalhead.club had to refer to metalheadclub-media.cdn.650thz.de in the metalhead.club zone file:

media IN CNAME 3600 metalheadclub-media.cdn.650thz.de.

(Again, pay attention to the final point!)

To test the CDN after the DNS change (and the previously entered TTL), I quickly ran an nslookup on media.metalhead.club from each of the servers:

From s3.650thz.de: Returns own IP
From cdn-us.650thz.de: Returns own IP
From cdn-ap.650thz.de: Returns own IP

So the dynamic allocation worked!

How well does it work?

To get a broader picture, I ran further tests using the following tools, for example:

CDN HTTP Response Time Test: https://tools.keycdn.com/performance
CDN Test with IPs: https://www.uptrends.com/tools/cdn-performance-check
BunnyCDN Ping Test: https://tools.bunny.net/latency-test
CDNPerf Test: https://www.cdnperf.com/tools/cdn-latency-benchmark

I also asked metalhead.club users about this. A user from Australia had previously reported these ping times to media.metalhead.club:

64 bytes from 5.1.72.141: icmp_seq=0 ttl=38 time=369.765 ms
64 bytes from 5.1.72.141: icmp_seq=1 ttl=38 time=350.830 ms
64 bytes from 5.1.72.141: icmp_seq=2 ttl=38 time=469.047 ms
64 bytes from 5.1.72.141: icmp_seq=3 ttl=38 time=379.882 ms
64 bytes from 5.1.72.141: icmp_seq=4 ttl=38 time=400.998 ms

(That’s an average of just under 400 ms!)

After that, the latencies were significantly lower because he was correctly redirected to the US server instead of the EU server:

64 bytes from 5.223.63.2: icmp_seq=0 ttl=50 time=150.384 ms
64 bytes from 5.223.63.2: icmp_seq=1 ttl=50 time=230.931 ms
64 bytes from 5.223.63.2: icmp_seq=2 ttl=50 time=360.083 ms
64 bytes from 5.223.63.2: icmp_seq=3 ttl=50 time=142.986 ms
64 bytes from 5.223.63.2: icmp_seq=4 ttl=50 time=138.416 ms
64 bytes from 5.223.63.2: icmp_seq=5 ttl=50 time=300.561 ms

(264 ms on average).

Not earth-shattering, but still significantly less. The fact that this user still has a relatively high latency to the media storage is due to the fact that the distance from Australia to Singapore is still not negligible. This is where my small CDN immediately shows its weakness: With only three locations, you can’t achieve optimal performance - but you can at least achieve a small improvement.

For another Australian user, the latency improved from an average of 346 ms to 108 ms!

And for the user from the US I mentioned at the beginning? He was able to play the video in question smoothly and without any problems after the switch.

I used the BunnyCDN test to do a before-and-after comparison:

Before:

After:

Goal achieved!

Latency times have improved significantly for most locations around the US, Australia, Korea, and Japan. Europe remains unchanged, as expected. However, there are also some unexpected outliers that may have fallen victim to incorrect IP localization, for example in Singapore itself (interesting!), Turkey, India, the US state of Texas, and Chile.

Of course, the results also depend heavily on the IP addresses used for testing and the local connection. Turkey may already have been classified as an “Asian region,” and the location from which testing was conducted in Texas may have been identified as European.

I will continue to monitor the results. Since the measurements here were primarily taken from data centers (and not from private homes in the corresponding ISP areas), the results may not reflect the whole truth.

So, one might rightly ask: …

Can you rely on GeoIP?

When testing my CDN with the test websites mentioned above, I discovered that the server contacted didn’t always correctly match the test location (see the result for the Singapore location!), so I surveyed my users. The results from these web tools are probably not particularly accurate. I suspect that the mechanism works much better for end-user address ranges than for data center address ranges. After all, GPS and other location information can be “fused” much more easily for residential IP addresses.

And indeed: In almost all cases, users were assigned the correct media server based on their origin. See: https://metalhead.club/@thomas/114676148254141233 (I probably should have added a poll option to this post…)

The last manual count on June 15, 2025, showed:

For 83 users (most of them from Germany), the allocation worked.
For 4 users, it didn’t work. 3 of them were redirected to the USA instead of Frankfurt - one user was even connected to Singapore, even though they were in Germany.

I asked Scaleway Support which GeoIP database was used and how often it was updated. I received the following response:

There can occasionally be discrepancies, such as the one you experienced with the Singapore data center being identified as European. It can take time for GeoIP databases to reflect changes like IP reallocation. Our GeoIP database provider involve frequent updates, often on a weekly or bi-weekly basis. While we don’t publicly disclose the specific third-party GeoIP database provider we use, we rely on a reputable industry-standard provider to ensure the highest possible accuracy for our customers. We understand that accurate GeoIP routing is crucial for latency-sensitive applications. If you encounter significant and persistent inaccuracies, we encourage you to report them to our support team so we can investigate further.

I also asked whether Scaleway GeoIP uses only the resolver address for positioning, or whether it also evaluates the EDNS ECS field. This would improve accuracy:

Our product team has got back to us to indicate that you are right. First we use the EDNS/ECS feature and if it’s not available we use resolver’s IP address.

What’s next?

An Anycast-based CDN could certainly achieve significantly better results, but for the reasons mentioned above, that’s not a viable solution for me at the moment. Since my small GeoIP CDN achieves an improvement in most, but not all, cases, I’ll continue the experiment. Perhaps there are still some adjustments I can make to improve localization.

Otherwise, I’m also toying with the idea of possibly switching to a more professional, third-party hosted AnyCast CDN. However, then Cloudflare would be the only provider I could rely on. I’m not willing to give up the “100% Green Energy” label of my services for a CDN.

Overall, I can say that my GeoIP CDN helps improve access for most remote users. However, there are limitations:

The appropriate CDN server is not assigned in all cases (inaccuracy of the GeoIP database).
To achieve a latency advantage, a specific content must already have been accessed by another user in the region (content is not (yet?) kept synchronously across all CDN servers => more storage required).
Only media content is hosted by the CDN. API services and the web frontend are still hosted only in the EU.

Further Improvements: DNS Optimization

After rolling out my CDN, I discovered through performance tools that DNS name resolution, especially for remote users, accounts for a large portion of the overall load time. I described how to improve this in this follow-up article: “Speeding up global DNS resolution by eliminating CNAMES”

Icinga DB fails after update

thomas.leister@mailbox.org (Thomas Leister) — Mon, 24 Mar 2025 16:22:33 +0100

Since I have been using Icinga DB (instead of just PostgreSQL) as the backend for my Icinga2 instance, it has happened to me twice that Icinga no longer worked correctly after an update of the Icinga DB package. On closer inspection, it also becomes clear why:

× icingadb.service - Icinga DB
Loaded: loaded (/lib/systemd/system/icingadb.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-01-22 21:42:50 CET; 27s ago
Duration: 13ms
Main PID: 1853805 (code=exited, status=1/FAILURE)
CPU: 13ms
Jan 22 21:42:50 monitor systemd[1]: Starting icingadb.service - Icinga DB...
Jan 22 21:42:50 monitor systemd[1]: Started icingadb.service - Icinga DB.
Jan 22 21:42:50 monitor icingadb[1853805]: Starting Icinga DB daemon (v1.2.1)
Jan 22 21:42:50 monitor icingadb[1853805]: Connecting to database at 'pgsql://icingadb@localhost:5432/icingadb'
Jan 22 21:42:50 monitor icingadb[1853805]: unexpected database schema version: v3 (expected v4), please make sure you have applied all database migrations after upgrading Icinga DB
Jan 22 21:42:50 monitor systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 21:42:50 monitor systemd[1]: icingadb.service: Failed with result 'exit-code'.

Apparently the software was updated with the package update, but the database was left out and not automatically upgraded to the new DB schema. It is unclear to me why this does not happen automatically with an APT update. Fortunately, the problem can be solved quickly by hand:

psql -U icingadb icingadb < /usr/share/icingadb/schema/pgsql/upgrades/1.2.1.sql

(If in doubt, select the most recent .sql file in the directory. In my case this was 1.2.1.sql)_

… and then a

sudo systemctl restart icingadb

afterwards. Icinga should be able to connect to the Icinga DB again.

Creating a Windows 10 USB boot medium on Fedora / Linux

thomas.leister@mailbox.org (Thomas Leister) — Wed, 02 Oct 2024 11:52:12 +0200

I recently refurbished a used and old Acer Aspire E774 laptop as a donation for the Computertruhe e.V.. Of course, this also involves overwriting the hard disk with random data so that old data can no longer be reconstructed. Because “deleted” is not the same as “securely deleted”. I used the shred tool in a Fedora Live environment for secure deletion. However, it works just as well with any other Linux distribution.

Since an Ubuntu on the laptop was not immediately executable or reasonable, because the touchpad did not work and there were always strange problems during boot, I decided to simply reinstall Windows 10 after my deletion. Just as it was running perfectly on the laptop before.

Without further ado, I downloaded a WIndows 10 64-bit image from Microsoft and wanted to copy it to my USB stick:

sudo dd if=~/Downloads/Win10.img of=/dev/sdc bs=8M

After the image was burned, I plugged the stick into the laptop and called up the boot manager via one of the F keys at startup. Normally, the boot medium can be adjusted here before the OS starts if the UEFI setting is not already correct. After all, I wanted to boot from the USB stick and not from one of the two built-in hard disks.

But: Nothing! There was no entry for the USB stick. A look into the BIOS revealed that booting from a USB medium did not have the highest priority, but even after I had changed this, the BIOS or UEFI only showed me “No boot medium found”.

Oh well. Maybe my boot medium was defective. I tried the Fedora Media Writer - a graphical tool that is included by default in every Fedora installation. The Media Writer can be used to copy Fedora images to USB sticks, as well as any other operating system images. But even with the Fedora Media Writer I was unable to get a bootable USB stick: same mistake. Plugging it into other USB slots was also unsuccessful. Finally, I even tried formatting the USB stick manually and then copying the files manually. I followed these instructions: “Create Windows 10 boot stick”. But again no success.

Finally, I came across the tool woeusb. I liked the fact that it could be installed from the Fedora package sources and - unlike Unetbootin and other alternatives - ran from the command line. So it also worked in my Wayland environment (unlike Unetbootin!).

I copied my Windows 10 image to my USB stick as follows:

sudo woeusb --device ~/Downloads/Win10_22H2_German_x64v1.iso /dev/sda

… and was successful!

Now - why did this work and my previous attempts did not? The log from woeusb provides an explanation:

WoeUSB v5.2.4
==============================
Info: Mounting source filesystem...
Info: Wiping all existing partition table and filesystem signatures in /dev/sda...
/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 8 bytes were erased at offset 0x734ffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sda: calling ioctl to re-read partition table: Erfolg
Info: Ensure that /dev/sda is really wiped...
Info: Creating new partition table on /dev/sda...
Info: Creating target partition...
Info: Making system realize that partition table has changed...
Info: Wait 3 seconds for block device nodes to populate...
mkfs.fat 4.2 (2021-01-31)
mkfs.fat: Warning: lowercase labels might not work properly on some systems
Info: Mounting target filesystem...
Info: Copying files from source media...
Splitting WIM: 4900 MiB of 4900 MiB (100%) written, part 2 of 26%
Finished splitting "./sources/install.wim"
Info: Installing GRUB bootloader for legacy PC booting support...
i386-pc wird für Ihre Plattform installiert.
installation beendet. Keine Fehler aufgetreten.
Info: Installing custom GRUB config for legacy PC booting...
Info: Done :)
Info: The target device should be bootable now
Info: Unmounting and removing "/tmp/woeusb-source-20240930-085645-Sunday.0twgOL"...
Info: Unmounting and removing "/tmp/woeusb-target-20240930-085645-Sunday.2DBYAG"...
Info: You may now safely detach the target device

There are two interesting points here:

Firstly, the USB stick is apparently formatted with FAT (mkfs.fat) and not - as can be read in other instructions - with NTFS
And: install.wim is “split”?!

The split of the install.wim file explains why it did not work for me with the other methods: This step splits the large image file (> 4 GB) into smaller files. This is necessary because the FAT file system can only handle files up to a size of 4 GB.

And why does Microsoft deliver such a large file in its image, which I tried to copy via dd? Because the Microsoft Windows 10 ISO is NTFS formatted. And not FAT formatted. NTFS can handle such large files without any problems - which is why I was advised in various instructions to format the USB stick with NTFS rather than FAT.

However, another peculiarity comes into play here: The Acer laptop apparently does not support NTFS boot media. … How can that be?

The laptop originally came onto the market with Windows 8. At that time, the Windows boot medium did not contain any files larger than 4 GB. Accordingly, the FAT file system was sufficient. Consequently, the Acer E774 laptop only supports FAT in the UEFI and not NTFS. This also explains why it booted without any problems from all the live Linux USB sticks I gave it. FAT is still used here.

To summarize: with the woeusb tool I finally had success because:

The Dell laptop only supports FAT media
The woeusb tool uses FAT
And makes sure that none of the files is larger than 4 GB!

So the key is woeusb’s “split” function.

On newer laptops, copying the Win10 image should also work simply via dd. But for my older laptop I found a great solution with woeusb.

T-Shirts for my Mastodon instance metalhead.club

thomas.leister@mailbox.org (Thomas Leister) — Tue, 16 Apr 2024 18:05:00 +0200

Two things are usually in short supply when you run a Mastodon instance: Funding and public attention. But both are important so that the instance can continue to operate and - if desired - achieve growth or reach.

My goal with metalhead.club is to offer a professionally hosted platform for everyone who feels at home in the metal music genre. In order for such a theme-based instance to be viable and its users to benefit from it to the greatest extent, it must achieve a certain level of popularity. Nevertheless, the usual advertising media are only of limited use - either because they do not implement my idea of data protection or because they are currently not readily financially viable. Precious donations have to be used sparingly.

In addition, the target group is large but relatively specific: while you can probably sell an LED light bulb to anyone, it’s not quite so easy to sell a social network for metal fans. The right people have to be addressed in their “language”.

… and what could be better than selling something that has a high status in the scene and is highly visible? T-shirts! As band shirts, they can be seen everywhere at home, at festivals or concerts and show which bands the wearer likes to listen to. It should work in a similar way with the Instanz T-shirts: As someone wearing a t-shirt of their Instanz, you can function as a kind of mobile advertising medium and at the same time promote what you yourself find worth supporting - the metalhead.club Mastodon Instanz!

After some metalhead.club members had been enthusiastically demanding them for a long time, I announced the first T-shirt order campaign in summer 2023 and ordered 100 metalhead.club T-shirts and sold them to the members. Now - in spring 2024 - it was time for the second campaign with a different T-shirt model.

In the following, I will describe how I approached the “metalhead.club T-shirts” project and give a few tips. Perhaps it will help some people who are also thinking about offering merchandise for their Mastodon instance.

Do it yourself or lean back?

There are numerous stores on the internet and in cities that offer a wide variety of merchandise and often even ship directly to the buyer. Examples include Spreadshirt, Red Bubble and Printful. In some cases, the suppliers can even be commissioned automatically via online store plugins, so that the seller of the items no longer has to do much: The customer orders in an online store, the supplier carries out the production, processes the payment and sends the goods to the customer. The store operator collects a few euros from the sale at the end.

However, I want to take a different approach with my T-shirts. As long as the quantities (approx. 100 items per campaign) allow it, I want to be involved in the ordering process myself: I take payment and do the shipping myself - even if it is sometimes tedious work. In this way, I can avoid unnecessary money flowing to third parties in the form of commissions and fees and I am also in a position to enable data protection-friendly purchasing: Only I know the customers and their address data. They can be deleted after use. I can also check the quality myself and choose the packaging, as well as add any extras I like, such as metalhead.club business cards.

Although I initially toyed with the idea of printing the T-shirts myself, I quickly abandoned the idea. Because in addition to the right equipment, you also need a lot of time and space - all things that I don’t have. I have my T-shirts purchased and printed by a supplier. I then sell the result myself.

The ingredients: T-shirts and (screen) printing

When it comes to merchandise or team clothing with printing, there is a big name in the industry: Stanley/Stella. The company has been producing high-quality clothing since 2012, which can then be further customized by third parties - a kind of “blank clothing”. A special feature is the organic cotton certification.

The brand’s T-shirts were highly recommended to me in conversation with some metalhead.club members, who confirmed their high quality. In the meantime, I can also give an absolute recommendation - the clothing has proven itself and the price is right. So the name was set.

The choice of model is a matter of taste. The first order campaign focused on the “Presenter” and “Evoker” models - for the second campaign, I limited myself to the “Crafter” model to avoid additional effort. I also limited the color to black in order to have only the dress sizes as the only variable. This makes processing easier and therefore less error-prone. It is advisable to order a few more pieces of all sizes. This may be to compensate for quality defects, to replace lost items or simply to achieve a better unit price with a larger quantity. I got rid of surplus T-shirts relatively quickly during the 2023 order campaign, as not everyone who was interested found out about the campaign or missed it for other reasons.

In terms of the printing process, it was clear to me pretty quickly that I wanted a screen print. In comparison, it is not necessarily the cheapest or fastest method, but it promises a very high durability, especially with frequent washing, as the color can penetrate deep into the fabric and hold there well. What would a metalhead.club T-shirt be that can’t withstand Mosphits and is no longer legible after a few washes?

I have my T-shirts printed by Promoyard - a regional company that impressed me from the very first order. They print with OekoTex-certified, skin-friendly inks.

The ordering process

Once I had decided on the T-shirt manufacturer and printer, I started to advertise the campaign. On the one hand, I wrote a post on my 650thz.de Services Blog summarizing the information about the T-shirts and ordering information - on the other hand, I also wrote posts about the campaign on Mastodon and posted an admin announcement for my instance metalhead.club. Every member will be informed about the T-shirt promotion via a separate message in the web and app.

After the first order campaign in summer 2023, it was clear that I would have to draw attention to the campaign more frequently in order to reach as many interested parties as possible. Although the order period was 6 weeks long, not everyone found out about the order campaign due to the vacation period and relatively concentrated information.

The second time in spring 2024, I sent out a post every morning at the beginning to reach members (at least in Germany) in the morning before work or on the way if they had time for Mastodon. Apparently I was still able to reach some interested parties this way. I reduced my tips towards the end of the six-week order period so as not to bore anyone, but posted more shortly before the end of the period to catch those who made a last-minute decision.

Technically, I handled the orders very simply: Anyone who was interested in a T-shirt wrote me an e-mail with the number of T-shirts, size and address. The customer was then sent an e-mail with a confirmation of receipt and the bank details as well as a Stripe.com link for payment. I manually reconciled and noted the payments on a daily basis. Once payment had been received, another payment confirmation was sent by email. I recorded all orders and the payment status in an unspectacular LibreOffice Calc spreadsheet.

After the first T-shirt campaign at the latest, I had a simple Django-based order management interface in mind for a while, but I didn’t have enough time to develop it. For orders in the order of 100 pieces, a simple spreadsheet application was also fine - albeit less practical.

Incidentally, I chose to pay in advance to minimize the risk on my side, for example through late or non-payment. Fortunately, the members on metalhead.club (and those who ordered from other instances) proved to be extremely conscientious and reliable. There were no problems with payment.

The dispatch

Once the orders had been collected and the order period had ended, I commissioned my supplier to produce the T-shirts. The T-shirts were delivered within about 2 weeks and I set about sending them out to the members one by one.

Shipping material

I put a lot of thought into the shipping material, because it should not only get the goods to their destination dry and safe, but also be as environmentally friendly and reusable as possible. With Biobiene.com I have found exactly the right supplier for this! The store offers various grass paper-based shipping materials and other environmentally friendly alternatives.

The following materials were used for the first metalhead.club T-shirt campaign:

With their DIN A3 dimensions, the garment covers are a little too large, but they are all the more convenient to fill with the T-shirts. The alternative - “polybags” made of paper are cheaper per item, but can only be bought in quantities of 250 or more. So I opted for the former.

The grass paper mailing bags can be used a second time after being torn open: A second, internal adhesive strip allows them to be used several times.

The delivery note envelopes are intended for invoices, which are required for customs in the case of international shipments (non-EU countries). A customs declaration must also be affixed to the shipping envelopes - with details of the contents, weight and value.

I initially printed out the recipient addresses on paper and stuck them on, but unsurprisingly this turned out to be very time-consuming. Later, I ordered printable stickers in A4 size, printed the addresses on them and cut them out. When entering several addresses online, Deutsche Post combines all the addresses into one PDF page (max. 10 addresses per page) and offers this for printing. Once printed, the addresses can then be cut out and stuck on directly.

For the second campaign, I also wanted to use environmentally friendly address labels and ordered Heisap paper labels HEI027.

Shipping rates with Deutsche Post and DHL

I have used Deutsche Post for shipping to recipients in Germany. The Warensendung 500 can also be used to send 3 T-shirts in one envelope without any problems.

I have sent shipments to other EU countries and worldwide via DHL. The “Päckchen EU / or International XS 2 kg” is suitable for this.

Shipments within Germany were usually delivered to the recipients within 3 working days. However, international shipments sometimes took several weeks. Patience was particularly required for shipments to Canada and Mexico. But items to Spain and Hungary also seemed to be lost at first, before they were found in post offices many days after the expected date of receipt. It is therefore worth being patient (and, if necessary, instructing the recipient to ask again and have the parcel searched for).

Sending the T-shirts

As soon as the T-shirts and the shipping material have arrived, you can get started. Last time, I stretched out the dispatch of the almost 100 T-shirts over approx. 3 weeks so as not to lose concentration. This is because mistakes in shipping can quickly become relatively expensive - especially when it comes to recipients abroad.

The 2023 T-shirt campaign was a complete success and the order phase for the spring 2024 campaign ended last week. I’m looking forward to seeing the many photos of metalhead.club members with their T-shirts! It’s nice to see the members in a kind of common uniform in the photos!

Switching Mastodon from Scaleway S3 to self-hosted Minio S3 media storage

thomas.leister@mailbox.org (Thomas Leister) — Sun, 14 May 2023 19:00:00 +0200

Since the user flood of November 2022 I’ve been using Scaleway’s S3 storage for media file caching and storage of my metalhead.club Mastodon instance. It was easy to set up and has been working reliably for me. Back then the media cache size increased so much that my server’s internal storage could not keep up with the increasing demand. I didn’t want to shrink down the cache duration too much and therefore left it at 14 days. At the time, the cache was about 800 GB in size - a big mass of image files that could not be handled by my aged server itself.

metalhead.club was moved to a more powerful, new server in April. The new server was built to have plenty of storage for Mastodon media storage. I planned to move the S3 storage back to my own infrastructure to save costs. Although media file requests have always been cached by my own HTTP proxy and Scaleway did not get any user-related metadata, it also feels better to have more control and not depend on any 3rd party services for such an essential part of my social network software.

(… and there have been short service interruptions that we’ve had several times in the past. The overall experience has been decent, but I hoped to achieve better availability by running S3 myself ;-) )

As an S3 compatible storage software I chose Minio. I’ve never setup and run my own Minio instance before, but I’ve heard good things about the project that is usually used in big cloud computing setups. A quick glimpse at the documentation revealed that running my own simple Minio instance would just be a matter of minutes - given that the software can easily deployed via Docker. The “single node - single drive” operating mode of Minio is sufficient for now. I’m not expecting any exponential growth of my Mastodon instance anywhere in the future, so I kept the setup as small and simple as possible. No redundancies (already covered by a lower-layer storage backend) and just a single node.

This article does not aim to be a complete guide, because several days have passed since I implemented my S3 server and I might not remember every detail. Nevertheless I’d like to put some notes about the setup here. Just in case it is interesting for anybody.

Basic Minio setup using Podman

Like I mentioned - Minio can bew run as a Docker container. I’m not that much a fan of Docker deployments, so chose Podman instead. Podman lets my run Minio “root-less”, which adds extra security. As the root user, I installed Podman and created a new user for Minio on my system:

apt install podman uidmap slirp4netns
mkdir /opt/minio
useradd -s /usr/sbin/nologin --home-dir /opt/minio minio
chown minio:minio /opt/minio

Now create a directory for Minio to store its data, e.g.:

sudo mkdir /var/lib/minio
sudo chown minio:minio /var/lib/minio

The minio user is not allowed to log in via a login shell. But you can switch to it by using this command as root:

su -s /bin/bash - minio

The minio user will run the Podman-managed container. As the minio user, the Podman container for Minio is downloaded:

podman pull quay.io/minio/minio

If you see this message:

Error: command required for rootless mode with multiple IDs: exec: “newuidmap”: executable file not found in $PATH

log out from your account and log back in. The problem should be fixed then. If not, make sure that package newuidmap is installed.

Next the environment variable file is created. It will be located on the container host, but be mounted inside the Minio container so Minio can read it and configure itself according to it.

Create the file /opt/minio/config.env:

MINIO_SERVER_URL="https://s3.mydomain.tld"
MINIO_DOMAIN="s3.mydomain.tld"
MINIO_ROOT_USER="myadminuser"
MINIO_ROOT_PASSWORD="mysupepoassword"
MINIO_REGION=myregion

MINIO_SERVER_URL: Public URL of your S3 service
MINIO_DOMAIN: Domain part of the Public URL. Used my Minio to enable addressing Buckets via subdomain, e.g. mybucket.s3.mydomain.tld
MINIO_ROOT_USER and MINIO_ROOT_PASSWORD: Admin user and password for Minio Console.
MINIO_REGION: “Availablity region” of this S3 server. I used my physical server’s hostname here.

Afterwards, it’s time to start up the container for the first time:

podman run -dt \
-p 9000:9000 \
-p 9090:9090 \
-v /var/lib/minio:/mnt/data \
-v /opt/minio/config.env:/etc/config.env \
-e MINIO_CONFIG_ENV_FILE=/etc/config.env \
--name minio \
quay.io/minio/minio \
server /mnt/data --console-address ":9090"

/var/lib/minio is the path to the Minio S3 storage. In my case it’s just the path to the mount point of an ext4 formatted virtual drive. All S3 files will reside there.

After starting up the container you should already be able to see container details via podman ps and connect to the admin console at http://s3.mydomain.tld:9090 and log in as your admin user (given that no firewall is blocking access).

Letting systemd start the container

Podman is a daemon-less container manager and cannot start or restart containers by itself. To start the Minio container after a reboot, I use a systemd user service that can easily be generated by podman. To make systemd user services work, enable user lingering (switch to root user for the next two lines):

loginctl enable-linger $(id -u minio)
systemctl start user@$(id -u minio).service

and put the following lines into ~/.profile: (return to minio user before, using su -s /bin/bash - minio!)

export XDG_RUNTIME_DIR="/run/user/$(id -u)"
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus"

Next, create the user service file:

mkdir -p ~/.config/systemd/user/
podman generate systemd --new --name minio > ~/.config/systemd/user/container-minio.service

… and enable the new Minio service:

systemctl --user daemon-reload
systemctl --user enable container-minio.service

if you ever want to modify the podman run parameters, make your changes in ~/.local/systemd/user/container-minio.service and restart the Minio service using

systemctl --user daemon-reload
systemctl --user restart container-minio.service

Your Minio container should automatically start after a reboot from now on.

Nginx reverse proxy config

I use Nginx for handling HTTPS connections. A suitable configuration might look like this:

upstream minio {
server 127.0.0.1:9000;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl;
listen [::]:443 ssl;
# regex: Make bucket subdomains work
server_name ~^([^.]+).s3.mydomain.tld s3.mydomain.tld;
include snippets/tls-common.conf;
ssl_certificate /etc/acme.sh/s3.mydomain.tld/fullchain.pem;
ssl_certificate_key /etc/acme.sh/s3.mydomain.tld/privkey.pem;
# Allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# Disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio; # This uses the upstream directive definition to load balance
}
}

This Nginx virtual host at s3.mydomain.tld will serve all requests to the Minio S3 server.

Nginx reverse proxy for Minio console

There is an extra Nginx virtual host config.s3.mydomain.tld for the Minio console:

server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name console.s3.mydomain.tld;
include snippets/tls-common.conf;
ssl_certificate /etc/acme.sh/s3.mydomain.tld/fullchain.pem;
ssl_certificate_key /etc/acme.sh/s3.mydomain.tld/privkey.pem;
# Allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# Disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websockets in MinIO versions released after January 2023
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://127.0.0.1:9090; # This uses the upstream directive definition to load balance
}
}

Creating an S3 bucket for Mastodon

After logging in to Minio admin dashboard / console at console.s3.mydomain.tld, I created a new S3 bucket for my metalhead.club metalheadclub-media. It is important to set the bucket access rules correctly:

I’ve set a a new policy in the “Administrator” => “Policies” menu. “Create Policy”, then:

Name: “masto-public-read-nolist”

In the bucket settings, I’ve set the “Access Policy” to “Custom” and entered this configuration:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::metalheadclub-media/*"
]
}
]
}

It allows all users from the internet to access the files in the bucket. But it does not allow to list the files in the bucket!. This is important because once the file list is available to the public, everyone will be able to download the full S3 bucket, which can (and probably will) contain sensitive information that must not be available to anyone except the information owners. If the file listing is turned off, it it simply not possible to guess all the random URLs and such an attack is avoided. This is the exact security issue that Mastodon.social had a while back. We definitely want to avoid that!

Creating an API key for Mastodon S3 access

The Mastodon software needs a new API key and secret to access the S3 bucket. Other than any anonymous public user, Mastodon must be allowed write access to the S3 bucket. In the Minio console go to “Access Keys” and create a new key. (Click “create”).

Copy the API key and secret to your password safe - the secret will never be displayed again!

Then click on the API key that was just created to edit its permissions. Set the following permissions:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::metalheadclub-media"
]
},
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::metalheadclub-media/*"
]
}
]
}

(as always - adapt to your own bucket name!)

Save the bucket key and secret for later - they will be needed inside Mastodon’s live/.env.production file.

Caching Mastodon media files

But there is another Nginx virtual host specifically for my Mastodon instance: media.metalhead.club. I’ve used the domain for my media files on Scaleway before. My own media proxy does not only allow me to protect my users’ data towards 3rd party S3 services, but also cache the media files (and thus reducing the load/traffic to my S3 bucket) and make user-transparent S3 migrations:

proxy_cache_path /tmp/nginx-cache-metalheadclub-media levels=1:2 keys_zone=s3_cache:10m max_size=10g
inactive=48h use_temp_path=off;
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name media.metalhead.club;
include snippets/tls-common.conf;
ssl_certificate /etc/acme.sh/media.metalhead.club/fullchain.pem;
ssl_certificate_key /etc/acme.sh/media.metalhead.club/privkey.pem;
client_max_body_size 100M;
# Register backend URLs
set $scaleway_backend 'https://metalheadclub-media.s3.fr-par.scw.cloud';
set $minio_backend 'https://metalheadclub-media.s3.650thz.de';
# Note: We cannot use both @minio and @scaleway here, because only one named location is accepted with try_files
# Workaround: proxy_intercept_errors section and forwarding to @scaleway if @minio throws 404
# See: https://stackoverflow.com/questions/21286850/nginx-try-files-with-multiple-named-locations
location / {
access_log off;
try_files $uri @minio;
}
#
# Own Minio S3 server (new)
#
location @minio {
limit_except GET {
deny all;
}
resolver 8.8.8.8;
proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_pass $minio_backend$uri;
#proxy_intercept_errors off;
# Caching to avoid S3 access
proxy_cache s3_cache;
proxy_cache_valid 200 304 48h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_lock on;
proxy_cache_revalidate on;
expires 1y;
add_header Cache-Control public;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
# Workaround: Forward request to @scaleway if @minio returns 404
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 = @scaleway;
}
#
# Scaleway S3 server (legacy)
#
location @scaleway {
limit_except GET {
deny all;
}
resolver 8.8.8.8;
proxy_set_header Host 'metalheadclub-media.s3.fr-par.scw.cloud';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_pass $scaleway_backend$uri;
proxy_intercept_errors off;
proxy_cache s3_cache;
proxy_cache_valid 200 304 48h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_lock on;
proxy_cache_revalidate on;
expires 1y;
add_header Cache-Control public;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
}
}

There are a few notable lines here:

proxy_cache_path adds a cache directory for requests to any S3 resource. proxy_cache s3_cache will enable the cache. Thsi reduces load and traffic to the S3 buckets. Since media files are immutable once they are in the S3 storage, we can keep them in the cache them for a long time.
set $scaleway_backend and set $minio_backend set the URLs of the S3 storage backends.
try_files $uri @minio; will try to read a file from the local web root (root directive) first and try the minio backend second.
@minio and scaleway location blocks: There is a proxy configuration for each of the S3 backends. The @minio “named location” will point to my own new Minio based S3 service. @scaleway will point to the 3rd party S3 bucket hosted by my previous S3 provider.

If Minio is not able to serve a certain file, Scaleway will be asked to deliver the file.

By the way: You might think “Why not use something like try_files $uri @minio @scaleway; to implement a Fall-Back to Scaleway? Well, turns out that try_files only supports a single “named location(@)”, so this does not work. Instead I used another method to fall back to Scaleway:

# Workaround: Forward request to @scaleway if @minio returns 404
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 = @scaleway;

Another important note: the line

proxy_set_header Host 'metalheadclub-media.s3.650thz.de';

is important since it will set the Host header to a fictional / internal subdomain that is not used in public - but Minio will expect to receive this header so it knows which bucket (metalheadclub-media) is to be addressed. If the Host header is not set correctly, it will not know and will return an error.

The migration plan

I wanted to make the S3 migration from Scaleway to Minio as seamless as possible. My Mastodon users should not notice the switch - and here’s how I did it:

I configured Mastodon to use the new Minio based S3 storage for any media uploads (live/.env.production on the Mastodon server):

S3_ENABLED=true
S3_BUCKET=metalheadclub-media
AWS_ACCESS_KEY_ID=secretkeyid
AWS_SECRET_ACCESS_KEY=secretaccesstoken
S3_ALIAS_HOST=media.metalhead.club
S3_REGION=hyper2
S3_PROTOCOL=https
S3_HOSTNAME=media.metalhead.club
S3_ENDPOINT=https://s3.650thz.de
S3_SIGNATURE_VERSION=v4

(then restarted all Mastodon services to apply changes).

I switched the DNS from my old Nginx media proxy to the new one (with Scaleway + Minio backend): The media proxy will now check if the requested media file exists on Minio S3 - if not, it will download from the legacy Scaleway storage. => New uploads get served by Minio, old ones by Scaleway.
I triggered a S3 files sync using rclone for media that was uploaded by metalhead.club users (and therefore is not “cached remote media”)
There will be less S3 requests to Scaleway day after day, since the number of requests to the old Mastodon media cache will decline over time.
At some point the Scaleway S3 storage will not be accessed at all anymore, because all media files have either been transferred to Minio (user owned file uploads) or are not relevant anymore (cached media thumbnails, … - get invalid after a couple of days anyway)

Cloning existing instance user data to the new storage

Rclone is an awesome tool for copying files from one S3 bucket to another. I’ve created S3 storage configurations for both, Scaleway S3 and Minio S3 by running the rclone config guide, and here’s the result as a text configuration file (in ~/.config/rclone/rclone.conf):

[scaleway]
type = s3
provider = Scaleway
env_auth = false
access_key_id = <access_key_id>
secret_access_key = <secret_access_key>
region = fr-par
endpoint = s3.fr-par.scw.cloud
acl = public-read
storage_class = STANDARD
[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = <access_key_id>
secret_access_key = <secret_access_key>
region = hyper2
endpoint = https://s3.650thz.de
location_constraint = hyper2
acl = public-read

Copying the existing user data from Scaleway to Minio was as simple as:

rclone copy --progress --transfers=8 scaleway:metalheadclub-media/custom_emojis/ minio:metalheadclub-media/custom_emojis/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/accounts/ minio:metalheadclub-media/accounts/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/site_uploads/ minio:metalheadclub-media/site_uploads/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/media_attachments/ minio:metalheadclub-media/media_attachments/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/imports/ minio:metalheadclub-media/imports/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/cache/accounts/ minio:metalheadclub-media/cache/accounts/
rclone copy --progress --transfers=8 scaleway:metalheadclub-media/cache/custom_emojis/ minio:metalheadclub-media/cache/custom_emojis/

I didn’t copy Mastodon’s cache/preview_cards and cache/media_attachments directories on purpose, since these directories make the majority of files and they will lose their validity in a couple of days anyway.

The file copying took several hours, esp. due to the large amount of files (not the file size!) and was fully transparent to the user. In the mean time, almost every request is served by my own Minio server and the traffic to my Scaleway bucket has drastically declined:

Scaleway bucket network traffic. Declining after a short period of intense file copying ;-)

The Scaleway bucket will not be needed anymore soon and I will drop it or use it as a backup storage.

acme.sh fails with error 404 - ACME challenge not found (Nginx)

thomas.leister@mailbox.org (Thomas Leister) — Wed, 03 May 2023 17:27:04 +0200

Recently I can into an issue with acme.sh / Let’s Encrypt and a failing ACME validation

Error 404 when running acme.sh --renew -d mydomain.tld

[Wed May 3 15:31:45 UTC 2023] Pending, The CA is processing your order, please just wait. (1/30)
[Wed May 3 15:31:49 UTC 2023] mydomain.tld:Verify error:<ipaddress> Invalid response from https://mydomain.tld/.well-known/acme-challenge/5GmSwd0P0ukTtX302yHHhAuZMCEDJx7MmAaBBoPIKtk: 404
[Wed May 3 15:31:49 UTC 2023] Please add '--debug' or '--log' to check more details.
[Wed May 3 15:31:49 UTC 2023] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

The problem:

Probably your Nginx config has two segments:

server {
listen 80;
listen [::]:80;
...(some redirect to HTTPS)...
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
..etc etc ...
}

Cause of the error:

acme.sh will only insert its temporary acme-related redirection snippet into the first “server” section but not in the second one. Still it will try to connect to the server via HTTPS, first. There will be no ACME redirection (via temporary inserted “location” block), so the ACME server will not find the challenge file (thus: Error 404).

Solution:

Put everything into one server block, like this:

server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
# HTTP redirect
if ($scheme = http) {
return 301 https://$server_name$request_uri;
}
}

Then:

systemctl reload nginx

And try again:

acme.sh --renew -d mydomain.tld

If the renewal keeps failing and you want to try different configurations, make sure to use the “staging” API of Letsencrypt in the meantime. The main API is rate-limited and will block you after a few tries per hour. Enable the staging API by using the --staging flag. If the renewal finally works, go back to the production API to receive a production certificate, e.g. acme.sh --renew --force -d mydomain.tld.

=> Profit :-)

Mastodon: Adding S3 based cloud storage to your instance

thomas.leister@mailbox.org (Thomas Leister) — Wed, 23 Nov 2022 18:19:00 +0100

My Mastodon instance metalhead.club exists since summer 2016 and seen several waves of new users - but never as many new users as in early November 2022. This has not only led to heavy CPU work on the servers (see my post about scaling up Mastodon’s Sidekiq Workers), but also to greater load on storage space. Mastodon uses a media cache that not only stores copies of preview images for posts containing links - but also copies of all media files that the server knows of. Before the user wave of late 2020 metalhead.club’s media cache was about 350 GB in size with a cache retention time of 60 days. Quickly the numbers escalated and after a few days we were already at 400 GB - and after about 3 weeks we had more than 550 GB of cached media files. Not with 60 days retention time - but with 30 only.

Despite I added hundreds of GB of new storage space, the cache showed no signs of shrinking in the near future, so decided to offload the storage to an S3 storage provider. The local disks would have been full a few days later.

… and here’s how it works:

Choosing an S3 compatible provider

S3 is a storage type / protocol which origins from Amazon, but as I wanted a EU-based company to host my files, I looked for a compatible alternative. Luckily there are several S3 compatible storage providers in the EU, as this list reveals: https://european-alternatives.eu/alternative-to/amazon-s3

My top choices were Scaleway and IONOS. Scaleway is a French Hosting company and since their prices are even more affordable compared to IONOS’s prices, I picked Scaleway as my S3 bucket provider. Reviews on Scaleway are very mixed and you might get the impression that their customer service is not very helpful - but for the last 2 weeks my S3 bucket has been running just fine and there was no reason to call for help. We’ll see if Scaleway stays my S3 bucket provider or I will switch to IONOS instead.

The following steps will relate to Scaleway S3, but should be more or less applicable to other storage providers as well. You might need to change some details only.

Creating an S3 bucket

Creating a bucket is easy: Pick a name for your S3 bucket and make it look like a suitable subdomain or directory name. If you name your bucket “myinstance-media”, you will be able to access the Bucket via those two URLs:

(… assuming you’ve picket the fr-par availability region)

The Bucket visibility setting should be set to private instead of public. This will not change the accessibility of any media file! The private setting just makes sure that there is not list of file contents available to the public. While we want individual files to be publicly accessible by URL, we don’t want to help scrapers by handing them out a list of all available files.

Creating an Nginx Proxy for the S3 bucket

Usually web browsers would receive the corresponding S3 URL of a media file and directly download the file from one of the URLs mentioned earlier. Because every file would need to be downloaded on every user device at least once, a lot of traffic would be generated - and outgoing traffic costs money. Therefore I implemented an S3 proxy with Nginx. Media assets are not directly downloaded from the S3 storage, but the webbrowsers will download from https://media.metalhead.club. If a new media file should be downloaded, the Nginx proxy will pass the request to the S3 storage and cache the contents locally. The next time a request for the same file comes in, Nginx will directly send the cached version of the file instead of re-downloading it from S3. This way a lot of traffic from the S3 bucket can be saved.

Here’s my Nginx config:

proxy_cache_path /tmp/nginx-cache-instance-media levels=1:2 keys_zone=s3_cache:10m max_size=10g
inactive=48h use_temp_path=off;
server {
listen 80;
listen [::]:80;
server_name media.metalhead.club;
access_log off;
error_log /var/log/nginx/media.metalhead.club-error.log;
root /home/mastodon/live/public/system;
set $s3_backend 'https://instance-media.s3.fr-par.scw.cloud';
keepalive_timeout 30;
location = / {
index index.html;
}
location / {
try_files $uri @s3;
}
location @s3 {
limit_except GET {
deny all;
}
resolver 9.9.9.9;
proxy_set_header Host 'instance-media.s3.fr-par.scw.cloud';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_pass $s3_backend$uri;
proxy_intercept_errors off;
proxy_cache s3_cache;
proxy_cache_valid 200 304 48h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_cache_lock on;
proxy_cache_revalidate on;
expires 1y;
add_header Cache-Control public;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
}
}

(HTTPS is handled by another upstream Nginx instance)

Make sure to update both instance-media.s3.fr-par.scw.cloud URLs and the FQDN media.metalhead.club according to your environment. My configuration will keep media files cached for 48 hours before they will be dropped from the cache.

Two noteworthy lines are root /home/mastodon/live/public/system; and try_files $uri @s3;. They will make Nginx first look for a file in the local non-S3 storage. If a file exists there, S3 and the cache will not be tried. This mechanism enables a smooth transition between the local media cache and the remote S3 cache, since data transfer / sync can take hours or even days, depending on the Mastodon media cache size.

Enabling S3 stroage support in Mastodon

I enabled S3 support in Mastodon by editing the .env.production file in /home/mastodon/live:

S3_ENABLED=true
S3_BUCKET=instance-media
AWS_ACCESS_KEY_ID=<accesskey>
AWS_SECRET_ACCESS_KEY=<secretkey>
S3_ALIAS_HOST=media.metalhead.club
S3_HOSTNAME=media.metalhead.club
S3_REGION=fr-par
S3_ENDPOINT=https://s3.fr-par.scw.cloud

accesskey and secretkey need to be replaced by proper API access keys. You can find more information about the keys here: https://www.scaleway.com/en/docs/console/my-project/how-to/generate-api-key/

After having modified the config file, restart all mastodon processes!

Syncing your local data to S3

It’s now time to sync your local Mastodon media cache data to your S3 bucket. Follow the steps of the Scaleway How-To to set up the aws tool. When done, run the following steps to sync your data (better use a tmux / screen session - this make take a looong time!):

cd /home/mastodon/live
aws s3 sync --acl public-read public/system/ s3://instance-media --endpoint=https://s3.fr-par.scw.cloud

The --acl public-read is especially important, because uploaded files will only be available publically if the public-read acl is set for each file individually. If you miss to do that, you will need to re-upload all files. Guess how I learned that … ;-)

No worries, if the upload takes many hours. As mentioned before, Nginx will keep serving the media files from the local storage if it doesn’t find any suitable file in the S3 bucket, so the transition will be transparent to the users.

Checking if it works

Apart from the obvious (“do all the media files on my instance still load properly?”) you can also check if your web browser fetches images files from the correct location, e.g. media.metalhead.club, and if caching is successful. Open the web developer tools and reload the page while observing the network resources being loaded:

A media resource being loaded for the first time

Check the response headers: The x-cache-status attribute should be miss the first time an image os loaded from the server. If you pick the URL of the image and download it in another tab (or reload the page), the attribute should switch to hit. That means that the image was not delivered by the S3 bucket, but by the local Nginx cache - exactly what we wanted to achieve!

Useful resources: