Transfer Mastodon S3 storage from self-hosted Minio to Hetzner S3
On behalf of a customer, I recently had the task of transferring an S3 bucket for a Mastodon instance from a self-hosted Minio instance to Hetzner S3. I would like to explain the procedure and the configurations I used here.
The client’s Mastodon instance had previously been supplied with media and media cache by a Minio instance on the same server. However, the media cache continued to grow as the Mastodon instance became increasingly networked. Even by reducing the cache retention period to a few days, the costs for S3 storage were too high. This was because it was located on a powerful but relatively expensive external volume of a Hetzner cloud server.
At the end of 2024, Hetzner introduced Hetzner S3 storage to its product portfolio, opening up an attractive alternative to the Minio instance. An alternative that stands out above all for its low cost. The cost for a 300 GB volume was just under €16/month. Not very much for a business – but certainly a lot for a small, donation-funded Mastodon instance. In comparison, the price for 1000 GB (!) of Hetzner S3 storage is just under €6/month. So the decision to switch was quickly made.
Step 1: Parallel operation of two buckets
To make the transition to the new storage as smooth as possible for users and to keep downtime to a minimum, I decided to run both buckets in parallel during the data migration.
- Old bucket “minio”: Delivers the media files stored to date (read-only)
- New bucket “hetzner-s3”: Receives new media files, stores them, and delivers them
The upstream Nginx proxy was configured so that the new bucket was queried first when a file request was made. If the requested file could not be found there, the old S3 bucket was queried.
Proxy configuration before:
upstream minio {
server 127.0.0.1:9000;
}
server {
[...]
location / {
proxy_set_header Host 'mastodon-media.s3.domain.tld';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
add_header X-Content-Type-Options nosniff;
add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
proxy_pass https://minio;
}
}
Proxy configuration afterwards - with parallel operation of both buckets (fallback to old bucket):
upstream hetzner {
server mastodon-media.nbg1.your-objectstorage.com;
}
upstream minio {
server 127.0.0.1:9000;
}
server {
[...]
resolver 9.9.9.9;
location / {
proxy_set_header Host 'mastodon-media.nbg1.your-objectstorage.com';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
add_header X-Content-Type-Options nosniff;
add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
proxy_pass https://hetzner; # This uses the upstream directive definition to load balance
# Fallback to oldbucket
proxy_intercept_errors on;
error_page 404 403 = @oldbucket;
}
location @oldbucket {
proxy_set_header Host 'mastodon-media.s3.domain.tld';
proxy_set_header Connection '';
proxy_set_header Authorization '';
proxy_hide_header Set-Cookie;
proxy_hide_header 'Access-Control-Allow-Origin';
proxy_hide_header 'Access-Control-Allow-Methods';
proxy_hide_header 'Access-Control-Allow-Headers';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header x-amz-bucket-region;
proxy_hide_header x-amzn-requestid;
proxy_ignore_headers Set-Cookie;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
add_header Cache-Control public;
add_header 'Access-Control-Allow-Origin' '*';
add_header X-Cache-Status $upstream_cache_status;
add_header X-Content-Type-Options nosniff;
add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
proxy_pass http://minio;
}
}
The new “location” block “@oldbucket” and the reference to it if files cannot be found in the new bucket are decisive here:
# Fallback to oldbucket
proxy_intercept_errors on;
error_page 404 403 = @oldbucket;
It is important at this point not only to catch 404 errors, but also 403 errors (“forbidden”). Hetzner S3 Storage apparently responds with this error code when a resource is requested whose parent directories do not (yet) exist.
Step 2: Saving new media in the new bucket
Up to this point, users of the Mastodon instance should hardly have noticed anything. The proxy change takes effect in a fraction of a second and is completely transparent to the user. However, there will be a brief downtime of several seconds as the Mastodon configuration is adjusted and all Mastodon services must be restarted.
The following settings were used as an example for the new bucket in Mastodon’s .env.production:
S3_ENABLED=true
S3_BUCKET=mastodon-media
AWS_ACCESS_KEY_ID=<myaccessid>
AWS_SECRET_ACCESS_KEY=<myaccesskey>
S3_ALIAS_HOST=media.domain.tld
S3_REGION=nbg1
S3_PROTOCOL=https
S3_HOSTNAME=media.domain.tld
S3_ENDPOINT=https://nbg1.your-objectstorage.com
S3_SIGNATURE_VERSION=v4
The changes will take effect once the Mastodon services have been restarted. New media files will now be stored exclusively in the new S3 bucket:
cd /etc/systemd/system
systemctl restart mastodon-*
After a few seconds, Mastodon is ready to go again.
Step 3: Migrate old data to new bucket
The critical part is now complete. Now you can relax and transfer the old data to the new bucket. To do this, I use the “rclone” tool, which makes it very easy to synchronize two S3 buckets with each other. To do this, configurations for the two S3 providers are first created under ~/.config/rclone/rclone.conf. Here is another example:
[hetzner-s3]
type = s3
provider = Other
access_key_id = <accessid>
secret_access_key = <accesskey>
endpoint = nbg1.your-objectstorage.com
acl = public-read
region = nbg1
[minio-s3]
type = s3
provider = Minio
access_key_id = <accessid>
secret_access_key = <accesskey>
endpoint = s3.domain.tld
acl = public-read
region = server1
The endpoint, region, access ID, and access key must, of course, be adapted to the respective bucket. Another important setting is acl = public-read, which ensures that transferred files have the correct permissions. If this setting is not set correctly, the files may not be publicly readable and the media storage cannot be used in the sense of Mastodon.
Now follows a series of rclone commands that move the media files from the old minio-s3 to the new hetzner-s3 storage. Please note that the copy subcommand of rclone is used here. This does not synchronize deletions, but only transports non-existent files from A to B. Unlike sync! This would delete newly arrived files in the new bucket. So caution is advised. copy is the right command for our purpose.
The most important files first:
rclone copy --progress --transfers=8 minio-s3:mastodon-media/custom_emojis/ hetzner-s3:mastodon-media/custom_emojis/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/accounts/ hetzner-s3:mastodon-media/accounts/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/site_uploads/ hetzner-s3:mastodon-media/site_uploads/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/media_attachments/ hetzner-s3:mastodon-media/media_attachments/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/imports/ hetzner-s3:mastodon-media/imports/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/accounts/ hetzner-s3:mastodon-media/cache/accounts/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/custom_emojis/ hetzner-s3:mastodon-media/cache/custom_emojis/
This is followed by the less important media files. These are only cached media and preview images for websites. If you want to save time and traffic, you can also omit these (or reduce the retention time for these media in Mastodon to a few days in advance—this will significantly reduce the amount of data to be transferred!).
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/preview_cards/ hetzner-s3:mastodon-media/cache/preview_cards/
rclone copy --progress --transfers=8 minio-s3:mastodon-media/cache/media_attachments/ hetzner-s3:mastodon-media/cache/media_attachments/
This process can take up to half a day, depending on the performance of the servers involved and the size of the bucket.
Step 4: Disconnect the old bucket
Once all data has been transferred, it is advisable to keep the old bucket for a while in case any data has not been transferred correctly. However, you can disconnect the old bucket from the proxy on a trial basis and check whether the Mastodon instance runs correctly without the old bucket. To do this, remove the @oldbucket location block and comment out (or remove) the following lines:
#upstream minio {
# server 127.0.0.1:9000;
#}
[...]
# Fallback to oldbucket
#proxy_intercept_errors on;
#error_page 404 403 = @oldbucket;
After running systemctl reload nginx, only the new bucket will be used and the website’s functionality can be checked. Don’t forget: For serious testing, clear your browser cache!
If everything is still running smoothly after a few days, the old bucket can be deleted. Done!