Migrate Mastodon Media Storage from Minio S3 to SeaweedFS S3

🇩🇪 [Deutsch]

Until now, I have enjoyed using the Minio S3 Server together with my Mastodon instance metalhead.club to store media files. However, since last summer, the Minio project has unfortunately taken a turn that has prompted me to look for alternatives: First, the internal file browser was removed from the administration interface. Shortly thereafter, it was announced that Minio would no longer be maintained as an open source project. In addition, the distribution of Minio binaries was discontinued. This meant that there were no more updates. Minio has effectively been dead ever since at least as far as the community version is concerned.

However, I found a great alternative in SeaweedFS, which I recently switched my Mastodon instance to. Here, I describe the problems I encountered and how I solved them.

Install SeaweedFS

Installation is very simple: SeaweedFS consists of a single binary, which is downloaded and stored in /usr/local/bin:

curl -L https://github.com/seaweedfs/seaweedfs/releases/download/4.07/linux_amd64_full.tar.gz | tar xvz -C /usr/local/bin/

Next, a new SeaweedFS system user is created:

useradd --system seaweedfs

and a corresponding Systemd service is created:

/etc/systemd/system/seaweedfs.service:

[Unit]
Description=SeaweedFS Server
After=network.target

[Service]
Type=simple
User=seaweedfs
Group=seaweedfs

ExecStart=/usr/local/bin/weed server -s3 -ip 127.0.0.1 -ip.bind=127.0.0.1 -volume.max=0 -dir /mnt/s3storage1/seaweedfs
WorkingDirectory=/usr/local/bin
SyslogIdentifier=seaweedfs

[Install]
WantedBy=multi-user.target
  • /usr/local/bin/weed server -s3 starts the SeaweedFS binary weed and with it the components master server, volume server, filer, and the s3 frontend.
  • -ip 127.0.0.1 -ip.bind=127.0.0.1: Internal communication interfaces and the S3 API should remain internal to localhost, as we are not running SeaweedFS in a cluster and the S3 interface should only be accessible to the internal Nginx proxy.
  • -volume.max=0 ensures that the number of storage volumes is not limited. Otherwise, you could set the maximum amount of storage SeaweedFS is allowed to use (default: 1 volume = 30 GB). In my case, I want to use all available storage.
  • -dir /mnt/s3storage1/seaweedfs: This is where the volumes are stored in the form of files.

Activate and start the service:

systemctl enable --now seaweedfs.service

SeaweedFS should start after a few seconds. See also:

systemctl status seaweedfs

Create a bucket

A new S3 bucket can now be created and configured using the “weed shell.” In my case, the new bucket is called “metalheadclub-media.”

Open shell:

weed shell

Create bucket:

s3.bucket.create -name metalheadclub-media

Set permissions (dynamically, instead of a json config file):

s3.configure -access_key=ACCESS_KEY -secret_key=SECRET_KEY -buckets=metalheadclub-media -user=mastodon -actions=Read,Write,List -apply

The following is displayed for confirmation:

{
    "identities":  [
    {
        "name":  "mastodon",
        "credentials":  [
        {
            "accessKey":  "ACCESS_KEY",
            "secretKey":  "SECRET_KEY",
            "status":  ""
        }
        ],
        "actions":  [
        "Read:metalheadclub-media",
        "Write:metalheadclub-media",
        "List:metalheadclub-media"
        ],
        "account":  null,
        "disabled":  false,
        "serviceAccountIds":  [],
        "policyNames":  []
    }
    ],
    "accounts":  [],
    "serviceAccounts":  []
}

Anonymous (public) users only have read access to the media files:

s3.configure -buckets=metalheadclub-media -user=anonymous -actions=Read -apply

Output:

{
    "identities":  [
    {
        "name":  "mastodon",
        "credentials":  [
        {
            "accessKey":  "ACCESS_KEY",
            "secretKey":  "SECRET_KEY",
            "status":  ""
        }
        ],
        "actions":  [
        "Read:metalheadclub-media",
        "Write:metalheadclub-media",
        "List:metalheadclub-media"
        ],
        "account":  null,
        "disabled":  false,
        "serviceAccountIds":  [],
        "policyNames":  []
    },
    {
        "name":  "anonymous",
        "credentials":  [],
        "actions":  [
        "Read:metalheadclub-media"
        ],
        "account":  null,
        "disabled":  false,
        "serviceAccountIds":  [],
        "policyNames":  []
    }
    ],
    "accounts":  [],
    "serviceAccounts":  []
}

You can then exit the shell by typing “quit.”

Migrating data from Minio to SeaweedFS

The SeaweedFS server is now active, the bucket has been created and assigned the necessary permissions. Time to migrate the data from Minio to SeaweedFS!

There were two things to keep in mind for the migration:

  • The migration should be as “silent” as possible and have no impact on the users of my Mastodon instance.
  • Since I wanted to use the same virtual machine with the same storage for SeaweedFS and storage was limited, the data should be moved from Minio to SeaweedFS, not copied!

As with previous migrations, the strategy was as follows:

  1. Put the new backend into operation
  2. Configure the frontend proxy with both backends => Frontend first queries the new backend, falling back to the old backend in case of error
  3. Reconfigure the Mastodon instance: Stores new media in the new backend, while old media remains accessible from the old backend
  4. Media is slowly moved from the old storage to the new storage in the background
  5. … until the old storage is finally empty and can be removed
  6. Done!

Two S3 backends for a smooth transition

I have visualized the configuration for data migration here:

The graphic shows the dependencies between the Nginx proxies and the S3 servers. It starts with a media.metalhead.club server, which forms the front end. Behind it, one path leads directly to Minio, while another path leads to the SeaweedFS backend via another proxy, s3.650thz.de.

The graphic shows the dependencies between the Nginx proxies and the S3 servers. It starts with a media.metalhead.club server, which forms the front end. Behind it, one path leads directly to Minio, while another path leads to the SeaweedFS backend via another proxy, s3.650thz.de.

The upper row (media.metalhead.club, Minio server) already existed – the SeaweedFS backend and the s3.650thz.de server are new additions. The media.metalhead.club Nginx proxy was to play a special role again (similar to the migration from Scaleway S3 to Minio) and switch flexibly between the two S3 backends during the migration of the data. The new SeaweedFS backend should primarily respond to requests. In the event that the new backend was unable to deliver the data, a fallback was implemented to ensure that the request could then be answered by the old Minio backend.

The key lies in these lines:

# During data migration: Forward request to @minio if @seaweedfs returns 404 or 403 (access denied)
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 403 = @minio;

Virtual Host style addressing for SeaweedFS S3

An important difference in the handling of the SeaweedFS backend compared to Minio is that SeaweedFS does not support virtual host addressing of S3 buckets! With Minio, a bucket can be addressed in two ways:

  • bucket.s3server.tld or
  • s3server.tld/bucket

The former is therefore not possible with SeaweedFS. Instead, the SeaweedFS Wiki suggests rewriting addresses in the latter form:

...
server_name ~^(?:(?<bucket>[^.]+)\.)?s3\.yourdomain\.com;
...
# If bucket subdomain is not empty,
# rewrite request to backend.
if ($bucket != “”) {
    rewrite (.*) /$bucket$1 last;
}
...

However, I was unable to implement this type of configuration in my media.metalhead.club VirtualHost because it led to an error:

[error] 1388933#1388933: *5700433 rewrite or internal redirection cycle while redirect to named location "@seaweedfs", client: xxx, server: media.metalhead.club, request: "GET /cache/custom_emojis/images/xxx.png HTTP/2.0", host: "media.metalhead.club"

So I quickly moved this mechanism to another vHost called “s3.650thz.de”:

upstream seaweedfs {
    hash $arg_uploadId consistent;
    server 127.0.0.1:8333 fail_timeout=0;
    keepalive 20;
}

server {
   listen 80;
   listen [::]:80;
   listen 443 ssl;
   listen [::]:443 ssl;

   # Enable http2
   http2 on;

   server_name ~^(?<bucket>[^.]+)\.s3\.650thz\.de s3.650thz.de;

   ssl_certificate /etc/acme.sh/s3.650thz.de/fullchain.pem;
   ssl_certificate_key /etc/acme.sh/s3.650thz.de/privkey.pem;

   ignore_invalid_headers off;
   client_max_body_size 0;
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      if ($bucket != "") {
          rewrite ^/(.*)$ /$bucket/$1 break;
      }

      proxy_pass http://seaweedfs; # This uses the upstream directive definition to load balance
   }
}

Ultimately, the vHost only serves to convert “virtual host style addressing” into conventional “path style addressing.”

(The “vHost style addressing” in the media.metalhead.club vHost requires the proxy_pass directives, as paths such as proxy_pass http://seaweedfs/bucket cannot be specified here.)

My media.metalhead.club for the migration phase now looks like this:

upstream minio {
        server 127.0.0.1:9000;
        keepalive 2;
}

server {
   listen 80;
   listen [::]:80;
   listen 443 ssl;
   listen [::]:443 ssl;

   server_name  media.metalhead.club;

   # Enable http2
   http2 on;

   # SSL
   ssl_certificate /etc/acme.sh/media.metalhead.club/fullchain.pem;
   ssl_certificate_key /etc/acme.sh/media.metalhead.club/privkey.pem;

   client_max_body_size 100M;

   root /var/www/media.metalhead.club;

   location / {
      access_log off;
      try_files $uri @seaweedfs;
   }

   # Display index.html if user is browsing https://media.metalhead.club
   location = / {
        alias /var/www/media.metalhead.club/;
   }


   # 
   # Own SeaweedFS S3 server (new)
   #

   location @seaweedfs {
        # Set headers
        proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
        proxy_set_header Connection '';
        proxy_set_header Authorization '';

        # Hide headers
        proxy_hide_header Set-Cookie;
        proxy_hide_header 'Access-Control-Allow-Origin';
        proxy_hide_header 'Access-Control-Allow-Methods';
        proxy_hide_header 'Access-Control-Allow-Headers';
        # Hide amazon S3 headers, which are sent by Minio for compatibility
        proxy_hide_header x-amz-id-2;
        proxy_hide_header x-amz-request-id;
        proxy_hide_header x-amz-meta-server-side-encryption;
        proxy_hide_header x-amz-server-side-encryption;
        proxy_hide_header x-amz-bucket-region;
        proxy_hide_header x-amzn-requestid;
        proxy_hide_header x-amz-meta-mtime;

        # ignore headers
        proxy_ignore_headers Set-Cookie;

        # Pass headers
        proxy_pass_header  Date;
        proxy_pass_header  Last-Modified;
        proxy_pass_header  ETag;
        proxy_pass_header  Cache-Control;
        proxy_pass_header  Expires;

        # Connection to backend server  
        proxy_pass http://[::1];
        proxy_connect_timeout 1s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;

        # Buffer settings
        proxy_request_buffering off;
        proxy_buffering on;
        proxy_buffers 16 64k;
        proxy_busy_buffers_size 128k;
        proxy_buffer_size 32k;

        # Migration Workaround: Forward request to @minio if @seaweedfs returns 404 or 403 (access denied)
        proxy_intercept_errors on;
        recursive_error_pages on;
        error_page 404 403 = @minio;

        add_header 'Access-Control-Allow-Origin' '*';
        add_header X-Cache-Status $upstream_cache_status;
        add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
        add_header X-Served-By "s3.650thz.de SeaweedFS";
   }

   #
   # Own Minio S3 server (old)
   #
   location @minio {
        limit_except GET {
                deny all;
        }

        # Set headers
        proxy_set_header Host 'metalheadclub-media.s3.650thz.de';
        proxy_set_header Connection '';
        proxy_set_header Authorization '';

        # Hide headers
        proxy_hide_header Set-Cookie;
        proxy_hide_header 'Access-Control-Allow-Origin';
        proxy_hide_header 'Access-Control-Allow-Methods';
        proxy_hide_header 'Access-Control-Allow-Headers';
        # Hide amazon S3 headers, which are sent by Minio for compatibility
        proxy_hide_header x-amz-id-2;
        proxy_hide_header x-amz-request-id;
        proxy_hide_header x-amz-meta-server-side-encryption;
        proxy_hide_header x-amz-server-side-encryption;
        proxy_hide_header x-amz-bucket-region;
        proxy_hide_header x-amzn-requestid;
        proxy_hide_header x-amz-meta-mtime;

        # ignore headers
        proxy_ignore_headers Set-Cookie;

        # Pass headers
        proxy_pass_header  Date;
        proxy_pass_header  Last-Modified;
        proxy_pass_header  ETag;
        proxy_pass_header  Cache-Control;
        proxy_pass_header  Expires;

        # Connection to backend server  
        proxy_pass http://minio;
        proxy_connect_timeout 1s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;

        # Buffer settings
        proxy_request_buffering off;
        proxy_buffering on;
        proxy_buffers 16 64k;
        proxy_busy_buffers_size 128k;
        proxy_buffer_size 32k;

        add_header 'Access-Control-Allow-Origin' '*';
        add_header X-Cache-Status $upstream_cache_status;
        add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
        add_header X-Served-By "s3.650thz.de Minio";
    }
}

Using add_header X-Served-By “s3.650thz.de ...”, I also have the HTTP header indicate whether a file was delivered by Minio or SeaweedFS. This allows me to check later whether the data migration worked as hoped.

Configuring the Mastodon instance

In order for Mastodon to store the new media files in SeaweedFS, the correct parameters had to be set in the .env.production configuration:

S3_ENABLED=true
S3_BUCKET=metalheadclub-media
AWS_ACCESS_KEY_ID=ACCESS_KEY
AWS_SECRET_ACCESS_KEY=SECRET_KEY
S3_ALIAS_HOST=media.metalhead.club
S3_REGION=hyper2
S3_PROTOCOL=https
S3_HOSTNAME=media.metalhead.club
S3_ENDPOINT=https://s3.650thz.de
S3_SIGNATURE_VERSION=v4

After restarting all Mastodon services, the change was active:

cd /etc/systemd/system
systemctl restart mastodon*

I could easily check whether my change was successful: I waited for new posts with images to appear in the global Mastodon timeline and opened the image in a new tab. The Firefox developer tools then allow you to view the HTTP headers for the file. As hoped, the X-Served-By header was set to “SeaweedFS.” :) …

and the old graphics could still be accessed. They were delivered with X-Served-By “Minio.”

Data transfer

Great! Now it was time to start the actual data migration. Away from Minio – over to SeaweedFS.

Actually, such a migration is not rocket science. I configured rclone with both S3 buckets:

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = hyper2
endpoint = http://127.0.0.1:9000
location_constraint = hyper2
acl = public-read

[seaweedfs]
type = s3
provider = SeaweedFS
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
endpoint = http://127.0.0.1:8333
acl = public-read

I tested access to both buckets with a simple file listing:

rclone lsd seaweedfs:metalheadclub-media
rclone lsd minio:metalheadclub-media

I then began testing by first copying only smaller directories such as site_uploads from Minio to SeaweedFS (instead of moving them!):

rclone sync minio:metalheadclub-media/site_uploads/ seaweedfs:metalheadclub-media/site_uploads/

Once again, I used browser tools to test how the affected graphics (e.g., banner graphics at https://metalhead.club/about) were delivered. While they were initially delivered by Minio, they were delivered by SeaweedFS after the rclone sync.

After this test was also successful, I migrated the Mastodon media data piece by piece to SeaweedFS (this time by moving it with move instead of copying it with sync!).

List of directories to be moved:

root@s3 /root # rclone lsd minio:metalheadclub-media/  
           0 2026-01-16 14:01:17        -1 accounts
           0 2026-01-16 14:01:17        -1 backups
           0 2026-01-16 14:01:17        -1 cache
           0 2026-01-16 14:01:17        -1 custom_emojis
           0 2026-01-16 14:01:17        -1 imports
           0 2026-01-16 14:01:17        -1 media_attachments
           0 2026-01-16 14:01:17        -1 site_uploads

My commands for moving:

rclone move --delete-empty-src-dirs minio:metalheadclub-media/accounts seaweedfs:metalheadclub-media/accounts
rclone move --delete-empty-src-dirs minio:metalheadclub-media/backups seaweedfs:metalheadclub-media/backups
rclone move --delete-empty-src-dirs minio:metalheadclub-media/custom_emojis seaweedfs:metalheadclub-media/custom_emojis
rclone move --delete-empty-src-dirs minio:metalheadclub-media/imports seaweedfs:metalheadclub-media/imports
rclone move --delete-empty-src-dirs minio:metalheadclub-media/media_attachments seaweedfs:metalheadclub-media/media_attachments
rclone move --delete-empty-src-dirs minio:metalheadclub-media/site_uploads seaweedfs:metalheadclub-media/site_uploads

If you pay close attention, you will notice that I have omitted one directory: the media cache directory /cache! This is by far the largest and receives special treatment …

Trouble with the large /cache directory

This directory contains all files that do not originate from your own Mastodon server. In other words, cached remote media data and preview images. Depending on the cache retention time, the directory can become very large. In the case of metalhead.club, I was dealing with about 700 GB and approximately 3 million small files. In fact, it’s not the total size that’s the problem, but the number of files.

Of course, I also tried to move this directory to the new storage:

rclone move --delete-empty-src-dirs minio:metalheadclub-media/cache seaweedfs:metalheadclub-media/cache

… but after a few minutes, it failed due to insufficient RAM. I added more RAM (about 16 GB), but again, the rclone tool was so RAM-hungry that it was terminated by the OOM killer.

All right. If I couldn’t move all the data from the cache directory, I could just leave it in the old Minio bucket until it had accumulated! This is because the /cache directory only contains files that are not very important for operation and are normally cleaned up by Mastodon after a specified period of time anyway (at metalhead.club: cache retention 15 days).

“Just leave it and wait” sounds like an appealing strategy, but it’s not that simple. This is because the switch of the Mastodon software to the new bucket means that Mastodon’s internal deletion routines no longer work. The bucket remains as full as it was last used. And new files are pushing their way onto the system. The result: memory consumption increases.

If you have a lot of free memory, you may actually be able to simply wait until 15 days have passed and then delete the entire bucket. However, I only had about 170 GB of free storage. This would have been used up well before 15 days by new media added to the new bucket. So waiting was not an option. The old media had to be deleted bit by bit.

First, I tried using rclone commands. This command deletes all files older than 15 days:

rclone -v --min-age 15d delete --rmdirs minio:metalheadclub-media/cache

… - actually. Once again, the RAM requirements were too high (in some cases, 15 GB consumption by rclone alone!).

With a few optimizations, you can significantly reduce consumption:

GOGC=20 rclone --list-cutoff 100000 --min-age 15d delete --rmdirs minio:metalheadclub-media/cache

See also: https://rclone.org/faq/# rclone-is-using-too-much-memory-or-appears-to-have-a-memory-leak

Just when I thought I had solved the problem, another error popped up:

ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 10, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get “http://127.0.0.1:9000/metalheadclub-media?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=cache%2F”: net/http: timeout awaiting response headers

Apparently, listing the files took so long that the S3 server itself timed out (after about 10 minutes). How should I solve this? There were simply too many files in the bucket.

I also had no luck with Minio’s own tool mc:

./mc rm --recursive --force --older-than 15d minio/metalheadclub-media/cache
mc: <ERROR> Failed to remove `minio/metalheadclub-media/cache` recursively. Get "http://localhost:9000/metalheadclub-media/? delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=cache": read tcp [::1]:53662->[::1]:9000: i/o timeout

How could I get rid of 3 million small files? As it turned out, it was actually very simple…

Deleting files using S3 Lifecycle Rules…

because I had completely forgotten about a practical S3 mechanism: Lifecycle Rules. These rules allow you to specify what should happen to files during or after their lifecycle. One use case is automatic deletion. Exactly what I needed! And the best part is that this deletion takes place internally in the backend and does not have to be controlled by external tools. This allowed me to cleverly circumvent my timeout problem with file listing.

So I used the mc tool to expire all files in /cache after 15 days:

mc ilm add minio/metalheadclub-media/cache --expire-days 15

Done!

The current status can be viewed with

mc admin scanner status minio/

To make the Minio internal file scanner work faster and thus delete files more quickly when they have exceeded their expiration date, the process can be accelerated a little further:

mc admin config set minio/ scanner speed=fastest

Conclusion

After 15 days, all old files had been deleted and I was able to delete the Minio bucket, remove Minio, and also remove the old bucket from my Nginx configurations. The lines

# Migration Workaround ...
proxy_intercept_errors on;
recursive_error_pages on;
error_page 404 403 = @minio;

and

location @minio {
    ...
}

were removed and Nginx was reloaded. Metalhead.club was now completely migrated to SeaweedFS.

By the way, thanks to Stefano Marinelli (@stefano@bsd.cafe) for the SeaweedFS article under FreeBSD. This ultimately convinced me to give SeaweedFS a try and tackle the migration.