Since the big Twitter wave that flooded the Mastodon network and, more broadly, the Fediverse in the fall and winter of 2022, international users have been playing a bigger role for metalhead.club. The service is hosted entirely in Germany, and that was still the case until recently. However, with the increasing number of international members come new challenges: for example, the rapid delivery of content.
As long as users are mainly located in Germany and Europe, latency times to the “Full Metal Server” in Frankfurt are low. However, the situation is different for users from Canada, the US, and Australia, for example, of whom there are a significant number on metalhead.club. For these users, using metalhead.club was sometimes a bit of a test of patience, as videos and larger images in particular appeared on the website with a slight delay. I can only simulate the situation in the browser, but even a ping of more than 200 ms spoils the fun of scrolling through the timeline in some places.
My Mastodon instance metalhead.club exists since summer 2016 and seen several waves of new users - but never as many new users as in early November 2022. This has not only led to heavy CPU work on the servers (see my post about scaling up Mastodon’s Sidekiq Workers), but also to greater load on storage space. Mastodon uses a media cache that not only stores copies of preview images for posts containing links - but also copies of all media files that the server knows of. Before the user wave of late 2020 metalhead.club’s media cache was about 350 GB in size with a cache retention time of 60 days. Quickly the numbers escalated and after a few days we were already at 400 GB - and after about 3 weeks we had more than 550 GB of cached media files. Not with 60 days retention time - but with 30 only.
Despite I added hundreds of GB of new storage space, the cache showed no signs of shrinking in the near future, so decided to offload the storage to an S3 storage provider. The local disks would have been full a few days later.