As you might know, I’m running metalhead.club, a Mastodon instance for metalheads. Due to the increasing storage and computing demand (and because I wanted to drop my old host) I decided to move the instance to my new, more powerful host. Luckily I’ve packed the whole instance and all its dependencies into a LXC container (with LXD as container manager) a couple of months ago. Usually you would restore your Ruby / NodeJS environment on your new host, transfer database and application files as well as media files and make sure everything fits. In my case it was basically just a file system transfer and re-import on the new LXD host: Much easier and less error prone.
In this post I’ll show you the exact steps how I moved my Mastodon instance yesterday.
In this guide I assume that:
- You’re using LXD via the official snap package
- Your’re using LXD with ZFS
- LXD is set up on the new host
- A trusted SSH connection is available from the old host to the new one
Step #1: Decrease DNS TTL
Usually the TTL for DNS records is somewhere between 3600 (1 hour) and 86400 (24 h), which will cause a mayor downtime for some clients that just refreshed their DNS data. Before you move your instance, you should decrease the TTL to one minute (60) or even less. After the TTL was changed, wait according to the TTL until the new TTL is spread in the DNS. If your previous TTL was 86400, wait 24 hours until you continue with the next steps.
Step #2: First data migration (foundation)
To keep downtime low and not miss any new or updated data, the data migration will be done in two parts: First a basic copy of the container’s file system will be transferred. This might take a while, depending on the dat asize. Then, shortly before IP switching, updated data will be synchronized to the new host with the help of ZFS snapshots. This process will be much faster because older data is already there.
Let’s first create a snapshot of the container’s filesystem on the old host:
root@oldhost# zfs snapshot default/containers/mastodon@mig1
Then transfer the snapshot to the new host’s ZFS filesystem:
root@oldhost# zfs send default/containers/mastodon@mig1 | pv -ps 35g | ssh newhost zfs recv default/containers/mastodon
pv -ps 35g command shows the overall progress, where 35g is the expected data size (shown by e.g.
zfs list). The ZFS volume
default/containers/mastodon must not exist on the new host!
After the transfer, mount the file system in the namespace context of the LXD process:
root@newhost# nsenter -t $(pgrep daemon.start) -m root@newhost# mkdir /var/snap/common/lxd/storage-pools/default/containers/mastodon root@newhost# mount -t zfs default/containers/mastodon /var/snap/common/lxd/storage-pools/default/containers/mastodon root@newhost# exit
Then import the new filesystem into LXD:
root@newhost# lxd import --force mastodon
--force option is needed because the new LXD instance can’t find any earlier snapshots in the ZFS dataset although backup.yml is announcing them.
lxc list should now display the imported Mastodon container. You can try and start it via
lxc start mastodon - just as usual.
Step #3: Final data sync
Let’s start with the final dara migration. To make sure the container file system is in a consistent state and no more data is written to the file systems, both Mastodon instances must now be shut down!
root@oldhost# lxc stop mastodon root@newhost# lxc stop mastodon
Create a new, updated snapshot on the old host and send an incremental snapshot (
mig2) to the new server:
zfs snapshot default/containers/mastodon@mig2 zfs send -i default/containers/mastodon@mig1 default/containers/mastodon@mig2 | ssh newhost zfs recv -F default/containers/mastodon
-F option at
zfs recv makes ZFS ignore changes that were caused by the previous startup of the mastodon container.
Step #4: Update DNS to new IP address
It is now time to update your DNS settings and publish the new server’s IP address for the Mastodon instance. Start the new instance:
root@newhost# lxc start mastodon
… and keep the old instance stopped!
Container migration is now complete! :-)