Matrix Synapse: migrating from Cloudron to ansible

Date

Tags
#selfhosting #cloudron #matrix

Maybe, like me, you tried to selfhost a Matrix Synapse server, miserably failed because it's just not quite that easy and then settled for Cloudron's Synapse app because it just works. Pay a bit more, worry a little less.

Sure, it works but you still introduced a middle man in your pristine homelab setup and the itch never goes away to get rid of it.

Time for round 2.

The ansible playbook

The last few years, I have been slowly learning how to work with ansible and ever since I found the matrix-docker-ansible-deploy project, I knew this would be my chance for redemption.

So, I found myself a spare machine (SBC, NUC, VPS, all should work), installed Debian 11, enabled SSH and went to work. Here's what I did to finally make the Cloudron migration happen.

Decommissioning Cloudron

!Read through the whole guide first! Understanding why you do certain things will help you do those things correctly or do them do differently: just because I did them a certain way doesn't mean it's the best way for everyone.

First, I moved some important files from the Cloudron server to my new server. There are two folders of interest, both can be obtained by visiting the Cloudron admin panel, going to the settings for the Matrix-Synapse app and clicking on Storage.

In the /home/yellowtent/appsdata/<APP-ID> folder, you'll find the postgresqldump file that was generated during the last Cloudron app backup — so make sure to run a backup right before migrating to have the latest data!

In the /mnt/data/apps/<APP-NAME>/data folder, you'll find the all-importand media_store folder.

Additionally, in the /mnt/data/apps/<APP-NAME>/configs folder, you will find the homeserver.yaml config file for the Matrix Synapse server — use this for inspiration for the new one but more importantly, make a note of the database/args/user value. There is also the signing.key file.

Using your method of choice (rsync, wormhole…), copy the postgresqldump file, the media_store folder and the signing.key over to the new server in — for example — the /migration folder.

This is the moment to power down the Cloudron machine (or simply the app if you wish to keep Cloudron running) and update the DNS records.

Fixing the database dump

Before we touch ansible, we need to go our postgresqldump file and replace all instances of the previous database user (the one you found in the homeserver.yaml on the Cloudron server, it should look like user1a2b3c4d5e6f7g8h9i) with matrix.

According to the Importing an existing Postgres database, this should be synapse and not matrix. I ran into permission issues when using synapse but that may have been due to a configuration error I made elsewhere. Feel free to attempt with synapse.

Setting up the new server with ansible

I am going to skip the majority of the ansible process here, namely the Configuring the Ansible playbook part and only focus on what is relevant when migrating from a Cloudron instance. There is a special Importing an existing Postgres database but I had to change the steps a bit to make them work for me.

I followed the Configuring the Ansible playbook steps to obtain my ansible configuration file.

In the end, I added to vars.yml:

# Set up synapse database connection
matrix_synapse_database_user: 'matrix'
matrix_synapse_database_password: 'SUPERSECRETPASSWORD'
matrix_synapse_database_database: 'matrix'

# Set up postgres
matrix_postgres_connection_username: 'matrix'
matrix_postgres_connection_password: 'SUPERSECRETPASSWORD'
matrix_postgres_db_name: 'matrix'

If you chose in the previous section to try and import the database dump into the synapse database instead of the matrix, make sure to update those values here.

Let's let ansible set up everything on the server:

ansible-playbook -i inventory/hosts setup.yml --tags=setup-all -K

Do not start just yet! First, let's import the database dump:

ansible-playbook -i inventory/hosts setup.yml \
  --extra-vars='server_path_postgres_dump=/migration/postgresqldump' \
  --tags=import-postgres -K

If there weren't any errors, let's import the media_store folder as well:

ansible-playbook -i inventory/hosts setup.yml \
  --extra-vars='server_path_media_store=/migration/media_store/' \
  --tags=import-synapse-media-store -K

If still no errors, great! Now let's take a look at that signing key we copied over. What I should have done was follow the instructions here to add the previous signing key to the list of old signing keys. But I didn't know of these instructions when performing the setup, so I simply used the old key to overwrite the signing key ansible had generated and stored in /matrix/synapse/config/matrix.your.domain.signing.key. Not as elegant, I admit, but it works.

It's time to start the server (and run the setup again for the new signing key):

ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start -K

Done!

Again, I skipped a lot of important steps like setting up a reverse proxy — this playbook includes Nginx — so make sure to read all the documentation to end up with a fully functional instance.

Something went wrong

Surely, something went wrong in one of the steps above, it happens.

If something went wrong during the importing of the postgresqldump, you can't just repeat the step as postgres will now complain that some import steps were already performed (see the Importing an existing Postgres database guide).

I followed the exact steps they propose. So, on the new server, run:

systemctl stop matrix-postgres
rm -rf /matrix/postgres/data/*
systemctl start matrix-postgres

Then, back on the ansible controller, run:

ansible-playbook -i inventory/hosts setup.yml \
  --tags=setup-postgres -K

You now have an empty database ready for a fresh import!

Checking the import process succeeded

To check if the data was imported correctly, here's how to log into the database and query the user table:

/usr/local/bin/matrix-postgres-cli
# List the databases
\l
# Connect to the matrix database
\c matrix
# Query the users table
select * from users;

If that last query returns the list of users you expect to see, we should be good! Well, almost.

Resetting passwords

For some explanation that is as of yet beyond me, the old passwords won't work. That is, we haven't changed the passwords during the import process, but we also cannot log in as the server will now complain the password is incorrect.

Luckily, we can simply reset a user's password to fix this:

ansible-playbook -i inventory/hosts setup.yml \
  --extra-vars='username=USER password=SUPERSECRETPASSWORD' \
  --tags=update-user-password -K

This user can now log in again.

If like me, you host a Synapse server for a small number of people that trust you, this step is not an issue. If you host for a lot of people, I am not really sure how to proceed. Hopefully someone can help out here, explain what step I missed to make passwords work after the migration and I can update the guide accordingly.

Conclusion

There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.

So now… dendrite when?