Matrix Synapse: migrating from Cloudron to ansible
Maybe, like me, you tried to selfhost a Matrix Synapse server, miserably failed because it's just not quite that easy and then settled for Cloudron's Synapse app because it just works. Pay a bit more, worry a little less.
Sure, it works but you still introduced a middle man in your pristine homelab setup and the itch never goes away to get rid of it.
Time for round 2.
The ansible playbook
So, I found myself a spare machine (SBC, NUC, VPS, all should work), installed Debian 11, enabled SSH and went to work. Here's what I did to finally make the Cloudron migration happen.
!Read through the whole guide first! Understanding why you do certain things will help you do those things correctly or do them do differently: just because I did them a certain way doesn't mean it's the best way for everyone.
First, I moved some important files from the Cloudron server to my new server. There are two folders of interest, both can be obtained by visiting the Cloudron admin panel, going to the settings for the Matrix-Synapse app and clicking on
/home/yellowtent/appsdata/<APP-ID> folder, you'll find the
postgresqldump file that was generated during the last Cloudron app backup — so make sure to run a backup right before migrating to have the latest data!
/mnt/data/apps/<APP-NAME>/data folder, you'll find the all-importand
Additionally, in the
/mnt/data/apps/<APP-NAME>/configs folder, you will find the
homeserver.yaml config file for the Matrix Synapse server — use this for inspiration for the new one but more importantly, make a note of the
database/args/user value. There is also the
Using your method of choice (rsync, wormhole…), copy the
postgresqldump file, the
media_store folder and the
signing.key over to the new server in — for example — the
This is the moment to power down the Cloudron machine (or simply the app if you wish to keep Cloudron running) and update the DNS records.
Fixing the database dump
Before we touch ansible, we need to go our
postgresqldump file and replace all instances of the previous database user (the one you found in the
homeserver.yaml on the Cloudron server, it should look like
According to the Importing an existing Postgres database, this should be
synapse and not
matrix. I ran into permission issues when using
synapse but that may have been due to a configuration error I made elsewhere. Feel free to attempt with
Setting up the new server with ansible
I am going to skip the majority of the ansible process here, namely the Configuring the Ansible playbook part and only focus on what is relevant when migrating from a Cloudron instance. There is a special Importing an existing Postgres database but I had to change the steps a bit to make them work for me.
I followed the Configuring the Ansible playbook steps to obtain my ansible configuration file.
In the end, I added to
# Set up synapse database connection matrix_synapse_database_user: 'matrix' matrix_synapse_database_password: 'SUPERSECRETPASSWORD' matrix_synapse_database_database: 'matrix' # Set up postgres matrix_postgres_connection_username: 'matrix' matrix_postgres_connection_password: 'SUPERSECRETPASSWORD' matrix_postgres_db_name: 'matrix'
If you chose in the previous section to try and import the database dump into the
synapse database instead of the
matrix, make sure to update those values here.
Let's let ansible set up everything on the server:
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all -K
Do not start just yet! First, let's import the database dump:
ansible-playbook -i inventory/hosts setup.yml \ --extra-vars='server_path_postgres_dump=/migration/postgresqldump' \ --tags=import-postgres -K
If there weren't any errors, let's import the
media_store folder as well:
ansible-playbook -i inventory/hosts setup.yml \ --extra-vars='server_path_media_store=/migration/media_store/' \ --tags=import-synapse-media-store -K
If still no errors, great! Now let's take a look at that signing key we copied over. What I should have done was follow the instructions here to add the previous signing key to the list of old signing keys. But I didn't know of these instructions when performing the setup, so I simply used the old key to overwrite the signing key ansible had generated and stored in
/matrix/synapse/config/matrix.your.domain.signing.key. Not as elegant, I admit, but it works.
It's time to start the server (and run the setup again for the new signing key):
ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start -K
Something went wrong
Surely, something went wrong in one of the steps above, it happens.
If something went wrong during the importing of the
postgresqldump, you can't just repeat the step as postgres will now complain that some import steps were already performed (see the Importing an existing Postgres database guide).
I followed the exact steps they propose. So, on the new server, run:
systemctl stop matrix-postgres rm -rf /matrix/postgres/data/* systemctl start matrix-postgres
Then, back on the ansible controller, run:
ansible-playbook -i inventory/hosts setup.yml \ --tags=setup-postgres -K
You now have an empty database ready for a fresh import!
Checking the import process succeeded
To check if the data was imported correctly, here's how to log into the database and query the user table:
/usr/local/bin/matrix-postgres-cli # List the databases \l # Connect to the matrix database \c matrix # Query the users table select * from users;
If that last query returns the list of users you expect to see, we should be good! Well, almost.
For some explanation that is as of yet beyond me, the old passwords won't work. That is, we haven't changed the passwords during the import process, but we also cannot log in as the server will now complain the password is incorrect.
Luckily, we can simply reset a user's password to fix this:
ansible-playbook -i inventory/hosts setup.yml \ --extra-vars='username=USER password=SUPERSECRETPASSWORD' \ --tags=update-user-password -K
This user can now log in again.
If like me, you host a Synapse server for a small number of people that trust you, this step is not an issue. If you host for a lot of people, I am not really sure how to proceed. Hopefully someone can help out here, explain what step I missed to make passwords work after the migration and I can update the guide accordingly.
There you have it, this is what I did to obtain a working fresh Synapse server with all the data from the Cloudron server. Cloudron was nice to work with but I am glad I have everything working again without the need for a middle man.
So now… dendrite when?