What is "Sidekiq" and why is it important?
Sidekiq is the application that processes all of the "jobs" on a Mastodon server. These "jobs" are every action your server performs including sending e-mail, creating posts, pulling posts for users to view, uploading media, updating profiles, setting filters, cleaning up the database, and so many more actions. Basically, if something happens on your server or on a server your server knows about, there's most likely a Sidekiq job for it. This means that Sidekiq is one of the most crucial parts of the Mastodon server and it needs to be able to process as many jobs as it can as fast as possible to ensure your server is running smoothly.
About the queues.
There are multiple Sidekiq queues and they vary in importance and usage. The Mastodon documentation actually outlines this very well, so I'm just going to borrow the information from there:
The only other super important note is that the scheduler queue can only be running once. This is very important so to avoid people skipping over this, I'm going to post another screenshot from the documentation:
The scheduler queue is used for running scheduled jobs so if you have more than one running, it can trigger jobs out of order, multiple times, or miss them entirely. Don't do that.
How to configure multiple Sidekiq services.
One of the biggest performance improvements you can make to a Mastodon server is to break out your Sidekiq queues into multiple services (and eventually giving them their own servers). To do this, you'll need to create multiple files in the /etc/systemd/system/ directory.
NOTE: These instructions are for adding additional services, I recommend keeping the original Mastodon service file in use since it already runs all of the queues including the scheduler queue. If you remove/replace the original service file, you'll need to make sure to add a scheduler queue to one of the new files you create.
- Navigate to /etc/systemd/system/ on your server. > cd /etc/systemd/system/
Create a new file called mastodon-sidekiq-default.service
vi mastodon-sidekiq-default.service
Copy and paste the code from this GitHub gist: mastodon-sidekiq-default.service (github.com)
Reload the systemd daemon.
systemctl daemon-reload
Enable and start the service.
systemctl enable mastodon-sidekiq-default.service --now
And that's all there is to it to add another service for your Sidekiq queues! You can keep adding services depending on your server's resources. The rule of thumb is 1 service per CPU core, but I tend to leave at least 1 CPU core available for handling media encoding (ffmpeg/convert processes).
For a single user or a small instance, 2 services should be plenty. You'll notice the second service file only handles the default, push, pull, ingress, and mailer queues because, as mentioned before, the scheduler queue can only be running once.
If you do decide to add more services, you can keep creating files using the steps above. It's usually recommended to have a queue for each service so that the ExecStart line on the file is in a different order. For example, here's my mastodon-sidekiq-push.service file: mastodon-sidekiq-push.service (github.com)
NOTE: Keep the thread count to 25, anything lower or higher can hurt performance. If you need more threads create a new service file.
To give you an idea of how my server is configured, here's a screenshot of the queues from the Sidekiq dashboard:
From this image you can see I have multiple services on 2 different servers, I have 4 services on one server (sidekiq00) and 3 services on the other (sidekiq01). I keep my scheduler queue on a single service and, because I had issues with e-mails previously, I keep my mailer queues limited to 2 services on the same server.
If you want to learn more about this whole process, there are other really good resources out there, here's an article with a really good breakdown of the queues and what-not (also other good info here, but mostly focused on Docker): Scaling Mastodon in the Face of an Exodus | Nora Codes
Running Sidekiq services on remote servers
I almost didn't include this because there's a lot of variables involved including network latency, media IO speed, database performance, etc...
But since you're here, let's go over the basics.
- Sidekiq queues need access to the PostgreSQL and Redis databases so this only works if you have your database servers configured for remote connections.
- Sidekiq queues need access to your media storage, you will need to have network storage configured (NFS will not work due to latency, go with an S3 compatible object storage platform).
The actual process:
- You will need to basically have Mastodon installed on each Sidekiq server, follow the install guide but do not configure PostgreSQL, Redis, Nodejs, Puma, or nginx. The only steps you need to follow are the Prerequisites (skipping the nginx, PostgreSQL, and Redis parts) and the Setting up Mastodon steps.
- Copy over your .env.production file from your main server.
- Once you have Mastodon installed on the new server, follow the steps above to configure the Sidekiq services, do this last since it will need the .env.production file.
Simple right? It can be, but for me it took a while to figure out and even longer before a lot of the stuff "clicked" for me so I understood it.