Hi, someone told me that lemmy.world is running a similar setup to mine and that you might have some insight.

For background, I’ve got my instance running on 3 servers, I’ve disabled the scheduled tasks on the Lemmy containers that are load balanced (I’ve got an extra container that does the scheduled tasks).

If you started up the Lemmy containers and they all tried to do a database migration, how do you handle this? Is the best option to just run a single process on an upgrade and wait until the migration is finished before starting up the others?

  • PriorProject@lemmy.world
    3·
    1 year ago

    I’m not running my own Lemmy, but I do run non-trivial systems elsewhere. Things I’d be looking for would be:

    • Are concurrent migrations actually a problem? If Lemmy does a good job of taking out an exclusive transaction that checks if the migration is needed and then does it, it might be that several containers can race safely and you don’t need to do anything. I’d want to review the schema migration code before I trusted this myself though, since few people rely on it.
    • A mode in Lemmy to do nothing other than run migrations and exit. You could then execute this as a job or exit-on-terminate container during the upgrade while other Lemmy’s are down.
    • Create my own DB migration script based on the Lemmy schema diffs from release tag to release tag.
    • Is there a config-flag on Lemmy to enable/disable db migrations? I don’t have a link handy but I feel like I heard about lemmy.world having a Lemmy container just for doing async jobs, separate from the ones serving requests. It would be nice if one could run this single separate container and let it handle the migrations… but the other Lemmy’s would have to be clever enough to wait until the migration is complete.
    • As you note, run just one Lemmy container for “a while” when upgrading. The issue here is knowing when the migrations are complete, which feels finicky to automate. Though if you do this manually as part of your upgrade process I’m sure it can work ok.
    • Ellie@silkky.pubOP
      2·
      1 year ago

      Are concurrent migrations actually a problem?

      That might be the case, I wouldn’t be confident reviewing it myself though since I don’t know Rust that well and haven’t used diesel before but I could ask the developers about it.

      A mode in Lemmy to do nothing other than run migrations and exit.

      I don’t think there’s a way to just run migrations like this.

      Create my own DB migration script based on the Lemmy schema diffs from release tag to release tag.

      Yeah maybe this could work?

      Is there a config-flag on Lemmy to enable/disable db migrations?

      There’s a flag to disable scheduled jobs like you described but not for database migrations. I use that flag on all my Lemmy containers that serve requests, I have a separate container to handle the scheduled jobs and since it’s just normal instance of Lemmy it runs database migrations on startup, so maybe one option would be to upgrade the jobs container first, wait for it to finish the migrations and then upgrade the rest. Obviously to do that with no issues or downtime the migration would need to be backwards compatible. The docs seem to indicate that this is possible.

      As you note, run just one Lemmy container for “a while” when upgrading.

      Yeah I feel like it would be a bit cumbersome and it would be nice to be able to do a rolling update with no downtime.

      • PriorProject@lemmy.world
        2·
        1 year ago

        Seems to me like you’re thinking clearly about this. Unless someone else with hands on experience jumps in with a pro-tip I suspect you may just have to pick your poison and give it a go.

        Multi-lemmy isn’t a common config, though. Lemmy.world and I think lemmy.ca do it, but even lemmy.ml which was the biggest instance 2mo ago doesn’t. There might be half a dozen people in the world with hands on experience at this.