Hi, someone told me that lemmy.world is running a similar setup to mine and that you might have some insight.

For background, I’ve got my instance running on 3 servers, I’ve disabled the scheduled tasks on the Lemmy containers that are load balanced (I’ve got an extra container that does the scheduled tasks).

If you started up the Lemmy containers and they all tried to do a database migration, how do you handle this? Is the best option to just run a single process on an upgrade and wait until the migration is finished before starting up the others?

  • Ellie@silkky.pubOP
    2·
    1 year ago

    Are concurrent migrations actually a problem?

    That might be the case, I wouldn’t be confident reviewing it myself though since I don’t know Rust that well and haven’t used diesel before but I could ask the developers about it.

    A mode in Lemmy to do nothing other than run migrations and exit.

    I don’t think there’s a way to just run migrations like this.

    Create my own DB migration script based on the Lemmy schema diffs from release tag to release tag.

    Yeah maybe this could work?

    Is there a config-flag on Lemmy to enable/disable db migrations?

    There’s a flag to disable scheduled jobs like you described but not for database migrations. I use that flag on all my Lemmy containers that serve requests, I have a separate container to handle the scheduled jobs and since it’s just normal instance of Lemmy it runs database migrations on startup, so maybe one option would be to upgrade the jobs container first, wait for it to finish the migrations and then upgrade the rest. Obviously to do that with no issues or downtime the migration would need to be backwards compatible. The docs seem to indicate that this is possible.

    As you note, run just one Lemmy container for “a while” when upgrading.

    Yeah I feel like it would be a bit cumbersome and it would be nice to be able to do a rolling update with no downtime.

    • PriorProject@lemmy.world
      2·
      1 year ago

      Seems to me like you’re thinking clearly about this. Unless someone else with hands on experience jumps in with a pro-tip I suspect you may just have to pick your poison and give it a go.

      Multi-lemmy isn’t a common config, though. Lemmy.world and I think lemmy.ca do it, but even lemmy.ml which was the biggest instance 2mo ago doesn’t. There might be half a dozen people in the world with hands on experience at this.