Hey folks

This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.

We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.

I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.

As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!


Edit: upgrade complete!

I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.

As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner’s Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I’m really happy with the solution now.

In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.

    • vsh@lemm.ee
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      17
      ·
      edit-2
      11 months ago

      Yeah, he has no obligation to post changes or even maintain this server as well. Would it mean he cares for the community? No. This admin sucks balls and you can’t change my mind.

      He’s keeping it secret probably because our server looks like this:

        • vsh@lemm.ee
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          10
          ·
          11 months ago

          Having a good stable server is seen as luxury today smh.

          • TexMexBazooka@lemm.ee
            link
            fedilink
            arrow-up
            11
            arrow-down
            1
            ·
            11 months ago

            As stated, spin up your own, show us how it’s done. You get no compensation, and you better fucking get it right.

            • vsh@lemm.ee
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              10
              ·
              11 months ago

              Is this a threat? I’ve been administrating several servers in my free time over the past few years. You know nothing kid.

      • Neve8028@lemm.ee
        link
        fedilink
        arrow-up
        2
        ·
        11 months ago

        It literally doesn’t matter if people know or not. As long as things run smoothly, it’s fine.

        • vsh@lemm.ee
          link
          fedilink
          English
          arrow-up
          0
          arrow-down
          5
          ·
          11 months ago

          It literally doesn’t matter if people know or not. As long as things run smoothly, it’s fine.

          Except it doesn’t