It looks like the lack of persistent storage for the federated activity queue is leading to instances running out of memory in a matter of hours. See my comment for more details.

Furthermore, this leads to data loss, since there is no other consistency mechanism. I think it might be a high priority issue, taking into account the current momentum behind growth of Lemmy…

  • King@vlemmy.net
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Can you provide a link? Only thing I see on Couchbase is for NoSQL databases.

    • Gashole711@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      It is a NoSQL database but the SQL syntax is ANSI SQL compliant. If you moved the queues to Couchbase and let it handle the replication and consistency, you wouldn’t have to code for it.

      • King@vlemmy.net
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        Thanks for checking.
        We don’t need full database replication. We are just replicating activities that other servers are subscribed to. So the Comments table, only some of the rows will be “replicated”. Not sure if/how CouchBase handles this.

        • Gashole711@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Normally with XDCR you can specify which documents to replicate out of a bucket. It doesn’t have to be the entire bucket. So if you had certain types (comments, upvotes, etc) then only those would sync when the target comes online.

          I did check into Apache CouchDB, the open source upstream, and replication is there. We use Enterprise Couchbase at work and it’s a dream but there are some tools that I use that only use Apache CouchDB (Inkdrop for example). It’s worth looking into.

    • Gashole711@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Actually XDCR is not available in the community edition so that option is gone. Sorry about that.