I can’t say for sure- but, there is a good chance I might have a problem.

The main picture attached to this post, is a pair of dual bifurcation cards, each with a pair of Samsung PM963 1T enterprise NVMes.

It is going into my r730XD. Which… is getting pretty full. This will fill up the last empty PCIe slots.

But, knock on wood, My r730XD supports bifurcation! LOTS of Bifurcation.

As a result, it now has more HDDs, and NVMes then I can count.

What’s the problem you ask? Well. That is just one of the many servers I have laying around here, all completely filled with NVMe and SATA SSDs…

Figured I would share. Seeing a bunch of SSDs is always a pretty sight.

And- as of two hours ago, my particular lemmy instance was migrated to these new NVMes completely transparently too.

  • Decronym@lemmy.decronym.xyzB
    link
    fedilink
    English
    arrow-up
    94
    arrow-down
    1
    ·
    edit-2
    1 year ago

    Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

    Fewer Letters More Letters
    NVMe Non-Volatile Memory Express interface for mass storage
    PCIe Peripheral Component Interconnect Express
    SATA Serial AT Attachment interface for mass storage
    SSD Solid State Drive mass storage

    4 acronyms in this thread; the most compressed thread commented on today has 3 acronyms.

    [Thread #13 for this sub, first seen 8th Aug 2023, 21:55] [FAQ] [Full list] [Contact] [Source code]

    • brygphilomena@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      1 year ago

      I think I’m at 7x 18tb drives. I’m slowly replacing all the smaller 8tb disks in my server. Only 5 more to go. After that it’s a new server with more bays and/or a jbod shelf.

      • iesou@lemm.ee
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        That’s my next step. I have 8 8tb drives I need to start swapping, 2x512 NVMEs for system/app cache, and 1 2tb NVME for media cache.

  • Millie@lemm.ee
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 year ago

    I dream of this kind of storage. I just added a second m.2 with a couple of TB on it and the space is lovely but I can already see I’ll fill it sooner than I’d like.

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      6
      ·
      1 year ago

      I will say, it’s nice not having to nickel and dime my storage.

      But, the way I have things configured, redundancy takes up a huge chunk of the overall storage.

      I have around 10x 1T NVMe and SATA SSDs in a ceph cluster. 60% storage overhead there.

      Four of those 8T disks are in a ZFS Striped Mirror / Raid 10. 50% storage overhead.

      The 4x 970 evo / evo plus drives are also in a striped mirror ZFS pool. 50% overhead.

      But, still PLENTY of usable storage, and- highly available at that!

        • HTTP_404_NotFound@lemmyonline.comOP
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 year ago

          The two ZFS pools are only 4 devices. One pool is spinning rust, the other is all NVMe.

          I don’t use raid 5 for large disks, and instead go for raid6/z2. Given z2 and striped mirrors both have 50% overhead with only 4 disks- striped mirrors has the advantage of being much faster, double the IOPs, and faster rebuilds. For these particular pools, performance was more important than overall disk space.

          However, before all of these disks were moved from TrueNAS to Unraid- there was a 8x8T Z2 pool, which worked exceptionally well.

  • Heggico@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    I’m confused. Why do those cards have a heatsink? I needed a card like that because my motherboard did not support bifurcation. So had to use a splitting card. The cards I know that require bifurcation do not even need a controller or heatsink. They are just wired pretty much directly to the pci-e bus.

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      I actually looked up the chip numbers, and its a “splitter”.

      I, don’t know WHY there is a splitter, as a splitter isn’t needed, and these cards are advertised to only work on motherboards supporting bifurcation. However, there is indeed, a splitter.

      The documentation is also, REALLY horribly translated.

      Note: Without pcie splitter function in this host adapter (ASM1182E chip), so motherboard must support PCIe Bifurcation. Otherwise, only one M.2 PCIe SSD will be recognized. If you are not sure PCIe Bifurcation of your motherboard, please consult motherboard munufacture or contact us via amazon message

      Here is the documentation for the chip itself: https://www.asmedia.com.tw/product/213yQcasx8gNAzS4/b7FyQBCxz2URbzg0

      I, am not 100% certain how, where, or why it fits in there. Perhaps, its for link power management? Or something.

      But, I can confirm, these cards DO require bifurcation to be enabled. Without bifurcation, you only see the first drive.

      • Heggico@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Wierd, but thanks for the info. The card i’ve used before pretty much only has 2 slots and a few random components on it l, like capacitors and such. So I assumed it was never needed. My motherboard didn’t support bifurcation, so I never got that to work though, so maybe it couldn’t work at all… only found that out after installing it.

  • joel@aussie.zone
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    Love this. Apart from hosting an instance, what are you using it for? Self-cloud?

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 year ago

      I host a few handfuls of websites, some discord bots.

      I hoard Linux isos. I use it for general purpose learning and experimentation.

      There is also kubernetes running, source control, and a bit of everything else.

        • HTTP_404_NotFound@lemmyonline.comOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Backups backups backups.

          Anything you don’t want to lose, follow the 3.2.1. rule.

          Snapshots / Raid are not backups.

          Also, unraid is fantastic for handling bulk media. ZFS is fantastic for keeping things safe. (and fast).

          And ceph is great for squeezing 20k IOPs out of 6 million IOPs worth of enterprise SSDs!

  • webuge@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    Well this seems to be a good problem to have hahah. If you need to get rid of some of those ssds count with me.

  • krolden@lemmy.ml
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    Having a large flash pool really makes your life so much better.

    Until you fill up all your space and have to buy more :p

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Hopefully that doesn’t happen soon! I don’t have too much room for more flash, lol.

      But, I have quite a bit of available space, so, there shouldn’t be any concerns. Also- tomorrow, after a few adapters arrives, I’ll be adding another 2x 1T flash drives my Optiplex 5060 SFF.

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      6
      ·
      edit-2
      1 year ago

      Whats the problem?

      Each NVMe uses 4 lanes. For each of these x8 slots, they have two NVMes, for a total of 8 lanes.

      The x16 slot already has 4x NVMe in it, lol. The other x16 slot has a GPU, which is located in that particular slot due to the lovely 3d-printed fan shroud.

      One of the other full-height x8 slots also has a PLX switch, and is loaded with 4 more NVMes.

      • feitingen@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Does the plx introduce noticeable latency, and does it get hot?

        I want to get a few, but I don’t really have the airflow you do, so I’m a bit worried.

        • HTTP_404_NotFound@lemmyonline.comOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          I have not noticed any issues with it.

          And- prior to Jan of this year, I used two of them in an r720xd because it didn’t support bifurcation. And- can’t say I ran into any issues.

          I also, have not checked to see if it was hot either though.

  • Vake@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Wondering what software you’re running to have all the storage managed and then your containers and things on top? Is it all on the 730XD?

  • alligatorSoup
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Ive been interested in this, but i havnt figured out if my motherboard supports bifurcation. I have an AORUS B450 i mini itx, so just the 1 PCIe slot (gree as i use an APU 4650G ). Is bifurcation more of a premium feature for more expensive boards?

    • HTTP_404_NotFound@lemmyonline.comOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      I believe it’s much more common places these days.

      Although, of the… 10 or so computers I have laying around my house, only two of them supports bifurcation… although, most of my computers are a decade old.