Choosing NVMe drives for CM3588 Nas kit

[DragonFly]

The specs of the CM3588 says:

up to 4 x PCIe interfaces

2 x PCIe 2.1 x1 and 2 x PCIe 3.0 x2
or 2 x PCIe 2.1 x1 and 1 x PCIe 3.0 x4
or 1 x PCIe 2.1 x1, 1 x PCIe 3.0 x2, and 2 x PCIe 3.0 x1
or 4 x PCIe 3.0 x1

So, If I understand that correctly and I want 8TB of storage, Its better to choose:

2 x 4TB drives which both could use 2 PCIe lanes over 4 x 2TB in which each only can use 1 PCIe lane?

Has anyone meassured the difference in read and write speeds between using 1 or 2 lanes?

[fetcher]

It appears that only one PCIe 3.0 lane is wired through to each of the four NVMe m.2 slots, limiting the benefit of higher-performance drives. See the first-page block diagram of this schematic PDF for the NAS breakout board:

https://wiki.friendlyelec.com/wiki/imag ... 09_SCH.PDF

[DragonFly]

Thanks fetcher,

Got this answer from Techsupport:
"4 x 2TB is better, This way you can use the full bandwidth"

Makes any sense? (given your technical knowledge)

I realize the drives won't be able to give their full potential. Just need to get the maximum R/W speeds from what's left on those available PCIe 3.0 lanes.

[yaazzz]

Here is the result with Iozone3 as numbers are often better than spec

Code: Select all

iozone -t1 -i0 -i2 -r1k -s1g /tmp

Code: Select all

Iozone: Performance Test of File I/O
                Version $Revision: 3.489 $
                Compiled for 64 bit mode.
                Build: linux 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa,
                     Alexey Skidanov, Sudhir Kumar.

        Run began: Sun Feb  4 18:12:05 2024

        Record Size 1 kB
        File size set to 1048576 kB
        Command line used: iozone -t1 -i0 -i2 -r1k -s1g /tmp
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 1 process
        Each process writes a 1048576 kByte file in 1 kByte records

        Children see throughput for  1 initial writers  =  478763.97 kB/sec
        Parent sees throughput for  1 initial writers   =  198851.92 kB/sec
        Min throughput per process                      =  478763.97 kB/sec 
        Max throughput per process                      =  478763.97 kB/sec
        Avg throughput per process                      =  478763.97 kB/sec
        Min xfer                                        = 1048576.00 kB

        Children see throughput for  1 rewriters        =  147268.70 kB/sec
        Parent sees throughput for  1 rewriters         =  101467.37 kB/sec
        Min throughput per process                      =  147268.70 kB/sec 
        Max throughput per process                      =  147268.70 kB/sec
        Avg throughput per process                      =  147268.70 kB/sec
        Min xfer                                        = 1048576.00 kB

        Children see throughput for 1 random readers    =  446310.19 kB/sec
        Parent sees throughput for 1 random readers     =  444966.13 kB/sec
        Min throughput per process                      =  446310.19 kB/sec 
        Max throughput per process                      =  446310.19 kB/sec
        Avg throughput per process                      =  446310.19 kB/sec
        Min xfer                                        = 1048576.00 kB

        Children see throughput for 1 random writers    =  108246.30 kB/sec
        Parent sees throughput for 1 random writers     =   73021.19 kB/sec
        Min throughput per process                      =  108246.30 kB/sec 
        Max throughput per process                      =  108246.30 kB/sec
        Avg throughput per process                      =  108246.30 kB/sec
        Min xfer                                        = 1048576.00 kB

[DragonFly]

Nice!

Yeah, As you mentioned before, that's about the same values (or a bit slower) as you would get from SATA SSD

No troubles copying large files over Ethernet?

[yaazzz]

I am more limited by my wireless connection for the moment. Did not add a change to make a direct 1Gb Ethernet connection to check.

[Homwin]

Forgot to make the adjustment for a direct 1Gb Ethernet connection while checking.

[seamon]

So I was considering the CM3588 Kit along with 4x Nextorage NVMe SSDs to make the ultimate portable NAS build.
However, the CM3588 page says that the max power draw from the AC adapter is 15W.
However, according to Nextorage page, these drives can draw 11W+ per drive.
https://www.nextorage.net/en/ssds/nem-pa/#spec

I suppose 4 drives will not work?
Does this board only work with low power drives?

[Apokalypz]

... the CM3588 page says that the max power draw from the AC adapter is 15W.
However, according to Nextorage page, these drives can draw 11W+ per drive...

The CM3588 page only seems to mention the electrical characteristics of the CPU module that plugs into the IO daughter board. Basically, the power rail that powers the SoC/DRAM is limited to 15W. The IO board with the 4x NVME slots have it's own electrical characteristics. Looking at the specifications, EACH nvme slot gets its own 3.3v 3Amp regulator coming directly from the 12v barrel jack. So as long as your nvme cards don't draw more than 9.9W, or total + SOC + IO + supporting circuitry more than your power brick can handle, you should be fine.

It's also worth considering the fact that each NVME slot only gets 1 pcie lane at pcie gen3. It would be very likely that a high performance card that is forced to use the 50% slower speeds of pcie gen3 and a single lane will use less power than it would with it's native pcie generation at 4x lanes. How much less power exactly? I have no idea, but I suspect the pcie bus accounts for a significant portion of the total NVME drives power usage... Just something to think about.