Now that NexentaStor is fully configured. Time to do some benchmarks. From eperience random IO was always wicked fast but I found that sequential performance was really dependent on the CPU speed in terms of CPU frequency. In my previous builds typical speeds were about 300MB/s writes and 500MB/s reads. Let’s see if this new server can beat that.
All drives are 146GB 15K SAS Drives.
During these tests Deduplication and Compression are turned off in order to only factor the disks in the equation.
Config: 3 Groups, 2 Drive Mirrors = 6 Drives on SAS6IR. Sync On.
Using DD to generate a sequential write in 16k blocks.
> dd if=/dev/zero of=ddfile1 bs=16k count=3M
^C62850+0 records in
62850+0 records out
1029734400 bytes (1.0 GB) copied, 100.685 seconds, 10.2 MB/s
Ended up aborting this via Ctrl-C as at 10MB/s the benchmark would take a LONG time. 10MB/s is a bit slow. In fact, that’s helluva slow. 10MB writes are actually quite pathetic.
Let’s try to step things up a bit and include all the drives attached.
Config: 10 Groups, 2 Drive Mirros = 20 Drives on SAS6IR and H200 Combined.
> dd if=/dev/zero of=ddfile1 bs=16k count=3M
^C79104+0 records in
79104+0 records out
1296039936 bytes (1.3 GB) copied, 72.4045 seconds, 17.9 MB/s
Wow 18MB/s. I got USB drives that write faster than this.
Ugh. Without a SLOG (or ZIL), performance is going to be crap. NexentaStor is basically waiting on each drive to confirm commitment to disk before attempting to write the next block. Great for data integrity, terrible for performance.
Let’s disable Sync and try again. This time again with only 6 drives.
> dd if=/dev/zero of=ddfile1 bs=16k count=3M
^C1185778+0 records in
1185778+0 records out
19427786752 bytes (19 GB) copied, 63.966 seconds, 304 MB/s
That’s what I’m talking about! Now let’s try it with all the drives.
> dd if=/dev/zero of=ddfile1 bs=16k count=3M
^C1713095+0 records in
1713094+0 records out
28067332096 bytes (28 GB) copied, 68.8867 seconds, 407 MB/s
Success!
Now, a little warning about disabling Sync. Unless the server is on permanent power (UPS, backup genny), disabling is Sync is downright dangerous for your data. A sudden power failure guarantees loss or corruption of uncommitted data. In my case, with my custom built UPS system, I have guaranteed 4 hours of uptime which is plenty of time to commit any writes to disk and gracefully shutdown all servers.
A quick read test
> dd if=ddfile1 of=/dev/null bs=16k
1713094+0 records in
1713094+0 records out
28067332096 bytes (28 GB) copied, 25.3297 seconds, 1.1 GB/s
Whoa! In this case NexentaStor uses BOTH drives in a mirror for reads. Results in INSANE read speeds.
Update: I realized during the benchmarks that one drive was being utilized 90% while others were hovering around 60%. Replacing the drive and resilvering the array yielded absolutely unbelievable numbers.
> dd if=/dev/zero of=ddfile1 bs=16k count=3M
3145728+0 records in
3145728+0 records out
51539607552 bytes (52 GB) copied, 85.7514 seconds, 601 MB/s
And a quick read test.
> dd if=ddfile1 of=/dev/null bs=16k
3145728+0 records in
3145728+0 records out
51539607552 bytes (52 GB) copied, 46.07 seconds, 1.1 GB/s
Am I impressed? Hell yeah. One thing is for certain. the NICs are definitely going to be the bottleneck. Even with 4 way Active/Active MPIO. I’m going to have to start thinking about moving to 10Gb network next year. What good is all that speed if I’m bottlenecked at the NIC.