sas io shredding

fio.txt

root@localhost:~# fio --rw=randwrite --name=TP-write --bs=256k --direct=1 --filename=thiz --numjobs=4 --ioengine=libaio --iodepth=256 --refill_buffers --group_reporting --runtime=60 --time_based  --size=1G
TP-write: (g=0): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=584MiB/s][w=2334 IOPS][eta 00m:00s]
TP-write: (groupid=0, jobs=4): err= 0: pid=2406: Fri Jun 21 01:29:48 2024
  write: IOPS=2623, BW=656MiB/s (688MB/s)(38.6GiB/60294msec); 0 zone resets
    slat (usec): min=12, max=401142, avg=198.77, stdev=6193.70
    clat (msec): min=20, max=1284, avg=389.84, stdev=96.90
     lat (msec): min=20, max=1284, avg=390.04, stdev=96.86
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[  251], 10.00th=[  330], 20.00th=[  376],
     | 30.00th=[  380], 40.00th=[  384], 50.00th=[  388], 60.00th=[  388],
     | 70.00th=[  397], 80.00th=[  401], 90.00th=[  430], 95.00th=[  575],
     | 99.00th=[  743], 99.50th=[  785], 99.90th=[  877], 99.95th=[  902],
     | 99.99th=[ 1133]
   bw (  KiB/s): min=332288, max=1017886, per=99.84%, avg=670535.88, stdev=32340.09, samples=480
   iops        : min= 1298, max= 3976, avg=2619.22, stdev=126.34, samples=480
  lat (msec)   : 50=0.42%, 100=1.46%, 250=3.13%, 500=87.76%, 750=6.33%
  lat (msec)   : 1000=0.88%, 2000=0.02%
  cpu          : usr=4.01%, sys=1.76%, ctx=140812, majf=0, minf=51
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,158185,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=656MiB/s (688MB/s), 656MiB/s-656MiB/s (688MB/s-688MB/s), io=38.6GiB (41.5GB), run=60294-60294msec

Disk stats (read/write):
  vda: ios=0/157718, merge=0/49, ticks=0/57900848, in_queue=57901726, util=99.89%

sar.txt

01:29:19          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
01:29:24       606.60      0.00  73602.40      0.00    121.34      0.16      0.26      8.74 nvme0n1
01:29:24         5.40      0.00     33.60      0.00      6.22      0.00      0.33      0.14 sda
01:29:24     13662.80      0.40 109200.00      0.00      7.99     23.95      1.75    100.00 sdb

01:29:24          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
01:29:29       607.40      0.00  73650.40      0.00    121.26      0.17      0.29      9.60 nvme0n1
01:29:29         2.40      0.00     23.20      0.00      9.67      0.00      0.33      0.14 sda
01:29:29     13805.00      0.00 110279.20      0.00      7.99     23.73      1.72    100.00 sdb
^C

Average:          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
Average:       159.20    792.00  16971.77      0.00    111.58      0.04      0.26      2.54 nvme0n1
Average:         3.28      0.00     27.57      0.00      8.42      0.00      0.40      0.14 sda
Average:      3500.71    812.33  27133.02      0.00      7.98      6.08      1.74     25.45 sdb
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/sdb/queue/nr_
nr_requests  nr_zones
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/sdb/queue/nr_
nr_requests  nr_zones
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/sdb/queue/nr_requests
256
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/nvme0n1/queue/nr_requests
256

same queue size
max size better @ SAS

(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/sdb/queue/max_hw_sectors_kb
512
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/nvme0n1/queue/max_hw_sectors_kb
128

One difference at optimal IO size

(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/nvme0n1/queue/optimal_io_size
0
(M)node090fc:/var/lib/one/datastores/1# cat /sys/block/sdb/queue/optimal_io_size
8192

enable write cache using sdparm --set=WCE --save
checked that it got enabled.

performance / distribution / behaviour? stable/unchanged ;p

01:39:28          tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util DEV
01:39:33       611.20      0.00  73592.80      0.00    120.41      0.15      0.24      8.00 nvme0n1
01:39:33         2.60      0.00     14.40      0.00      5.54      0.00      0.23      0.14 sda
01:39:33     13266.20      0.00 106094.40      0.00      8.00     24.15      1.82    100.00 sdb