SAS vs. SSD, Сравнительные тесты дисковых подсистем.

Ок, напиши ОС и конфиг машины сюда или в ЛС.

Ну как бы да - идиоты. HP дала им батарейку - используй батарейку, а они надеятся, что и без батарейки будет толк.

Очень многие считают кеширование чтения важными, а гораздо чаще типичный "корпоративный" диск, то есть, не бешено раздающий фильмы, а хранящий офисные данные, занимается в основном тем, что фиксирует изменения рабочего набора данных. Все, что нужно прочитать уже давно прочитано и сидит в кеше ОЗУ.

В такой специфике даже один SSD сравнительно даже неплохо выглядит. Кстати, в некоторых "батарейка" тоже есть и она позволяет при отключении питания все-таки записать содержимое кеша.

И по-моему с вами мы уже это обсуждали.

Небольшой тест на неравномерно нагруженной системе

FreeBSD *.azar-a.net 9.2-RC4 FreeBSD 9.2-RC4 #0 r+750678a-dirty: Thu Sep 26 15:19:27 MSK 2013 root@*.azar-a.net:/usr/obj/usr/src/sys/RACT amd64

ZFS zpool v28-ext zfs v5

[global]
bs=4k
size=256M
filename=test.file
direct=1
buffered=0
ioengine=posixaio
iodepth=16
thread

[seq-read]
rw=read
stonewall
name=Sequential reads

[rand-read]
rw=randread
stonewall
name=Random reads

[seq-write]
rw=write
stonewall
name=Sequential writes

[rand-write]
rw=randwrite
stonewall
name=Random writes

Результаты

Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
fio-2.1.1
Starting 4 threads
Jobs: 1 (f=1)
Sequential reads: (groupid=0, jobs=1): err= 0: pid=102596: Thu Oct 17 18:28:01 2013
  read : io=262144KB, bw=1662.4MB/s, iops=425558, runt=   154msec
    slat (usec): min=0, max=10, avg= 1.44, stdev= 0.57
    clat (usec): min=9, max=109, avg=24.43, stdev= 9.44
     lat (usec): min=10, max=111, avg=25.87, stdev= 9.42
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   13], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   21], 50.00th=[   23], 60.00th=[   25],
     | 70.00th=[   28], 80.00th=[   32], 90.00th=[   39], 95.00th=[   41],
     | 99.00th=[   50], 99.50th=[   52], 99.90th=[   59], 99.95th=[   61],
     | 99.99th=[   77]
    lat (usec) : 10=0.08%, 20=34.91%, 50=63.98%, 100=1.03%, 250=0.01%
  cpu          : usr=14.38%, sys=84.31%, ctx=189, majf=0, minf=0
  IO depths    : 1=0.1%, 2=6.3%, 4=28.7%, 8=57.7%, 16=7.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.3%, 8=0.1%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
Random reads: (groupid=1, jobs=1): err= 0: pid=101018: Thu Oct 17 18:28:01 2013
  read : io=262144KB, bw=1560.1MB/s, iops=399609, runt=   164msec
    slat (usec): min=0, max=39, avg= 1.43, stdev= 0.59
    clat (usec): min=9, max=111, avg=26.61, stdev=11.19
     lat (usec): min=10, max=112, avg=28.04, stdev=11.18
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   14], 20.00th=[   17],
     | 30.00th=[   19], 40.00th=[   22], 50.00th=[   25], 60.00th=[   27],
     | 70.00th=[   30], 80.00th=[   37], 90.00th=[   43], 95.00th=[   47],
     | 99.00th=[   59], 99.50th=[   66], 99.90th=[   82], 99.95th=[   86],
     | 99.99th=[   98]
    lat (usec) : 10=0.02%, 20=30.17%, 50=66.63%, 100=3.16%, 250=0.01%
  cpu          : usr=19.63%, sys=79.75%, ctx=233, majf=0, minf=0
  IO depths    : 1=0.1%, 2=3.5%, 4=28.3%, 8=60.6%, 16=7.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.0%, 8=0.0%, 16=7.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=65536/w=0/d=0, short=r=0/w=0/d=0
Sequential writes: (groupid=2, jobs=1): err= 0: pid=100883: Thu Oct 17 18:28:01 2013
  write: io=262144KB, bw=1213.3MB/s, iops=310597, runt=   211msec
    slat (usec): min=0, max=24, avg= 1.97, stdev= 1.15
    clat (usec): min=12, max=73, avg=36.24, stdev=11.75
     lat (usec): min=15, max=75, avg=38.21, stdev=11.77
    clat percentiles (usec):
     |  1.00th=[   16],  5.00th=[   20], 10.00th=[   23], 20.00th=[   26],
     | 30.00th=[   29], 40.00th=[   31], 50.00th=[   34], 60.00th=[   36],
     | 70.00th=[   40], 80.00th=[   50], 90.00th=[   55], 95.00th=[   57],
     | 99.00th=[   61], 99.50th=[   63], 99.90th=[   66], 99.95th=[   66],
     | 99.99th=[   69]
    lat (usec) : 20=3.94%, 50=75.29%, 100=20.77%
  cpu          : usr=36.67%, sys=59.05%, ctx=5050, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.5%, 4=27.3%, 8=63.3%, 16=7.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=92.7%, 8=0.0%, 16=7.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
Random writes: (groupid=3, jobs=1): err= 0: pid=102595: Thu Oct 17 18:28:01 2013
  write: io=262144KB, bw=1024.0MB/s, iops=262144, runt=   250msec
    slat (usec): min=0, max=334, avg= 2.21, stdev= 5.13
    clat (usec): min=9, max=2146, avg=43.55, stdev=49.80
     lat (usec): min=13, max=2149, avg=45.77, stdev=50.71
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   20], 10.00th=[   22], 20.00th=[   26],
     | 30.00th=[   29], 40.00th=[   32], 50.00th=[   34], 60.00th=[   37],
     | 70.00th=[   45], 80.00th=[   52], 90.00th=[   57], 95.00th=[   65],
     | 99.00th=[  262], 99.50th=[  358], 99.90th=[  588], 99.95th=[  740],
     | 99.99th=[ 1512]
    lat (usec) : 10=0.01%, 20=4.33%, 50=70.12%, 100=22.33%, 250=2.09%
    lat (usec) : 500=0.96%, 750=0.13%, 1000=0.02%
    lat (msec) : 2=0.03%, 4=0.01%
  cpu          : usr=21.69%, sys=65.46%, ctx=3807, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.1%, 4=26.5%, 8=64.2%, 16=8.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=92.6%, 8=0.1%, 16=7.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=65536/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=262144KB, aggrb=1662.4MB/s, minb=1662.4MB/s, maxb=1662.4MB/s, mint=154msec, maxt=154msec

Run status group 1 (all jobs):
   READ: io=262144KB, aggrb=1560.1MB/s, minb=1560.1MB/s, maxb=1560.1MB/s, mint=164msec, maxt=164msec

Run status group 2 (all jobs):
  WRITE: io=262144KB, aggrb=1213.3MB/s, minb=1213.3MB/s, maxb=1213.3MB/s, mint=211msec, maxt=211msec

Run status group 3 (all jobs):
  WRITE: io=262144KB, aggrb=1024.0MB/s, minb=1024.0MB/s, maxb=1024.0MB/s, mint=250msec, maxt=250msec

Небольшой тест на неравномерно нагруженной системе

Оказывается:

pool: fd

state: DEGRADED

status: One or more devices has been removed by the administrator.

Sufficient replicas exist for the pool to continue functioning in a

degraded state.

action: Online the device using 'zpool online' or replace the device with

'zpool replace'.

scan: none requested

config:

NAME STATE READ WRITE CKSUM

fd DEGRADED 0 0 0

mirror-0 DEGRADED 0 0 0

gpt/disk0 ONLINE 0 0 0

1934153058 REMOVED 0 0 0 was /dev/gpt/disk1

errors: No known data errors

zpool DEGRADED

диск только один, второй отвалился

Но давайте чуть увеличим до 1536 мегабайт

Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
fio-2.1.1
Starting 4 threads
Sequential reads: Laying out IO file(s) (1 file(s) / 1536MB)
Jobs: 1 (f=1): [___w] [99.4% done] [0KB/14585KB/0KB /s] [0/3646/0 iops] [eta 00m:03s]  
Sequential reads: (groupid=0, jobs=1): err= 0: pid=101006: Thu Oct 17 18:47:19 2013
  read : io=1536.0MB, bw=1761.5MB/s, iops=450935, runt=   872msec
    slat (usec): min=0, max=439, avg= 1.38, stdev= 1.02
    clat (usec): min=8, max=701, avg=23.11, stdev= 9.59
     lat (usec): min=9, max=702, avg=24.49, stdev= 9.61
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   13], 20.00th=[   15],
     | 30.00th=[   17], 40.00th=[   20], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   38], 95.00th=[   40],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   56], 99.95th=[   62],
     | 99.99th=[  145]
    bw (MB  /s): min= 1758, max= 1758, per=99.85%, avg=1758.91, stdev= 0.00
    lat (usec) : 10=0.24%, 20=39.35%, 50=60.12%, 100=0.27%, 250=0.02%
    lat (usec) : 500=0.01%, 750=0.01%
  cpu          : usr=16.07%, sys=83.58%, ctx=475, majf=0, minf=0
  IO depths    : 1=0.1%, 2=7.1%, 4=28.5%, 8=57.2%, 16=7.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.3%, 8=0.1%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Random reads: (groupid=1, jobs=1): err= 0: pid=101032: Thu Oct 17 18:47:19 2013
  read : io=1536.0MB, bw=634731KB/s, iops=158682, runt=  2478msec
    slat (usec): min=0, max=7426, avg= 4.51, stdev=15.39
    clat (usec): min=8, max=11078, avg=57.39, stdev=97.06
     lat (usec): min=11, max=11080, avg=61.90, stdev=100.13
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   18], 20.00th=[   23],
     | 30.00th=[   27], 40.00th=[   32], 50.00th=[   38], 60.00th=[   45],
     | 70.00th=[   55], 80.00th=[   72], 90.00th=[  115], 95.00th=[  179],
     | 99.00th=[  294], 99.50th=[  334], 99.90th=[  426], 99.95th=[  516],
     | 99.99th=[ 3888]
    bw (KB  /s): min=254016, max=1024184, per=87.32%, avg=554270.00, stdev=349118.42
    lat (usec) : 10=0.02%, 20=13.30%, 50=51.40%, 100=22.80%, 250=10.44%
    lat (usec) : 500=2.00%, 750=0.02%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=26.36%, sys=36.21%, ctx=106755, majf=0, minf=0
  IO depths    : 1=0.9%, 2=5.9%, 4=27.6%, 8=58.3%, 16=7.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.2%, 8=0.1%, 16=6.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Sequential writes: (groupid=2, jobs=1): err= 0: pid=101422: Thu Oct 17 18:47:19 2013
  write: io=1536.0MB, bw=253402KB/s, iops=63350, runt=  6207msec
    slat (usec): min=0, max=12557, avg= 2.44, stdev=30.18
    clat (usec): min=8, max=192112, avg=233.52, stdev=1854.71
     lat (usec): min=9, max=192117, avg=235.96, stdev=1855.05
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   21], 10.00th=[   23], 20.00th=[   27],
     | 30.00th=[   30], 40.00th=[   33], 50.00th=[   37], 60.00th=[   42],
     | 70.00th=[   50], 80.00th=[   56], 90.00th=[   68], 95.00th=[  996],
     | 99.00th=[ 3984], 99.50th=[ 5984], 99.90th=[11968], 99.95th=[15040],
     | 99.99th=[30080]
    bw (KB  /s): min=23256, max=505528, per=93.47%, avg=236857.92, stdev=169635.96
    lat (usec) : 10=0.01%, 20=3.15%, 50=66.19%, 100=22.66%, 250=0.74%
    lat (usec) : 500=0.15%, 750=0.01%, 1000=2.42%
    lat (msec) : 2=2.13%, 4=1.61%, 10=0.79%, 20=0.12%, 50=0.01%
    lat (msec) : 250=0.01%
  cpu          : usr=7.40%, sys=15.45%, ctx=44731, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.4%, 4=24.6%, 8=64.6%, 16=9.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=92.4%, 8=0.9%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
Random writes: (groupid=3, jobs=1): err= 0: pid=102612: Thu Oct 17 18:47:19 2013
  write: io=1536.0MB, bw=3241.4KB/s, iops=810, runt=485244msec
    slat (usec): min=0, max=4157, avg= 2.04, stdev=11.65
    clat (usec): min=9, max=1633.2K, avg=19722.59, stdev=136467.97
     lat (usec): min=10, max=1633.3K, avg=19724.63, stdev=136468.03
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   41], 10.00th=[   52], 20.00th=[   97],
     | 30.00th=[  159], 40.00th=[  203], 50.00th=[  255], 60.00th=[  330],
     | 70.00th=[  466], 80.00th=[  980], 90.00th=[ 2960], 95.00th=[ 6560],
     | 99.00th=[970752], 99.50th=[1073152], 99.90th=[1253376], 99.95th=[1335296],
     | 99.99th=[1564672]
    bw (KB  /s): min= 1544, max=16235, per=100.00%, avg=3291.87, stdev=1092.16
    lat (usec) : 10=0.01%, 20=0.11%, 50=8.93%, 100=11.27%, 250=28.78%
    lat (usec) : 500=22.39%, 750=5.51%, 1000=4.68%
    lat (msec) : 2=6.06%, 4=5.08%, 10=3.51%, 20=1.24%, 50=0.45%
    lat (msec) : 100=0.08%, 250=0.01%, 500=0.01%, 750=0.11%, 1000=0.95%
    lat (msec) : 2000=0.83%
  cpu          : usr=0.18%, sys=0.25%, ctx=38177, majf=0, minf=0
  IO depths    : 1=1.6%, 2=5.8%, 4=16.0%, 8=58.0%, 16=18.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.4%, 8=1.8%, 16=3.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=393216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1536.0MB, aggrb=1761.5MB/s, minb=1761.5MB/s, maxb=1761.5MB/s, mint=872msec, maxt=872msec

Run status group 1 (all jobs):
   READ: io=1536.0MB, aggrb=634731KB/s, minb=634731KB/s, maxb=634731KB/s, mint=2478msec, maxt=2478msec

Run status group 2 (all jobs):
  WRITE: io=1536.0MB, aggrb=253401KB/s, minb=253401KB/s, maxb=253401KB/s, mint=6207msec, maxt=6207msec

Run status group 3 (all jobs):
  WRITE: io=1536.0MB, aggrb=3241KB/s, minb=3241KB/s, maxb=3241KB/s, mint=485244msec, maxt=485244msec

ENELIS, а диск какой?

Kingston SSD 120GB V300

Да и файловая система zfs себя чуть по другому ведет.

3x SAS | FreeBSD

2x SAS | Linux

Это на линуксе был zfs или в BSD?

Я ARC обошел выбрав файл 1536 мегабайт, т.к. в ARC уже не было места и памяти тоже не было.

И система нагружена mysqlом и ядреными perl скриптами с 100-200 запросов в страничке.

Поправим пул позже посмотрим еще раз, т.к. в DEGRADED падает производительность заметно на запись и чтение.

---------- Добавлено 17.10.2013 в 17:54 ----------

Вот тест такого же пула (идентичная копия), только сейчас он забит чуть менее чем полностью, сервер в свапе на 1 процент и под постоянной нагрузкой la 8 на 8 тредовом сервере.

Вот его стандартная нагрузка

last pid: 39740;  load averages:  7.59,  8.82,  7.35                                                                                                                                                                                                 up 20+02:57:08  19:53:17
51 processes:  4 running, 46 sleeping, 1 zombie
CPU: 37.5% user,  0.0% nice, 11.6% system,  0.4% interrupt, 50.5% idle
Mem: 219M Active, 1072M Inact, 5536M Wired, 194M Cache, 853M Free
ARC: 3702M Total, 1945M MFU, 1177M MRU, 10M Anon, 90M Header, 480M Other
Swap: 8192M Total, 143M Used, 8049M Free, 1% Inuse

  pool: fd
 state: ONLINE
  scan: none requested
config:

	NAME                                            STATE     READ WRITE CKSUM
	fd                                             ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/58b4bfaf-141e-11e3-8b5a-00259097e6e2  ONLINE       0     0     0
	    gptid/557f3637-141e-11e3-8b5a-00259097e6e2  ONLINE       0     0     0

errors: No known data errors

Во время теста нагрузка выросла в 2 раза

Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16
fio-2.1.1
Starting 4 threads
Sequential reads: Laying out IO file(s) (1 file(s) / 1536MB)
Jobs: 1 (f=1): [___w] [100.0% done] [0KB/40414KB/0KB /s] [0/10.2K/0 iops] [eta 00m:00s]
Sequential reads: (groupid=0, jobs=1): err= 0: pid=102785: Thu Oct 17 19:49:50 2013
  read : io=1536.0MB, bw=131941KB/s, iops=32985, runt= 11921msec
    slat (usec): min=0, max=181386, avg=17.78, stdev=770.91
    clat (usec): min=5, max=870244, avg=316.33, stdev=7370.62
     lat (usec): min=11, max=870254, avg=334.11, stdev=7411.51
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   13], 10.00th=[   20], 20.00th=[   29],
     | 30.00th=[   38], 40.00th=[   51], 50.00th=[   66], 60.00th=[   84],
     | 70.00th=[  104], 80.00th=[  126], 90.00th=[  151], 95.00th=[  169],
     | 99.00th=[ 1880], 99.50th=[ 6560], 99.90th=[43776], 99.95th=[82432],
     | 99.99th=[317440]
    bw (KB  /s): min=10017, max=336712, per=97.62%, avg=128799.90, stdev=81324.43
    lat (usec) : 10=0.49%, 20=9.32%, 50=29.15%, 100=28.84%, 250=30.07%
    lat (usec) : 500=0.61%, 750=0.26%, 1000=0.09%
    lat (msec) : 2=0.19%, 4=0.33%, 10=0.29%, 20=0.15%, 50=0.13%
    lat (msec) : 100=0.04%, 250=0.03%, 500=0.01%, 1000=0.01%
  cpu          : usr=5.81%, sys=9.51%, ctx=313519, majf=1, minf=0
  IO depths    : 1=4.3%, 2=10.8%, 4=24.7%, 8=53.4%, 16=6.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.7%, 8=0.3%, 16=6.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Random reads: (groupid=1, jobs=1): err= 0: pid=101044: Thu Oct 17 19:49:50 2013
  read : io=1536.0MB, bw=178572KB/s, iops=44643, runt=  8808msec
    slat (usec): min=0, max=56156, avg=10.10, stdev=150.53
    clat (usec): min=4, max=318976, avg=272.95, stdev=1893.59
     lat (usec): min=7, max=318978, avg=283.05, stdev=1901.16
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[   11], 10.00th=[   14], 20.00th=[   23],
     | 30.00th=[   35], 40.00th=[   49], 50.00th=[   67], 60.00th=[   88],
     | 70.00th=[  114], 80.00th=[  143], 90.00th=[  181], 95.00th=[  231],
     | 99.00th=[ 5024], 99.50th=[ 8096], 99.90th=[21632], 99.95th=[33024],
     | 99.99th=[60160]
    bw (KB  /s): min=20024, max=421488, per=100.00%, avg=178736.35, stdev=136780.57
    lat (usec) : 10=2.57%, 20=12.55%, 50=24.99%, 100=24.47%, 250=30.82%
    lat (usec) : 500=1.03%, 750=0.28%, 1000=0.13%
    lat (msec) : 2=0.28%, 4=1.32%, 10=1.19%, 20=0.26%, 50=0.10%
    lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
  cpu          : usr=10.04%, sys=15.41%, ctx=323551, majf=0, minf=0
  IO depths    : 1=2.5%, 2=7.5%, 4=18.7%, 8=56.0%, 16=15.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.3%, 8=1.1%, 16=4.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=393216/w=0/d=0, short=r=0/w=0/d=0
Sequential writes: (groupid=2, jobs=1): err= 0: pid=101856: Thu Oct 17 19:49:50 2013
  write: io=1536.0MB, bw=162469KB/s, iops=40617, runt=  9681msec
    slat (usec): min=0, max=23439, avg= 4.54, stdev=40.53
    clat (usec): min=6, max=1132.1K, avg=356.47, stdev=6989.74
     lat (usec): min=14, max=1132.1K, avg=361.01, stdev=6989.93
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   24], 10.00th=[   29], 20.00th=[   35],
     | 30.00th=[   43], 40.00th=[   51], 50.00th=[   61], 60.00th=[   75],
     | 70.00th=[   99], 80.00th=[  474], 90.00th=[  612], 95.00th=[  804],
     | 99.00th=[ 4512], 99.50th=[ 6624], 99.90th=[20864], 99.95th=[31616],
     | 99.99th=[70144]
    bw (KB  /s): min=30288, max=352504, per=100.00%, avg=169479.35, stdev=97958.20
    lat (usec) : 10=0.07%, 20=2.18%, 50=35.88%, 100=31.90%, 250=8.56%
    lat (usec) : 500=2.25%, 750=13.42%, 1000=2.53%
    lat (msec) : 2=1.16%, 4=0.90%, 10=0.88%, 20=0.16%, 50=0.09%
    lat (msec) : 100=0.01%, 250=0.01%, 1000=0.01%, 2000=0.01%
  cpu          : usr=5.52%, sys=12.96%, ctx=137201, majf=0, minf=0
  IO depths    : 1=0.3%, 2=4.9%, 4=19.7%, 8=62.4%, 16=12.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.3%, 8=1.6%, 16=5.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=393216/d=0, short=r=0/w=0/d=0
Random writes: (groupid=3, jobs=1): err= 0: pid=100163: Thu Oct 17 19:49:50 2013
  write: io=1536.0MB, bw=9098.4KB/s, iops=2274, runt=172874msec
    slat (usec): min=0, max=166956, avg= 6.89, stdev=510.56
    clat (usec): min=6, max=2331.7K, avg=6994.06, stdev=38807.83
     lat (usec): min=12, max=2331.7K, avg=7000.95, stdev=38815.75
    clat percentiles (usec):
     |  1.00th=[   20],  5.00th=[   31], 10.00th=[   45], 20.00th=[   85],
     | 30.00th=[  147], 40.00th=[  270], 50.00th=[  796], 60.00th=[ 2512],
     | 70.00th=[ 4128], 80.00th=[ 5792], 90.00th=[ 9920], 95.00th=[22144],
     | 99.00th=[91648], 99.50th=[183296], 99.90th=[651264], 99.95th=[765952],
     | 99.99th=[1003520]
    bw (KB  /s): min=   80, max=67072, per=100.00%, avg=9450.67, stdev=7550.18
    lat (usec) : 10=0.01%, 20=0.88%, 50=10.60%, 100=11.35%, 250=15.95%
    lat (usec) : 500=8.03%, 750=2.81%, 1000=2.87%
    lat (msec) : 2=5.13%, 4=11.46%, 10=21.06%, 20=4.35%, 50=3.45%
    lat (msec) : 100=1.15%, 250=0.56%, 500=0.21%, 750=0.09%, 1000=0.05%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=0.82%, sys=1.43%, ctx=377431, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.6%, 4=3.0%, 8=55.4%, 16=40.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.7%, 8=2.4%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=393216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1536.0MB, aggrb=131940KB/s, minb=131940KB/s, maxb=131940KB/s, mint=11921msec, maxt=11921msec

Run status group 1 (all jobs):
   READ: io=1536.0MB, aggrb=178572KB/s, minb=178572KB/s, maxb=178572KB/s, mint=8808msec, maxt=8808msec

Run status group 2 (all jobs):
  WRITE: io=1536.0MB, aggrb=162469KB/s, minb=162469KB/s, maxb=162469KB/s, mint=9681msec, maxt=9681msec

Run status group 3 (all jobs):
  WRITE: io=1536.0MB, aggrb=9098KB/s, minb=9098KB/s, maxb=9098KB/s, mint=172874msec, maxt=172874msec

Оба тестируемые пулы с dedup=on,compression=lz4,checksum=fletcher4

Поправил пост выше.

А какая BSD? Там была куча исправлений в 8 и 9ой версии.