Вот на ней же 1536 мегабайт, чтобы в кеши не влезало, в принципе мы видим реальную ситуацию с random writes.
Если что это RAID 10 из 4 дисков по 146гб 10k rpm.
Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16fio-2.1.1Starting 4 threadsSequential reads: Laying out IO file(s) (1 file(s) / 1536MB)Jobs: 1 (f=1): [___w] [97.3% done] [0KB/248KB/0KB /s] [0/62/0 iops] [eta 00m:23s] Sequential reads: (groupid=0, jobs=1): err= 0: pid=101280: Thu Oct 17 21:16:13 2013 read : io=1536.0MB, bw=205603KB/s, iops=51400, runt= 7650msec slat (usec): min=1, max=157162, avg=15.03, stdev=440.70 clat (usec): min=11, max=458218, avg=158.15, stdev=1662.10 lat (usec): min=21, max=458221, avg=173.18, stdev=1721.49 clat percentiles (usec): | 1.00th=[ 13], 5.00th=[ 26], 10.00th=[ 35], 20.00th=[ 54], | 30.00th=[ 72], 40.00th=[ 92], 50.00th=[ 110], 60.00th=[ 131], | 70.00th=[ 155], 80.00th=[ 183], 90.00th=[ 221], 95.00th=[ 278], | 99.00th=[ 564], 99.50th=[ 964], 99.90th=[ 4048], 99.95th=[11456], | 99.99th=[85504] bw (KB /s): min=27033, max=354312, per=99.56%, avg=204698.73, stdev=96811.42 lat (usec) : 20=4.07%, 50=13.24%, 100=26.85%, 250=49.22%, 500=5.40% lat (usec) : 750=0.55%, 1000=0.19% lat (msec) : 2=0.28%, 4=0.10%, 10=0.04%, 20=0.03%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=7.77%, sys=33.14%, ctx=273751, majf=0, minf=0 IO depths : 1=3.9%, 2=11.2%, 4=26.0%, 8=52.3%, 16=6.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.9%, 8=0.1%, 16=6.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Random reads: (groupid=1, jobs=1): err= 0: pid=101280: Thu Oct 17 21:16:13 2013 read : io=1536.0MB, bw=181666KB/s, iops=45416, runt= 8658msec slat (usec): min=2, max=1501.1K, avg=17.77, stdev=2403.34 clat (usec): min=11, max=1502.3K, avg=163.63, stdev=4196.58 lat (usec): min=23, max=1502.7K, avg=181.40, stdev=4836.62 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 30], 10.00th=[ 41], 20.00th=[ 60], | 30.00th=[ 79], 40.00th=[ 99], 50.00th=[ 119], 60.00th=[ 145], | 70.00th=[ 173], 80.00th=[ 217], 90.00th=[ 270], 95.00th=[ 334], | 99.00th=[ 540], 99.50th=[ 668], 99.90th=[ 1592], 99.95th=[ 2768], | 99.99th=[23680] bw (KB /s): min= 8765, max=341208, per=100.00%, avg=208128.40, stdev=98893.63 lat (usec) : 20=3.24%, 50=11.69%, 100=25.21%, 250=46.40%, 500=12.22% lat (usec) : 750=0.90%, 1000=0.16% lat (msec) : 2=0.11%, 4=0.05%, 10=0.02%, 50=0.01%, 100=0.01% lat (msec) : 250=0.01%, 2000=0.01% cpu : usr=10.33%, sys=29.38%, ctx=226297, majf=0, minf=0 IO depths : 1=3.5%, 2=10.4%, 4=26.3%, 8=53.1%, 16=6.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.8%, 8=0.1%, 16=6.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Sequential writes: (groupid=2, jobs=1): err= 0: pid=101280: Thu Oct 17 21:16:13 2013 write: io=1536.0MB, bw=44670KB/s, iops=11167, runt= 35211msec slat (usec): min=1, max=644645, avg=36.35, stdev=1422.03 clat (usec): min=9, max=6385.9K, avg=1127.88, stdev=44379.24 lat (usec): min=25, max=6385.9K, avg=1164.23, stdev=44409.66 clat percentiles (usec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 34], 20.00th=[ 68], | 30.00th=[ 102], 40.00th=[ 129], 50.00th=[ 161], 60.00th=[ 193], | 70.00th=[ 231], 80.00th=[ 266], 90.00th=[ 334], 95.00th=[ 908], | 99.00th=[ 5536], 99.50th=[11200], 99.90th=[156672], 99.95th=[250880], | 99.99th=[1036288] bw (KB /s): min= 36, max=216896, per=100.00%, avg=60334.73, stdev=58099.83 lat (usec) : 10=0.01%, 20=5.50%, 50=7.39%, 100=16.51%, 250=46.05% lat (usec) : 500=17.50%, 750=1.44%, 1000=1.07% lat (msec) : 2=1.78%, 4=1.36%, 10=0.86%, 20=0.19%, 50=0.13% lat (msec) : 100=0.06%, 250=0.10%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.78%, sys=5.98%, ctx=346971, majf=0, minf=0 IO depths : 1=4.5%, 2=10.7%, 4=23.6%, 8=53.7%, 16=7.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.8%, 8=0.2%, 16=6.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Random writes: (groupid=3, jobs=1): err= 0: pid=101214: Thu Oct 17 21:16:13 2013 write: io=1536.0MB, bw=2062.6KB/s, iops=515, runt=762601msec slat (usec): min=1, max=580663, avg=159.66, stdev=2470.08 clat (usec): min=6, max=4596.7K, avg=29807.89, stdev=218886.77 lat (usec): min=23, max=4596.7K, avg=29967.56, stdev=218911.52 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 70], 20.00th=[ 215], | 30.00th=[ 350], 40.00th=[ 490], 50.00th=[ 636], 60.00th=[ 812], | 70.00th=[ 1064], 80.00th=[ 1848], 90.00th=[20096], 95.00th=[104960], | 99.00th=[460800], 99.50th=[1433600], 99.90th=[3522560], 99.95th=[4046848], | 99.99th=[4423680] bw (KB /s): min= 2, max=28717, per=100.00%, avg=3474.92, stdev=6039.38 lat (usec) : 10=0.07%, 20=4.59%, 50=3.31%, 100=4.23%, 250=10.49% lat (usec) : 500=18.04%, 750=16.06%, 1000=11.35% lat (msec) : 2=12.62%, 4=3.98%, 10=3.16%, 20=2.09%, 50=2.62% lat (msec) : 100=2.20%, 250=3.16%, 500=1.09%, 750=0.24%, 1000=0.09% lat (msec) : 2000=0.19%, >=2000=0.41% cpu : usr=0.19%, sys=0.31%, ctx=361515, majf=0, minf=0 IO depths : 1=4.6%, 2=10.2%, 4=22.6%, 8=53.7%, 16=8.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.9%, 8=0.6%, 16=5.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Run status group 0 (all jobs): READ: io=1536.0MB, aggrb=205603KB/s, minb=205603KB/s, maxb=205603KB/s, mint=7650msec, maxt=7650msecRun status group 1 (all jobs): READ: io=1536.0MB, aggrb=181665KB/s, minb=181665KB/s, maxb=181665KB/s, mint=8658msec, maxt=8658msecRun status group 2 (all jobs): WRITE: io=1536.0MB, aggrb=44669KB/s, minb=44669KB/s, maxb=44669KB/s, mint=35211msec, maxt=35211msecRun status group 3 (all jobs): WRITE: io=1536.0MB, aggrb=2062KB/s, minb=2062KB/s, maxb=2062KB/s, mint=762601msec, maxt=762601msec
Тогда очень странно что write performance низкий настолько.
Вот данные на очень нагруженной системе (полностью, памяти тоже нет)
FreeBSD 8.2 zfs соответственно 15
Файл 256 мегабайт
Кеш райда 256 мегабайт...
конфиг простой zpool поверх hw raid 1
Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16fio-2.1.1Starting 4 threadsSequential reads: Laying out IO file(s) (1 file(s) / 256MB)Jobs: 1 (f=1): [___w] [62.5% done] [0KB/181.7MB/0KB /s] [0/46.5K/0 iops] [eta 00m:03s] Sequential reads: (groupid=0, jobs=1): err= 0: pid=101280: Thu Oct 17 20:54:23 2013 read : io=262144KB, bw=243628KB/s, iops=60907, runt= 1076msec slat (usec): min=1, max=487, avg=11.85, stdev= 8.21 clat (usec): min=11, max=1343, avg=132.11, stdev=83.53 lat (usec): min=22, max=1354, avg=143.96, stdev=84.57 clat percentiles (usec): | 1.00th=[ 13], 5.00th=[ 18], 10.00th=[ 31], 20.00th=[ 54], | 30.00th=[ 78], 40.00th=[ 103], 50.00th=[ 126], 60.00th=[ 149], | 70.00th=[ 173], 80.00th=[ 197], 90.00th=[ 227], 95.00th=[ 266], | 99.00th=[ 386], 99.50th=[ 446], 99.90th=[ 620], 99.95th=[ 740], | 99.99th=[ 1112] bw (KB /s): min=243080, max=247984, per=100.00%, avg=245532.00, stdev=3467.65 lat (usec) : 20=5.18%, 50=11.39%, 100=22.02%, 250=54.82%, 500=6.29% lat (usec) : 750=0.24%, 1000=0.03% lat (msec) : 2=0.02% cpu : usr=18.79%, sys=27.81%, ctx=63115, majf=0, minf=0 IO depths : 1=5.8%, 2=12.4%, 4=25.1%, 8=50.3%, 16=6.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=94.1%, 8=0.0%, 16=5.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0Random reads: (groupid=1, jobs=1): err= 0: pid=101280: Thu Oct 17 20:54:23 2013 read : io=262144KB, bw=303407KB/s, iops=75851, runt= 864msec slat (usec): min=1, max=518, avg= 9.25, stdev= 9.01 clat (usec): min=13, max=760, avg=114.40, stdev=70.25 lat (usec): min=23, max=766, avg=123.64, stdev=71.98 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 29], 10.00th=[ 39], 20.00th=[ 55], | 30.00th=[ 70], 40.00th=[ 86], 50.00th=[ 102], 60.00th=[ 117], | 70.00th=[ 137], 80.00th=[ 163], 90.00th=[ 211], 95.00th=[ 247], | 99.00th=[ 334], 99.50th=[ 386], 99.90th=[ 506], 99.95th=[ 548], | 99.99th=[ 652] bw (KB /s): min=366632, max=366632, per=100.00%, avg=366632.00, stdev= 0.00 lat (usec) : 20=2.11%, 50=14.53%, 100=31.94%, 250=46.84%, 500=4.47% lat (usec) : 750=0.11%, 1000=0.01% cpu : usr=20.86%, sys=46.81%, ctx=24847, majf=0, minf=0 IO depths : 1=2.3%, 2=10.7%, 4=26.7%, 8=53.6%, 16=6.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.7%, 8=0.1%, 16=6.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0Sequential writes: (groupid=2, jobs=1): err= 0: pid=101280: Thu Oct 17 20:54:23 2013 write: io=262144KB, bw=237020KB/s, iops=59254, runt= 1106msec slat (usec): min=1, max=908, avg=13.16, stdev=15.95 clat (usec): min=13, max=1480, avg=150.96, stdev=89.10 lat (usec): min=28, max=1484, avg=164.12, stdev=89.89 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 35], 10.00th=[ 53], 20.00th=[ 74], | 30.00th=[ 94], 40.00th=[ 114], 50.00th=[ 137], 60.00th=[ 159], | 70.00th=[ 185], 80.00th=[ 223], 90.00th=[ 274], 95.00th=[ 310], | 99.00th=[ 402], 99.50th=[ 454], 99.90th=[ 612], 99.95th=[ 732], | 99.99th=[ 1208] bw (KB /s): min=231568, max=246096, per=100.00%, avg=238832.00, stdev=10272.85 lat (usec) : 20=2.67%, 50=5.47%, 100=24.73%, 250=52.82%, 500=14.00% lat (usec) : 750=0.26%, 1000=0.03% lat (msec) : 2=0.02% cpu : usr=18.46%, sys=37.01%, ctx=28379, majf=0, minf=0 IO depths : 1=2.9%, 2=7.3%, 4=24.9%, 8=57.6%, 16=7.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.3%, 8=0.2%, 16=6.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0Random writes: (groupid=3, jobs=1): err= 0: pid=101280: Thu Oct 17 20:54:23 2013 write: io=262144KB, bw=233848KB/s, iops=58462, runt= 1121msec slat (usec): min=2, max=935, avg=13.31, stdev=15.23 clat (usec): min=13, max=1339, avg=150.84, stdev=90.10 lat (usec): min=27, max=1344, avg=164.15, stdev=91.05 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 34], 10.00th=[ 53], 20.00th=[ 73], | 30.00th=[ 93], 40.00th=[ 113], 50.00th=[ 137], 60.00th=[ 159], | 70.00th=[ 187], 80.00th=[ 227], 90.00th=[ 282], 95.00th=[ 314], | 99.00th=[ 378], 99.50th=[ 422], 99.90th=[ 588], 99.95th=[ 780], | 99.99th=[ 1304] bw (KB /s): min=231504, max=233392, per=99.40%, avg=232448.00, stdev=1335.02 lat (usec) : 20=2.96%, 50=5.53%, 100=25.32%, 250=50.78%, 500=15.20% lat (usec) : 750=0.16%, 1000=0.02% lat (msec) : 2=0.04% cpu : usr=9.38%, sys=44.29%, ctx=30635, majf=0, minf=0 IO depths : 1=3.1%, 2=7.3%, 4=24.9%, 8=57.5%, 16=7.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.3%, 8=0.2%, 16=6.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0Run status group 0 (all jobs): READ: io=262144KB, aggrb=243628KB/s, minb=243628KB/s, maxb=243628KB/s, mint=1076msec, maxt=1076msecRun status group 1 (all jobs): READ: io=262144KB, aggrb=303407KB/s, minb=303407KB/s, maxb=303407KB/s, mint=864msec, maxt=864msecRun status group 2 (all jobs): WRITE: io=262144KB, aggrb=237019KB/s, minb=237019KB/s, maxb=237019KB/s, mint=1106msec, maxt=1106msecRun status group 3 (all jobs): WRITE: io=262144KB, aggrb=233848KB/s, minb=233848KB/s, maxb=233848KB/s, mint=1121msec, maxt=1121msec
А какая BSD? Там была куча исправлений в 8 и 9ой версии.
Это на линуксе был zfs или в BSD?
Я ARC обошел выбрав файл 1536 мегабайт, т.к. в ARC уже не было места и памяти тоже не было.
И система нагружена mysqlом и ядреными perl скриптами с 100-200 запросов в страничке.
Поправим пул позже посмотрим еще раз, т.к. в DEGRADED падает производительность заметно на запись и чтение.---------- Добавлено 17.10.2013 в 17:54 ----------Вот тест такого же пула (идентичная копия), только сейчас он забит чуть менее чем полностью, сервер в свапе на 1 процент и под постоянной нагрузкой la 8 на 8 тредовом сервере.
Вот его стандартная нагрузка
last pid: 39740; load averages: 7.59, 8.82, 7.35 up 20+02:57:08 19:53:1751 processes: 4 running, 46 sleeping, 1 zombieCPU: 37.5% user, 0.0% nice, 11.6% system, 0.4% interrupt, 50.5% idleMem: 219M Active, 1072M Inact, 5536M Wired, 194M Cache, 853M FreeARC: 3702M Total, 1945M MFU, 1177M MRU, 10M Anon, 90M Header, 480M OtherSwap: 8192M Total, 143M Used, 8049M Free, 1% Inuse
pool: fd state: ONLINE scan: none requestedconfig: NAME STATE READ WRITE CKSUM fd ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/58b4bfaf-141e-11e3-8b5a-00259097e6e2 ONLINE 0 0 0 gptid/557f3637-141e-11e3-8b5a-00259097e6e2 ONLINE 0 0 0errors: No known data errors
Во время теста нагрузка выросла в 2 раза
Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16fio-2.1.1Starting 4 threadsSequential reads: Laying out IO file(s) (1 file(s) / 1536MB)Jobs: 1 (f=1): [___w] [100.0% done] [0KB/40414KB/0KB /s] [0/10.2K/0 iops] [eta 00m:00s]Sequential reads: (groupid=0, jobs=1): err= 0: pid=102785: Thu Oct 17 19:49:50 2013 read : io=1536.0MB, bw=131941KB/s, iops=32985, runt= 11921msec slat (usec): min=0, max=181386, avg=17.78, stdev=770.91 clat (usec): min=5, max=870244, avg=316.33, stdev=7370.62 lat (usec): min=11, max=870254, avg=334.11, stdev=7411.51 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 38], 40.00th=[ 51], 50.00th=[ 66], 60.00th=[ 84], | 70.00th=[ 104], 80.00th=[ 126], 90.00th=[ 151], 95.00th=[ 169], | 99.00th=[ 1880], 99.50th=[ 6560], 99.90th=[43776], 99.95th=[82432], | 99.99th=[317440] bw (KB /s): min=10017, max=336712, per=97.62%, avg=128799.90, stdev=81324.43 lat (usec) : 10=0.49%, 20=9.32%, 50=29.15%, 100=28.84%, 250=30.07% lat (usec) : 500=0.61%, 750=0.26%, 1000=0.09% lat (msec) : 2=0.19%, 4=0.33%, 10=0.29%, 20=0.15%, 50=0.13% lat (msec) : 100=0.04%, 250=0.03%, 500=0.01%, 1000=0.01% cpu : usr=5.81%, sys=9.51%, ctx=313519, majf=1, minf=0 IO depths : 1=4.3%, 2=10.8%, 4=24.7%, 8=53.4%, 16=6.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.7%, 8=0.3%, 16=6.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Random reads: (groupid=1, jobs=1): err= 0: pid=101044: Thu Oct 17 19:49:50 2013 read : io=1536.0MB, bw=178572KB/s, iops=44643, runt= 8808msec slat (usec): min=0, max=56156, avg=10.10, stdev=150.53 clat (usec): min=4, max=318976, avg=272.95, stdev=1893.59 lat (usec): min=7, max=318978, avg=283.05, stdev=1901.16 clat percentiles (usec): | 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 14], 20.00th=[ 23], | 30.00th=[ 35], 40.00th=[ 49], 50.00th=[ 67], 60.00th=[ 88], | 70.00th=[ 114], 80.00th=[ 143], 90.00th=[ 181], 95.00th=[ 231], | 99.00th=[ 5024], 99.50th=[ 8096], 99.90th=[21632], 99.95th=[33024], | 99.99th=[60160] bw (KB /s): min=20024, max=421488, per=100.00%, avg=178736.35, stdev=136780.57 lat (usec) : 10=2.57%, 20=12.55%, 50=24.99%, 100=24.47%, 250=30.82% lat (usec) : 500=1.03%, 750=0.28%, 1000=0.13% lat (msec) : 2=0.28%, 4=1.32%, 10=1.19%, 20=0.26%, 50=0.10% lat (msec) : 100=0.02%, 250=0.01%, 500=0.01% cpu : usr=10.04%, sys=15.41%, ctx=323551, majf=0, minf=0 IO depths : 1=2.5%, 2=7.5%, 4=18.7%, 8=56.0%, 16=15.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=94.3%, 8=1.1%, 16=4.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Sequential writes: (groupid=2, jobs=1): err= 0: pid=101856: Thu Oct 17 19:49:50 2013 write: io=1536.0MB, bw=162469KB/s, iops=40617, runt= 9681msec slat (usec): min=0, max=23439, avg= 4.54, stdev=40.53 clat (usec): min=6, max=1132.1K, avg=356.47, stdev=6989.74 lat (usec): min=14, max=1132.1K, avg=361.01, stdev=6989.93 clat percentiles (usec): | 1.00th=[ 17], 5.00th=[ 24], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 43], 40.00th=[ 51], 50.00th=[ 61], 60.00th=[ 75], | 70.00th=[ 99], 80.00th=[ 474], 90.00th=[ 612], 95.00th=[ 804], | 99.00th=[ 4512], 99.50th=[ 6624], 99.90th=[20864], 99.95th=[31616], | 99.99th=[70144] bw (KB /s): min=30288, max=352504, per=100.00%, avg=169479.35, stdev=97958.20 lat (usec) : 10=0.07%, 20=2.18%, 50=35.88%, 100=31.90%, 250=8.56% lat (usec) : 500=2.25%, 750=13.42%, 1000=2.53% lat (msec) : 2=1.16%, 4=0.90%, 10=0.88%, 20=0.16%, 50=0.09% lat (msec) : 100=0.01%, 250=0.01%, 1000=0.01%, 2000=0.01% cpu : usr=5.52%, sys=12.96%, ctx=137201, majf=0, minf=0 IO depths : 1=0.3%, 2=4.9%, 4=19.7%, 8=62.4%, 16=12.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.3%, 8=1.6%, 16=5.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Random writes: (groupid=3, jobs=1): err= 0: pid=100163: Thu Oct 17 19:49:50 2013 write: io=1536.0MB, bw=9098.4KB/s, iops=2274, runt=172874msec slat (usec): min=0, max=166956, avg= 6.89, stdev=510.56 clat (usec): min=6, max=2331.7K, avg=6994.06, stdev=38807.83 lat (usec): min=12, max=2331.7K, avg=7000.95, stdev=38815.75 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 31], 10.00th=[ 45], 20.00th=[ 85], | 30.00th=[ 147], 40.00th=[ 270], 50.00th=[ 796], 60.00th=[ 2512], | 70.00th=[ 4128], 80.00th=[ 5792], 90.00th=[ 9920], 95.00th=[22144], | 99.00th=[91648], 99.50th=[183296], 99.90th=[651264], 99.95th=[765952], | 99.99th=[1003520] bw (KB /s): min= 80, max=67072, per=100.00%, avg=9450.67, stdev=7550.18 lat (usec) : 10=0.01%, 20=0.88%, 50=10.60%, 100=11.35%, 250=15.95% lat (usec) : 500=8.03%, 750=2.81%, 1000=2.87% lat (msec) : 2=5.13%, 4=11.46%, 10=21.06%, 20=4.35%, 50=3.45% lat (msec) : 100=1.15%, 250=0.56%, 500=0.21%, 750=0.09%, 1000=0.05% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=0.82%, sys=1.43%, ctx=377431, majf=0, minf=0 IO depths : 1=0.1%, 2=0.6%, 4=3.0%, 8=55.4%, 16=40.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.7%, 8=2.4%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Run status group 0 (all jobs): READ: io=1536.0MB, aggrb=131940KB/s, minb=131940KB/s, maxb=131940KB/s, mint=11921msec, maxt=11921msecRun status group 1 (all jobs): READ: io=1536.0MB, aggrb=178572KB/s, minb=178572KB/s, maxb=178572KB/s, mint=8808msec, maxt=8808msecRun status group 2 (all jobs): WRITE: io=1536.0MB, aggrb=162469KB/s, minb=162469KB/s, maxb=162469KB/s, mint=9681msec, maxt=9681msecRun status group 3 (all jobs): WRITE: io=1536.0MB, aggrb=9098KB/s, minb=9098KB/s, maxb=9098KB/s, mint=172874msec, maxt=172874msec
Оба тестируемые пулы с dedup=on,compression=lz4,checksum=fletcher4
Kingston SSD 120GB V300
Небольшой тест на неравномерно нагруженной системе
Оказывается:
pool: fd
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
fd DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gpt/disk0 ONLINE 0 0 0
1934153058 REMOVED 0 0 0 was /dev/gpt/disk1
errors: No known data errors
zpool DEGRADED
диск только один, второй отвалился
Но давайте чуть увеличим до 1536 мегабайт
Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16fio-2.1.1Starting 4 threadsSequential reads: Laying out IO file(s) (1 file(s) / 1536MB)Jobs: 1 (f=1): [___w] [99.4% done] [0KB/14585KB/0KB /s] [0/3646/0 iops] [eta 00m:03s] Sequential reads: (groupid=0, jobs=1): err= 0: pid=101006: Thu Oct 17 18:47:19 2013 read : io=1536.0MB, bw=1761.5MB/s, iops=450935, runt= 872msec slat (usec): min=0, max=439, avg= 1.38, stdev= 1.02 clat (usec): min=8, max=701, avg=23.11, stdev= 9.59 lat (usec): min=9, max=702, avg=24.49, stdev= 9.61 clat percentiles (usec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 38], 95.00th=[ 40], | 99.00th=[ 44], 99.50th=[ 47], 99.90th=[ 56], 99.95th=[ 62], | 99.99th=[ 145] bw (MB /s): min= 1758, max= 1758, per=99.85%, avg=1758.91, stdev= 0.00 lat (usec) : 10=0.24%, 20=39.35%, 50=60.12%, 100=0.27%, 250=0.02% lat (usec) : 500=0.01%, 750=0.01% cpu : usr=16.07%, sys=83.58%, ctx=475, majf=0, minf=0 IO depths : 1=0.1%, 2=7.1%, 4=28.5%, 8=57.2%, 16=7.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.3%, 8=0.1%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Random reads: (groupid=1, jobs=1): err= 0: pid=101032: Thu Oct 17 18:47:19 2013 read : io=1536.0MB, bw=634731KB/s, iops=158682, runt= 2478msec slat (usec): min=0, max=7426, avg= 4.51, stdev=15.39 clat (usec): min=8, max=11078, avg=57.39, stdev=97.06 lat (usec): min=11, max=11080, avg=61.90, stdev=100.13 clat percentiles (usec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 27], 40.00th=[ 32], 50.00th=[ 38], 60.00th=[ 45], | 70.00th=[ 55], 80.00th=[ 72], 90.00th=[ 115], 95.00th=[ 179], | 99.00th=[ 294], 99.50th=[ 334], 99.90th=[ 426], 99.95th=[ 516], | 99.99th=[ 3888] bw (KB /s): min=254016, max=1024184, per=87.32%, avg=554270.00, stdev=349118.42 lat (usec) : 10=0.02%, 20=13.30%, 50=51.40%, 100=22.80%, 250=10.44% lat (usec) : 500=2.00%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=26.36%, sys=36.21%, ctx=106755, majf=0, minf=0 IO depths : 1=0.9%, 2=5.9%, 4=27.6%, 8=58.3%, 16=7.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.2%, 8=0.1%, 16=6.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=393216/w=0/d=0, short=r=0/w=0/d=0Sequential writes: (groupid=2, jobs=1): err= 0: pid=101422: Thu Oct 17 18:47:19 2013 write: io=1536.0MB, bw=253402KB/s, iops=63350, runt= 6207msec slat (usec): min=0, max=12557, avg= 2.44, stdev=30.18 clat (usec): min=8, max=192112, avg=233.52, stdev=1854.71 lat (usec): min=9, max=192117, avg=235.96, stdev=1855.05 clat percentiles (usec): | 1.00th=[ 17], 5.00th=[ 21], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 33], 50.00th=[ 37], 60.00th=[ 42], | 70.00th=[ 50], 80.00th=[ 56], 90.00th=[ 68], 95.00th=[ 996], | 99.00th=[ 3984], 99.50th=[ 5984], 99.90th=[11968], 99.95th=[15040], | 99.99th=[30080] bw (KB /s): min=23256, max=505528, per=93.47%, avg=236857.92, stdev=169635.96 lat (usec) : 10=0.01%, 20=3.15%, 50=66.19%, 100=22.66%, 250=0.74% lat (usec) : 500=0.15%, 750=0.01%, 1000=2.42% lat (msec) : 2=2.13%, 4=1.61%, 10=0.79%, 20=0.12%, 50=0.01% lat (msec) : 250=0.01% cpu : usr=7.40%, sys=15.45%, ctx=44731, majf=0, minf=0 IO depths : 1=0.1%, 2=1.4%, 4=24.6%, 8=64.6%, 16=9.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=92.4%, 8=0.9%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Random writes: (groupid=3, jobs=1): err= 0: pid=102612: Thu Oct 17 18:47:19 2013 write: io=1536.0MB, bw=3241.4KB/s, iops=810, runt=485244msec slat (usec): min=0, max=4157, avg= 2.04, stdev=11.65 clat (usec): min=9, max=1633.2K, avg=19722.59, stdev=136467.97 lat (usec): min=10, max=1633.3K, avg=19724.63, stdev=136468.03 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 41], 10.00th=[ 52], 20.00th=[ 97], | 30.00th=[ 159], 40.00th=[ 203], 50.00th=[ 255], 60.00th=[ 330], | 70.00th=[ 466], 80.00th=[ 980], 90.00th=[ 2960], 95.00th=[ 6560], | 99.00th=[970752], 99.50th=[1073152], 99.90th=[1253376], 99.95th=[1335296], | 99.99th=[1564672] bw (KB /s): min= 1544, max=16235, per=100.00%, avg=3291.87, stdev=1092.16 lat (usec) : 10=0.01%, 20=0.11%, 50=8.93%, 100=11.27%, 250=28.78% lat (usec) : 500=22.39%, 750=5.51%, 1000=4.68% lat (msec) : 2=6.06%, 4=5.08%, 10=3.51%, 20=1.24%, 50=0.45% lat (msec) : 100=0.08%, 250=0.01%, 500=0.01%, 750=0.11%, 1000=0.95% lat (msec) : 2000=0.83% cpu : usr=0.18%, sys=0.25%, ctx=38177, majf=0, minf=0 IO depths : 1=1.6%, 2=5.8%, 4=16.0%, 8=58.0%, 16=18.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=94.4%, 8=1.8%, 16=3.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=393216/d=0, short=r=0/w=0/d=0Run status group 0 (all jobs): READ: io=1536.0MB, aggrb=1761.5MB/s, minb=1761.5MB/s, maxb=1761.5MB/s, mint=872msec, maxt=872msecRun status group 1 (all jobs): READ: io=1536.0MB, aggrb=634731KB/s, minb=634731KB/s, maxb=634731KB/s, mint=2478msec, maxt=2478msecRun status group 2 (all jobs): WRITE: io=1536.0MB, aggrb=253401KB/s, minb=253401KB/s, maxb=253401KB/s, mint=6207msec, maxt=6207msecRun status group 3 (all jobs): WRITE: io=1536.0MB, aggrb=3241KB/s, minb=3241KB/s, maxb=3241KB/s, mint=485244msec, maxt=485244msec
FreeBSD *.azar-a.net 9.2-RC4 FreeBSD 9.2-RC4 #0 r+750678a-dirty: Thu Sep 26 15:19:27 MSK 2013 root@*.azar-a.net:/usr/obj/usr/src/sys/RACT amd64
ZFS zpool v28-ext zfs v5
[global]bs=4ksize=256Mfilename=test.filedirect=1buffered=0ioengine=posixaioiodepth=16thread[seq-read]rw=readstonewallname=Sequential reads[rand-read]rw=randreadstonewallname=Random reads[seq-write]rw=writestonewallname=Sequential writes[rand-write]rw=randwritestonewallname=Random writes
Результаты
Sequential reads: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random reads: (g=1): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Sequential writes: (g=2): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16Random writes: (g=3): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=16fio-2.1.1Starting 4 threadsJobs: 1 (f=1)Sequential reads: (groupid=0, jobs=1): err= 0: pid=102596: Thu Oct 17 18:28:01 2013 read : io=262144KB, bw=1662.4MB/s, iops=425558, runt= 154msec slat (usec): min=0, max=10, avg= 1.44, stdev= 0.57 clat (usec): min=9, max=109, avg=24.43, stdev= 9.44 lat (usec): min=10, max=111, avg=25.87, stdev= 9.42 clat percentiles (usec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 28], 80.00th=[ 32], 90.00th=[ 39], 95.00th=[ 41], | 99.00th=[ 50], 99.50th=[ 52], 99.90th=[ 59], 99.95th=[ 61], | 99.99th=[ 77] lat (usec) : 10=0.08%, 20=34.91%, 50=63.98%, 100=1.03%, 250=0.01% cpu : usr=14.38%, sys=84.31%, ctx=189, majf=0, minf=0 IO depths : 1=0.1%, 2=6.3%, 4=28.7%, 8=57.7%, 16=7.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.3%, 8=0.1%, 16=6.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0Random reads: (groupid=1, jobs=1): err= 0: pid=101018: Thu Oct 17 18:28:01 2013 read : io=262144KB, bw=1560.1MB/s, iops=399609, runt= 164msec slat (usec): min=0, max=39, avg= 1.43, stdev= 0.59 clat (usec): min=9, max=111, avg=26.61, stdev=11.19 lat (usec): min=10, max=112, avg=28.04, stdev=11.18 clat percentiles (usec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 30], 80.00th=[ 37], 90.00th=[ 43], 95.00th=[ 47], | 99.00th=[ 59], 99.50th=[ 66], 99.90th=[ 82], 99.95th=[ 86], | 99.99th=[ 98] lat (usec) : 10=0.02%, 20=30.17%, 50=66.63%, 100=3.16%, 250=0.01% cpu : usr=19.63%, sys=79.75%, ctx=233, majf=0, minf=0 IO depths : 1=0.1%, 2=3.5%, 4=28.3%, 8=60.6%, 16=7.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=93.0%, 8=0.0%, 16=7.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=65536/w=0/d=0, short=r=0/w=0/d=0Sequential writes: (groupid=2, jobs=1): err= 0: pid=100883: Thu Oct 17 18:28:01 2013 write: io=262144KB, bw=1213.3MB/s, iops=310597, runt= 211msec slat (usec): min=0, max=24, avg= 1.97, stdev= 1.15 clat (usec): min=12, max=73, avg=36.24, stdev=11.75 lat (usec): min=15, max=75, avg=38.21, stdev=11.77 clat percentiles (usec): | 1.00th=[ 16], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 34], 60.00th=[ 36], | 70.00th=[ 40], 80.00th=[ 50], 90.00th=[ 55], 95.00th=[ 57], | 99.00th=[ 61], 99.50th=[ 63], 99.90th=[ 66], 99.95th=[ 66], | 99.99th=[ 69] lat (usec) : 20=3.94%, 50=75.29%, 100=20.77% cpu : usr=36.67%, sys=59.05%, ctx=5050, majf=0, minf=0 IO depths : 1=0.1%, 2=1.5%, 4=27.3%, 8=63.3%, 16=7.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=92.7%, 8=0.0%, 16=7.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0Random writes: (groupid=3, jobs=1): err= 0: pid=102595: Thu Oct 17 18:28:01 2013 write: io=262144KB, bw=1024.0MB/s, iops=262144, runt= 250msec slat (usec): min=0, max=334, avg= 2.21, stdev= 5.13 clat (usec): min=9, max=2146, avg=43.55, stdev=49.80 lat (usec): min=13, max=2149, avg=45.77, stdev=50.71 clat percentiles (usec): | 1.00th=[ 17], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 34], 60.00th=[ 37], | 70.00th=[ 45], 80.00th=[ 52], 90.00th=[ 57], 95.00th=[ 65], | 99.00th=[ 262], 99.50th=[ 358], 99.90th=[ 588], 99.95th=[ 740], | 99.99th=[ 1512] lat (usec) : 10=0.01%, 20=4.33%, 50=70.12%, 100=22.33%, 250=2.09% lat (usec) : 500=0.96%, 750=0.13%, 1000=0.02% lat (msec) : 2=0.03%, 4=0.01% cpu : usr=21.69%, sys=65.46%, ctx=3807, majf=0, minf=0 IO depths : 1=0.1%, 2=1.1%, 4=26.5%, 8=64.2%, 16=8.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=92.6%, 8=0.1%, 16=7.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0Run status group 0 (all jobs): READ: io=262144KB, aggrb=1662.4MB/s, minb=1662.4MB/s, maxb=1662.4MB/s, mint=154msec, maxt=154msecRun status group 1 (all jobs): READ: io=262144KB, aggrb=1560.1MB/s, minb=1560.1MB/s, maxb=1560.1MB/s, mint=164msec, maxt=164msecRun status group 2 (all jobs): WRITE: io=262144KB, aggrb=1213.3MB/s, minb=1213.3MB/s, maxb=1213.3MB/s, mint=211msec, maxt=211msecRun status group 3 (all jobs): WRITE: io=262144KB, aggrb=1024.0MB/s, minb=1024.0MB/s, maxb=1024.0MB/s, mint=250msec, maxt=250msec
near-line sas это диск sata с плюшками SAS (то есть и очередь 256 там тоже)
А так в SAS порт можно и sata воткнуть.
Это Вам не scsi vs ide
Можно узнать, что такое обусловленное ценообразование?
Почему его кто-то должен обуславливать? Разве клиент выбирает по обуславлимоести услуги? Или по соотношениям цена-скорость-качество все-таки?
Я это к чему, тема напоминает нытье:
"Вот "Ресторан" в России не дают рефиллить бесплатно напитки, а в ЕС/США Макдаки дают, я никому и никогда не рекомендую брать еду в "Ресторане" в России, за каждый стаканчик колы Вы заплатите 150 рублей!!!1!111 "
А на разумные аргументы только и слышно "Обоснуйте".
Жлобы не жлобы, никто Романа не заставлял платить под дулом пистолета (а такое в ресторанах в ЕС происходит частенько - хорошо, между прочим, поставленный бизнес).
С большинством серверов IPKVM не идет бесплатно, так почему хостер должен его отдать бесплатно?
Он докупает его и должен отдать бесплатно? А почему сервер не дать бесплатно?
И да в Windows Вы за терминалки будете платить, ой-ой-ой сколько будете платить. (это к ssh и ftp за деньги)
