linux io测试

http://space.itpub.net/441887/viewspace-630694

 

近来想了解一下开发环境的IO性能，分别用dd/orion/iozone/bonnie++四种工具测试了一下

开 发环境系统配置如下:

Intel SR1625 server , 2 CPU, 32GB内存, 用主板自带卡做了raid1+0，8个7200转SATA硬盘

操作系统是RHEL 5.3 64位

因为物理内存是32GB，因此整个过程都选用了60GB+的数据量来测试，以避免cache的影响

1. 首先用自带的dd命令先测一下, 块大小为8k

dd只能提供一个大概的测试结果，而且是连续IO 而不是随机IO

读测试

# time dd if=/dev/sda2 of=/dev/null bs=8k count=8388608

8388608+0 records in

8388608+0 records out

68719476736 bytes (69 GB) copied, 516.547 seconds, 133 MB/s

real    8m36.926s

user    0m0.117s

sys     0m55.216s

写测试

# time dd if=/dev/zero f=/opt/iotest bs=8k count=8388608

8388608+0 records in

8388608+0 records out

68719476736 bytes (69 GB) copied, 888.398 seconds, 77.4 MB/s

real    14m48.743s

user    0m3.678s

sys     2m47.158s

读 写测试

# time dd if=/dev/sda2 f=/opt/iotest bs=8k count=8388608

8388608+0 records in

8388608+0 records out

68719476736 bytes (69 GB) copied, 1869.89 seconds, 36.8 MB/s

real    31m10.343s

user    0m2.613s

sys     3m25.548s

2. 接下来用Oracle 的orion 工具来测一下

解压即可使用

# gzip orion_linux_x86-64.gz

测异步IO时需要libaio库

# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64

# echo $LD_LIBRARY_PATH

:/opt/oracle/product/10.2.0/lib:/usr/lib64

创 建配置文件mytest.lun,列出要测试的分区即可. 注意文件名前缀要跟下面的 testname一致

# vi mytest.lun

查看mytest.jun内容

# cat mytest.lun

/dev/sda2

先来个simple test

# ./orion_linux_x86-64 -run simple -testname mytest -num_disks 8

查 看测试结果

# cat mytest_20081111_1431_summary.txt

ORION VERSION 11.1.0.7.0

Commandline:

-run simple -testname mytest -num_disks 8

This maps to this test:

Test: mytest

Small IO size: 8 KB

Large IO size: 1024 KB

IO Types: Small Random IOs, Large Random IOs

Simulated Array Type: CONCAT

Write: 0%

Cache Size: Not Entered

Duration for each Data Point: 60 seconds

Small Columns:,      0

Large Columns:,      0,      1,      2,      3,      4,      5,      6,      7,      8,      9,     10,     11,     12,     13,     14,     15,     16

Total Data Points: 38

Name: /dev/sda2 Size: 629143441920

1 FILEs found.

Maximum Large MBPS=56.97 @ Small=0 and Large=7

Maximum Small IOPS=442 @ Small=40 and Large=0

Minimum Small Latency=14.62 @ Small=1 and Large=0

最大MBPS为56.97，最大IOPS为442

再 测一下8k随机读操作

# ./orion_linux_x86-64 -run advanced -testname mytest -num_disks 8 -size_small 8 -size_large 8 -type rand &

看 看结果

# cat mytest_20081111_1519_summary.txt

ORION VERSION 11.1.0.7.0

Commandline:

-run advanced -testname mytest -num_disks 8 -size_small 8 -size_large 8 -type rand

This maps to this test:

Test: mytest

Small IO size: 8 KB

Large IO size: 8 KB

IO Types: Small Random IOs, Large Random IOs

Simulated Array Type: CONCAT

Write: 0%

Cache Size: Not Entered

Duration for each Data Point: 60 seconds

Small Columns:,      0

Large Columns:,      0,      1,      2,      3,      4,      5,      6,      7,      8,      9,     10,     11,     12,     13,     14,     15,     16

Total Data Points: 38

Name: /dev/sda2 Size: 629143441920

1 FILEs found.

Maximum Large MBPS=3.21 @ Small=0 and Large=13

Maximum Small IOPS=448 @ Small=38 and Large=0

Minimum Small Latency=15.16 @ Small=1 and Large=0

最大MBPS为3.21(这么低？？)，最大IOPS为448

再 测一下1M顺序读操作, 失败了, 原因不明...

# ./orion_linux_x86-64 -run advanced -testname mytest -num_disks 8 -size_small 1024 -size_large 1024 -type seq

ORION: ORacle IO Numbers -- Version 11.1.0.7.0

mytest_20081114_1349

Test will take approximately 73 minutes

Larger caches may take longer

rwbase_run_test: rwbase_reap_req failed

rwbase_run_process: rwbase_run_test failed

rwbase_rwluns: rwbase_run_process failed

orion_warm_cache: Warming cache failed. Continuing

看 看结果

# cat mytest_20081111_1620_summary.txt

ORION VERSION 11.1.0.7.0

Commandline:

-run advanced -testname mytest -num_disks 8 -size_small 1024 -size_large 1024 -type seq

This maps to this test:

Test: mytest

Small IO size: 1024 KB

Large IO size: 1024 KB

IO Types: Small Random IOs, Large Sequential Streams

Number of Concurrent IOs Per Stream: 4

Force streams to separate disks: No

Simulated Array Type: CONCAT

Write: 0%

Cache Size: Not Entered

Duration for each Data Point: 60 seconds

没 结果，失败

3. 用iozone 来测一下

安装

# tar -xvf iozone3_345.tar

# make linux-AMD64

指定 64G的文件，只测read/reread和write/rewrite，记录大小从4k-16k.同时生成一个excel文件iozone.wks

# ./iozone -Rab iozone.wks -s64G -i 0 -i 1 -y 4k -q 16k

        Iozone: Performance Test of File I/O

                Version $Revision: 3.345 $

                Compiled for 64 bit mode.

                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,

                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,

                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,

                     Fabrice Bacchella, Zhenghua Xue, Qin Li.

        Run began: Tue Nov 11 10:23:25 2008

        Excel chart generation enabled

        Auto Mode

        File size set to 67108864 KB

        Using Minimum Record Size 4 KB

        Using Maximum Record Size 16 KB

        Command line used: ./iozone -Rab iozone.wks -s64G -i 0 -i 1 -y 4k -q 16k

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

                                                            random  random    bkwd   record   stride                                 

              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread

        67108864       4   72882   69470   104898   125512

        67108864       8   72083   69256   133689   109061

        67108864      16   73375   69155   142019   116034

iozone test complete.

Excel output is below:

"Writer report"

        "4"  "8"  "16"

"67108864"   72882  72083  73375

"Re-writer report"

        "4"  "8"  "16"

"67108864"   69470  69256  69155

"Reader report"

        "4"  "8"  "16"

"67108864"   104898  133689  142019

"Re-Reader report"

        "4"  "8"  "16"

"67108864"   125512  109061  116034

可 以看到，8k的写是72M/s左右，读是133M/s左右，跟dd的结果比较接近

测一下64G文件 8k随机读写

# ./iozone -Rab iozone.wks -s64G -i 2 -y 8k -q 8k

        Iozone: Performance Test of File I/O

                Version $Revision: 3.345 $

                Compiled for 64 bit mode.

                Build: linux-AMD64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins

                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss

                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,

                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,

                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,

                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,

                     Fabrice Bacchella, Zhenghua Xue, Qin Li.

        Run began: Fri Nov 14 15:52:01 2008

        Excel chart generation enabled

        Auto Mode

        File size set to 67108864 KB

        Using Minimum Record Size 8 KB

        Using Maximum Record Size 8 KB

        Command line used: ./iozone -Rab iozone.wks -s64G -i 2 -y 8k -q 8k

        Output is in Kbytes/sec

        Time Resolution = 0.000001 seconds.

        Processor cache size set to 1024 Kbytes.

        Processor cache line size set to 32 bytes.

        File stride size set to 17 * record size.

                                                            random  random    bkwd   record   stride                                 

              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread

        67108864       8

Error reading block at 6501007360

read: Success

出错了(??)

4. 最后用bonnie++ 测 一下

安装

# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64

# ./configure

# make

# make install

开始测试，默认文件大小是内存的2 倍

# bonnie++ -d /opt/IOTest/ -m sva17 -u root

Using uid:0, gid:0.

Writing with putc()...done

Writing intelligently...done

Rewriting...done

Reading with getc()...done

Reading intelligently...done

start 'em...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

sva17           63G 52391  84 35222   7 34323   6 56362  88 131568  10 176.7   0

                    ------Sequential Create------ --------Random Create--------

                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

sva17,63G,52391,84,35222,7,34323,6,56362,88,131568,10,176.7,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

顺 序写: 按字符是52391KB/s，CPU占用率84%；按块是35222KB/s，CPU占用率7%

顺序读: 按字符是56362KB/s, CPU占用率88%；按块是131568KB/s，CPU占用率10%

随机读写: 176.7次/s，CPU占用率0%

后两项全是++ (没结果?)

结论:不 同测试工具构建出来的测试环境不同，侧重点也不一样，得到的结果可能相差比较大。

MBPS:

dd 和iozone比较接近，读写分别是130+和70+。

orion读57左右，写没测(会删掉分区内所有文件!)

bonnie++ 按块读是130左右，写是35左右；按字符读是56左右，写是52左右

IOPS:

dd 无结果

orion 440左右(只读)

iozone 出错

bonnie++ 176.7 (读写)

Sysbench:

sysbench --test=fileio --file-total-size=30G prepare

sysbench --test=fileio --file-total-size=30G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrw prepare

最专业的是iometer和hd tach