ZFS random IOPS performance
Materials
- Supermicro X9SCM-F mainboard
- two 8-port IBM M1015 SAS/SATA controllers
- 16GB ECC RAM
- Intel Xeon E3-1230
- 15x Hitachi Deskstar 7K1000
- CentOS 6.2, kernel 2.6.32-279.9.1.el6.x86_64
- zfs-0.6.0-rc14, spl-0.6.0-rc14
- fio I/O benchmark tool
- a custom BASH script to iterate through a number of benchmark parameters
Method
Like the sequential read/write zfs benchmarks the bash script creates zfs vdevs of a certain size, ashift, volblocksize and strategy (raidz1, raidz2, raidz3, mirror and so on). Second it starts fio with a number of parameters and repeats this 5 times. Third, results are collected in a .csv file for analysis.
Because every pool need to be written first almost entirely, an acceptable volblocksize and io blocksize is determined. This saves a lot of time. Further iops benchmarks are carried out with this volblocksize and io blocksize only
In Open Office, each set of 5 measurements was averaged. In some charts, because of jitter, the standard deviation for writing is presented in an errorbar.
The results are adjusted for data disks, so for example a raidz1 of 5 disks will appear as 4 data disks in the charts.
The y-axis show scaling instead of iops, to make the results less machine specific.
Raw data
The resulting .csv data sits here. The Open Office version with aggregated results is there
This is
the data file for gnuplot, made from the .ods file, and that is
the file you can load in gnuplot to have the scaling graphs in
PNG format.
Which volblocksize to use as shortcut?
Benchmarking every combination of array strategy and volblocksize over a range of disks is just too time consuming. For realistic random (read) iops a zvol needs to be written entirely. Have a month? So is there a single volblocksize we can use for benchmarking? If so, this would save writing terabytes of data for each combination. The results below are for a 8-disk zfs stripe with various volblocksizes and io blocksizes.
For an 8 disk stripeset, it would appear 32k is the best volblocksize for random iops at any io blocksize. However, given the fact that a 64k volblocksize performs better for sequential throughput, 64k might be a much better real-world choice for the other benchmarks. Obviously, this generalizes these findings to other pool strategies and at other sizes. This may may not be true. Indeed, to be certain, a month of benchmarking time, or more, would be needed.
Random read/write IOPS scaling comparison
