Like the sequential read/write zfs benchmarks the bash script creates zfs vdevs of a certain size, ashift, volblocksize and strategy (raidz1, raidz2, raidz3, mirror and so on). Second it starts fio with a number of parameters and repeats this 5 times. Third, results are collected in a .csv file for analysis.
Because every pool need to be written first almost entirely, an acceptable volblocksize and io blocksize is determined. This saves a lot of time. Further iops benchmarks are carried out with this volblocksize and io blocksize only
In Open Office, each set of 5 measurements was averaged. In some charts, because of jitter, the standard deviation for writing is presented in an errorbar.
The results are adjusted for data disks, so for example a raidz1 of 5 disks will appear as 4 data disks in the charts.
The y-axis show scaling instead of iops, to make the results less machine specific.
Benchmarking every combination of array strategy and volblocksize over a range of disks is just too time consuming. For realistic random (read) iops a zvol needs to be written entirely. Have a month? So is there a single volblocksize we can use for benchmarking? If so, this would save writing terabytes of data for each combination. The results below are for a 8-disk zfs stripe with various volblocksizes and io blocksizes.
For an 8 disk stripeset, it would appear 32k is the best volblocksize for random iops at any io blocksize. However, given the fact that a 64k volblocksize performs better for sequential throughput, 64k might be a much better real-world choice for the other benchmarks. Obviously, this generalizes these findings to other pool strategies and at other sizes. This may may not be true. Indeed, to be certain, a month of benchmarking time, or more, would be needed.