I went to replace an aging drive with a brand new one, so I issued a
zpool replace poolname olddisk1 newdisk1
Sweet. It worked. Err….
7.19G scanned out of 10.2T at 46.0M/s, 64h23m to go
1.73G resilvered, 0.07% done
What the hell? That’s ridiculously slow.
Turns out there is a default priority setting in the ZFS kernel module that is tunable via sysfs. It defaults to 2 clock ticks of delay and setting it to 0 essentially sets it at the same priority as other I/O:
echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay
54.6G scanned out of 10.2T at 87.8M/s, 33h35m to go
13.2G resilvered, 0.52% done
Still slow, but not as bad as 46MB/s. Sheesh.
There is another parameter that greatly increases the resilvering throughput:
echo 0 > /sys/module/zfs/parameters/zfs_scan_idle
The effect it had on my latest drive replacement was enormous:
885G scanned out of 8.99T at 431M/s, 5h29m to go