Prioritizing OpenZFS resilvering so that it isn’t so slow (ZFS on Linux)

I went to replace an aging drive with a brand new one, so I issued a

zpool replace poolname olddisk1 newdisk1

Sweet. It worked. Err….

zpool status

7.19G scanned out of 10.2T at 46.0M/s, 64h23m to go
1.73G resilvered, 0.07% done

What the hell? That’s ridiculously slow.

Turns out there is a default priority setting in the ZFS kernel module that is tunable via sysfs. It defaults to 2 clock ticks of delay and setting it to 0 essentially sets it at the same priority as other I/O:
echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay

54.6G scanned out of 10.2T at 87.8M/s, 33h35m to go
13.2G resilvered, 0.52% done

Still slow, but not as bad as 46MB/s. Sheesh.


There is another parameter that greatly increases the resilvering throughput:
echo 0 > /sys/module/zfs/parameters/zfs_scan_idle

The effect it had on my latest drive replacement was enormous:
885G scanned out of 8.99T at 431M/s, 5h29m to go