10566 Multiple DVA Scrubbing Fix

Review Request #1572 - Created March 19, 2019 and submitted

Toomas Soome

Currently, there is an issue in the sequential scrub code which
prevents self healing from working in some cases. The scrub code
will split up all DVA copies of a bp and issue each of them
separately. The problem is that, since each of the DVAs is no
longer associated with the others, the self healing code doesn't
have the opportunity to repair problems that show up in one of the
DVAs with the data from the others.

This patch fixes this issue by ensuring that all IOs issued by the
sequential scrub code include all DVAs. Initially, only the first
DVA of each is attempted. If an issue arises, the IO is retried
with all available copies, giving the self healing code a chance
to correct the issue.

To test this change, this patch also adds the ability for zinject
to specify individual DVAs to inject read errors into. We then
add a new test case that utilizes this functionality to ensure
scrubs and self-healing reads can handle and transparently fix
issues with individual copies of blocks.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8453

Pre-and post fixes:
551905dd4 vdev_mirror: kstat observables for preferred vdev
d6c6590c5 vdev_mirror: load balancing fixes
9f500936c FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
fb40095f5 Disable LBA weighting on files and SSDs
4770aa064 Fix vdev_open_child() race on updating vdev_parent->vdev_nonrot
13d9a004f Fix taskq creation failure in vdev_open_children()
7dcd31883 Cleanup nits from ab7615d92

I assume we will land after https://www.illumos.org/issues/10809,
so the avl.h macros presented here will be implemented by 10809

Jerry Jelinek
Toomas Soome
Jerry Jelinek
Jerry Jelinek
Toomas Soome
Toomas Soome
Jerry Jelinek
Jerry Jelinek
Jerry Jelinek
Jerry Jelinek
Toomas Soome
Review request changed

Status: Closed (submitted)