1

I'm having a performance problem with my ZFS array that i think to be due to defragmentation. I have a striped array of 20 devices with 2 mirrored drives each. I've recently added some devices to this striped array that are bigger (2x) than the original ones.

Using zpool iostats shows that even while some of the old devices are 90% full and the new ones only 50% full, writes are equally distributed between all devices, indipendently from the free % or space of every drive.

Does ZFS expects devices of all the same size for a striped array? Is this the reason why the disks are not similarly used, or this is completely irrelevant?

Improve this question

1 Answer 1

Reset to default
1

I tested this out using zdb -ddddd [see http://www.cuddletech.com/blog/pivot/entry.php?id=980 ]

I created two files called disk1 and disk2 that were 400M and then disk3 and disk4 at 650M, and I created a zpool "tank" using them:

zpool create tank mirror disk1 disk2
zpool add mirror disk3 disk4

(I'm assuming that you used zpool add mirror to include the disks in the pool, and did not zpool attach the disks to an existing mirror vdev; the latter is a universally bad idea and will not increase capacity anyway, would just add another mirror device)

As I filled the pool up 100M at a time w/ mkfile, I checked zdb -ddddd tank <inode#> to see which vdevs were being written to, and zfs did balance the load between each one; out of 400 block addresses, on average about 300-350 were written to the smaller "disks" and 450-500 to the larger, even in the last file that filled the pool.

That means ZFS will balance out the blocks it writes between each vdev in a pool. (In this case, between each of your 11 mirrors). OTOH, if you were to add a larger drive to an existing mirror using zpool attach, ZFS would not be able to utilize that extra space, since it writes the same blocks to each disk in a mirror.

It is typically recommended to use the same disks across a pool, but that's apparently not necessary. Don't bother adding a faster drive to a pool with slower disks, though; in a pool, your slowest disk is your bottleneck (although adding an SSD log for the ZIL may mitigate that problem).

Also, I wouldn't worry about defragmentation in ZFS, it's the nature of the beast -- since ZFS is a Copy-on-Write filesystem, it's unavoidable. (Meaning, instead of overwriting a block w/ new data, it writes the new data to a new block, then updates metadata pointers and checksums to use the new block instead.) I wouldn't expect that to drag down performance.

What will drag down performance is being 90% full (interferes w/ Copy-on-Write). It's advised to keep utilization at or under 80% for peak performance.

Which makes me think of something else -- your pool already existed, and then you added two new disks -- the existing data WILL NOT be striped across those disks, not until it's re-written, so I would expect to see less total data on the new disks for some time.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .