Welcome back to me! I’ve gone dark for a while as I welcomed my first child into the world. It has been a very busy and confusing few months. 🙂
In this blog post I’d like to cover some rules around good storage pool construction and hopefully shed some light on what the storage pool “recommended numbers” really mean.
Let’s walk through a scenario. You’ve got a VNX array, and you’ve got a hankerin’ for some RAID1/0. You’ve got 18 unbound disks burning a hole in your pocket and nothing can stop you now. You fire up Unisphere and away you go!
You start out with this:
And end up with this:
Looks awesome! Unfortunately you’ve actually made a pretty bad mistake. Under the covers, things are really out of balance now. The private RAID groups behind the storage pool now look like this:
One RG has 1/4th of the IOPs capability of the other two.
There is kind of an interesting density mechanic here at play in that it is likely that the 4+4 will generate more I/O overall because there is more usable capacity…this metric would be something like I/O per GB. But the fact remains that the 1+1 behind the scenes has a good chance of being a bottleneck at some point.
Even worse, the VNX doesn’t understand that the RGs have a performance imbalance. So while there is less space for it to move slices into via FAST, it is still going to try to balance out the I/O within the tier. Edit: Today I learned (always learning!) that Flare32 and later VNXes actually will recognize the performance imbalance of the private RGs and allocate less I/O to them. However there are still less total IOPs within that PRG to serve the slices on it. And because the private RAID groups aren’t made terribly obvious unless you dig in to the array, you might not ever notice the misconfiguration either until you start asking questions like, “why is my array performing like a Yugo instead of a Corvette?”
What could I have done?
I see this mistake made pretty frequently on arrays where perhaps an admin who wasn’t intimately familiar with VNX mechanics created or expanded some pools.. It isn’t always this severe…sometimes it is a 3 or 4 disk RAID5 RG in a 4+1 pool. And really it is kind of counter-intuitive, since you probably (and rightly) feel like you really should be able to use any amount of disks you want…after all you paid for them! And sadly it is not reversible without destroying and recreating the entire pool. But the important takeaway is that it never needs to happen. You have two ways to avoid this.
One way is to always add disks in “recommended” chunks (more on this below), which also translates into purchasing them in recommended chunks. If you had 18 disks available on your array, you should have planned on using them for a 4+4 and two 4+1’s. Or two 8+1’s. You should be aware of the number of disks you are adding and whether they correspond with the recommended amounts.
Another way is to build the entire pool in smaller chunks. This is a very manual, very annoying process. It is also likely to bite your company in the rear after you win the lottery and leave, and another admin comes in after you. But for some reason you just REALLY want to use all 18 disks in RAID 1/0. In order to do this properly, you want to create the pool with 6 disks in RAID1/0. Then add another 6 disks in RAID1/0. Then add the final 6 disks in RAID1/0. This will create 3 x 3+3 balanced RGs in the pool. Now, unless the next guy coming in to fill your lottery-winning shoes is a sharpshooter, he’s probably not going to know this and may just start adding in 8 disks groups to the pool, which again unbalances the back end.
What about those “recommended” numbers?
This is another interesting misconception behind the storage pool numbers. The recommended numbers per RAID type are:
- R1/0: 4+4
- R5: 4+1 or 8+1
- R6: 6+2 or 14+2
Some people feel like there is some magic behind these choices. There isn’t. These are simply some good balances between performance, capacity, and the fault domain of the RAID sets. If you’ve read my epic on RAID you know that a 4+1 has less performance (because of less spindles) than a 15+1, but also has less of a chance of a dual disk failure which would cause data loss.
So they are good choices but they aren’t perfect choices. You may experience a dual disk failure with 4+1 and corrupt the entire storage pool. Having a R5 5+1 isn’t going to guarantee data destruction. Having a R5 3+1 isn’t going to guarantee abysmal performance.
Specific disk counts may also enable more Full Stripe Writes, but truthfully if something is writing a LOT of data to the array, it is probably doing it sequentially anyway so the disk count becomes less of a factor.
Based on this it isn’t really accurate to say go with the recommended disk counts for pools because they are amazing, but it does guarantee back-end balance and provides a good balance of performance and capacity.
raid-zero.com 