VMAX is a truly amazing piece of hardware with truly amazing features and unfortunately some truly mind-boggling concepts behind it. I think really this comes with the territory – most often the tools with the biggest capabilities and flexibilities require a lot of knowledge to configure and understand.
One concept that I struggled with on the VMAX was “geometry limited” via Federated Tiered Storage so I thought I would provide some info for anyone else who is having trouble with it. This also gives me the opportunity to talk briefly about some other VMAX topics.
Federated Tiered Storage (FTS) is the ability to have the VMAX leverage third-party storage arrays behind it. So as a simple example, I could connect a VNX to the back of a VMAX, and then present usable storage to the VMAX, and then present that storage to a host. The VMAX in this case acts more like a virtual front end than a storage array itself.
FTS is a complex subject that could itself be the subject of multiple posts. Instead I want to provide a high level overview of concepts, and then go over the ‘geometry limited’ part. There are two ways that FTS manages the other arrays: external provisioning and encapsulation.
External provisioning is the easiest to understand. In this method you present some amount of storage from the external array as a LUN (or multiple LUNs) to the VMAX, and the VMAX interprets this as being special disks. It then uses these disks in much the same manner as it would use a direct attached disk. I say the disks are special because it relies on the assumption that the external array is going to RAID protect the data. Therefore, there is no need to once again RAID protect them on a VMAX like it would normally do – this would waste space and likely degrade performance. Because of this, the VMAX manipulates them in unprotected form. And because they are just seen as attached disks, it also formats them which destroys any data that is on them.
From here you can do most of the stuff you would do with a normal set of disks. You can create a thin pool out of them and even use them as a tier in FAST VP. This is a good way to really maximize the benefits of FTS and leverage your older storage arrays. Or simply just consolidate your arrays into one point of management. Cool feature, but again any data on the LUNs will evaporate.
Encapsulation is the focus of this post and is a little more complicated. Encapsulation allows you to preserve data on a LUN presented for the purposes of FTS. For example if you had an existing LUN on a VNX that a host was using (say oracle database data) and you wanted to present that LUN through the VMAX, you probably wouldn’t want to use External Provisioning because the VMAX would wipe out the data. Instead you do what is known as encapsulation.
When you encapsulate the external LUN, the VMAX in either a thick way or a thin way preserves all the data on the LUN. So you could then connect your oracle database server to the VMAX, attach the encapsulated LUN to it, and viola! all your data is available. It preserves the data by creating a “dummy LUN” of sorts on the VMAX and passing through from the external array to a host.
Encapsulation is neat but there are some restrictions around it. For instance, an encapsulated LUN can’t be the target of a VLUN migration (though you can VLUN migrate it somewhere else) and an encapsulated LUN can’t participate in FAST (whether geometry limited or not).
Some encapsulated LUNs are geometry limited and some aren’t.
In order to understand what geometry limited means, you must first understand how the VMAX sees device sizes. VMAX sizing is always done in cylinders (15 tracks) which are 960KB. This means that what I would consider common LUN sizes (100GB, 500GB, 1TB, 2TB) don’t actually “fit” on a VMAX. Instead, if you ask it to create a 100GB LUN, it rounds up to the nearest cylinder(ish). You can kind of think of this as a “Do No Harm” rule. If you request a device size that falls exactly onto a cylinder boundary, you get that exact device size. If you request one that falls outside of a cylinder boundary, the VMAX rounds up in order to make sure that you get all the space you originally requested.
We won’t get a 100GB LUN because 100GB doesn’t fall onto a cylinder boundary:
100GB * 1024 * 1024 = 104857600KB / 960KB = 109226.6 Cylinders
So we might end up with a device that is actually 109227 cylinders, which would be (109227 * 960KB / 1024 /1024) 100.000305GB.
During “normal” operations this difference is not particularly meaningful (unless you are trying to match device sizes for things like replication, in which case it becomes tremendously important), but it is important to understand.
It is also important to understand that there is a maximum device size on a VMAX, and that is 262668 cylinders, or 240.479GB. In order for a device to be larger than this, you must create what is known as a meta device, or several devices bundled together into a larger virtual device. For instance, if I needed an 800GB LUN, I could “meta” four 200GB regular devices together and have an 800GB device.
So finally, what does geometry limited mean? Geometry limited is what happens when you encapsulate a LUN into a device that does not match up exactly with a VMAX device from a size perspective. In other words, the “dummy LUN” on the VMAX is larger than the actual LUN on the remote array. Again remember the “Do No Harm” philosophy here. You are asking the VMAX to preserve data from an external array, and there is a good chance that external device will not align with cylinder boundaries. The VMAX in this case can’t round down, because it would effectively be chopping off the last parts of what you asked it to preserve – not good! Instead, because it needs to preserve the entire external LUN, and it is required that device sizes align to cylinder boundaries, the VMAX device size is larger than the actual LUN on the external array – this is exactly what causes a device to be geometry limited. If it happens that the external LUN matches up precisely with the cylinder boundary, it is not geometry limited.
With geometry limited devices, the VMAX is going to “fake” the sizing to the host. So no matter how large the VMAX device is, the host is only going to see exactly what is on the original array.
To demonstrate, there are two specific instances where this will happen.
Scenario the First
The first occurrence of geometry limited would be when the external device size does not align with a cylinder boundary. This happens exactly like my previous example with the 100GB LUN. Any device size where the VMAX would have to round-up to match a cylinder boundary would be geometry limited. For instance, if I were to encapsulate a 15GB LUN, this device would not be geometry limited. This is because 15GB fits exactly into 960KB cylinders (16384 cylinders = 16384 * 960KB / 1024 / 1024 = 15GB).
But if I were to encapsulate a 50GB LUN, the VMAX needs to preserve the entire 50GB even though it doesn’t align with cylinder values. Similar to my 100.000305GB LUN above, the VMAX “dummy LUN” must be slightly larger than the 50GB LUN on the external array.
50GB * 1024 * 1024 = 52428800KB / 960KB = 54613.3 Cylinders
So the “dummy LUN” in this case needs to be 54614 cylinders in order to preserve all of the 50GB of data, and 54614 cylinders is larger than the original 50GB device. Hence this encapsulated LUN would be geometry limited.
Scenario the Second
The second occurrence of geometry limited happens due to meta configuration. Let’s encapsulate a 300GB LUN.
300GB * 1024 * 1024 = 314572800KB / 960KB = 327680 cylinders
OK good news! The device falls exactly onto a cylinder boundary, so no geometry limited feature here right? Well, maybe.
Remember that the max device size on a VMAX is around 240GB, so in this case we need a meta configuration to create the VMAX device. Whether this device is geometry limited or not revolves around exactly how that meta is created.
- Sometimes the meta settings force all the members to be a specific size
- Sometimes the meta settings force all the members to be the same size
These conditions can cause a geometry limited condition. In this case, imagine that our meta member size was 125GB and we forced all members to be the same size. In this case we would end up with 3 members and a 375GB meta – 75GB larger than the original device. Again this is a geometry limited device.
Another weird situation can arise when the original device might fall on a cylinder boundary but the meta member count causes it to deviate. For instance if we tried to do a 7 member meta for the 300GB device. Even with the proper meta settings, this is going to be a geometry limited device because 300GB / 7 will not align onto 960KB.
What does it mean and what can I do about it?
Geometry limited devices have several restrictions (do your own research to validate).
- Can only be source devices for local replication
- Can only be R1 devices for SRDF
- Can’t be expanded
Getting rid of geometry limited is possible but still strange. For instance, you can VLUN migrate to an internal pool. This will fix the geometry limited part! What’s sort of bizarre is that the size mismatch still exists. Further, the VMAX will continue to “fake” the sizing to the host even if the VMAX device is expanded. In order to fix this you need to reset the LUN geometry, which requires that you unmount/unmap the device from all FAs…so this is disruptive.
There are a lot of potential use cases for FTS and it is some really sweet technology. However, if you are going to use encapsulation, you should understand these limitations and make sure that you aren’t painting yourself into a corner.