- Part 1: Intro
- Part 2: Hardware, Protocols, and Platforms, Oh My!
- Part 3: File Systems
- Part 4: The Layer Cake
- Part 5: Summary
Last post we covered the differences between NFS and iSCSI (NAS and SAN) and determined that we saw a different set of commands when interacting with a file. The NFS write generated an OPEN command, while the iSCSI write did not. In this post we’ll cover the layering of NAS (file or file systems) on top of SAN (SCSI or block systems) and how that interaction works.
Please note! In modern computing systems there are MANY other layers than I’m going to talk about here. This isn’t to say that they don’t exist or aren’t important, but just that we are focusing on a subset on them for clarity. Hopefully.
First, take a look at the NFS commands listed here: https://tools.ietf.org/html/rfc1813
Compare this with the SCSI reference: http://www.t10.org/lists/op-alph.htm
Notice that in the SCSI case, we still have read and write, but there is no mention of files (other than “filemarks”). There is no way to delete a file with SCSI – because again we are working with a block device which is a layer below the file system. There is no way to delete a file because there is no file. Only addresses where data is stored.
As a potentially clumsy analogy (like I often wield!) think about your office desk. If it’s anything like mine, there is a lot of junk in the drawers. File storage is like the stuff in a drawer. The space in a drawer can have a lot of stuff in it, or it can have a little bit of stuff in it. If I add more stuff to the drawer, it gets more full. If I take stuff out of the drawer, it gets less full. There is meaning to how much stuff is in an individual drawer as a relation to how much more stuff I can put in the drawer.
Block storage, on the other hand, is like the desk itself. There are locations to store things – the drawers. However, whether I have stuff in a drawer or I don’t have stuff in a drawer, the drawer still exists. Emptying out my desk entirely doesn’t cause my desk to vanish. Or at least, I suspect it wouldn’t…I have never had an empty desk in my life. There is no relationship to the contents of the drawers and the space the desk occupies. The desk is a fixed entity. An empty drawer is still a drawer.
To further solidify this file vs block comparison, take a look at this handsome piece of artwork depicting the layers:
Here is a representation of two files on my computer, a word doc and a kitty vid, and their relationship to the block data on disk. Note that some disk areas have nothing pointing to them – these are empty but still zero filled (well…maybe, depending on how you formatted the disk). In other words, these areas still exist! They still have contents, even if that content is nothing.
When I query a file, like an open or read, it traverses the file system down to the disk level. Now I’m going to delete the word doc. In most cases, this is what is going to happen:
My document is gone as far as I can “see.” if I try to query the file system (like look in the directory it was stored in) it is gone. However on the disk, it still exists. (Fun fact: this is how “undelete” utilities work – by restoring data that is still on disk but no longer has pointers from the file system.) It isn’t really relevant that it is still on the disk, because from the system’s perspective (and the file system’s perspective) it doesn’t exist any more. If I want to re-use that space, the system will see it as free and store something else there, like another hilarious kitten video.
Sometimes this will happen instead, either as you delete something (rarely) or later as a garbage collection process:
The document data has been erased and replaced with zeros. (Fun fact: this is how “file shredder” programs work – by writing zeros (or a pattern) once (or multiple times) to the space that isn’t being actively used by files.) Now the data is truly gone, but from the disk perspective it still isn’t really relevant because something still occupies that space. From the disk’s perspective, something always occupies that space, whether it is kitty video data, document data, or zeros. The file system (the map) is what makes that data relevant to the system.
This is a really high level example, but notice the difference in the file system level and the disk level. When I delete that file, whether the actual disk blocks are scrubbed or left intact, the block device remains the same except for the configuration of the 1’s and 0’s. All available addresses are still in place. Are we getting closer to understanding our initial question?
Let’s move this example out a bit and take a look at an EMC VNX system from a NAS perspective. This is a great example because there are both SAN/block (fibre channel) and NAS/file (cifs/nfs) at the same time. The connections look like this:
From my desktop, I connect via NFS to an interface on the NAS (the datamover) in order to access my files. And the datamover has a fibre channel connection to the block storage controllers which is where the data is actually stored. The datamover consumes block storage LUNs, formats them with appropriate file systems, and then uses that space to serve out NAS. This ends up being quite similar to the layered file/disk example above when we were looking at a locally hosted file system and disk.
What does it look like when I read and write? Simply like this:
Reads and writes are supported by SCSI, but what happens when I try to do something to a file like open or delete?
The same command conversion happens, but it is just straight reads and writes at the SCSI level. It doesn’t matter whether the NAS is SAN attached like this one, or it just has standard locally attached disks. This is always what’s going to happen because the block protocol and subsystems don’t work with files – only with data in addresses.
By understanding this layering – what file systems (NAS) do vs what disks (SAN) do – you can better understand important things about their utility. For instance, file systems have various methods to guarantee consistency, in spite of leveraging buffers in volatile memory. If you own the file system, you know who is accessing data and how. You have visibility into the control structure. If the array has no visibility there, then it can’t truly guarantee consistency. This is why e.g. block array snapshots and file array snapshots are often handled differently. With NAS snapshots, the array controls the buffers and can easily guarantee consistent snapshots. But for a block snapshot, the array can only take a picture of the disk right now regardless of what is happening in the file system. It may end up with an inconsistent image on disk, unless you initiate the snapshot from the attached server and properly quiesce/clean the file system.
Back to the idea of control, because NAS systems manage the file side of things, they also have a direct understanding of who is trying to access what. Not only does this give it the ability to provide some access control (unlike SAN which just responds happily to any address requests it gets), it also explains why NAS is often ideal for multi-access situations. If I have users trying to access the same share (or better yet, the same file), NAS storage is typically the answer because it knows who has what open. It can manage things on that level. For the SAN, not so much. In fact if you want two hosts to access the same storage, you need to have some type of clustering (whether direct software or file system) that provides locks and checks. Otherwise you are pretty much guaranteed some kind of data corruption as things are reading and writing over top of one another. Remember SAN and SCSI just lets you read and write to addresses, it doesn’t provide the ability to open and own a file.
In part 5 I’ll provide a summary review and then some final thoughts as well.