vRealize Automation and UCS Director Interaction via REST API – Part 1

You might ask yourself, “self, why do I care so much about automation?  Why so much fuss over APIs?”  The truth is the world is changing – this is true everywhere, and especially true in IT.  Automation and APIs are changing the game, allowing you to answer the question, “what can we do?” with “pretty much anything.”

There are a lot of references to VMware vRealize Automation/Orchestration (vRA/vRO) and Cisco UCS Director (UCSD) REST APIs from the vendors, but I thought I would post some findings and methodologies around rolling your own integrated solution.  I’m specifically going to work through calling a UCSD workflow with a vRA XaaS (a.k.a. vRO workflow) blueprint, but you can use this methodology to add any REST functionality through vRA for most anything.  APIs allow us the opportunity for endlessly integrated solutions…as long as you’ve got the time and patience to develop them! Continue reading

Plan, Execute, THEN Drink the Beer!

This topic has been simmering in my mind for years now.  Everyone’s favorite thing to hate: planning.  I don’t have all the answers but I want to try to provide some guidance to someone specific.  Yes, you, the dude who recognizes the need for planning when nobody around does.  The dudette who is tired of having projects go awry Every. Single. Time.  Beers are great to celebrate a project victory, but a lot of you are using (abusing?) them throughout the project cycle to help numb the pain.  I get it; I’ve been there.

There are a lot of things that this post is not.  It is not a crash course in project management.  I don’t talk about cool buzzwords like agile or scrum.  I’m not going to talk about change management processes or anything like that.  It is simply a collection of my observations about the multitude of businesses I’ve worked in or with, and the multitude of projects I’ve seen succeed and fail.  But read on, I think you’ll still find it useful. Continue reading

The Amazingness That Will Be vSphere 6.5

In case you haven’t heard the news, VMware announced vSphere 6.5 at VMworld Europe this year.  As usual I wouldn’t recommend anyone jumps on the bandwagon at the .0 release (unless you have test environments).  Nobody wants to be that first gazelle across the river!

Make no mistake, there are a TON of new features and enhancements in this version, which you should be checking out on VMware’s press releases.  I don’t want to cover them all, as you’d need a gallon of coffee to make it through, but I did want to talk about a few of them that I’m personally excited about. Continue reading

Back to Basics – vSphere Maintenance Mode

I’ve noticed over the years that while VMware admins tend to really understand maintenance mode, a lot of others in adjacent spaces (storage, network, etc.) have a very murky perspective on it.  In fact, I’d bet that if you sat down a storage person at vCenter and told them to evacuate a host with MM, odds are they would be really confused when the task hung or failed.  I know I was the first time I tried it. Continue reading

vSphere Client Console Black Screen

Or, more accurate than the title, a black area on the border of the console window.

I got a new laptop running Windows 10 and when I brought up the console in my home lab, this is what I’d get.


It actually ended up being worse even than it looks as the mouse is pretty much unusable on the screen and a huge chunk of the screen (bottom and right side) is cut off.  It is virtually impossible to do anything with VMs including installations of operating systems.

I tried several fixes like compatibility mode, update drivers, etc. without success.  Then I stumbled upon this scaling setting which is mentioned on several sites dealing with mouse usage in the VDI space.

Browse to C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\ and right click on VpxClient.exe and select Properties.


On this screen, select the Disable Display Scaling box indicated.  Then restart your client.

As a annoying side effect, now some of the text isn’t scaled right so some things in the client look wrong.  But it is still usable and the console is usable.

Weirdly, the vCenter console from the web client has no such issue that I noticed.  But if you are being pestered by this “feature” try this setting and see if it helps.

Shedding Light on Storage Encryption

I’ve been noticing some fundamental misunderstandings around storage encryption – I see this most when dealing with XtremIO although plenty of platforms support it (VNX2 and VMAX).  I hope this blog post will help someone who is missing the bigger picture and maybe make a better decision based on tradeoffs.  This is not going to be a heavily technical post, but is intended to shed some light on the topic from a strategic angle.

Hopefully you already know, but encryption at a high level is a way to make data unreadable gibberish except by an entity that is authorized to read it.  The types of storage encryption I’m going to talk about are Data At Rest Encryption (often abbreviated DARE or D@RE), in-flight encryption, and host-based encryption.  I’m talking in this post mainly about SAN (block) storage, but these concepts also apply to NAS (file) storage.  In fact, in-flight encryption is probably way more useful on a NAS array given the inherent security of FC fabrics.  But then, iSCSI, and it gets cloudier.

Before I start, security is a tool and can be used wisely or poorly with equivalent results.  Encryption is security.  All security, and all encryption, is not great.  Consider the idea of cryptographic erasure, by which data is “deleted” merely because it is encrypted and nobody has the key.  Ransomware thrives on this.  You are looking at a server with all your files on it, but without the key they may as well be deleted.  Choosing a security feature for no good business reason other than “security is great” is probably a mistake that is going to cause you headaches.


Here is a diagram with 3 zones of encryption.  Notice that host-based encryption overlaps the other two – that is not a mistake as we will see shortly.

Data At Rest Encryption

D@RE of late is typically referring to a storage arrays ability to encrypt data at the point of entry (write) and decrypt on exit (read).  Sometimes this is done with ASICs on an array or I/O module, but it is often done with Self Encrypting Drives (SEDs).  However the abstract concept of D@RE is simply that data is encrypted “at rest,” or while it is sitting on disk, on the storage array.

This might seem like a dumb question, but it is a CRUCIAL one that I’ve seen either not asked or answered incorrectly time and time again: what is the purpose of D@RE?  The point of D@RE is to prevent physical hardware theft from compromising data security.  So, if I nefariously steal a drive out of your array, or a shelf of drives out of your array, and come up with some way to attach them to another system and read them, I will get nothing but gibberish.

Now, keep in mind that this problem is typically far more of an issue on a small server system than it is a storage array.  A small server might just have a handful of drives associated with it, while a storage array might have hundreds, or thousands.  And those drives are going to be in some form of RAID protection which leverages striping.  So even without D@RE the odds of a single disk holding meaningful data is small, though admittedly it is still there.

More to the point, D@RE does not prevent anyone from accessing data on the array itself.  I’ve heard allusions to this idea that “don’t worry about hackers, we’ve got D@RE” which couldn’t be more wrong, unless you think hackers are walking out of your data center with physical hardware.  If the hackers are intercepting wire transmissions, or they have broken into servers with SAN access, they have access to your data.  And if your array is doing the encryption and someone manages to steal the entire array (controllers and all) they will also have access to your data.

D@RE at the array level is also one of the easiest to deal with from a management perspective because usually you just let the array handle everything including the encryption keys.  This is mostly just a turn it on and let it run solution.  You don’t notice it and generally don’t see any fall out like performance degradation from it.

In-Flight Encryption

In-flight encryption is referring to data being encrypted over the wire.  So your host issues a write to a SAN LUN, and that traverses your SAN network and lands on your storage array.  If data is encrypted “in-flight,” then it is encrypted throughout (at least) the switching.

Usually this is accomplished with FC fabric switches that are capable of encryption.  So the switch that sees a transmission on an F port will encrypt it, and then transmit it encrypted along all E ports (ISLs) and then decrypt it when it leaves another F port.  So the data is encrypted in-flight, but not at rest on the array.  Generally we are still talking about ASICs here so performance is not impacted.

Again let’s ask, what is the purpose of in-flight encryption?  In-flight encryption is intended to prevent someone who is sniffing network traffic (meaning they are somehow intercepting the data transmissions, or a copy of the data transmissions, over the network) from being able to decipher data.

For local FC networks this is (in my opinion) not often needed.  FC networks tend to be very secure overall and not really vulnerable to sniffing.  However, for IP based or WAN based communication, or even stretched fabrics, it might be sensible to look into something like this.

Also keep in mind that because data is decrypted before being written to the array, it does not provide the physical security that D@RE does, nor does it prevent anyone from accessing data in general.  You also sometimes have the option of not decrypting when writing to the array.  So essentially the data is encrypted when leaving the host, and written encrypted on the array itself.  It is only decrypted when the host issues a read for it and it exits the F port that host is attached to. This results in you having D@RE as well with those same benefits.  A real kicker here becomes key management, because in-flight encryption can be removed at any time without issue.  You can remove or disable in-flight encryption and not see any change in data because at the ends it is unencrypted.  However, if the data is written encrypted on the array, then you MUST have those keys to read that data.  If you had some kind of disaster that compromised your switches and keys, you would have a big array full of cryptographically erased data.

Host Based Encryption

Finally, host-based encryption is any software or feature that encrypts LUNs or files on the server itself.  So data that is going to be written to files (whether SAN based or local files) is encrypted in memory before the write actually takes place.

Host-based encryption ends up giving you both in-flight encryption and D@RE as well.  So when we ask the question, what is the purpose of host-based encryption?, we get the benefits we saw from in-flight and D@RE, as well as another one.  That is the idea that even with the same hardware setup, no other host can read your data.  So if I were to forklift your array, fabric switches, and get an identical server (hardware, OS, software) and hook it up, I wouldn’t be able to read your data.  Depending on the setup, if a hacker compromises the server itself in your data center, they may not be able to read the data either.

So why even bother with the other kinds of encryption?  Well for one, generally host-based encryption does incur a performance hit because it isn’t using ASICs.  Some systems might be able to handle this but many won’t be able to.  Unlike D@RE or in-flight, there will be a measurable degradation when using this method.  Another reason is that key management again becomes huge here.  Poor key management and a server having a hardware failure can lead to that data being unreadable by anyone.  And generally your backups will be useless in this situation as well because you have backups of encrypted data that you can’t read without the original keys.

And frankly, usually D@RE is good enough.  If you have a security issue where host-based encryption is going to be a benefit, usually someone already has the keys to the kingdom in your environment.

Closing Thoughts

Hopefully that cleared up the types of encryption and where they operate.

Another question I see is “can I use one or more at the same time?”  The answer is yes, with caveats.  There is nothing that prevents you from using even all 3 at the same time, even though it wouldn’t really make any sense.  Generally you want to avoid overlapping because you are encrypting data that is already encrypted which is a waste of resources.  So a sensible pairing might be D@RE on the array and in-flight on your switching.

A final HUGELY important note – and what really prompted me to write this post – is to make sure you fully understand the effect of encryption on all of your systems.  I have seen this come up in a discussion about XtremIO using D@RE paired with host-based encryption.  The question was “will it work?” but the question should have been “should we do this?”  Will it work?  Sure, there is nothing problematic about host-based encryption and XtremIO D@RE interacting, other than the XtremIO system encrypting already encrypted data.  What is problematic, though, is the fact that encrypted data does not compress, and most encrypted data won’t dedupe either…or at least not anywhere close to the level of unencrypted data.  And XtremIO generally relies on its fantastic inline compression and dedupe features to fit a lot of data on a small footprint. XtremIO’s D@RE happens behind the compression and deduplication, so there is no issue.  However host-based encryption will happen ahead of the dedupe/compression and will absolutely destroy your savings. So if you wanted to use the system like this, I would ask, how was it sized?  Was it sized with assumptions about good compression and dedupe ratios?  Or was it sized assuming no space savings?  And, does the extra money you will be spending for the host-based encryption product and the extra money you will be spending on the additional required storage justify the business problem you were trying to solve?  Or was there even a business problem at all?  A better fit would probably be something like a tiered VNX2 and FAST cache which could easily handle a lot of raw capacity and use the flash where it helps the most.

Again, security is a tool, so choose the tools you need, use them judiciously, and make sure you fully understand their impact (end-to-end) in your environment.

VMware vSphere – Guest clock, replication, and squirrels


That’s me blowing the cobwebs off of my blog. 🙂

Time marches on and things are always changing.  I too have recently made several changes both in my personal and professional life.  I am now a delivery engineer for CDI Southeast, though we are going through another transition period ourselves.  I am also doing a bit of retooling, attempting to become a little less storage focused and branching out into other areas.  Right now I’m trying to focus more on vSphere but also working with some DevOps stuff as well.  I also released a course on Pluralsight last October which I am very proud of.  If you are interested in hearing me talk to you about VNX for hours, I encourage you to check it out.

So, great things hopefully coming down the pipe for me in the future and I hope to continue to share with the community at large that has given so much to me.

In the meantime, here is a quick nugget that I turned up a couple of weeks ago.  I was doing a pretty straightforward implementation of vSphere replication (and I hope to do a series on that soon), but ran into an oddity which I initially wrote off as a squirrel.

A squirrel is a situation where I walk into your building to do an implementation, and shortly thereafter you (or your co-worker, or your boss) comes up to me and says, “you know ever since you got here, the squirrels have been going crazy outside.  What did you do?”

The squirrel can be anything really, but most of the time is comically unrelated to anything I’m actually working on. And sometimes it is just people messing around with me.  “Hey you know Exchange is down?  JUST KIDDING!”

Side note – I’ve been on site with a jokey client where something really did go down and it took us a minute or two to figure out that there was really a problem. 

I’m used to squirrels now.  Honestly they rarely have merit but I always take them seriously “just in case.”

In this case, in the midst of replicating VMs in the environment, an admin asked me if anything I was doing would mess with the clock on a guest server.  I racked my brain for a moment and replied that I wouldn’t think so.  The admin didn’t think so either so he went off to investigate more.  I did some more thinking, and then went back and got some more information.  What exactly was happening to the clock?

In this case the server had a purposefully mis-set clock.  As in, it wasn’t supposed to read current time but it kept getting set to the current time.  VMware tools dawned on me, because there is a clock sync built into tools that has to be disabled.  We double checked but it was disabled. It made sense because we didn’t do anything with tools (no new or updated tools install).

So later that night I was playing around in my lab.  I recreated the setup as best I could. Installed a guest with tools, disabled time sync, and set the clock back a year and some months.  Then I started replication.  And instantly, the clock was set forward.

So it turns out that even if you tell the guest “don’t sync the clock to the host,” it will STILL sync the clock to the host in certain situations.


While I understand the rationale (certain operations have the potential to skew the clock, ergo syncing the clock up after those will help prevent ongoing skew) I really feel like if time sync is disabled, it shouldn’t sync the clock.  Ever.  Or there should be another check box that says “No really, never sync the clock.”  Nevertheless, I don’t work for VMware so I can’t tell them how to run their product.

In this case the fix is pretty simple though it does require downtime.  Shutdown the guest and add some lines to the .vmx:

time.synchronize.continue = "0"
time.synchronize.restore = "0"
time.synchronize.resume.disk = "0"
time.synchronize.shrink = "0"
time.synchronize.tools.startup = "0"
time.synchronize.tools.enable = "0"
time.synchronize.resume.host = "0"

Now it will really, really never mess with the clock on the guest.

This might be common knowledge to VMware admins but I had no idea and I suppose I’ve never dealt with a purposefully skewed clock before.