VMware vSphere – Guest clock, replication, and squirrels

****PPFFFFFFFFFFPFPPPFFFPFFFFPPFFFP****

That’s me blowing the cobwebs off of my blog. 🙂

Time marches on and things are always changing.  I too have recently made several changes both in my personal and professional life.  I am now a delivery engineer for CDI Southeast, though we are going through another transition period ourselves.  I am also doing a bit of retooling, attempting to become a little less storage focused and branching out into other areas.  Right now I’m trying to focus more on vSphere but also working with some DevOps stuff as well.  I also released a course on Pluralsight last October which I am very proud of.  If you are interested in hearing me talk to you about VNX for hours, I encourage you to check it out.

So, great things hopefully coming down the pipe for me in the future and I hope to continue to share with the community at large that has given so much to me.

In the meantime, here is a quick nugget that I turned up a couple of weeks ago.  I was doing a pretty straightforward implementation of vSphere replication (and I hope to do a series on that soon), but ran into an oddity which I initially wrote off as a squirrel.

A squirrel is a situation where I walk into your building to do an implementation, and shortly thereafter you (or your co-worker, or your boss) comes up to me and says, “you know ever since you got here, the squirrels have been going crazy outside.  What did you do?”

The squirrel can be anything really, but most of the time is comically unrelated to anything I’m actually working on. And sometimes it is just people messing around with me.  “Hey you know Exchange is down?  JUST KIDDING!”

Side note – I’ve been on site with a jokey client where something really did go down and it took us a minute or two to figure out that there was really a problem. 

I’m used to squirrels now.  Honestly they rarely have merit but I always take them seriously “just in case.”

In this case, in the midst of replicating VMs in the environment, an admin asked me if anything I was doing would mess with the clock on a guest server.  I racked my brain for a moment and replied that I wouldn’t think so.  The admin didn’t think so either so he went off to investigate more.  I did some more thinking, and then went back and got some more information.  What exactly was happening to the clock?

In this case the server had a purposefully mis-set clock.  As in, it wasn’t supposed to read current time but it kept getting set to the current time.  VMware tools dawned on me, because there is a clock sync built into tools that has to be disabled.  We double checked but it was disabled. It made sense because we didn’t do anything with tools (no new or updated tools install).

So later that night I was playing around in my lab.  I recreated the setup as best I could. Installed a guest with tools, disabled time sync, and set the clock back a year and some months.  Then I started replication.  And instantly, the clock was set forward.

So it turns out that even if you tell the guest “don’t sync the clock to the host,” it will STILL sync the clock to the host in certain situations.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189

While I understand the rationale (certain operations have the potential to skew the clock, ergo syncing the clock up after those will help prevent ongoing skew) I really feel like if time sync is disabled, it shouldn’t sync the clock.  Ever.  Or there should be another check box that says “No really, never sync the clock.”  Nevertheless, I don’t work for VMware so I can’t tell them how to run their product.

In this case the fix is pretty simple though it does require downtime.  Shutdown the guest and add some lines to the .vmx:

time.synchronize.continue = "0"
time.synchronize.restore = "0"
time.synchronize.resume.disk = "0"
time.synchronize.shrink = "0"
time.synchronize.tools.startup = "0"
time.synchronize.tools.enable = "0"
time.synchronize.resume.host = "0"

Now it will really, really never mess with the clock on the guest.

This might be common knowledge to VMware admins but I had no idea and I suppose I’ve never dealt with a purposefully skewed clock before.