Fair warning – this is not meant to be a technical article, but hopefully it will inspire you to better your organization! This is based on my observations for companies I have worked for and companies I have worked with.
Data center ghosts. Or relics. Or demons. Or zombies. You should know what I’m talking about, because almost every company has them.
Sometimes there are fascinating stories about them that read like something out of a Stephen King novel.
Oh yea don’t go near rack C-39! One time a site ops guy spent too much time fixing up cables in there, and the entire infrastructure went down. Some say he went mad and is now locked up in a velcro tie straight jacket…
Sometimes it is ancient hardware that people joke about…but always in a respectful way!
Yep this is the oldest server in the data center…takes up 24U of rack space and is less powerful than my cell phone. Ol’ Loudy McDiesALot we call him. Sure we roast marshmallows on his rear exhaust but he’ll outlive us all.
And worse sometimes you might not really know they are around you.
What that network hub on the desk in the corner? Naa that isn’t managing our entire inbound/outbound network. I’m sure if we unplugged it we totally wouldn’t lose entire connectivity to the outside world. I mean, we never have unplugged it because we don’t really know what it is doing…still don’t look at it too long.
Often there is a lot of pain in dealing with these entities. It takes a concentrated effort and a lot of buy-in by key players. They require a massive overhaul to remove, or a herculean migration effort to swap out. There are people in the organization who will fight it. Maybe they are stuck in the past. Maybe they are lazy. Or maybe they’ve been around long enough to see the scorched earth after the demon rears its ugly head – usually once you’ve seen the fallout of trying to correct a problem and causing an outage, you don’t want to be around when they push that button again.
It can seem like a “good thing” to just leave it be, but by not dealing with these entities head on problems simply cascade through the environment.
These ghosts in your environment are toxic. They are short term heart attacks and long term ulcers for employees. They make admins hate the infrastructure and the data center. They make managers and C-levels unsure of the stability of their enterprise.
They make any “new” people in your environment very dangerous. Whether that is someone like myself who is coming in to implement some new hardware, or just some new employee you hired on. Ghosts are especially problematic because they usually don’t have giant red warning signs on them. I don’t know about that Windows NT server you’ve got stowed away in the corner managing your database connections. The new guy doesn’t know that disconnecting the hub in the corner is going to take down your entire business. Unless you make that part of your onboarding process…in which case I’d expect that if you are bringing in some intelligent new hires they won’t be with you too long…
Some admins wear ghosts like a badge of honor. It has always been really strange for me to listen to someone say, “yessir, this widget is my responsibility and IT CAN NEVER GO DOWN!” with a big beaming smile. I mean, I get what you are saying, but all I’m hearing you proudly declare is, “yep, I first designed and now admin this terrible solution!” You can design a SERVICE to (likely) never go down, but not a SERVER.
It is no exaggeration to say that there are hundreds of good solutions out there for virtually any problem. As much as orgs like to think they are beautiful, unique snowflakes (and in some tiny ways almost all of them are) most problems fall into general buckets and have been tackled before. By not availing yourself of these solutions it is simply doing a disservice to your company. Solutions come in many shapes and sizes (which mostly just means complexity and cost), from new hardware, to updated methodologies, to new ways of thinking like automation or public cloud (let someone else worry about hardware problems).
The pay off for eradicating your ghosts is immeasurable. The infrastructure will be stable, but it is something even more than that. It is the admins being able to stroll in to work in the morning, grab their coffee, and saunter over to their desks without dreading what horrific email alerts are going to greet them. It is the data center tech who is no longer terrified to open up a rack door. It is a manager having a firm understanding of what is in his environment and all the redundancy he could ever want.
Nobody wants to work in, around, or above infrastructure that is fragile. Find the fault lines and seal them up permanently. Make it your New Years resolution. Your stress levels will thank me.