The Superman Conundrum

I used to collect comic books when I was a kid, so superheroes are close and dear to my heart, but just not as crazy about the expectation that technical support experts be held to the expecation of infallible like Superman.
I’ve both experienced and have discussed with other technical folks regarding this situation that, as of late and in all common sense, just doesn’t make any sense in the technical world.

I’ve nicknamed this technical environment situation the “Superman Conundrum”.   In my history I’ve  *survived* three companies who’ve treated their technical environment in this manner.  This is the kind of company where hardware, software, applications and processes fail on a consistent basis with the expectation that DBA’s and other support staff will come in and save the day/night/weekends….you get the picture…

The reason for the environment being in such poor condition is commonly due to the following:

• Lack of consistent, demanded maintenance schedule.
• Inability or refusal to replace hardware/software/code before it becomes an impacting problem.
• Inefficient resource head count to address problems pro-actively, placing the staff in a constant fire-fighting mode as neglected systems start to unravel.
• Poor planning of projects, (weak development/testing/production implementation phase.)
• An expectation when issues arise, (and often this is not in writing or even spoken, but when fires erupt and they say jump…) that someone will come in and do whatever necessary to save the day, (often employing duct tape and baling wire to get them through the demand to get the system back up and running after the impacting issue…)
• Lack of understanding from business management on what it takes to build a strong foundation to support the technical environment for the long run.

For a DBA in this type of environment, it can be both invigorating and terrifying-  kind of like being on the high wire without a net.  It’s thrilling to come in and save the day, to be able to bring back a system or save a process that everyone thought was doomed.  Users and Managers think you walk on water, so it is something akin to an adrenaline high.  If it starts to become a repeat performance…often, you then begin to realize, just how foolish and dangerous it is to depend on any human being to be this infallible just because something in the environment should have been automated/built with redundancy/mirrored/designed without such flaws and that’s when you recognize that missing net below you….  Management and peers outside the demands to be Supermen do not see the harm of the missing net, you are Supermen, what’s a little fall?  You are infallible, right?

Many of these environments also start to play what I like to call the “blame game”.  It first appears that the people involved are attempting to help ensure that processes are put in place to keep the problem from repeating itself.  The slight difference is that it results in simply pointing fingers at the “Supermen” they once praised, instead telling them what they did wrong, what they should have done, (in hindsight of course) differently to keep the problem from occurring, (and of course, regularly demands more checks, double checks, etc. to compensate for what flaws are in the environment/software/processes, etc assigned to, you guessed it- Supermen…) or just outright complain without even any suggestions.  When it comes to anyone who actually takes the time to suggest an option to correcting the problem at its source, (i.e. fix the code, replace the insufficient hardware, upgrade, tune, etc…) this is quickly and quietly turned down or refused due to time/resource/budget constraints.

There is a backlash condition post the blame game.  Little by little, the environment, if on it’s destined course, becomes more and more overwhelming for the “supermen”.  This is an exhausting situation, as personal life is intruded upon and professionally it’s not so great to constantly be in fire-fighting mode.  In turn, their morale is whittled away at by the blame game until the “supermen” start playing the “why even try anymore game”.  You begin to see the looks of hopelessness on the supermen faces.  It’s there when they walk in the office, it’s there with the first sarcastic joke of the day and it’s there when you receive email at 2am in the morning in regards to the latest fire they had to use their “super breath” on to put out.

Soon, the turnover starts.  The supermen know it’s coming, some may warn you, most won’t say a word and will just quietly leave.  The one thing you should know is that the technical world is a small place and even if they don’t let you in on the chaos that exists in your environment, that doesn’t mean they won’t share this information with peers in their field when asked.  This can be detrimental to companies when they are looking for new “supermen” and can’t figure out why they can’t get any solid resumes.  I have a “list” of companies that I refuse to interview with.  Trusted sources have listed them as poor work environments and I have surprised more than one technical recruiter when I’ve been aware of an odd quirk or two in a company that they later on verified.  Being a DBA is tough enough, so if we can avoid a poorly managed environment, we’ll commonly do so! 

So as a manager, what can you do to try to quell the “Superman Conundrum”? 

• Challenge the culture, this is a constant task that must be repeated.  When the business/management expects technical personnel to make “supermen leaps”, demand that the requester find a solid, long term solution by the next request or that it will be turned down.
• Be a buffer between your “Supermen” and the business.  Do not let them be placed in a “blame game” situation or take heat for what logically is a failure in design/hardware/process.  Work alone with the superman held accountable to find out what they think needs to change to deter the breakdown and push for a solid resolution.
• Demand full project plans and full project cycles that include padded development, test and user testing schedules.  Use the “Pay Forward” model to show value in time invested before technical projects go to production vs. the cost post going production for outages and breakdowns.
• Design, build and code for the LEAST OPTIMAL scenario.  Anything built for only the most optimal scenario should be expected to experience outages and endure high support cost in both man hours and upgrades.
• Always remember “You get what you pay for” and “Nothing is free”.  In this technical onslaught of open source and free tools, applications, etc. there needs to be an honest look at what you are really getting for the money.  If you require an enterprise environment and believe you can get by on the freebie version, you are likely lying to yourself and making unreasonable demands of your people.  You should not go on the sales pitch-  try the product out, it might live up to your needs and then again, it might not, so don’t go production with it the first week!

If you’re a DBA in this type of environment, one who, like Superman, thinks you are not just going to save the day, but going to stabilize and productionize it, I can tell you from going through this myself, it’s not a whole lot of fun.  You often feel like you’ve taken two steps forward only to be thrusted 2 1/2 steps back.  You may implement automation, request meetings, document processes and standards.  The problem always lies in that this is not where management’s focus is.  They are sure, that if you were just more perfect, more like Superman, then the environment would run just peachy.  It will always come down to either you or the database that is at fault and you must either change that culture of thinking or hope you have a manager around who can help fight this battle for you so you can simply do your job.

It takes a lot of power to take on this kind of culture fight, but heck, you’re Superman, right? 🙂