Category: DBA Rants


It’s been such a long time since I posted anything on my blog.  I had this post, last save time was on Dec. 13th, which is ironic considering the subject and what ended up happening that day.  I’m going to ramble a bit, but there is a message here folks… :)

I’ve always been a strong believer of the old Hebrew saying, “Fall down seven times, get up eight”.  I’ve lived by it, which most who know me well enough, realize that it’s not just part of my DBA methodology, it’s also an integral part of my life.  When I was in my 20′s, I wasn’t a techie at all.  I was working as an auditor for a company in Colorado when I started experiencing strokes between the ages of 21-26 years of age.  I had five documented strokes, that until a very dedicated and brilliant Neurologist diagnosed the cause of, I was told no one could deter from occurring and that I’d be blind before I was 40.  Through this doctor’s excellent care and help from a Rhuematologist and Hematologist, they figured out the odd auto-immune issue, combined with a natural tendency to have very low blood pressure so that I was stroke-free for almost 19 years.  I’m now onto my way to turning 46, was left with 48% of my visual field gone, (eyes are fine, it’s the striate cortex of the right hemisphere that isn’t able to “translate” data any longer, showing up as damaged in MRI scans.)  I have very, very few, long-term memories from about the age of 19-26 years of age.  All other damage that occurred was addressed with physical therapy and speech therapy.  If you consider the years that I am impacted memory wise, yes, had to start looking for a new career, wasn’t like I could perform my previous one any longer.  I started first by selling shoes, then computers, which outside of being a woman in her late 20′s selling computers, which meant I did very, very well, I found out I had a knack for software.  That was 1995 and by 1997 I was doing desktop support for a large telecom company who ended up putting me through my Oracle 8 certification classes and made me a DBA by 1999.

Now with the years gone by, children born, (by oldest just turned 18, youngest is 12 yrs old)  I thought I was pretty safe and had no worries.  I had a couple surgeries and lasik without any issue and was secure in the fact that I understood everything I needed to know about my condition, knew it hindered me from receiving care as much as it helped me to know when I did enter the hospital what to keep doctors from doing to me that might kill me, (no, you can’t thin my blood…no there is no concern about clots, I have a BLEEDING problem, not a clotting one, pretty much get the heck away from me before you kill me…)  I have an excellent primary care physician that I appreciate for his knowledge and natural skill in the medical field.

I went in for surgery  on November 10th.  It was unexpected, but I checked into a hospital that was very aware of my medical history, even went over it with me before they checked me in.  As it was unexpected, they had decided to do exploratory surgery due to my abdominal pain and discovered I had a ruptured ovarian cyst and simply removed it all in one swoop to avoid any complications.  The standard procedure when many have any type of bleeding, is for the anesthesiologist to lower your blood pressure to lessen bleeding.  Many aren’t aware of this- I know I wasn’t.  Unfortunately, my old medical condition surprised the anesthesiologist as my blood pressure “dipped” twice, very low and upon recovery, I found I had some new areas that were missing, including “spots” across my central right visual field that were just plain annoying.  My vision seemed a bit blurry, too, but when I squinted to clear the vision, which would commonly work for anyone who knows what it’s like to have blurry vision, it just didn’t improve.  It should be noted, as with any of my children’s births, previous surgeries, again, didn’t partake in any pain medications.  I have a high threshold for pain and found no need of them.  I bounced back quickly, but the vision issue and what caused it, was of high concern.

My primary care physician followed up with an eye exam, (always have to prove it’s not the eyes first before you go to the brain…) and then onto an MRI scan once that came back as “eyes all good!”  I received a great set of glasses that addressed the distortion and for the first time in my life, the unending glare that happens to many with visual field damage.  With the new glasses, I was able once again, to drive at night.

During this follow-up time, I was the lucky recipient on Dec. 13th of kidney stones and had to go into the same hospital ER for these.  I had been directed there, even after the surgery, as the best place to go due to a urologist on site.  After a visit that shall just go down as the “one of the worst care given at a hospital in my life and lucky I got home alive” situations, I then had to see my doctor for a referral to a urologist to actually get the kidney stone broken up, as my ER trip just ended up in me going in with severe pain and then being released in severe pain, drugged up and in worse shape than when I entered the place.  My MRI’s are not back yet and we go in to discuss my procedure to break up the stone with the urology specialist.  The nursing assistant comes back to the room and says, “I’m sorry, but I can’t get anyone to perform the procedure on you without a medical release from your doctor due to the assumed stroke from last month.”  Which translated to me, returns as “Yes, we know what needs to be done, but due to a mistake that no one in the medical industry will admit to, no one will now perform the procedure you need to have done now unless someone else takes responsibility for any mistakes we might make.”

There are times when you realize that figuring out some medical mysteries are more threatening to your future health than just addressing the results and going forward to get the care you need.  My primary physician went after them the next day and we took the whole stroke issue off the table immediately.  I ended up going a full week with a kidney stone that required a surgical procedure because so much of the medical profession is too worried about red tape, managing pain vs. addressing the problem and forget about caring for the unique patient in front of them.  I will never give up my primary care physician.  I have no doubt Dr. Michael Archer will keep me alive for a long time.  His “strong-arming” of the medical staff at different locations not only got me what I needed, it also made me realize I needed to request copies of my medical records to see just how poorly I was cared for.  If I have to return to an ER or hospital, I’m pretty sure all bets are off.  You can be sure I will never return to St. Anthony’s North Hospital and I recommend anyone in the Northern Denver area to avoid it.  I am now recovering at home after the surgical procedure being performed on the 23rd of December and have one more follow-up visit to complete the process.  The delay in my care put me more behind, caused me more lost time in work- thank God it was the holidays, first time I’ve ever actually take pain medication, yes, kidney stone procedures beat out child birth and other surgeries I’ve had for amount of pain.  It resulted in a request for more recovery time-  It would have been much more difficult had I not worked for a great company like Enkitec and worked from home.  I’ve lost out on almost six weeks of writing time on the EM12c book, playing lead author and helping out with other areas for the book, RMOUG Training Days work that requires my time has gone pretty well, thanks to Tim’s involvement, he’s often been able to answer issues for me when I’ve been laid up.

We, as patients need to be ever vigilant and constantly push for the care we deserve from the medical profession.  The damage that can be done because someone didn’t take the time to read a full chart or a nurse didn’t have time to take a full medical history of the patient or understand the patient’s unique physiology, pain thresholds, etc. put us at risk. Removing the humanity from health care is true cause for most of our failures in this area of American healthcare, which we in turn have just responded with more insurance choices to address.

So my final message is this-  The medical profession has a few bright stars in it and everyone else is just following what it says in a medical book on an issue or the policies of a hospital.  To protect your loved ones when they enter a  hospital is an incredibly demanding task.  Ask every nurse, every doctor to explain what they are doing and if anything doesn’t function, (the call button, the phone, etc…) demand that they correct the situation immediately.  If you aren’t getting the care you need, demand to see the facilities patient rights documentation and file complaints.  Everyone deserves to receive the care that will help them live a long and healthy life.  If the medical staff can not perform their job, find someone else more qualified to do so.

Jessica Ridgeway

This is my first blog post that isn’t on a technical subject, but this one has been rolling around in my head the last couple days…

Jessica Ridgeway’s home is not far from my own.  My ex-husband owned a home just four blocks from hers and my dogs played at the dog park in the open space off of Simms St., two blocks over.  I’m 99% sure that Jessica went to the same McDonalds that my youngest son goes to each week before heading over to Boy Scout’s on 100th and Wadsworth and I just shopped at that King Soopers in the same location very recently.

I can’t imagine the pain the family is going through and yes, my children are on a shorter leash, even though they are in their teens.  These are all good neighborhoods.  There is a golf course just off of 108th in Jessica’s neighborhood and like I said, a really cool open space with a dog park just west of it.  North Metropolitan airport is less than a mile on Simms and then you can turn east and head towards the Oracle campus and roads that lead to Flat Irons Mall.

What spurred this blog post?  I’ve been obsessed with this story as is the rest of the country and as it’s close to home, maybe even more so.  As I’ve read the stories, comments at the bottom of many of the news sites have caught my attention from time to time.  We wonder about the monsters that take children and yet we can’t even act with sympathy when leaving comments in news stories.

Some of the less offensive ones, that I can list here:

“The Republicans are to blame..”

“Obama must have taken her…”

“Who would take her, she’s too ugly!”

“I’ve noticed that all parents of abducted kids are obese…”

What is the matter with people?  We ask for kindness, sympathy, assistance and for a better world, yet we can’t even have the common decency to act appropriately in a comment section of a news story?

My faith in humanity is again questioned…

 

Yes, still catching up on Blog posts… :)

Monday is the official start of Oracle Open World.  I planned to be onsite at the Enkitec booth from the time the exhibitors hall opened and I was a few minutes early.  I hadn’t received an exhibitors ribbon, (one of the few missing from my extended display from my badge…) and so I waited patiently and chatted with a couple of folks who had attended one of my sessions the day before.
Upon entering the exhibition hall, you realize why vendors flock to this conference in hopes of promoting their company.  The south hall of Moscone was a stampede upon the opening of the doors and I have no doubt the other exhibition areas received the same welcome by attendees at those entrances.
I proceeded to spend a few hours chatting with folks at the booth and scanning their badges for an opportunity to receive one of the books we offered in drawings twice a day.
Igloo a break from time to time and headed over to the Oak Table World event.  This was a great set of specialized talks put on by the wonderful Oak Table folks, sponsored by Delphix, Miracle and others in the great venue of the Children’s Discovery Museum.
I had the opportunity to catch a took on Monday by Tanel Poder and Doug Burns before sitting down and locking in the keynote speaker for RMOUG Training Days 2013.  I had also received a  nod from another to announce the keynote for us, as well.  During all of these days, I also continued to work on another task as the database track coordinator for KSCOPE, serving ODTUG and filling the seats on that committee.  I want to thank those that have signed up to help- Galo Balda, Martin Berger, Randy Johnson, Don Seiler, Kent Graziano and Bobby Curtis.

Tuesday was pretty much a replay-  I went to the booth, spent time with the Oak Table sessions, then attended the Tweet Meet.    It was a good time and I was able to take some time out and interview with the Oracle Social Media Network group.

We ended the evening out with a few friends at the Stinking Rose, a garlic intensive restaurant.  I can honestly say that a good time was had by all and one of the best prime ribs was consumed… :)

How many sessions did I attend so far, outside of the Oak Table ones?  None, there’s just not enough time to meet with all those that want a bit of your time, spend the time needed at the booth and complete tasks on the side.  I have folks asking me for info on 12c database and the only time I’ve spent is investigating EM Express!

Tomorrow is another day-  Wednesday, to be exact…

As many know, I’ve taken on the mantle of Training Days Director this year for RMOUG’s Training Days 2013.  I don’t have much time in the month of August, but wanted to remind everyone that the submission for abstracts is open and can be found on the main RMOUG website page.

We are also looking for volunteers for the 2013 event.  Anyone who is interested, please see the website for details and the volunteer page.

I’m very thrilled to say that I think this will be the best Training Days conference yet!  I’ve attended a number of conferences this last year and I can verify for anyone that RMOUG’s is the “best bang for the buck” both in cost and for the amount of quality content we offer!  The conference has pretty much extended to a full three days now, as OTN Developer Day has enhanced what was once just our University Sessions on the first day.  Training Days is now scheduled for Feb. 11th-13th for the 2013 year, moving us to a Monday-Wednesday line up before the President’s Day holiday weekend.

Registration will be opening in the next week and attendees can already look forward to some nice extras in the registration options.  Cary Millsap will offer you the opportunity to sign up for his “Mastering Trace Data” class on the 11th of February.  I’ve taken this class in both its extended and one day formats and can tell you it’s well worth the extra cost.  We will also offer our excellent University sessions and although I can’t spill much info yet, I’m thrilled with the line-up that is already forming and with the small cost of these sessions, they are worth the chance to get in on these limited seating opportunities!

The 11th is also the much anticipated OTN Developer Day from Oracle.  This year Kris Rice and the Oracle Experts have not only created a new line of workshops for folks to participate in, we are going to include more for the DBA by having a workshop by Maria Colgan.  She’s one of my favorites and anyone who’s attended one of her presentations should really look to this great workshop offering on the first day.

We are already seeing a number of great abstracts submittals.  We expect over 300 this year and will need to trim that down to around 175 presentations in the two days of RMOUG’s main training event.  We’ve re-organized the tracks to better suit the changes in the database and development arena, including Big Data, Hadoop and Exadata into more advanced database topics with the “Database Deep Dive” track, along with separating out “Database Tools” from “Application Development” to give them more defined areas to the Database Technologist.

Our MySQL SIG should bring us more presentations on this highly successful group.  Their Quarterly Education Workshop sessions are packed, so I expect a high turn out at Training Days, too.

I’ve been thrilled with the feedback, suggestions and comments I’ve received from those who have attended, presented or volunteered.  This information has been very valuable in letting us know what to keep, what to change and where we can improve this fantastic conference.

There are a number of web/social media events coming up in preparation for the Training Days conference.  Keep an eye out for these by following @RMOUG_ORG on Twitter, using the #RMOUG hashtag, liking us on Facebook and joining the Linkedin Group under Rocky Mountain Oracle User Group

If you have any questions or just curious about how this great conference comes together, please drop me a line at TrainingDaysDir@rmoug.org or at dbakevlar@gmail.com.

Thank you again to everyone that have been so supportive to the incredible Rocky Mountain Oracle User Group Training Days Conference!

If you desire a subject that will invoke deep passion and
often combined with disgust from a group of DBA’s, disaster recovery is the
one.  It is the subject that rarely we
feel our butts are not out there hanging, no matter how much we’ve attempted to
secure our environment.

I’ve observed a consistent flow of articles, conversation
and email discussions on the subject and it is apparent that rarely is the
business as aware as the technical specialists, (aka the DBA) of just how
vulnerable their environments are.  Rarely
are the budget dollars allotted to the task of insuring that systems have the
proper disaster recovery hardware/software in place and/or testing performed in
a regular basis.

It’s easy for the business to see the value in the
production systems.  They create revenue
and their value is equal to the dollars they produce.  Development and test are more difficult for
them to understand, but most times, they can be justified the first time
production is undermined due to development or testing being performed in one
of them…:)

We now get into backup and recovery.  How many backups are impacted by 24X7 shops,
where the only thing viewed by the business is impact to revenue by having to
allocate resources to backing up production and placing it on disk/tape that
offers them no value to that revenue.
Yes, the DBA’s and technical management argue, “All it takes is one loss
of production and you will be thankful that we have that backup…”  Until that day comes, many business’ rely on
the robust nature of Oracle, the hardware it resides on and the technical
expertise of the folks they’ve hired to keep it running and never having to
rely on those backups.   DBA’s commonly fight on a regular basis for time
to allocate to testing, hardware to test recoveries to and explaining to the
business why it’s important.  The
business again looks on this as time that could be better allocated to creating
faster systems to create more revenue and again, impact to what the business is
there for- creating revenue.

The next level is then disaster recovery.  All DBA’s know this is the final
gauntlet.  We pray to the DBA Gods hoping
for a technical manager with the gift to motivate, sell and help the business
to understand why having standby’s of production databases to keep revenue
flowing in case of primary production going down is important.  We are willing to sacrifice small animals in
the name of a secondary data center for disaster recovery testing.  We want to know how long the business will be
down in case the unthinkable does happen and if all the documentation on what
it takes to create production will actually work when we do try to recreate it.  We would also like to have that answer
demanded of us when the unthinkable does happen and upper management is sitting
in front of us asking, “So HOW long are we down for???”

This is not rare, this is not uncommon, it is all too often
the norm for most DBA’s in the business world.
IT Managers, Network Administrators, Database Administrators often
battle day in and day out, not just for what they need to provide the growing
demands of the business, but what the business needs to survive in case of
disaster.  Our jobs are not just to
provide you with production, but to provide you with an ability to sustain your
business when the unthinkable happens.

A blog post by Simon Cooper sent up a reminder of a subject that has been at the forefront of my mind for the last couple weeks and is a follow up to my blog post The Superman Conundrum.  I’d had lunch with a previous coworker from years ago about  less than stellar management behavior directed towards him as an employee and also spoke last week to another previous co-worker about challenges in her current workplace.  I’m pretty content with how I am treated at my company.  Even though the company does not comprise of techies, they at least attempt to understand the demands of the IT industry, giving folks off time during the day when appointments and personal issues arise, knowing that we often “pay-it-forward”  within a week or so in after-hours work that is common for most operations support roles.

Simon’s post discusses what should be common sense in a business-   Give employees what they need to be as productive as possible, pick your battles and you will reap the benefit in return for the business.  You want loyal employees-  ones that come to work, look forward to coming to work, drive for new challenges with gusto and always give their best to the company that hired them.  Unfortunately, their morale and loyalty is often tested by management and management practices that have no common sense behind them.

The simple policy of treating your employees well and in return you will receive their best is a rarity in the work place.  Having this policy, with the understanding that if abused, then there was a mistake made in hiring them and then the company has the  opportunity to replace the individual should be in place everywhere when it comes to Information Technology.  Instead we commonly hear of companies mistreating skilled professionals and in return, spend ridiculous amounts of time repeatedly in the recruiting phase due to consistent turnover which is a huge waste of any company’s time and money.

The employees that a company such as this will retain are either:

1.  possessed of little self-esteem, which in turn shows itself in lack of initiative and personal growth, (doesn’t serve any business well in the long run.)

2. Aren’t shining examples of good employees anyway and are perfectly happy mistreating the company as they are mistreated, (also an additional burden to those that have stuck around with the low self-esteems…)

I think most techies will agree with me-  Information Technology is a peculiar industry.  Many of the folks that fall into the field have a broad range of skills, often quite intelligent and unique personality traits vs. other areas of business.  Since most companies prefer to promote from within the department that the group will be managed from,  this results in techies being turned into managers.  Where the problem of management promotion being based on control issues more often than true leadership skills, this is never more apparent than in IT.  Sorry folks, it’s just a major fact as it is a problem in business overall.  We say, “Our Manager is being unfair and effecting morale!” and someone is going to undeniably say, “It’s a job, you’re not supposed to like it, man up, (or woman up, depending on the case)!”

The deal is, most companies should be paying more attention to these types of complaints.  Often employees aren’t too keen to complain in fear of reprisal to begin with, let alone stating what the REAL problem is.  Keeping an employee happy and productive is actually quite easy and involves very little effort than simple respect.  It baffles the mind how often it is more important to a manager to “be right” than to do the right thing for the long haul.

Take the following scenarios:

Jennifer has worked a year for her company and has performed well in her manager’s eyes.  He relies on her often for last minute jobs and she has always come through.  She suddenly needs the afternoon off , her son is sick at school and has to pick him up. Jennifer’s boss asks her to take sick time to do so and follows up with an email asking her to correct the time in the system because it was short what he feels was a 1/2 hr from what she really took off.  Jennifer has only five sick days a year and often performs after-hour support for the company, interrupting her personal life to address issues, losing time with her kids and spouse to address them.  There is no “comp time” that she is given for this work-  it is just expected from employees at her company.

John has been employed for years by the same company and is in good standing with his current manager.  John lost his son in a car accident three months ago.  His family has had a very difficult time with the loss and John has a large cache of vacation time that he is pestered by HR to take.  When his wife has an extra difficult day due to the loss of their son, he has requested to take a vacation day.  John now has an interim manager who refuses to OK the vacation day and chooses to leave  John unpaid for those days.  John is the sole provider for his family and this is causing even more stress in his personal life.  John also performs after hour support and rarely is offered any type of comp time for this time away from his family.

These above scenarios were both opportunities to build employee loyalty but poor management choices, driven by control issues have ended up costing employee loyalty, morale and in the end, company time and money.

The company has taken the time to hire professionals-  treat them as you would wish to be treated if you are their managers.   Make them realize the value they have to the company as they ARE what comprises the company.  I have seen this wonderful pay-it-forward attitude in action when I worked for the Parent Company, formally known as eToys, Inc.  They took four months to hire me and once there, I walked around and was impressed to realize the quality of the folks in every department.  I give high kudos to what a great manager Steve Ridley, the director of technical operations, along with the Manager of the DataWarehouse group, Greg Sitzman and our CTO, Chris Cummings all were.  Steve, as my direct manager, always challenged me, treated me with respect and if I had an emergency come up or needed an afternoon off, he simply stated that my family must come first and he meant it.  I came in everyday wanting to work there, do my best and you better believe I put in any after hours support they needed from me.   I made sure I was always available and had a sense of pride for the company that I belonged to.  This was a direct result of feeling valued and important to the company for the part I played.  They understood that employee loyalty was earned by treating everyone with respect, no matter what part you played on the company ladder.

“Jennifer” and “John’s” scenarios above could easily have been turned into a win for everyone.  If “Jennifer’s” manager had supported her situation, stating, “I always appreciate everything you do for us and the after-hours support you put in, please, go take care of your family and I will see you tomorrow….”  how much time just in emails, time entry and frustration could have been saved to the company?  If “John’s” interim manager, no matter what his personal management style was, had put himself in “John’s” shoes for just a moment and said, “John, I know you’ve had a rough time the past couple months and I know you already have an agreement in place with your previous manager.  Please, go tend to your family’s needs…”  how much could he have earned John’s loyalty if he was interested in making that interim position permanent and in making John a more productive employee the next day when he was back after taking care of his family?

I hear time and time again from different companies and even through recruiters how exhausted certain companies are of finding good people, but I also hear from impressive candidates looking for employment stating that they won’t even phone interview with certain companies due to this type of behavior.  Before any company starts complaining about how lacking the hiring pool is, it might be a good idea to look internally and find out if you are losing perfectly good people due to the lack of care and feeding that should be common sense so that a company is able to retain those valuable resources once originally hired.

 

 

I used to collect comic books when I was a kid, so superheroes are close and dear to my heart, but just not as crazy about the expectation that technical support experts be held to the expecation of infallible like Superman.
I’ve both experienced and have discussed with other technical folks regarding this situation that, as of late and in all common sense, just doesn’t make any sense in the technical world.

I’ve nicknamed this technical environment situation the “Superman Conundrum”.   In my history I’ve  *survived* three companies who’ve treated their technical environment in this manner.  This is the kind of company where hardware, software, applications and processes fail on a consistent basis with the expectation that DBA’s and other support staff will come in and save the day/night/weekends….you get the picture…

The reason for the environment being in such poor condition is commonly due to the following:

  • Lack of consistent, demanded maintenance schedule.
  • Inability or refusal to replace hardware/software/code before it becomes an impacting problem.
  • Inefficient resource head count to address problems pro-actively, placing the staff in a constant fire-fighting mode as neglected systems start to unravel.
  • Poor planning of projects, (weak development/testing/production implementation phase.)
  • An expectation when issues arise, (and often this is not in writing or even spoken, but when fires erupt and they say jump…) that someone will come in and do whatever necessary to save the day, (often employing duct tape and baling wire to get them through the demand to get the system back up and running after the impacting issue…)
  • Lack of understanding from business management on what it takes to build a strong foundation to support the technical environment for the long run.

For a DBA in this type of environment, it can be both invigorating and terrifying-  kind of like being on the high wire without a net.  It’s thrilling to come in and save the day, to be able to bring back a system or save a process that everyone thought was doomed.  Users and Managers think you walk on water, so it is something akin to an adrenaline high.  If it starts to become a repeat performance…often, you then begin to realize, just how foolish and dangerous it is to depend on any human being to be this infallible just because something in the environment should have been automated/built with redundancy/mirrored/designed without such flaws and that’s when you recognize that missing net below you….  Management and peers outside the demands to be Supermen do not see the harm of the missing net, you are Supermen, what’s a little fall?  You are infallible, right?

Many of these environments also start to play what I like to call the “blame game”.  It first appears that the people involved are attempting to help ensure that processes are put in place to keep the problem from repeating itself.  The slight difference is that it results in simply pointing fingers at the “Supermen” they once praised, instead telling them what they did wrong, what they should have done, (in hindsight of course) differently to keep the problem from occurring, (and of course, regularly demands more checks, double checks, etc. to compensate for what flaws are in the environment/software/processes, etc assigned to, you guessed it- Supermen…) or just outright complain without even any suggestions.  When it comes to anyone who actually takes the time to suggest an option to correcting the problem at its source, (i.e. fix the code, replace the insufficient hardware, upgrade, tune, etc…) this is quickly and quietly turned down or refused due to time/resource/budget constraints.

There is a backlash condition post the blame game.  Little by little, the environment, if on it’s destined course, becomes more and more overwhelming for the “supermen”.  This is an exhausting situation, as personal life is intruded upon and professionally it’s not so great to constantly be in fire-fighting mode.  In turn, their morale is whittled away at by the blame game until the “supermen” start playing the “why even try anymore game”.  You begin to see the looks of hopelessness on the supermen faces.  It’s there when they walk in the office, it’s there with the first sarcastic joke of the day and it’s there when you receive email at 2am in the morning in regards to the latest fire they had to use their “super breath” on to put out.

Soon, the turnover starts.  The supermen know it’s coming, some may warn you, most won’t say a word and will just quietly leave.  The one thing you should know is that the technical world is a small place and even if they don’t let you in on the chaos that exists in your environment, that doesn’t mean they won’t share this information with peers in their field when asked.  This can be detrimental to companies when they are looking for new “supermen” and can’t figure out why they can’t get any solid resumes.  I have a “list” of companies that I refuse to interview with.  Trusted sources have listed them as poor work environments and I have surprised more than one technical recruiter when I’ve been aware of an odd quirk or two in a company that they later on verified.  Being a DBA is tough enough, so if we can avoid a poorly managed environment, we’ll commonly do so! 

So as a manager, what can you do to try to quell the “Superman Conundrum”? 

  • Challenge the culture, this is a constant task that must be repeated.  When the business/management expects technical personnel to make “supermen leaps”, demand that the requester find a solid, long term solution by the next request or that it will be turned down.
  • Be a buffer between your “Supermen” and the business.  Do not let them be placed in a “blame game” situation or take heat for what logically is a failure in design/hardware/process.  Work alone with the superman held accountable to find out what they think needs to change to deter the breakdown and push for a solid resolution.
  • Demand full project plans and full project cycles that include padded development, test and user testing schedules.  Use the “Pay Forward” model to show value in time invested before technical projects go to production vs. the cost post going production for outages and breakdowns.
  • Design, build and code for the LEAST OPTIMAL scenario.  Anything built for only the most optimal scenario should be expected to experience outages and endure high support cost in both man hours and upgrades.
  • Always remember “You get what you pay for” and “Nothing is free”.  In this technical onslaught of open source and free tools, applications, etc. there needs to be an honest look at what you are really getting for the money.  If you require an enterprise environment and believe you can get by on the freebie version, you are likely lying to yourself and making unreasonable demands of your people.  You should not go on the sales pitch-  try the product out, it might live up to your needs and then again, it might not, so don’t go production with it the first week!

If you’re a DBA in this type of environment, one who, like Superman, thinks you are not just going to save the day, but going to stabilize and productionize it, I can tell you from going through this myself, it’s not a whole lot of fun.  You often feel like you’ve taken two steps forward only to be thrusted 2 1/2 steps back.  You may implement automation, request meetings, document processes and standards.  The problem always lies in that this is not where management’s focus is.  They are sure, that if you were just more perfect, more like Superman, then the environment would run just peachy.  It will always come down to either you or the database that is at fault and you must either change that culture of thinking or hope you have a manager around who can help fight this battle for you so you can simply do your job.

It takes a lot of power to take on this kind of culture fight, but heck, you’re Superman, right? :)

As busy as I am these days, I am seeing a light at the end of the tunnel, (no, it’s not a train!)  We have a new DBA training that is doing bang up job and I do believe there is some lull in the demands of our busy season. As a database administrator, I’m happiest when I have a number of demanding tasks, along with mysteries to challenge me that often, the business isn’t even aware of the level of importance it is to having resolved until I’ve implemented the resolution and they have reaped the benefits. 

I have two solid managers that allow me a wide berth to allocate many of my own tasks, along with the ones that they designated high priority.  I have quite a wide bandwidth and my ability to multi-task at a dizzying degree allows me to focus on more tasks than most people are comfortable with.  I’m a very low-maintenance employee, so it is rare for them to have to come smack me with something hard for not working on what I should be, (although when I’m telling one of them what I think he needs to hear than what I think he wants to hear, I’m sure he would like to hit me up-side the head just for the fun of it…:))

One of these managers is gracious enough to call me his “pinch hitter in the ninth inning of the world series”.  As for many business’ that world series is now-  we are exceptionally busy, have been building to this time in the year when demands run high and systems have to perform top-notch.  This requires all of my energy, time (and often brain cells) to ensuring I’m in top-notch, gung-ho, fire-fighting mode.  The demand to place my project DBA skills on the back burner during these crucial service times where quick-thinking, ability to assess situations at blinding speed and low impact, high resolution skills are essential.

As rewarding as this is, (and shows you just what you are made of!), I’m looking forward to the opportunities ahead to bring my project skills back to a fore-front.  A quality technical project, well done is something to take great pride in.  Once the season slows a bit, I get to work with new technology, test and implement new projects that will offer our business more opportunities to grow that we identified during the last number of months.  To return to new feature and design projects that are put on hold to ensure that priorities are met for the business at it’s high time, is a nice change of pace and widens the area of interest for any technical resource.

I am about to take on a number of projects involving 11g features with ASM, OEMGC, data guard, Apex, partitioning and parallel execution.  I have another set of project work that involves SQL Server reporting services new features, including work that bridges to our Oracle environment and work more with our MySQL environments. This is when a DBA gets to stretch their technical legs and learn new features and gain knowledge that is put on hold while supporting the current production environment. 

While working on these projects, research will need to be performed to ensure we are making the best decisions every step of the way.  This then allows another aspect of education that we rarely get when we are in fire-fighting mode-  learning in a slower, more controlled environment vs. hurried and very specific knowledge to solve a specific problem that is creating the “fire”.  Each of these decisions will be carefully discussed with the DBA team and documented.  No hurried trouble-shooting document added to handbooks or emails sent with “up to the minute required” information.  These are quality documents that contain all of the data that is needed to support the project work being performed, including the goal, scope and requirements for the project, (and should for any company’s project!), not just the technical details behind the project.

As the adrenaline rush ceases and demands/hours come back more in line with where we all like to be,  there is a sense of satisfaction knowing that the business needs were satisfied and that we were able to deliver what people needed to be successful in their own positions to offer the company more success in turn.

As a DBA, you can’t ask for a whole lot more… :)

This question seems to pop into my mind consistently over the years as a DBA.  I’m a “build it right or don’t build it at all” kind of DBA, but due to my gift for finding problems and fixing them, I find myself more and more often performing the second build on processes/procedures/designs, which I often would like to avoid.  I went through this repeatedly at a previous shop and it’s still fresh in my mind, even today…

Don’t get me wrong-  I think it’s a noble cause when you first come into a new shop and it’s either been neglected or didn’t have the DBA support it may have needed and the update in TLC is always appreciated, but if it’s something I’m revisiting because of a lack of requirements or a production change that wasn’t thoroughly benchmark tested, I find me getting a little cranky-  (usually with myself since I expect that I should be able to read minds at this point! :))

I believe to utilize your DBA and development team efficiently, not wasting money on resources, building “right” is essential.  Designing the initial database, as well as future development should be reviewed repeatedly to verify you can answer the following questions with a resounding “YES!”:

  • Is the main schema design scalable and easily adaptable to our business?
  • Does the daily/weekly/monthly processing require no manual intervention by support staff?
  • Have we performed capacity planning and does the hardware meet the foreseen demand and growth of the environment?
  • Has the schema owners been secured and application security been set to the DBA satisfaction?
  • Are all database files secured on the server from access to anyone outside the database group?
  • Are naming conventions documented and followed in the database environment?
  • Have code reviews been adopted, new code/processing plan meeting requirements been adopted?
  • Have different groups been assigned responsibility for the aspects they must be accountable for that will impact the database environment, (i.e. application support on call for apps, network admins for network issues, etc.)

The companies I’ve experienced DBA’s overstressed and over-worked in the past are commonly places who can not answer yes to the questions above.  Without these requirements in place, the databases become kin to a house with a foundation made of Popsicle sticks, (often built on a side of a mountain prepped for a future mudslide, BTW…:)) 

If a DBA is not there for the initial build of a database, as often is the case, and is just the “unlucky home buyer” of the database, then there is a unique opportunity in front of you to attempt to correct and change the culture that has built this rickety structure. Changing a culture is a difficult challenge-  I won’t try to pull the wool over any one’s eyes.   It is attainable and if you have management support, you will already have a huge head start! 

If you do have management’s support, what steps should you take to stabilize an environment that is expecting it’s DBA’s to be infallible instead of it’s databases?

  • Communication-  Demand daily meetings.  10-15 minutes to discuss what is happening and who is doing what.  Often folks live in their own little world and are unaware of how many folks are impacting the database, thinking there’s in the “one little process” requiring manual intervention.
  • Automation-  What can be automated?  Anything and everything that requires manual work or intervention by a DBA or developer in production should be stopped as soon as possible.  Make it a priority to automate it with procedures, scripts, crons, whatever it takes, make it hands off!
  • “White Noise”-  This is what I call all the informational emails or “fluff” in important emails that can mask or make someone miss what we really need to know about, (like failures!)  Remove emails that are just to inform people processes/jobs “have completed”.  If they are concerned about failures, create scripts on secondary servers to monitor for the main ones, but all email should be “just the facts” and pages should be narrowed down to production issues that require a DBA to address.
  • Worst Case Scenario Development-  AKA, developers and DBA’s should develop all processes with the lightest footprint on the database.  Develop the code with the idea it is going to always be running against the heaviest load on it, the most limiting resources and the least amount of time requirements.  This is the code that will last-  this is the code that will not demand to be revisited by the DBA or developer in the near future, (we’re hoping we can have code that won’t have to be revisited at all, remember? :))
  • Database=Two year old Toddler-  What does this mean?  For me, this means I am either the  DBA Mommy or often feel like a daycare provider.  Poorly developed databases are like toddlers and they will throw tantrums-  A LOT!  I need to be vigil, I need to have the resource time in my schedule to spend time with them, monitor their behavior and know when something is wrong with them.  Now, if you build them right, they will mature quite nicely and require less time, but please remember, the teenage years are often still ahead of us.
  • 500,000 Miles without an Oil Change-  This is how some companies run their databases.  Down the road, 80 mph, and then ask their DBA’s, “Can you change the oil while we continue down the road?  Oh, and while you’re under the hood, could you replace the transmission with a new one, too?”  Designate maintenance windows-  they should be a priority, not an after-thought.  Understand all that is required daily in automated maintenance to ensure the system continues to run, too-  statistics collection, backups and cleanup.  These are required for the health of the database, not just for the DBA’s sanity.
  • Make a List, Check it Twice-  The last task is the hardest.  Keep an up to date list of what makes up that “foundation of popsicle sticks”.  What are the design/process flaws in the environment that need to be corrected to build a sound, solid database and design small projects to correct them.  This will pay-forward in so many ways.  The goal should be to address objects/processes in the system that are not scalable with the database or require intervention due to design flaws.  Selling these projects to the business should include dollars saved in support/hardware or revenue that can be generated post the fix.

The subject of building/designing robust database environments is one I take very seriously and I may take this subject a bit too much to heart at times.   I was raised with a tough, dedicated work ethic- something I received from both my parents-  rural Canadians are a pretty tough bunch…   Taking on a challenge is right up my alley and I’m just not one to give up.  Can I accomplish it?  Eh yah, you betcha! :P

RMOUG Deadbeat

Do you ever get that feeling that something just isn’t right, but you have no experience to go on and you’re just too busy to check up on it?  I thought it was a bit strange that there wasn’t more communication between the RMOUG management and speakers, but it’s my first time as a presenter, so I decided I was being too high maintenance and didn’t follow up, just checking the site periodically for anything that may pertain to persenters.

so this morning I noticed that I hadn’t received an email from someone in my Yahoo account and decided maybe I should check and see if it was caught by the spam filter.  As I sifted through the emails, I came across two emails from the “RMOUG Speakers” email address and I pretty much stopped breathing.  When they contacted me in October to say my abstract for my presentation had been approved, the reason they had contacted me by phone was that I had not received their emails.  We quickly changed my email address over from my Yahoo account to another, eliminating the problem, unfortunately, it appears it switched back to the registration email since then. 

The two emails were informing me of deadlines to submit my supporting white paper and presentation slides by, (both of which were due this last week…)  I contacted them as soon as I finished reading the emails and submitted first, my slides, then attained an extended deadline of Friday for my supporting white paper.  I already have 13 pages in a word document that was written as my pre-cursor to the slides, so it’s going to be pretty easy to finish it up.  Luckily there wasn’t anything else I was missing and I also made sure I have the exact URL to check for updates regularly since I will not be trusting email any longer!

I still feel like a deadbeat RMOUG presenter.  I wonder what kind of punishment goes with that?