As many know, I joke about being a strange, ADHD kid.  I didn’t really fit in with the other kids and was really, really small for my age.  I was bullied and very much a loner.  Growing up, my closest friend was my middle sister, Kristi who is 2 1/2 years younger.  She was the golden child-  blond hair, blue eyes and very, very sweet.  She was so easy-going that she was often frustrated by other kids, so my easy-going, but, hyper personality was preferable to them and we were happy to hang out together.

We grew up in a very rural area of upper Michigan and spent our days playing make believe in the fields and forests around our house on Lake Michigan.  When we did move to higher population areas, first in Oregon and then in Virginia, we still hung out with each other than our school mates, even though we attended different schools.

I was the maid of honor at Kristi’s wedding to her husband, Jim and we’d visit when I’d come out to Oregon to visit about once a year.  We’d pick up like we’d never been apart, which is common for sisters.  I was happy to see that she and our youngest sister, Kimberly, became very close as they had grown up and they have continued with that relationship.  It’s hard not to like Kristi, she is simply “likeable”.

She turned 43 this year and as she’s always been a bit of a health-nut, appeared to be in the prime of her life.  She had no clear “cancer indicators” when she started having symptoms in January and it wasn’t until February that the doctor’s discovered she had what appeared to be colon cancer.  Within weeks, scans came back showing she not only had colon cancer, it had spread to her liver.

Both my parents have had their challenges with cancer.  My Mother had a battle with uterine cancer and my father, a version of leukemia, but both bounced back rather quickly with treatment.  We respected Kristi’s desire to be private and expected her to make a full recovery.

This hasn’t happened.  She is now, just a couple months later, stage 4 colon cancer.  No treatment has hindered the progress of the cancer’s spread through her body and she is in her final battle.  The ravages of the disease are not just physical, they have brought a horrible financial burden as her family has struggled to save her.

The feeling of helplessness is difficult.  I, as many in my field can understand the need to “fix things” or at least make things better and there is so little I can do.  My sister’s friend’s have put together a contribution site to assist in their financial struggle to help with the cost of her treatments.  It makes the pain a little less knowing I can help ease the financial burden even if I can’t do anything to help save my sister.

If you would like to help Kristi’s and our family, click here.

Many of you know that I’m the mother of three, very bright, “equally ADHD as their mother” children.  As I’ve been tasked with a number of interesting challenges with both RMOUG’s and ODTUG’s WIT, (Women in Technology) events, I’ve started to investigate how, no matter what we do for women today, unless we start focusing on the women of tomorrow, we are lost.

I spent some time the last couple weeks investigating the offerings by my local school districts for technical education and am alarmed for both genders.  There are “Technology Plan Templates” that detail out the dismal specifications of what hardware will be provided in the classroom, but no mention of what to attain with it.  No one lists out what educational goals, what students hoped to achieve with the said technical hardware.

Now what were these marvel in educational opportunities offered in the way of software?

• Windows XP.
• Tutorials on how to use a USB flash drive.
• How to “Navigate the network”

The labs were beginning classes in Microsoft Office and how to use email.  What kid, by the time they are in mid-elementary, lack knowledge in9 basic tasks in MS Office??

I looked on the clubs link, as I remember computer and programming were often found in clubs, along with classes, in my high school.  This was what is now offered:

Sigh….

I assumed the technical classes had to be missing from the website, (I know, seemed like an oxy-moron to me, too, but I was willing to give the benefit of the doubt…. )  so I called the school.  I was politely informed that all classes for any student that are interested in technical educatiin are offered at Bollman Technical Education Center.

I remember this school, as it was what was recommended to my oldest when he was having challenges that were administration in nature, but had been classified by the school as “behavioral”.  If this doesn’t concern you right off, we’ll discuss it here in a minute, but first I wanted to make sure that there hadn’t been a “revamp” of the school.

No, Bollman is still considered a school that “benefits the student who is more productive when doing than learning in a more traditional academic setting.”

I was less than star-struckwhen I looked at their technical career programs:

Auto technology?  Do we really even have to discuss this one?

I was more pleased by the clubs, but the ones that are heading it up are often the same teachers that are failing the students in their primary schools!  I noted the names of three teachers that aren’t even technical in the three clubs that are mentioned in the image below!

One of the teachers listed as one of the sponsors teaches at my children’s high school and we’ve discussed her “ongoing challenges grasping today’s technology”.  Having some of these folks heading up tech clubs seems no better than the BBC show, “The IT Crowd” character, non-technical Jen Barber’s hilarious attempt at heading up IT, which is one of the consistent comedic story lines of the show.  This is what we have directing tomorrow’s technical minds?

Now we’re going to get into the down and dirty.  This school is an “alternative” school.  This is the school that they send the students that are not succeeding in standard, public schools.  What does this say about the future of technology in general?  What does this say for the number of folks in the industry that have a story similar to mine that started out in a VERY different field than the one they are highly successful in because they were unaware that the opportunity was even out there for them?

So, here’s where I’m at-  if the schools are going to continue to fail both genders in the technical arena, maybe this is a better alternative:  http://www.technologyreview.com/news/506466/given-tablets-but-no-teachers-ethiopian-children-teach-themselves/

How many performance specialists have experienced projects where reporting was horrid in the beginning, maybe didn’t even complete?  The focus of the project may have sat with data loads or due to the data architect’s design specialty, etc., the reporting aspect was just the last thing on everyone’s mind until it was too late. I come into this conversation with companies all the time and it is a difficult one, as there are so many database myths.

“Can’t you just do something to make the database go faster?  I don’t know, isn’t there a parameter that can be updated, maybe some partitioning or compression that will fix our problem?”

I feel there pain, I do understand…

When a project to design a warehouse or mart is undertaken, the groups involved may be quite distracted by following “best practice”, i.e. “my fact tables must be just so and the dim table design must follow the same format as I used at company XYZ that was so successful….”  You may find that the folks doing the actual design may not know as much about marts and warehouses and are learning as they go along.  You may have someone who is unable to move away from an OLTP design in the mart, “everything must be relational and I must follow normal form, so let me add primary keys, foreign keys, constraints everywhere!”  (Hey, it happens, I’ve seen it…:))

Never the less, as a development group learns, there is no place I’d rather be.  No project is a lost cause and working with the group to create proof of concepts, showing them to correct way to design a warehouse or mart, along with star schemas, the power of bitmap indexes, etc. is a valuable time for any database administrator to spend.  Ensuring you know what is causing the pain for an ETL load process, i.e. where time is consumed is essential, so that you are able to provide as many “wins” when making enhancements is pertinent, so you can get onto the reporting aspect of the project.

So why do we need to keep reporting on the horizon at all times?  Never lose sight of this important feature of the warehouse or mart as this will most likely be the user interface to the environment.  This is how the business will judge if its successful or not.  If they can’t view the data in an acceptable amount of time, it doesn’t matter how quickly it loads at night or how beautiful your star schema is, as far as they are concerned, it was all a waste of money and time.

With that said, where do I see the most common challenges for reporting?

Storing data in a very different format than the way reporting requires it to be presented in. 

I’m not talking about how a star schema is designed here.  I’m talking about the data having to be constantly cased, then summed then sorted, hashed, sorted again, so on and so forth, to present the data the way the customer requires vs. having materialized views, reporting and rollup tables that can lessen the performance hit on the database.

Why is it a performance hit?  I have another draft in my blog posts that I need to get out, but it’s a perfect example.  The business needs to see the data presented consistently in a way that is no way near the way its stored.  As the reporting queries perform well with simple hashes, bypassing nested loops, etc., there is no way that the amount of case statements, to sum, to sort and sort again, to fit within the allocation of PGA per process for it, even in parallel.  This results in huge uses of temp tablespace.  That means the process is now doing it’s work on disk and disk is just plain slow.

Why am I a temp tablespace nazi?  This above is why. 

I can move to temp tablespace groups and I can put my temp on SSD, but really, the best optimization enhancement will come from an materialized view or rollup table that will limit the amount of work that has to be done in PGA.   Do not say, “Well, can’t you just raise the PGA?”  These are on servers that the amount of PGA required is larger than the memory on the box and the temp tablespace is often the largest tablespace in the database, (remember, I work in VLDB’s and for someone to brag that they used 2TB of temp is not uncommon… )

No hardware will offer the same kind of performance gain as a small design change along with a few tweaks to some reporting SQL to do less in a report to produce a report.  No hardware will offer as long-term performance enhancements as the latter-  plain and simple.

I wish I did have a silver bullet to make the database “go faster”, but that’s not how it works if you really want the problem fixed vs. having someone offer you some duct tape.  Duct tape will last you a month, maybe a year if you are lucky, but the problem will continue to return until we fix it right.

OK, enough ranting for my first day as a ACE Director…

I won’t apologize to Oracle for saying that the new Incident Manager has some quirks to it, but it does, especially when it comes to managing incidents, events and how to clear out both.  I’m going to attempt in this post to explain how to check to see if you have alerts that can be cleared with an EMCLI command and then also take you through clearing out from the GUI, which is available in the Release 2 of EM12c.

If you want to know what kinds of stateless alerts that can be cleared manually, then once logged into the EMCLI, (Enterprise Manager Command Line Interface) you can do the following:

./emcli login -username=<login name>

./emcli get_metrics_for_stateless_alerts  -target_type=osm_instance

This command will return all metric types and alert types that can be cleared out for ASM alerts, (osm_instance).  You can do this for oracle_database or any of the other target_types.

As always, if you need help, EMCLI offers a good help option.  You can save off the file for reference and then gather more help info on specific verbs by doing the following:

./emcli help > emcli_verb.lst  <--creates the output file of the emcli verbs and descriptions.

./emcli help <verb>  <--gives you a descriptions, syntax and examples for a specific verb.

If you aren’t sure of Target Type names, you can query the database or you can do wild card searches through the EMCLI:

./emcli get_targets -targets=%database%

./emcli get_targets -targets=%osm%

./emcli get_targets -targets=%

This first option above returns a list of all targets that are database type targets.  The second returns all ASM targets and the third returns everything.  Some of the output from the third looks like the following:

ID      Status           Target Type           Target Name
0       Down             oracle_listener       LISTENER_SCAN2_exa1-cluster
1       Up               oracle_oms            oemdb02.orcl.com:48
1       Up               oracle_oms_console    oemdb02.orcl.com:48 
1       Up               oracle_oms_pbs        oemdb02.orcl.com:48
1       Up               oracle_database       orcl_db3.world_orcl2
1       Up               oracle_database       orcl_db4.world_orcl1
1       Up               oracle_exa_pdu        exa1-pdua.orcl.com
1       Up               oracle_exadata        exa1cel07.orcl.com
1       Up               osm_cluster           +ASM_exa1-cluster
1       Up               osm_cluster           +ASM_exa1-cluster
1       Up               osm_instance          +ASM1_exa1db01.orcl.com

Before attempting to run a clearstate on incidents, lets look at how to check for these.Note that we use two arguments here with the clear_stateless_alerts verb:

• Unacknowledged_only-  Only those alerts that have not been acknowledged, which helps distinguish from any that may be currently worked by someone.
• Preview-  I want to just check and see how many are available to be cleared.  The command will not clear anything this way, only tell me how many!

$ ./emcli clear_stateless_alerts -older_than=0 -target_type=oracle_database -target_name=<target name> -unacknowledged_only -ignore_notifications -preview

 The following alerts can be cleared:

Total Alerts
==============
1

Now, if you want to clear the one alert that is available to be cleared as stateless, you can:

$  ./emcli clear_stateless_alerts -older_than=0 -target_type=oracle_database -target_name=<target_name> 
1 alerts were cleared successfully.

Now if you have a huge quantity of alerts that are not available for clearing with this process, the EM12c console Incident interface offers the ability to mass clear alerts in release 2, (apologies to anyone still using BP1 and lower, this option will not work!)

From the EM12c console, there are a number of ways to access alerts, via the Incident Manager from the Enterprise drop down or from a database target summary page, but for this example, we will work from groups.

By clicking on All Targets, then Groups, we are brought to the groups summary page.  As we see, our Prod group has 932 open incidents identified with this group:

By clicking on the incidents link, (which is the number of incidents, so for this group, we would click on the “932″) we are then taken to the overall incidents summary for this group.

If you hover your cursor over the left hand side column, just left of the Severity column, you are able to hold your left mouse button and choose multiple alerts:

By dragging the mouse down, you can choose as many incidents/events as you wish.  Once you have chosen the amount you wish to remove, note that the “Rows Selected” will show in the left bottom corner, (for our example, we have 7.)  You can then click the “Clear” button, identified with the pencil eraser.

The confirmation pop-up screen will then ask you to confirm and click OK, but remember to change it to not send email notifications so as not to “spam” everyone.

The worst thing you can do is send ridiculous amounts of emails telling everyone you are removing each of these incidents.  Remember, we hate “white noise”…   Once you have confirmed this is what you want to do, click “OK”.  You can do this for as many or as little incidents to clear out the system as you would like at a time.

Remember-  Any incidents that are still an incident and need attention will recreate on the next metric collection for the target.

As many of you know, the WIT Session at RMOUG and subsequent planning for a WIT program has been very successful.  I just received my evaluations, as this was offered just like any technical session.  There was a wonderfully, honest comment that I hope the author will not mind me sharing anonymously and I hope my response will help her in return.

“to be honest, if I could [do] it all over again, I wouldn’t be a DBA. I have a degree in CIS, computers were the in thing when I was in college. I chose CIS for the money to support myself & my child. It was something new & exciting. Now it is stressful and being on call isn’t fun. If I could do it over again I would have done something in the medical field like my sister who is a doctor. A lot of my colleagues say the same thing about not doing IT field as well if they had to do it again. Honestly, I believe that is why you aren’t seeing young girls & boys getting into the field because they see what their parents do and how stressed they are, etc. in the IT world.”

I found this comment both sad, honest and interesting. Facebook’s COO, Sheryl Sandberg, who just wrote a fantastic book, “Lean In”  discusses women in the workforce, not just IT and one of the things she says is crucial is a solid support system for any woman, especially if she has a family. This includes her family, her company and her manager.

I also have a couple friends in different areas of the medical field, I find that their schedules can be as stressful as the worst IT one, especially nursing.  I wondered if her sibling’s job choice in companies vs. the attendees was more the reason behind satisfaction than the career.  In my experience and from what I’ve seen, a good company is worth twice the actual career choice.

I admit that I entered the DBA field because of the flexibility it offered me.  I first became a DBA when my middle of three children was just an infant. The kid’s father was raised in a household with a stay at home Mom.  I think its challenging for a father that lived with a Mom who’s job was to take care of the family, to then have a wife with split responsibilities and higher demands than what he may have experienced his father having.  Some men acclimate to the change fine, others do not and my ex-husband did have difficulty with this.  With a career in IT, I could take time off from the office and take the kids to doctor appointments or go to parent/teacher conferences.  I could work from home if one of the kids were sick that day.  The good pay and ability to telecommute for after-hours tasks worked well with my lifestyle and my children were comfortable with Mom being present, with a laptop as her constant companion.

As I mentioned that the kid’s father is my ex-husband, yes, I was a single parent for many of those years.  I have a good parenting relationship with my ex, so no doubt that really does help, but he has to admit that I’m the one that goes and gets them from school, the one that stays home when they are sick and the one that addresses all work hour requirements.  I would have found it difficult to locate another career such as database administration that offered me this type of flexible schedule, as well as the ability to telecommute full time, which I’ve now done for two of my positions.

In the last year, I’ve begun to travel to present at conferences, where before 2012, I only presented locally.  I now am presenting at about 4-6 remote conferences which requires me to travel and my ex-husband and his new wife care for the kids during that time.  The kids are now 12, 15 and 18 yrs old, so the care is minimal and they don’t complain too much about having their schedules interrupted for a couple days.  Outside of the few days I travel once every two months, I work from home.  I’m here every day when my kids get home and I’m able to pick them up from school or take them to appointments.  Pretty much whatever they need me for, I’m here.  I have two office areas in the home and even if I’m not home, I’m incredibly accessible via cell, email and chat.

Although I understand where the attendee to the session is coming from, I’m going to focus on where I think the real problem lies-  I’ve seen IT environments where they work people 12 hrs a day and require them onsite, refuse to comp time, etc.  This is a work culture issue and not a database administration career issue.  We see it in not just IT jobs, but in so many others as well.  IT often derives from poor management and poor work culture.

I would say to this wonderful woman, “Please, do not give up on this career!  Search out companies that support their staff’s lives, realizing the difference running a company hard and running a company smart!”

I’ve worked in these environments.  Interviewed where it sounded very wonderful and no hint of what was in store, but once you are in the doors, you are thinking, “My God, what have I gotten myself into?”  You are torn as your family has demands of your time and you have a boss that doesn’t understand why the family can’t come second, third or “can’t your Mother-n-law just take care of it?”  Sigh….

I made a pact with myself a while back.

• Only work for good bosses and for good companies.
• Work for companies that support you in your own goals and family life, not just for their priorities.  It’s should be a win-win with small compromises that pay forward.
• Companies that support their employees have more loyal employees and are more likely to succeed.  The employees work for the company because they want to, not because they have to.
• Realize your worth and work hard for what you want.
• Don’t listen to those that tell you that you can’t accomplish something or that something will never happen.
• Surround yourself with positive people and find mentors
• Yes, there is often oncall/after-hours work with a DBA career.  This is why I recommend any DBA’s first goal is to silence the pager and remove all “white noise”.
• Any manager/department that is a proponent of ridiculous tasks allocated to after hours work, extended hours where a resource is exhausted and not able to function well and/or “white noise” paging-  START LOOKING FOR A NEW JOB.

I don’t know if this will reach the woman from my WIT session, but for any woman in the industry that feels this way, I hope it helps them find their way to succeed.  I love my job, my company and my career.  I wouldn’t change my path and yes, I’ve experienced many bumps in the road, but to have the persistence to continue on the path that is right for me is the only way to success.

There’s an old Hebrew proverb-  “Fall down seven times, get up eight”

The eighth time rocks.

Exachk does a number of health checks.  One of them is to verify the connectivity, settings and such of the ethernet cables in the Exadata environment.  If one is found amiss, the following will be returned in the Exachk summary report:

To validate the Exachk findings, you are asked to run the following as root on the specific node that is reporting the issue:
for cable in `ls /sys/class/net | grep ^eth`; do  printf “$cable: “; cat /sys/class/net/$cable/carrier; done

This command, if run on one of the other nodes that were not showing up in Exachk without an issue, the following is returned:
eth0: 1
eth1: 1
eth2: 1
eth3: cat: /sys/class/net/eth3/carrier: Invalid argument
eth4: 1
eth5: 1

Running this on third node, where the report of the failure had occurred and the following is returned, clearly reporting the issue on eth1 and eth4 that requires a physical check of the connections reporting back with a value of “0″.

eth0: 1
eth1: 0
eth2: 1
eth3: cat: /sys/class/net/eth3/carrier: Invalid argument
eth4: 0
eth5: 1

“Invalid Argument” signifies that it is not in use, so disregard eth3.  The values of 0 are the issue.  This shows we have a problem.  Andy Colvin, (our very own Enkitec Exadata Ninja!) recommended doing the following to just check a bit further.

As root on the third node, issue the following: ethtool eth3

$ ethtool eth3
 Settings for eth3:
 Supported ports: [ TP ]
 Supported link modes:   10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Full
 Supports auto-negotiation: Yes
 Advertised link modes:  10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Full
 Advertised auto-negotiation: Yes
         Speed: Unknown!
         Duplex: Unknown! (255)
 Port: Twisted Pair
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: pumbg
 Wake-on: g
 Current message level: 0x00000003 (3)
   Link detected: no

Yep, the lights are on, but nobody’s home…

At this point, someone *physically* needs to check the hardware.  For the incident above, it was suspected that someone had yanked on the cable, damaging it and due to this it was replaced, ran through the above tests again to verify that all was well.

There have been some great blog posts from folks on the RMOUG Training Days 2013 Conference, (I’ll add some links to my favorites I’ve come across at the end of this post!)  Due to this, I thought I would leave mine in my drafts folder and focus for now on the Conference from my viewpoint as the Conference Director.  John Juenette was gracious enough to send me a beginning blog post he’d started last year and I need to review it and add my own data for future “victims”, but until then, here were my experiences during this year’s run as Training Days Conference Director.

The conference, of course, started for me back in May when I said that I would be the director of the conference.  I started looking through information back then, taking notes at conferences as I attended on what I liked, what I didn’t like and what might be feasible for a conference our size.

The end of July everything ramped up and we started preparing for the conference.  I gathered my wonderful conference committee members, John Juenette, Peggy King, Tim Gorman and Pat Van Buskirk.  Team YCC, the crucial part of our conference planning team, readied the website to open for abstract submissions once we configured tracks, schedules and design.

Once abstracts were submitted, we chose the abstract committee, which did a phenomenal job scoring out the abstracts, which then made it easy for us, come November, did choose the best of the best, then pepper in new speakers, filling 142 sessions out of over 300 abstract submissions.  There were so many fantastic submissions and it was difficult to choose!

I had a very defined list of wants and enhancements that I desired for this year.  I had presented at SQLPass #169 in September and was impressed with their mobile app, Guidebook.  This app did a really fantastic job and upon research, I found that they had just used the free version and that if you looked into some of the enhanced versions, it really put every mobile event app I’d seen in the past to shame.  I had a bit of concern, especially considering how poorly many of the mobile event apps had perfomed for folks in the past, but then Team YCC returned from attending a conference about conferences.  Guidebook had been all the buzz there and they had seen it demonstrated, which had impressed them greatly.  Arisant graciously came on board, thanks to Kathy Robb, to sponsor the app and the mobile app enhancement to RMOUG was made.

My second goal was to upscale our Speaker and Volunteer Reception.  In previous years, it had been at Wynkoop and although I loved the “local brewery” hangout, we’d done it year after year and wanted to do a bit more with the opportunity, but not too much of a change that we shocked everyone.  This year we moved to the Rock Bottom Brewery and rented out the back bar.  This room has multiple TV, pool tables, a much nicer bar and with the joint effort of the Oracle ACE program, we combined what is commonly the ACE dinner with the RMOUG Volunteer/Speaker Reception to form the VIP Reception.  The buffet was fantastic, noting the consumption of not just the food, but the wonderful dessert offerings, but the beer and wine was well received by the group.  We had the largest turnout we’ve ever had for one of these events and we appreciate Vikki Lira’s contribution to this RMOUG/ACE event!

My third goal also sourced partly from the SQLPass event, but had been a source of great frustration for me for some time.  I was quite aware of the ratio of men vs. women in the database field.  In my experience, I had noted it around 30% women, yet when I attended Training Days conferences, the percentage of women in attendance was always less than 10%.  I was amazed when I presented at SQLPass, where in my experience on the Microsoft side, the ratio of women had always been lower, yet here, at the conference, the ratio of attendance was almost 40%!!  I had the great fortune of presenting at the same conference as Jen and Sean McCown, (@MidnightDBA) who, as a married MSSQL DBA couple, Jen does a lot with the WIT, (Women in Technology) program with the MSSQL group.  I appreciated Jen’s attitude and took a number of notes on what she said was required to create a successful WIT program and went forward with these ideas for RMOUG.

The WIT session was held at the time of the vendor sessions so as not to take away from the technical sessions that many might see as clear competitors, (next year, I plan on putting in an hour of professional development, if we can handle it… )  I then followed these rules:

1. Don’t make it a “cake social” or a “book club”, this is for technical professionals.  If you wouldn’t have it for a session with an audience filled with male attendees, don’t do it here.  You will lose the truly serious women in technology.
2. Do not be afraid to discuss all topics that are holding women back or holding women out of the STEM career area.  This is NOT, however, a male-bashing session.  If anyone starts on this line, please intervene, thank them for their opinion, but also remind them that this is not the correct platform for this type of conversation.
3. Recognize how we hold ourselves back culturally, without even realizing it.  Jen told me how folks would come up to her at conferences and say, “Wow, you are here presenting and your husband is home babysitting the kids?”  She says it was frustrating as she’s explaining to them, “They’re his kids, no one is *babysitting*” and how they would never have had this conversations with her husband at the conference, (her husband was in the room agreeing furiously…)  We often build boxes around ourselves without even realizing it and I’m very aware that I do this to myself- still.
4. Understanding that although analytical skills are important to a career in a database field, there are a number of other areas that are just as important.  Knowing that without this “balance”, a team can become askew in skills and not perform as well as it would if it were to have a more diversified set of individuals.  When I’m interviewing a new DBA, along with the correct answer to a question, I’m also gauging how they communicate and if they will work well with their team members.  Do their skills help balance out the skills already present in the team?  Women bring a strong component of problem solving and “ability to see the big picture” where men often are highly focused and analytical.  The balance of these strengths are crucial to building a great database team.

Needless to say, the WIT session was hands down, a roaring success.  I thank my fellow panel members, Dominic Delmolino, Maria Colgan, Debra Lilley and Karen Morton.  The outreach of the RMOUG community to me after we completed the very successful session asking what would we do next for WIT was a clear indicator we did something very right in this session!

With my three enhancement goals attained-  Guidebook Mobile App, VIP Reception and WIT Session to launch a WIT Program, I continued with the final changes to the schedule as speakers schedules required and not panicking when things didn’t go exactly as planned…

The pre-conference on Monday went surprisingly well.  I’m not sure what I would have done without Peggy King ensuring that we had all our volunteers and Tim Gorman setting up the rides from and to the airport.  This is a nice little perk we do here, but we have so few *bodies* to help and so much to do, it can be over-whelming at times.  We arrived on Monday morning, my middle-child, Caitlyn along for the day to assist and all the volunteers started to put bags together for attendees, prepare speaker gifts and registration preparation was all worked to completion.  Speakers started to arrive and register and the OTN and University Sessions were underway.  Cary Millsap held his “Mastering Trace Data” and as usual, the students were thrilled with his class.  Maria Colgan’s session was packed her OTN workshop, (the other session with Kris Rice and Jeff Smith was nothing to sneeze at either!) but we feared for the fire marshall and ensured that no more than the room would hold was allowed.

We had a few cancellations for speakers due to the winter storms out east and one due to illness, so we had to do a bit of scrambling.  I’d like to thank Carlos Sierra and Frits Hoogland for stepping in at the last minute with sessions.  I also would like to thank all the others that offered to throw in a session at the last minute.  I’m always amazed at the support that RMOUG is given by the Oracle community.  You folks really come through over and over again.  I’d like to thank Mogens Norgaard, even with the bad sound quality, he was a warrior, in his infamous keynote bathrobe and stuck through till the end!  The folks at the first three rows of tables in the ballroom thank you, as we got a fantastic keynote!  I apologize to all those in the back of the room who I know had a difficult time hearing anything.  We will work to ensure that the sound quality is better next year.

Upon speaking with different exhibitors, the set up this year really seemed to work well, too.  All seemed exceptionally happy with their spots and the traffic.  Folks seemed to really take time to talk with them and were interested in their products.  We had incredible support this year from exhibitors and sponsors.  DatAvail and DBAK did joint platinum sponsorship this year, which was a first for RMOUG.  We had a number of new exhibitors this year, like Delphix and Firebox Training.

Lastly, I’d like to thank my employer, Enkitec.  I could not do all that I do without the support of a fantastic employer.  I so appreciate the support to me professionally as an employee, as well as they as an exhibitor at RMOUG and my coworkers that submitted abstracts, to those that were accepted and presented.  Kerry Osborne, Wade Nicolas and Veronica Stigers are just the best in all that they do.  They always have their eye on the ball, never falling for the false alarms or latest trends, but strong and steady with the right direction for the company.  I appreciate my boss, (he so hates when I call him that… )  Andy Colvin.  Nothing much fazes him and he just lets me be the best I can be with the time I have to do everything I can possibly do.

As promised, here’s the links to other blogs posts on this year’s RMOUG.  Thanks for sticking through this, I think it was a long one…

Carlos Sierra

Maria Colgan

Arup Nanda

Maxym Kharchenko

Kent Graziano  (Kent has a couple others out on his blog, too!)

Mike Dean

Karl Arao

The Enterprise Manager Diagnostics Kit, (EMDIAG) is a great addition to have installed and be familiar with on your Enterprise Manager 12c, (EM12c) environment BEFORE you need it.  There are three main kits:

• repvfy – Extracts data from the Grid Control or DB Control Repository to help you to maintain or diagnose problems in your Grid Control or DB Control.
  The repvfy Kit is partly included in the Cloud Control 12C Support Workbench for the OMS.
• agtvfy Kit – A set of command lines to help you to maintain and diagnose problems in your Cloud Control, Grid Control or DB Control Management Agents.
• omsvfy Kit – A set of command lines to help you to maintain and diagnose problems in your Cloud Control, Grid Control or DB Control Management Services.

For this post, we are going to work with the REPVFY kit from the EMDIAG and explore how functional this kit can be to diagnose and assist when issues arise in the EM12c OMS.

Installing the EM Diagnostics Kit

Download the zip file for the version that you need, per Oracle Support Note: MOS ID# 421053.1

This note will explain about each of the kits and take you through the download process.

Create the EMDIAG_HOME in the environment variables for the OMS OS User and owner of the EM12c.  This will create an easier support scenario for the administrator in time of stress when they require quick access to the EMDIAG tools.

unzip the EMDIAG zip file into the EMDIAG directory
unzip <file name> -d $ORACLE_HOME/emdiag

Perform the installation by changing to the newly created directory and installing EMDIAG
cd to the EMDIAG home, (that new directory you just unzipped the files into…)

cd $EMDIAG_HOME/bin

In the bin directory will be three common files, the executable, the batch file for windows support and the perl script that supports both:

-rwxrwxr-- 1 oracle dba   3331 Feb 23  2012 repvfy.bat

-rwxrwxr-- 1 oracle dba 198800 Aug 31 18:57 repvfy.pl

-rwxrwxr-x 1 oracle dba   7195 Aug 31 18:57 repvfy

./repvfy install -tns em12rep -pwd password

The installation will commence and you will end with the following:

PL/SQL procedure successfully completed.

COMPONENT            INFO
 -------------------- --------------------
 EMDIAG Version       2012.0831
 Repository Version   12.1.0.1.0
 Database Version     11.2.0.3.0
 Test Version         2012.0926
 Repository Type      CENTRAL
 Verify Tests         381
 Object Tests         157
 Deployment           SMALL

Checking the installation and verification of the version

./repvfy version -tns em12rep version

Please enter the SYSMAN password when prompted.

Component                      Version
 ------------------------------ ---------------
 repvfy                               2012.0831
 EMDIAG                                   12cR1
 Database                            11.2.0.3.0
 Repository                          12.1.0.1.0
 Tests                                2012.0926
 Linux                                   2.6.18
 x86_64-linux-thread-mult
 PERL                                    5.10.0
 SQL*Plus                            11.2.0.3.0
 Agent Zip                                  3.0
 Agent Unzip                               6.00
 ------------------------------ ---------------

Using the EM Diagnostics REPVFY KIT

Querying the following data from the Enterprise Manager will gather data that is required to then run the diagnostics for job level reports.  The example used will involve a backup job for APEX:

From SQL*Plus as a user with select privileges to the SYSMAN schema:

select mj.job_id, mj.job_name, mjh.execution_id, mjh.step_id
from sysman.mgmt_job mj, sysman.mgmt_job_history mjh
where mj.job_id = 'BFC78F41009876F3E04014AC08003231'
and mj.job_id=mjh.job_id
and mjh.start_time > sysdate-1;

Using the job name, “BACKUP_APEX.1″, an obvious APEX backup job, we can perform some diagnostics report gathering on the job itself, using the job_name field.   Due to the amount of diagnostic data that is offered,  redirecting the output to a file is recommended.

cd $EMDIAG_HOME/bin

./repvfy dump job -name BACKUP_APEX.1 –pwd w0rldw1d3 -tns em12rep > ./rep_apex_job.lst

The output file will then show you a full report on the job, starting with the information about the diagnostic tool, the database, test and repository.

From here, the report steps into information about the job-  run time, timezone and detail information:

You can view the parameters in the job, including multi-step jobs commands:

This report will also tell you about where the job runs from and the target it runs to:

The next section then let you know who owns the job and who has grants, along with what kind of grants on the job.  As this job is simply owned and run by the SYSMAN user only, only this user shows in the report.  The report presents the last 10 run times in history, including elapsed time, retries and success information.

For the example, the EMDIAG informs me that there is one job that was in a waiting state.  In the recent history, it shows that all have been successful, (0), but that a level 17, signifies that one of the runs did have to wait due to some undisclosed issue that isn’t shown in the execution summary:

If there were any errors in the job analysis, this is saved for the last section of the report.  Our example compiled successfully, showed no issues and so the output results in the following:

If a job has more than one step and you wish to gather detailed diagnostic information about that step ONLY, a secondary report can be generated by running the following:

./repvfy dump –tns <oms_sid> step –id <step_id>

GUID Target Diagnostics

For our next scenario, we are going to utilize the GUID for our EM12rep, (the OMS Repository for the EM12c) and gather EM environment data:

./repvfy dump -tns <oms_sid> execution -guid EFD92E6468B02696B602A650E085B5A9 -pwd <pwd> > ./rep_guid.lst

By running the GUID level command, (GUID is your target GUID and dumps out information on all the jobs, from all the targets)you get a one-stop shop kind of diagnostic report about your EM Jobs Service health.

Although the report starts out very similar to the job level report, listing out repository and EM information, once you get to target info, you will note a distinct difference, as this report will list out all information regarding monitored targets, which the job level report did not.
If there are any background jobs submitted by the EM Jobs Service that are experiencing issues, they are clearly reported in their own section:

My only complaint at this time, would be that the BP1 version  of the EM12c, when the EMDIAG is run against it at a full enironment level, experiences an error at the Job Analysis Summary step:

Due to this, if you are still on a BP1 version and need to pull this level of diagnostic data, I hope that you have your issue narrowed down to a single job or target that you can focus the report to, subsequently avoiding this error.

Global Diagnostics

In this example, we will run a level 9 diagnostic, which will run a report against the whole of the Enterprise Manager environment, checking the health and for issues:

./repvfy -tns em12rep verify -level 9 –detail

The first thing this level of diagnostic does is check the version of EMDIAG you are using and if there is a newer version available. The diagnostics will proceed to perform a full verification of the EM environment before logging in to SQL Plus to perform the secondary diagnostics on the EM testing, the installation information about the EM system and then proceed onto internal jobs and alerts from the EM environment.

The next step is to check all promoted and un-promoted targets, make recommendations on changes and then end the report by verifying any non-standard EM users.
The logs for the output from these reports will then be generated to the <EMDIAG_HOME>/log directory to be reviewed or uploaded to Oracle Support.  I do prefer to redirect the screen output to a file for easy review, as seen in my examples above.  This data should assist the administrator in answering questions regarding their EM12c environment without having to always open an SR, but if the need arises, this valuable diagnostic data can then be generated to save time and effort so that the administrator can perform other tasks than just answering Oracle Support questions.

As I spend more and more time optimizing Exadata environments, I seem to be answering the same questions repeatedly when it comes to database design, parameter settings, etc.

Here’s a few high level notes to always remember-

• Always have baselines from previous environments.  Don’t go into performance expectations blind.  Have AWR and ASH data from the previous environment and mine AWR data for anything when others start “guessing”.
• Look at the big picture-  those that focus on one thing, often find that it’s a combination of issues that brew the perfect storm.
• What are your top 10 goals by going to Exadata?  They had better tell you it wasn’t just because someone thought “it was cool”.
• Expectations should be listed clearly, “we are expecting a XX% average improvement in our ETL loads and XX% average improvement in our reporting.  We expect to have a XX week post optimization exercise to address any performance issues after the migration and know that this may involve query rewrites.”
• Do the challenges in your database environment coincide with the features that Exadata offers?  If not, you could have a really sweet engineered system that offers your current design minimal improvements and what that leaves you with is future enhancements that should be geared to utilizing smart scans, storage indexes, HCC, etc.
• Just because a statement offloaded and returned the data quickly, doesn’t mean you have unlimited PGA to then sort and aggregate that data in a report.  Cell nodes can not help you here.

“My goals are not your goals” is an opportunity for any DBA who hears these words…

• Have you identified your stakeholders in the optimization exercise?  What enhancements are important to you, may not be what is important to them.
• Discuss what change control is required.  Ensure you know what is required to move anything to production.
• Openly listen to all involved and realize that all input is valuable and in the end, if the user isn’t happy, nobody’s happy.
• Options located on web sites and are stated to be “best practices” should not be taken as gospel.  Research and verify everything.

Successful migrations and optimization on Exadata starts with good planning

• Create top 10 out of group’s goals.  If a goal appears out of context, discuss, find out what is important about the goal and if there is another option for attaining it.
• Create a simple SOW for the optimization exercise.
• Clearly state what will be included and what WILL NOT.  Ensure all stakeholders have signed off on the SOW.

Dispel Exadata myths.  I was told by one client that, “Tom Kyte said at the last Oracle Open World that parallel should never be used on Exadata…”  Although I was sure they had heard him incorrectly, I wanted to see the original slides so that I could show them the point he was actually trying to relate.  The outcome of a quick tweet in search of Tom’s slides from OOW resulted in a bizarre and interesting set of emails/tweets when I was hit with the “140 char misconstrued tweet” issue.  Yes, I am now responsible for a “Tom Kyte says you should never use parallel with Exadata” myth…sigh…

When you are making recommendations

• Backup everything with data.
• IF you come to me with proposed parameter changes, (the silver bullet syndrome…)  please let me apologize for my lack of enthusiasm.  Even when parameter changes made a big difference in earlier versions of Oracle, it was commonly only a 10% improvement, where with SQL rewrite, I could get at least 80%-  you choose…
• Inspect opportunities for SQL rewrite- ensuring others that often it is required as systems grow and change, not because someone wrote, “bad code”.  Pointing fingers is a waste of time and unprofessional.
• Are there opportunities to incorporate parallelism.
• Is plan stability an issue? (statistics, dynamic sampling and profiles.)

And Tyler Muth reminded me that  forgot, even after just using it today with a client-  Exachk is your FRIEND!!  It will verify that no undocumented or non-default parameter settings are present, along with perform a health check on your Exadata environment.

Truth be known:

• Lots of SGA is not always your friend in Exadata, (if it runs in memory when it would better be served by offloading to a cell node due to high SGA, is it a good thing?)
• Do not count on storage indexes.  The feature, when implemented by Oracle is one requiring the choice of a complicated set of algorithms.  They can be your savior or your devil.
• More parallel is not better parallel.  Find you “sweet spot”.  Most environments have a setting of DOP, (degree of parallelism) that works best with over 90% of the environment’s SQL and delivers over 90% of the overall performance improvement.  The DOP is also often much LOWER than many would guess, (how often 4 has been my magic DOP… )
• Know when it’s time to implement classic warehouse/mart strategies such as star schema, bit map indexing, partitioning, mviews.  The harder you make the Exadata work now with these kinds of tasks, the less smarter it’ll be working as the environment grows.

Last note.  Although for some, it may not seem believable, but if I hear one more debate about any 11g database and optimizer_index_caching  or optimizer_index_cost_adj parameter settings,  I may go postal…:)

See ya next week at RMOUG Training Days 2013!

While at conferences, I attend a lot of dinners/social events outside of the day time festivities and although the topics on the table are often technical, sometimes they veer off to those of each person’s interest.  Discussions with me often fall into one of my areas of interest, which happens to be history, (American, European or Russian…)  Many who know me realize that if I hadn’t ended up in the technical world, I might have become a history teacher.  The common question during these conversations is “Who is your favorite American president?”  When I answer “Andrew Johnson”, I always get a quizzical look, often followed by some hoping to correct me, “You mean Andrew Jackson, right?”  I’ll stand by my answer, stating no, although I can talk about Jackson, our 7th president for hours, it was our 17th president, Andrew Johnson that I was referring to.

So often we all hear the popular, the much conversed about, the “buzz words”.  Even if we take the topic of databases and I were enter an area with five folks who work with databases and say the following words:

“SQL*Plus, inline function, Mark Hurd and RAC”

They would all have some thoughts on most of the subjects above.  They’ve read the headlines and the white papers, even if they haven’t been introduced personally to any of it, then if I proceeded to say:

“csscan, mutual recursion, Charles Rozwat and emdiag”

I might not be so lucky and would have at least a few wondering what a few of these were.  I have a tendency to find value in what may impact our day to day world, even when we don’t realize it.  How many of us know Mark Hurd and his impact to us as DBA’s, but may not realize Charles Rozwat’s impact as he heads up My Oracle Support is understandable.  His impact is much less visible.  History is one of those things, if you may not know about it and in turn you don’t realize how much it impacts you or you fail to repeat it…

Andrew Johnson had a tough road.  When Abraham Lincoln, aware that the civil war was turning and coming into the Union’s favor, started to look towards the future and how he was going to embrace the south back to the union.  He had taken the 7th president, (speak of the devil..:)) Andrew Jackson’s “never forsake the union” stance to heart and knew that he had quite the challenge ahead of him in his second term with the Northern Republicans wanting nothing more than the South to pay for the civil war in blood….even after the war was about to be won.  These folks were angry and they wanted the confederacy to pay for a very long time…

To address this, he chose a Southern Democrat, former Tennessee Senator Andrew Johnson, who was one of the few who had stayed loyal to the Union even after his home state of Tennessee had seceded to the confederacy.  The 1864 election upon him, Lincoln counted on the next four years to take Johnson under his wing, have him help enfold the south back into the Union successfully, without the terrible plans of vengeance planned by those around him.  Many questioned Lincoln’s choice, but no one dared to challenge him on it.  Lincoln knew that without Johnson, a Southerner, along with him in the White House, the chance of the Confederacy trusting him, would be unlikely.  He foresaw Johnson as someone who might be able to ascend to the presidency after his second term and do so successfully and trusted him only as ol’ Abe would.

Johnson was known for drinking heavily, but few knew why or why he was so against slavery.  His pro-abolitionist views had sparked many a fight with his Southern neighbors.  What most didn’t realize is that Andrew Johnson was the only president to comprehend slavery from a personal perspective.  At the age of 8 yrs old, he and his older brother were  legally sold into apprenticeship to a local tailor due to the unfortunate loss of his father and financial difficulties of his mother to make ends meet even after remarrying.  For a number of years, he and his brother endured the situation until they found a chance to escape and traveled throughout the south to escape the bounty on his head.  He truly understood some of what slaves were experiencing and understood why it could not go on or allowed to simply “die out” as some of the Confederates had claimed would occur if just left alone.  Lincoln had become aware of his history and found it incredibly moving, as well as important to the cause.

There was one thing that Lincoln, nor Johnson considered though-  Lincoln’s assassination.  This unexpected turn of history rewrote all the Lincoln’s well-laid plans.  Johnson was left in a less than hospitable environment.  The Northern Republicans expected after Lincoln’s assassination that Johnson would just go back to Tennessee, but he didn’t.  He fought and won his right to be president- one, that many of the existing cabinet had accused under whispered breaths, might be behind Lincoln’s assassination, confederate plots and any other political espionage they could consider.  They were more taken aback when he planned on going forward with Lincoln’s plans to embrace the south and bring our American brothers back into the Union.  This was unacceptable for Northern Republicans and they chose at that time to put Andrew Johnson through, what might be, the most miserable four year term any president has endured.

Not only did they lock him out of his own cabinet meetings, they proceeded to leave him locked in the middle of a tug-of-war between the North and the old-South, attempting to rebuild what it once had.  The south had very few men left to lead, but many of the ones who could, had been those that had been part of the former south.  The north wanted none of this and demanded Union generals be put in place.  Johnson fought this with vetoes and battles on capital hill.  As the congressional fights intensified, with what they viewed as a “Pro-Southern” president, (while the south viewed Johnson as a traitor), The Congress passed measures so that Johnson couldn’t even elect or terminate his own cabinet officials.  This lead the way to Johnson being the only other president, outside of Bill Clinton to be impeached.  He only avoided removal from office by one vote.

Johnson has been long vilified as this president that was against the North post the Civil War and against civil liberties.  The truth was, he was one man against terrible odds that could not proceed with Lincoln’s plans or any plans for that matter.  His failure left a long line of former civil war generals as future presidents, more failures in the reconstruction of the south and left it open to the carpet-bagger area, one of the most historically interesting times in Southern history, but a time that shows its impacts even today.

Johnson’s time in office is as pivotal to American history as Lincoln’s in many ways.  We just choose to remember Lincoln with monuments and book upon book in libraries and book stores.  Johnson was remembered when he finally came home, a defeated man post his presidency to find that his home town, the one that had called him a traitor on his departure for Washington as vice-president, had hung a sign, “Welcome Home Andrew Johnson, Our Hero”.