I ended up so wrapped up in everything going on when I returned home that I never did get my Miracle Oracle World 2012 post out on my blog, (bad DBA Goth Princess!:)) so here it is!

Miracle Oracle World, (aka MOW2012, hashtag #MOW2012) is a yearly event held by Mogens Noorgard and his company Miracle, (http://www.miracleas.dk/ ) at Hotel Legoland in Billund, Denmark each year. This year was no exception and I was thrilled when Mogens asked me while in attendance at RMOUG’s Training Days if I would come to Denmark and do my presentation on EM12c, (then let me choose one more topic to speak on, which turned out to be ASH Analytics.)

I’d never been to Denmark and it sounded like a wonderful opportunity, but there were clashes in schedules personally and I wasn’t sure if I’d be able to attend at first, but Tim was adamant about me going to MOW this year and stayed home to take care of responsibilities so I could speak in Denmark.

Upon reaching Copenhagen, Mogens picked me up at the airport, (still impressed he was able to locate me in the chaos that is any airport!) and after a short time to recover from the long fight at Mogen’s house, I loaded up into the car, driven by Mogens and accompanied by Cary Millsap, his wife, Mindy and their five year old daughter, Cat.

I have to give a lot of credit to young, Miss Cat. She was so well behaved and content to just ride along, watching movies and playing games, etc. She’s very close to both her parents and was quite enjoying her *special* trip to Denmark. It’s a 3 ½ hour trip to Billund from the town near Copenhagen where Mogens resides and the trip was pleasant, good conversation and humor when Cary attempted to calculate the miles per hour of the windmills by circumference, length and speed of the blades.

Upon arriving in Billund, we toured the Hotel Legoland for a bit, but I was glad to have the Millsap’s drop me off at the house at Lilandia and let me recuperate. Lilandia is a huge waterpark across from Hotel Legoland with what looks like small cul de sac’s of houses, different sizes and designs. The one I was in was four bedrooms and I was to share my room with Heli Helskyaho, who I was so looking forward to catching up with since I hadn’t seen her since RMOUG training days! My fellow house guests were Alex Gorbachev, Christian Antognini and Jonathan Lewis.

Most of the folks attend the party house and I heard it was quite the event, but I won’t lie, I tried to catch up on some sleep after being up for 36 hrs straight! Not much jet lag but I was up and spoke to Jonathan when he came in at around 8pm, Alex when he showed up at Midnight and then chatted with Christian when he was up at 2am working on his presentation slides. Jet lag is a funny thing…

I did get enough sleep to be up and ready before 7am and Jonathan happened to be up at the same time. He asked if I would like some company on my walk over to the conference site and I was happy to have such an excellent fellow along for the 1 mile walk. As much as I enjoy Jonathan’s presentations, there has got to be something said for one on one discussions with him. He can converse on any number of subjects and we were at the hotel Legoland in no time at all.

I appreciate Cary Millsap and his daughter, Cat allowing me to sit with them at breakfast. This helped alleviate some of my separation anxiety from my own children just watching Cat interact with her father. I ran into Oyvind Isene from Norway’s Oracle User Group, who I’d had the pleasure of meeting last year at Oracle Open World and he was someone I could always count on to put up with my incessant chatting for the two days of the MOW conference…

Choosing sessions was more difficult for me than I think for most other folks at the conference. The thing that really impressed me about MOW was that it was a mixed platform conference. This doesn’t mean there were a few MySQL sessions scattered among the Oracle, but that there was a full set of SQL Server presentations. For me, having as many years in SQL Server as I do Oracle, it was a real treat. I missed out on a few I would have liked to attend, but as always happens, conflicted with others I had promised attendance to. To attend Ron Crisco’s session, (someone I had spoken to and emailed with working with the Method R tool suite, but had never actually met…) meant I missed Thomas Kejser’s “SQL 2012 OLTP Improvements” session. This went on for most of the day even my own presentations competing with a few sessions I wanted to attend.

Christian Antognini’s “End to End Response Time Analysis” was engaging- anything to do with waits and response is going to interest me. His data was flawless and everyone in the room left with a few more ways to address performance challenges after seeing his examples and demos.

I then attended Dan Norris presentation on Exa-environment platform integration techniques. Dan didn’t have a large crowd in his session but this did allow for the conversation to be very specific between the attendees and Dan, which in turn satisfied those that attended his session greatly. Dan has a lot of experience and insight and I was impressed with the level of interest from those that had questions about Exa* products.

I proceeded to attend a SQL Server session next by Alexei Khalyako on SQL 2012 High Availability and Disaster Recovery. I hadn’t worked in this area of SQL Server for a couple years and was quite impressed with some of the new features Microsoft has introduced into the SQL Server product, including Availability Groups Multiple Secondary’s for HA and DR, Advanced Replication and new clustering features.

The next session, by Thomas Kejser, I found very intriguing. It was Hadoop and Big Data vs. Datawarehousing from a SQL Server perspective. I’ve heard plenty of presentations and read a number of papers on it from the Oracle side, but really enjoyed hearing a view of it via SQL Server world. The data behind the presentation was accurate and Thomas is a solid presenter, so the session was one of my favorites. I made enough of an impact, (can you say mouthy?) that Thomas came over promptly after his presentation was done and we continued to talk in a small group about big data and what options DBA’s had to handle it, no matter what the platform or technology choice.

My ASH Analytics session was at 4pm, last session of the day and it went as well as I could have expected. I was missing my demo for this session, the server I’d set it up on was down for maintenance, (I know, why did I trust a server to be up or even available??) I’d already run into the complication when I discovered much of my data I had been collecting over the last couple months missing from my external drives. I unfortunately didn’t realize this until I went to enter all of the data I thought I had, on slides just two weeks prior to the conference. I came up with challenges recollecting this data as my current EM12c environments are on 10g RAC databases. Anyone who has attempted to use ASH Analytics on a 10g environment will learn- it’s not going to happen. You can install the package to bring up the ASH Analytics page, you just won’t have any data in 95% of the panes, so nice try. Lucky for me, I was able to capture most of the data I needed to come up with a presentation just in time- still not the quality I would have liked.

Of course, my ASH Analytics session was attended by Cary Millsap, Alex Gorbachev, Uri Shaft, John Beresniewicz and other folks, many on the DBA God list… I still appreciated their time, I was thrilled to have them attend, even if it was not my best session. Afterwards, we talked for quite some time, (missing demos have a tendency to grant that time… ) and Cary asked if I would like to attend his “Mastering Trace Data” Class that he was giving the next day. The opportunity to take it offsite was on the table for me, but to take it in person was too much to pass up, so I agreed.

The afternoon ended with after-first-day-conference drinks at Hotel Legoland’s bar while waiting for dinner to be served. Heli Helskyaho had arrived at that point, was happy to see her, (and this also meant I was no longer the only female presenter at the conference! ) I hung out with her, Oyvind Isene and Christian Antognini, speaking to many others throughout the time there until they informed us dinner was ready.

Dinner was a fun affair, great food, great company and again, Cary and Mindy Millsap’s daughter, Cat was well-behaved, (more so than some of the adults? ) Alex Gorbachev, along with our common banter, discussed his recent election to the IOUG Board of Directors and had a good conversation about RMOUG’s relationship with the IOUG.

Post dinner, Heli Helskyaho and I went over to the “Party House” which was four doors down from our own. There was eating and drinking, although not as much eating and drinking as the first night, as I heard there was a roast pig that I chose to miss to catch up on sleep. We spent most of the evening talking with John Beresniewicz, Tuomas Pystynen and Uri Shaft. There were many others, but so many, I gave up on remembering faces, let alone names. Alex Gorbachev took my “hoppy” beer off my hands at one point and replaced it with red wine, which turned into my drink of choice for the evening.

The second day started with a lovely breakfast with Heli, Tuomas Pystynen, Jonathan Lewis and others. I took the first hour off, prepped for my EM12c presentation in a quiet location and it was worth the time. My session was in full attendance and I was content with the outcome. The Q&A was not as intensive as my RMOUG session, but the questions were well thought out and discussion was actively participated in. The presentation was well received and only makes me more anxious for KSCOPE 2012 in San Antonio come June!

I attended Cary Millsap’s “Mastering Trace Data” class for the rest of the day. This was a great refresher course for me after taking the extended class over a year ago and reviewing the book used for the class back in December. Cary goes over the basic information so that anyone can easily utilize the Method R tool suite along with offering clear reasons to trace, showing how having this knowledge can make us better DBA’s.

The conference ended with a speech by Mogens Noorgard, awarding participants and those that came up with a creative way to explain the huge pool that had been a hit with the sauna partiers at the party house to Lilandia management.
Dan Norris had arranged with Mogens that I would drive back with him and his group. It was a good drive back to Malov, Denmark, near Copenhagen. His friend Anne is lovely, easy to talk to and Tuomas Pystynen is fun to be around, (except for those salted, black licorice fish candy he bought and had me try! )

I spent the last 24hrs at the Yellow Mansion, home of the Oak Table. Dan, Anne and Tuomas made sure I arrived at the airport on time to catch my plane and upon arriving at my connection at London/Heathrow, I entered the lift and heard a familiar voice. Looking over the crowd entering, I noticed Mark Rittman and yes, the person that was missing at MOW was on her way to Calloborate in Las Vegas- Debra Lilley. We shared a few words, hugs and kisses, then I was off on my last leg of my flight home to Denver.

My only regret regarding Miracle Oracle World is that I didn’t have more time in Denmark. Upon returning, viewed some of the pictures Dan Norris’ friend Anne took in Copenhagen after they dropped me off at the airport. Even one more day in the city would have been wonderful!

~ The least questioned assumptions are often the most questionable ~ Paul Broca

I’ve always found assumptions to be one of the most common cause of failures in software releases/maintenance windows.

If I had a dime for every time I’ve heard, “I thought you were going to take care of/do that…” or “I didn’t involve you earlier because I thought the steps you were responsible for were so easy…”, well, you know the saying and I’d be rich.

Assumptions causing participation of the DBA too late into a project, release or maintenance is widespread enough when you are onsite, but as a remote DBA, can take on a whole new dimension. Where being in the office area might allow you some over-heard conversation or privy to meetings that you realize you should be part of, working off-site can really set you up to miss out on important prep time to offer the best level of support.

It’s nothing new, not rare and its most obvious in hindsight, after the release or maintenance has completed. As paranoid as I think I am, causing me to involve myself pretty well, (I’m not a fan of surprises… ) I did experience it this last weekend as a new DBA for a client. Acclimating to a client, as well as they becoming comfortable and involving you as their new DBA takes time. Something we just didn’t have the opportunity to do much of, nor was it anyone’s fault. Enkitec was unaware of this year-end maintenance, so they assumed I would easily take-over ownership of the client from the previous DBA.

Friday there was some concern, after they sent an email with the tasks they needed my time allocated for that night and my “real” first time on the client’s systems, that there might be a disk space issue for the required backup post the special, once-a-year release upon completion.

I did some quick research after this was discovered and offered an option but the client’s Unix admin cleared off some disk space and assumed the space would be enough. Now the estimated space usage for the release was not that big, definitely not that large when you consider what I’m used to. we are talking gigabytes, not terabytes. Only being in the system for one day, I made my first mistake and assumed the problem was taken care of and proceeded to perform the duties I had been assigned me for that evening and let them work through the weekend.

The client had assumed the tasks were quite simple for a DBA- the previous DBA had been there the entire time they had been clients and the database growth had been of minimal concern. It was taken into consideration that I may require a little more time to find my way around the environment, become familiar with the specifics of design, jobs and backup scenarios, etc., but I had no issues with the pre-release work, so why would “reversing” the process for the post work result in any difficulties?

Post the weekend work, they contacted me and asked me to start the tasks for after year-end processing. Everything worked out well until close to midnight when the backup failed. We didn’t have enough space.

The backup strategy is not RMAN backup files, but image copies, level 1 incremental and the size of the database due to the release ALONG with additional UNDO, etc. caused the database to be too large to fit on the volume. Different strategies hadn’t helped, even splitting across multiple channels to multiple volumes was not enough, now I was having a formatting issue on the apply to the incremental. It did not like the change one bit and yes, it was after midnight, (have we discussed when a DBA’s horse-drawn carriage turns back into a pumpkin yet? )

The unique configuration and my newness to the environment meant that it did take me a bit longer to work and investigate issues, (although I do think this may be the best way to retain knowledge about an environment, it’s just more painful for everyone involved!) I finally had the answers I needed in the wee morning hours-
– how to retain the existing backup from before the release from the same volume as I needed more space on.
– change from an image copy incremental to a level 0, rman compressed backup.
– what parallelism was acceptable during business hours.
– what jobs had to be stopped to not impact production while all this occurred.

Now the second level of the problem- I’ve been up two nights of the last three, had been ill on Saturday and I was starting to feel my IQ slip away like the free space on the backup volume. Enkitec beleives in giving their clients valuable resources that are able to give them their best and I was in no way close to that. I really appreciate it that my secondary DBA to this client, Miranda was so helpful, so willing to take what I, as the brand new DBA, had and put the plan to action, (and make it sound so easy to me who had so little IQ left at that point! ) I promptly notified the client after I transitioned to her the information and then collapsed to a deep slumber.

Now we come to the moral of our story.
This should have been simple work for the DBA. It was assumed to be so by BOTH parties: the DBA and the client. This was our downfall in that we really should make only one assumption when it comes to maintenance and releases- If we assume, especially on the simple stuff, it will most likely be what causes our biggest challenges. The DBA should be involved from the beginning of any project, maintenance or release and then from there, once both sides have inspected the tasks/level of difficulty, can they both decide that the DBA is not required to be heavily involved. An open invitation should be available to the DBA to return if any “red flags” arise and all communication should still include the DBA to ensure that there are as few surprises as possible.

Two things to be happy about today!

1st-  Enkitec did a lovely announcement on Enkitec regarding my coming on board.  I just arrived last night after three days at the main office in Irving, Tx, (Dallas area is lovely with all the fields of Blue Bonnet flowers this time of year…) and am psyched about working remotely for them from my home, northwest of Denver.

2nd-  I have a wonderful opportunity to speak this evening at Colorado Springs SQL Pass group on “Oracle for the SQL Server DBA”.  SpringsSQL

It’s only going to get crazier the next two months from here, so stay tuned!