May 26th, 2017 by dbakevlar

I’m in sunny Phoenix this week at the Data Platforms 2017 Conference and looking forward to a break in the heat when I return to Colorado this evening.

As this event is for big data, I expected to present on how big data could benefit from virtualization, but was surprised to find that I didn’t have a lot of luck finding customers utilizing us for this reason, (yet).  As I’ve discussed in previous presentations, I was aware of what a “swiss army knife” virtualization is, resolving numerous issues, across a myriad of environments, yet often unidentified.

The Use Case

To find my use case, I went out to the web and found a great article, “The Case for Flat Files in Big Data Projects“, by Time.com interactive graphics editor, Chris Wilson.  The discussion surrounds the use of data created as part of ACA and used for another article, “How Much Money Does Your Doctor Get From Medical Companies“.  The data in the interactive graphs that are part of the article is publicly available from cms.gov and the Chris discusses the challenges created by it and how they overcame it.

Upon reading it, there was a clear explanation of why Chris’ team did what they did to consume the data in a way that complimented their skill set.  It resonated with anyone who works in any IT shop and how we end up with technical choices that we’re left to justify later on.  While observing conversations at the conference this week, I lost count of how often I accepted the fact that there wasn’t a “hadoop” shop or a “hive” shop, but everyone had a group of varied solutions that resulted in their environment and if you didn’t have it, don’t count it out-  Pig, Python, Kafka or others could show up tomorrow.

This results in a more open and accepting technical landscape, which I, a “legacy data store” technologist, was welcome.  When I explained my upcoming talk and my experience, no one turned up their nose at any technology I admitted to having an interest in.

With the use case found online, was also the data.  As part of the policies in the ACA, cms.gov site, (The Center for Medicare and Medicaid) you can get access to all of this valuable data and it can offer incredible insight into these valuable programs.  The Time article only focuses on the payments to doctors from medical companies, but the data that is collected, separated out by area and then zipped up by year, is quite extensive, but as noted by a third article, as anticipated as the data was, it was cumbersome and difficult to use.

The Requirements

I proceeded to take this use case and imagine it as part of an agile environment, with this data as part of the pipeline to produce revenue by providing information to the consumer.  What would be required and how could virtualization not only enhance what Chris Wilson’s team had built, but how could the entire pipeline benefit from Delphix’s “swiss army knife” approach?

  1.  I can’t assume this is the main data store.  These flat files are a supplement to legacy data stores.
  2. There would be a standard development environment-  development and testing would need their own environments, not just a production copy of these files, applications, etc.
  3. If it’s providing data to a consumer and data is in perpetual motion in the age of the internet, an agile development method would need to be in place, which means a short development cycle with many, small, “scrum like” development groups from different departments working on tasks.
  4. Automation and seamless deployment would assist in less human intervention and resource demands, along with more successful deployments.

Solution

There were four areas that I focused to solve and eliminate bottlenecks that I either experienced or foresaw an organization experiencing when having this data as part of their environment.

  1. Eliminate the need to have multiple copies of the files, slow and manual process to propagate files to targets for development, test, etc. with Delphix’s vFile option, this would include any applications or other non-relational database tier included in the scenario.
  2. Eliminate any legacy data stores copies and refreshes that big data was dependent from and create VDBs for all development, test and reporting.
  3. Protect all non-production environments by masking non-production databases and flat files.
  4. Containerize environments for easy deployment, delivery, testing and cloud migrations.

vFiles

Each of the files, compressed were just over 500M and uncompressed, 15-18G.  This took about over 4 minutes per file to transfer to a host and could add up to considerable space.

I used Delphix vFile to virtualize files.  This means that there is a single, “gold copy” host of the files at the Delphix engine and then there’s an NFS Mount that “projects” the file access to each target, which can be used for unique copies to as many development, test and reporting copies.

Fig. 1- Creating vFiles from dSource that flat files are sourced on.

If a refresh is required, then the parent is refreshed and an automated refresh to all the “children” can be performed.  Changes can be made at the child level and if catastrophic, Delphix would allow for easy recovery, allowing for data version control, not just code version control throughout the development and testing process.

Fig. 2- Demonstration of target vFile, showing NFS Mount, files available, (created in less than 10 seconds) and how easily disabled and proven to be “Projection” of files.

Its a pretty cool feature and one that is very valuable to the big data arena.  I heard countless stories of how often, due to lack of storage, data scientists and developers were taking subsets of data to test and then once to production, find out that their code wouldn’t complete or fail when presented against the full data.  Having the ability to have the FULL files without taking up more space for multiple environments would be incredibly beneficial and shorten development cycles.

Virtualize

Most big data shops are still dependent on legacy data stores.  These are legendary roadblocks due to their size, complexity and demands for refreshes.  I proposed that those be virtualized so that each developer could have a copy and instant refresh without storage demands to again, ease development deadline pressures and allow for full access of data towards the development success.

Protect

Most people know we mask relational databases, but did you know we have Agile Data Masking for  flat files?  If these files are going to be pushed to non-production systems, especially with as much as we’re starting to hear about GDPR, (General Data Protection Regulations) from the EU in the US now, shouldn’t we mask outside of the database?

What kind of files can be masked?

  • Multi-record
  • CSV
  • XML
  • Word
  • Excel
  • PowerPoint
  • Unstructured
  • EDI

Thats a pretty considerable and cool list.  The ability to go in and mask data from flat files is a HUGE benefit to big data folks.  Many of them were looking at file security from the permissions and encryption level, so the ability to render the data benign to risk is a fantastic improvement.

Containerize

The last step is in simple recognition that big data is complex and consists of a ton of moving parts.  To acknowledge how much is often home built, open source, consisting of legacy data stores, flat files, application and other dependent tiers, should be expected.

Fig. 3- A Container, created on-prem, then moved to the cloud and to as many environments as required for the development cycle to meet the business needs.

Delphix has the ability to create templates of our sources, (aka dSources) which is nothing more than creating a container.  In my use case enhancement, I took all of these legacy data stores, applications, (including any Ajax code) flat files and then create a template from it for simple refreshes, deployments via jenkins, Chef jobs or other DevOps automation.  The ability to then take these templates and deploy them to the cloud would make a migration from on-prem to the cloud a simpler process or from one cloud vendor to another.

Fig. 4- A look at the full scenario-  Delphix engines masking files, databases, creating containers and deploying it all on-prem and to the cloud.

The end story is that this use case could be any big data shop or start up in the world today.  So many of these companies are hindered by data and Delphix virtualization could easily let their data move at the speed of business.

I want to thank Data Platforms 2017 and all the people who were so receptive of my talk.  If you’d like access to the slide deck, it’s been uploaded to Slideshare. I had a great time in Phoenix and hope I can come back soon!

 

 

Posted in big data, Cloud, Data Masking, Oracle Tagged with: ,

May 9th, 2017 by dbakevlar

Different combination in the game of tech create a winning roll of the dice and other times create a loss.  Better communication between teams can offer a better opportunity to deter from holes in development cycle tools, especially when DevOps is the solution you’re striving for.

It doesn’t hurt to have a map to help guide you.  This interactive map from XebiaLabs can help offer a little clarity to the solutions, but there’s definitely some holes in multiple places that could be clarified a bit more.

The power of this periodic table of DevOps tools, isn’t just that they are broken up by the tool type, but that you’re able to easily filter by open source, freemium, paid or enterprise level tools.  This assists in reach the goals of your solution.  As we all get educated, I’ll focus horizontally in future posts, but today, we’ll take a vertical look at the Database Management tier and where I specialize first.

Apples aren’t Apfel

When comparisons are made, it’s common to have the inability to do apple to apples.  Focusing on the Database Management toolsets, such as Delphix, I can tell you that only Redgate I view as a competitor and this only happened recently with their introduction of SQL Clone.  The rest of the products shown don’t offer any virtualization, (our strongest feature) in their product and we consider Liquidbase and Datical partners in many use cases.

Any tool is better than nothing, even one that helps you choose tools.  So let’s first start to discern what the “Database Management”  tools are supposed to accomplish and then create one of our own.  The overall goal appears to be version control for your database, which is a pretty cool concept.

DBMaestro

The first product on the list is something I do like because the natural “control issues” I have as a DBA.  You want to know that changes to a database occurred in a controlled, documented and organized manner.  DBMaestro allows for this and has some pretty awesome ways of doing it.  Considering that DevOps is embracing agility at an ever increasing rate, having version control capabilities that will work with both Oracle SQL Developer and Microsoft Visual Studio are highly attractive.  The versioning is still as a code change level and not at the data level, but it’s still worthy of discussion.

That it offers all of this through a simple UI is very attractive to the newer generation of developers and DBAs will still want to be part of it.

Liquibase

This is the first of two companies we partner with that are in the list.  It’s very different from DBMaestro, as it’s the only open source in the database management products and is works with XML, JSON, SQL and other formats. You can build just about anything you require and the product has an extensive support community, so if you need to find an example, it’s pretty simple to do so online.

I really like the fact that Liquibase takes compliance into consideration and has the capability to delay SQL transactions from performing without the approval from the appropriate responsible party.  It may not be version control of your data, but at least you can closely monitor and time out the changes to it.

Where Liquibase partners with Delphix is that we can perform continuous delivery via Liquibase and Delphix can then version control the data tier.  We can be used for versioning, branching and refreshing from a previous snapshot if there was a negative outcome in a development or test scenario, making continuous deployment a reality without requiring “backup scripts” for the data changes.

Redgate SQL Control

Everybody love a great user interface and like most, there’s a pretty big price tag that goes along with the ease of use when adding up all the different products that’s offered.  There’s just a ton that you can do with Redgate and you can do most of it for Oracle, MSSQL and MySQL, which is way cool.  Monitor, develop, virtualize, but the main recognition that you’re getting with the periodic table for DevOps tools is for version control and comparisons.  This comes from the SQL Control product from Redgate and offers quite a full quite of products for the developer and the DBA.

Datical

This is another product that we’ve partnered with repeatedly.  The idea that we, as DBAs can review and monitor any and all changes to a database is very attractive to any IT shop. Having it simplified into a tool is incredibly beneficial to any business who wants to deliver continuously and when implemented with Delphix, then the data can be delivered as fast as the rest of the business.

Idera

Idera’s DB Change Manager can give IT a safety net to ensure that changes intended are the changes that happen in the environment.  Idera, just like many of the others on the list supports multiple database platforms, which is a keen feature of a database change control tool, but no virtualization or copy data management, (CDM) tool exists or at least, not one exists any longer.

Fitting in

So where does Delphix fit in with all of these products?  We touched on it a little bit as I mentioned each of these tools. Delphix is recognized for the ability to deploy and that it does so as part of continuous delivery is awesome, but as I stated, it’s not a direct apple to apples comparison as we not only offer versioning control, but we do so at the data level.

Delphix Jet Stream

So let’s create an example-

We can do versioning and track changes in releases in the way of our Jet Stream product.  Jet Stream is the much loved product for our developers and testers.

I’ve often appreciated any tool set that allowed others not only to fish instead of me fishing for them,  Offering the Developer or Tester access to the administration console meant for a DBA can only set them up to fail.

Jet Stream’s interface is really clean and easy to use.  It has a clear left hand panel with options to access and the interaction is direct on what the user will be doing.  I can create bookmarks, naming versions, which allows me the ability to

If a developer is using Jet Stream, they would make changes as part of a release and once complete, create a bookmark, (a snapshot in time) of their container, (A container here is made up of the database, application tier and anything else we want included that Delphix can virtualize.)

We’ve started our new test run of a new development deployment.  We’ve made an initial book mark singing the beginning of the test and then a second bookmark to say the first set of changes were completed.

At this time, there’s a script that removes 20 rows from a table.  The check queries all verified that this is the delete statement that should remove the 20 rows in question.

SQL> delete from kinder_tbl  where c2 like '%40%';

143256 rows deleted.

SQL> commit;

SQL> insert into kinder_tbl values (...

When the tester performs the release and hits this step, the catastrophic change to the data occurs.

Whoops, thats not 20 rows.

Now, the developer could grovel to the DBA to use archaic processing like flashback database or worse, import the data back into a second table and merge the missing rows, etc.  There’s a few ways to skin this cat, but what if the developer could recover the data himself?

This developer was using Jet Stream and can simply go into the console, where they’ve been taking that extra couple seconds to bookmark each milestone during the release to the database, which INCLUDES marking the changes to the data!

If we inspect the bookmarks, we can see that the second of three occurred before the delete of data.  This makes it simple to use the “rewind” option, (bottom right icon next to the trash can for removing the bookmark) to revert the data changes.  Keep in mind that this will revert the database back to the point in time when the bookmark was performed, so ALL changes will be reverted to that point in time.

Once that is done, we can verify quickly that our data is returned and no need to bother the DBA or the concern that catastrophic change to data has impacted the development or test environment.

SQL> select count(*) from kinder_tbl
  where c2 like '%40%';

  COUNT(*)
----------
    143256

I plan on going though different combinations of tools in the periodic table of DevOps tools and show what strengths and drawbacks there are to choices in implementation in upcoming posts, so until the next post, have a great week!

 

 

Posted in Delphix, devops Tagged with: ,

April 28th, 2017 by dbakevlar

I did a couple great sessions yesterday for the awesome Dallas Oracle User Group, (DOUG.)  It was the first time I did my thought leadership piece on Making Sense of the Cloud and it was a great talk, with some incredible questions from the DOUG attendees!

This points me to a great [older] post on things IT can do to help guarantee tech projects are more successful. DevOps is a standard in most modern IT shops and DBAs are expected to find ways to be part of this valuable solution.  If you inspect the graph, displaying the value of different projects in ROI, vs. how often these different types of projects run over budget and time, it may be surprising.

Where non-software projects are concerned, the project rarely runs over the schedule, but in the way of benefits, often comes up short.  When we’re dealing with software, 33% of project run over time, but the ROI is excruciatingly high and worthy of the investment.  You have to wonder how much of that over-allocation in time feeds into the percentage increase in cost?  If this could be deterred, think about how more valuable these projects would become?

The natural life of a database is growth.  Very few databases stay a consistent size, as companies prosper, critical data valuable to the company requires a secure storage location and a logical structure to report on that data is necessary for the company’s future.  This is where relational databases come in and they can become the blessing and the burden of any venture.  Database administrators are both respected and despised for their necessity to manage the database environment as the health of the database is an important part of the IT infrastructure and with the move to the cloud, a crucial part of any viable cloud migration project.

How much of that time, money and delay shown in those projects are due to the sheer size and complexity of the database tier?  Our source data shows how often companies just aren’t able to hold it together due to lacking skills, lacking estimates in time estimates and other unknowns that come back to bit us.

I can’t stress enough why virtualization is key to removing a ton of the overhead, time and money that ends up going into software projects that include a database.

Virtualizing non-production databases results in:

  1. Ability to deliver full copies of production for developers without extensive demands on storage.
  2. Ability to deliver those databases in a matter of minutes vs. days or weeks.
  3. Ability to refresh databases as needed for any project.
  4. Self-service user-interface so developers and testers can recover from a catastrophic issue in a database without having to grovel to a DBA to restore a virtual database.
  5. Ability to branch the VDB and do versioning, which is awesome for both developers and testers, (I know, we DBAs care very little about this feature… :))
  6. In migrations/cloud migrations, the ability to migrate databases in short periods of time and to limit the storage footprint to save company the money they were promised the cloud would deliver that most are finding out in the long run, is not occurring with traditional database scenarios.

It’s definitely something to think about and if you don’t believe me, test it yourself with a free trial!  Not enough people are embracing virtualization and it takes so much of the headache out of RDBMS management.

Posted in AWS, Azure, Cloud, Oracle, SQLServer Tagged with: , ,

April 18th, 2017 by dbakevlar

For over a year I’ve been researching cloud migration best practices.  Consistently there was one red flag that trips me that I’m viewing recommended migration paths.  No matter what you read, just about all of them include the following high level steps:

As we can see from above, the scope of the project is identified, requirements laid out and a project team is allocated.

The next step in the project is to choose one or more clouds, choose the first environments to test out in the cloud, along with security concerns and application limitations.  DBAs are tested repeatedly as they continue to try to keep up with the demand of refreshing or ensuring the cloud environments are able to keep in sync with on-prem and the cycle continues until a cutover date is issued.  The migration go or no-go occurs and the either non-production or all of the environment is migrated to the cloud.

As someone who works for Delphix, I focus on the point of failure where DBAs can’t keep up with full clones and data refreshes in cloud migrations or development and testing aren’t able to complete the necessary steps that could be if the company was using virtualization.  From a security standpoint, I am concerned with how few companies aren’t investing in masking with the sheer quantity of breeches in the news, but as a DBA, there is a whole different scenario that really makes me question the steps that many companies are using to migrate to the cloud.

Now here’s where they loose me every time- the last step in most cloud migration plans is to optimize.

I’m troubled by optimization being viewed as the step you take AFTER you migrate to the cloud.  Yes, I believe that there will undoubtedly be unknowns that no one can take into consideration before the physical migration to a cloud environment, but to take databases, “as is” when an abundance of performance data is already known about the database that could and will impact performance, seems to be inviting unwarranted risk and business impact.

So here’s my question to those investing in a cloud migration or have already migrated to the cloud-  Did you streamline and optimize your database/applications BEFORE migrating to the cloud or AFTER?

 

 

Posted in AWS, Azure, Oracle, SQLServer Tagged with: , ,

March 30th, 2017 by dbakevlar

Azure is the second most popular cloud platform to date, so it’s where Delphix naturally is going to support second on our road to the cloud.  As I start to work with the options for us deploying Delphix, there are complexities I need to educate myself on in Azure.  As we’re just starting out, there’s a lot to learn and a lot of automation we can take advantage of.  It’s an excellent time for me to get up to speed with this cloud platform, so hopefully everyone will learn right along with me!

We’ll be using Terraform to deploy to Azure, just as we prefer to use it for our AWS deployments.  It’s open source, very robust and has significant support in the community, so we’ll switch from cloud setup to Terraform prep in many of these posts.  Before we can do that, we need to set up our Azure environment after we’ve registered our subscription with Azure.

Azure Consoles

There are the New and the Classic consoles for Azure, but also ones in the modern, New console that are marked as “Classic” that aren’t part of the actual “Classic” one.  I found this a bit confusing, so it’s good to have the distinction.

Azure’s “New” Portal, with it’s modern, sleek design

Azure’s “Classic” management interface, with it’s pale blue and white schema, which still serves a very significant purpose

Once you’ve created your Azure account, you’ll find that you need access to the Classic console to perform many of the basic setup tasks, where the Modern console is better for advanced administration.

Preparation is Key

There are a number of steps you’ll need to perform in preparation for Delphix to deploy to Azure.  The delphix engine, a source and a target are out goal, so we’ll start simple and work our way out.  Let’s see how much I can figure out and how much I may need to look to others more experienced to get me through.  No matter what, you’ll need both consoles, so keep the links above handy and I’ll refer to the consoles by “New” and “Classic” to help distinguish them as I go along.  Know that in this post, we’ll spend most of our time in the Classic console.

Set up an Account and Add Web App

If you don’t already have one, Microsoft will let you set up an account and even give you $200 in free credits to use.  Once you sign up, then you need to know where to go next.  This is where the “Classic” console comes in, as you need to set up your “application” that will be used for your deployment.

Log into the “Classic” console and click on Active Directory and the Default Directory highlighted in blue.  This will open a new page and you will have the opportunity to click Add at the bottom of the page to add a new Active Directory application.

  • Name the Application, (open up a text editor, copy and paste the name of the app into it, you’ll need this data later)
  • The application type is web app or api
  • Enter a URL/URI and yes, they can be made up.  They don’t have to be real.

Client and Client Secret

Now that your application is created, you’ll see a tab called Configure.  Click on this tab and you’ll see the client ID displayed.  Copy the Client ID and add that to your text editor, also for later.

Scroll down and you’ll see a section called Keys.  Click on the button that says “Select Duration” and choose 1 or 2 years.  At the very bottom of the screen, you’ll see a Save button, click it and then the Secret passcode will be displayed for you to copy and paste into your text editor.  Do this now, as you won’t be able to get to it later.

Tenant ID

To the left of the Save button, you’ll see “View Endpoints”.  Click on this and you’ll see a number of entries.  The tenant ID is the repeat value shown in each of the entries at the end.  An example is shown below:

Copy and paste this into your text editor under a heading of tenant ID.

Add Application to the Active Directory

Now that you’ve created this framework, you need to grant permissions to use it all.  In the Configure tab, scroll to the very bottom where it says “Permissions to Other Applications” and click on Add Application.  Choose the Active Directory application from the list, (if you have a new account, you won’t have much to choose from) Azure Service Management API and click on the checkmark in the lower right corner of the pane.  This will return you to the previous page.  Click on the designated privileges and choose to grant it Access Azure Service Management as organization and then save.

Subscription Data

Now, log into the New portal and click on Subscriptions on the left hand side.  Click on the Subscription and it will open up to display your Subscription ID, which you’ll need to copy and paste into your text editor.

Click on Access Control, (IAM) and click on Add.  Now you may only see your username, but the applications are there-  they just won’t be displayed by default.  Type in your application name that you put in your text editor, (example, mine is Web_K_Terra.)  Reminder-  you must type in the name of your app, just as you did when you created it, (it is cap sensitive, etc.) Grant reader and contributor roles from the role list, saving between each additional role.

You should now see your user in the list with both roles assigned to it like the example below for Web_K_Terra app:

Our configuration is complete and ready to go onto the networking piece.

The first part of my terraform template is ready, too.  All the pertinent data that I required from my build out has been added to it and it looks something like the following:

provider “Web_K_Terra” {
subscription_id = “gxxxxxxx-db34-4gi7-xxxxx-9k31xxxxxxxxxp2”
client_id = “d76683b5-9848-4d7b-xxxx-xxxxxxxxxxxx”
client_secret = “trKgvXxxxxxXXXXxxxXXXXfNOc8gipHf-xxxxxxXXx=”
tenant_id = “xxxxxxxx-9706-xxxx-a13a-4a8363bxxxxx”

}

This is a great start to getting us out on Azure, in part II, we’ll talk about setting up connectivity between your desktop and Azure for remote access and recommendations for tools to access it locally.

 

Posted in Azure, Oracle Tagged with: , ,

February 20th, 2017 by dbakevlar

We’ve been working hard to create an incredible new trial version of Delphix that uses AWS, which is built with the open source product Terraform.  Terraform is a tool that anyone can use to build, version and manage a product effectively and seamlessly in a number of clouds.  We are currently using it to implement to AWS, but there is a bright future for these types of open source products and I’m really impressed with how easy its made it to deploy compute instances, the Delphix Engine and supporting architecture on AWS EC2.  If you’d like to read up on Terraform, check out their website.

The Delphix Admin Console and Faults

After building out the Delphix environment with the Engine and the a Linux source/target, the first step for many is to log into the Delphix Admin console.  You can view any faults during the build at the upper right corner under Faults. One error that I’ve noticed comes up in after a successful build is the following:

AWS Console to the Rescue

By logging into your AWS EC2 console, you can view the instances that are being used.  As you’ll note, the error says that the Delphix Engine is using an unsupported  instance type m4.large.  Yes in our EC2 console, we can see the Delphix Engine, (last in the list and with the name ending in “DE”) that no, actually it isn’t.

It’s actually a m4.xlarge instance type.  What’s even more interesting, is that the Linux Target, (LT) and Linux Source, (LS) are both m4.large instance types, yet no warning was issued for either of these instances as unsupported.

AWS EC2 Supported Instance Types

You can locate what types of instance types are supported for AWS EC2 with the following link.  At this page, we can also see that both the m4.large and the m4.xlarge instance type IS SUPPORTED.

Knowing that we’ve validated that the instance type is supported means that I can safely ignore it and proceed to work through the trial without worry.

If you’re planning on deploying a production Delphix Engine on AWS, inspect the following document to ensure you build it with the proper configuration.

Nothing to see here and thought I better let everyone know before someone lumps Amazon with CNN… 🙂

Posted in AWS Trial, Oracle Tagged with: ,

February 13th, 2017 by dbakevlar

When tearing down an AWS Delphix Trial, we run the following command with Terraform:

>terraform destroy

I’ve mentioned before that every time I execute this command, I suddenly feel like I’m in control of the Death Star in Star Wars:

As this runs outside of the AWS EC2 web interface, you may see some odd information in your dashboard.  In our example, we’ve run “terraform destroy” and the tear down was successful:

So you may go to your volumes and after verifying that yes, no volumes exist:

The instances may still show the three instances that were created as part of the trial, (delphix engine, source and target.)

These are simply “ghosts of instances past.”  The tear down was completely successful and there’s simply a delay before the instance names are removed from the dashboard.  Notice that they no longer are listed with a public DNS or IP address.  This is a clear indication that these aren’t currently running, exist or more importantly, being charged for.

Just one more [little] thing to be aware of… 🙂

Posted in AWS Trial, Delphix Tagged with: ,

February 13th, 2017 by dbakevlar

Now, for most of us, we’re living in a mobile world, which means as our laptop travels, our office moves and our IP address changes.  This can be a bit troubling for those that are working in the cloud and our configuration to our cloud relies on locating us via our IP Address being the same as it was in our previous location.

What happens if you’re IP Address changes from what you have in your configuration file, (in Terraform’s case, your terraform.tfvars file) for the Delphix AWS Trial?  I set my IP Address purposefully incorrect in the file to demonstrate what would happen after I run the terraform apply command:

It’s not the most descriptive error, but that I/O timeout should tell you right away that terraform can’t connect back to your machine.

Addressing the IP Address Issue

Now, we’ll tell you to capture your current IP address and update the IP address in the TFVARS file that resides in the Delphix_demo folder, but I know some of you are wondering why we didn’t just build out the product to adjust for an IP address change.

The truth is, you can set a static IP Address for your laptop OR just alias your laptop with the IP Address you wish to have.  There are a number of different ways to address this, but looking into the most common, let’s dig into how we would update the IP address vs. updating the file.

Static IP Address

You can go to System Preferences or Control Panel, (depending on which OS you’re on) and click on Network and configure your TCP/IP setting to manual and type in an IP Address there.  The preference is commonly to choose a non-competitive IP address, (often the one that was dynamically set will do as your manual one to retain) and choose to save the settings.  Restart the PC and you can then add that to your configuration files.  Is that faster than just updating the TFVARS file-  nope.

Setting the IP Address- Mac/Linux

The second way to do this is to create an Alias IP address to deter from the challenge of each location/WiFi move having it automatically assigned.

Just as above, we often will use the IP address that was given dynamically and just choose to keep this as the one you’ll keep each time.  If you’re unsure of what your IP is, there are a couple ways to collect this information:

Open up a browser and type in “What is my IP Address

or from a command prompt, with “en0” being your WiFi connection, gather your IP Address one of two ways:

$ dig +short myip.opendns.com @resolver1.opendns.com
$ ipconfig getifaddr en0

Then set the IP address and cycle the your WiFi connection: 


$ sudo ipconfig set en0 INFORM <IP Address>
$ sudo ifconfig en0 down 
$ sudo ifconfig en0 up

You can also click on your WiFi icon and reset it as well.  Don’t be surprised if this takes a while to reconnect.  Renewing and releasing of IP addresses can take a bit across the network and the time required must be recognized.

An Alias on Mac/Linux

Depending on which OS you’re on.  Using the IP Address from your tfvars file, set it as an alias with the following command:

$ sudo ifconfig en0 alias <IP Address> 255.255.255.0

Password: <admin password for workstation>

If you need to unset it later on:

sudo ifconfig en0 -alias <IP Address>

I found this to be an adequate option-  the alias was always there, (like any other alias, it just forwards everything onto the address that you’re recognized at in the file.) but it may add time to the build, (still gathering data to confirm this.)  With this addition, I shouldn’t have to update my configuration file, (for the AWS Trial, that means setting it in our terraform.tfvars in the YOUR_IP parameter.)

Setting your IP Address on Windows

The browser commands to gather your IP Address work the same way, but if you want to change it via the command line, the commands are different for Windows PC’s:

netsh interface ipv4 show config

You’ll see your IP Address in the configuration.  If you want to change it, then you need to run the following:

netsh interface ipv4 set address name="Wi-Fi" static <IP Address> 255.255.255.0 <Gateway>
netsh interface ipv4 show config

You’ll see that the IP Address for your Wi-Fi has updated to the new address.  If you want to set it to DHCP, (dynamic) again, run the following:

netsh interface ipv4 set address name="Wi-Fi" source=dhcp

Now you can go wherever you darn well please, set an alias and run whatever terraform commands you wish.  All communication will just complete without any error due to a challenging new IP address.

Ain’t that just jiffy? OK, it may be quicker to just gather the IP address and update the tfvars file, but just in case you wanted to know what could be done and why we may not have built it into the AWS Trial, here it is! 🙂

Posted in AWS Trial, Delphix Tagged with: , , ,

February 10th, 2017 by dbakevlar

I ran across an article from 2013 from Straight Talk on Agile Development by Alex Kuznetsov and it reminded me how long we’ve been battling for easier ways of doing agile in a RDBMS environments.

Getting comfortable with a virtualized environment can be an odd experience for most DBAs, but as soon as you recognize how similar it is to a standard environment, we stop over-thinking it and it makes it quite simple to then implement agile with even petabytes of data in an relational environment without using slow and archaic processes.

The second effect of this is to realize that we may start to acquire secondary responsibilities and take on ensuring that all tiers of the existing environment are consistently developed and tested, not just the database.

A Virtual Can Be Virtualized

Don’t worry-  I’m going to show you that its not that difficult and virtualization makes it really easy to do all of this, especially when you have products like Delphix to support your IT environment. For our example, we’re going to use our trusty AWS Trial environment and we have already provisioned a virtual QA database.  We want to create a copy of our development virtualized web application to test some changes we’ve made and connect it to this new QA VDB.

From the Delphix Admin Console, go to Dev Copies and expand to view those available.  Click on Employee Web Application, Dev VFiles Running.  Under TimeFlow, you will see a number of snapshots that have been taken on a regular interval.  Click on one and click on Provision.

Now this is where you need the information about your virtual database that you wish to connect to:

  1.  You will want to switch from provisioning to the source to the Linux Target.

Currently the default is to connect to the existing development database, but we want to connect to the new QA we wish to test on.  You can ssh as delphix@<ipaddress for linuxtarget> to connect to and gather this information.

2.  Gathering Information When You Didn’t Beforehand

I’ve created a new VDB to test against, with the idea, that I wouldn’t want to confiscate an existing VDB from any of my developers or testers.  The new VDB is called EmpQAV1.  Now, if you’re like me, you’re not going to have remembered to grab the info about this new database before you went into the wizard to begin the provisioning.  No big deal, we’ll just log into the target and get it:

[delphix@linuxtarget ~]$ . 11g.env
[delphix@linuxtarget ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.4/db_1
[delphix@linuxtarget ~]$ ps -ef | grep pmon
delphix  14816     1  0 17:23 ?        00:00:00 ora_pmon_devdb
delphix  14817     1  0 17:23 ?        00:00:00 ora_pmon_qadb
delphix  17832     1  0 17:32 ?        00:00:00 ora_pmon_EmpQAV1
delphix  19935 19888  0 18:02 pts/0    00:00:00 grep pmon

I can now set my ORACLE_SID:

[delphix@linuxtarget ~]$ export ORACLE_SID=EmpQAV1

Now, let’s gather the rest of the information we’ll need to connect to the new database by connecting to the database and gathering what we need.

[delphix@linuxtarget ~]$ lsnrctl services
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-FEB-2017 18:13:06
Copyright (c) 1991, 2013, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
Services Summary...
Service "EmpQAV1" has 1 instance(s).
  Instance "EmpQAV1", status READY, has 1 handler(s) for this service...

Provision Your VFile

Fill in all the values required in the next section of the provisioning setup:

Click Next and add any requirements to match your vfile configuration that you had for the existing environment.  For this one, there aren’t any, (additional NFS Mount points, etc.)  Then click Next and Finish.

The VFile creation should take a couple minutes max and you should now see an environment that looks similar to the following:

This is a fully functional copy of your web application, created from another virtual copy that can test against a virtual database, ensuring that all aspects of a development project are tested thoroughly before releasing to production!

Why would you choose to do anything less?

 

 

 

Posted in AWS Trial, Delphix, Oracle Tagged with: ,

February 10th, 2017 by dbakevlar

So you thought you were finished configuring your AWS target, eh?  I already posted a previous time on how to address a fault with the RMEM, but now we’re onto the WMEM.  Wait, WM-what?

No, I fear a DBAs work is never over and when it comes to the cloud, our skill set has just expanded from what it was when we worked on-premise!

Our trusty Delphix Admin Console keeps track of settings on all our sources and targets, informing us when the settings aren’t set to what is recommended, so that we’ll be aware of any less than optimal parameters that could effect performance.

As we address latency in cloud environments, network settings become more important.

How WMEM Differs from RMEM

RMEM= receive

WMEM=send

Where RMEM is quite easy to remember as receive settings, we get to thank

 

As root, we’ll add another line to the sysctl.conf file to reflect values other than defaults:

$ echo 'net.ipv4.tcp_wmem= 102404194304 12582912' >> /etc/sysctl.conf

Reload the values into the system:

$ sysctl -p /etc/sysctl.conf

Verify the settings are now active:

$ sysctl -a | grep net.ipv4.tcp_wmem

net.ipv4.tcp_wmem = 10240 4194304 12582912

That’s all there is to it.  Now you can mark the fault as resolved in the Delphix Admin Console.

Posted in AWS Trial, Delphix Tagged with: , ,

February 8th, 2017 by dbakevlar

There are more configurations for AWS than there are fish in the sea, but as the rush of folks arrive to test out the incredibly cool AWS Trial for Delphix, I’ll add my rendition of what to look for to know you’re AWS setup is prepped to successfully deploy.

The EC2 Dashboard View

After you’ve selected your location, set up your security user/group and key pairs, there’s a quick way to see, (at least high level) if you’re ready to deploy the AWS Trial to the zone in question.

Go to your EC2 Dashboard and to the location, (Zone) that you plan to deploy your trial to and you should see the following:

Notice in the dashboard, you can see that the key pairs, (1) and the expected Security Groups, (3) are displayed, which tells us that we’re ready to deploy to this zone.  If we double click on the Key Pair, we’ll see that its match to the one we downloaded locally and will use in our configuration with Terraform:

How Terraform Communicates with AWS

These are essential to deploying in an AWS zone that’s configured as part of your .tfvars file for terraform.  You’ll note in the example below, we have both designated the correct zone and the key pair that is part of the zone we’ll be using to authenticate:


#VERSION=004

#this file should be named terraform.tfvars

# ENTER INPUTS BELOW

access_key="XXXXXXX"

secret_key="XXXXXXXXXX"

aws_region="us-east-1"

your_ip="xxx.xx.xxx.xxx"

key_name="Delphix_east1" #don't include .pem in the key name 

instance_name="Delphix_AWS"

community_username="xxx@delphix.com"

community_password="password"

Hopefully this is a helpful first step in understanding how zones, key pairs and security groups interact to support the configuration file, (tfvars) file that we use with the Delphix deployment via Terraform into AWS.

 

Posted in AWS Trial, Delphix, Oracle Tagged with: ,

February 6th, 2017 by dbakevlar

Ah, yes, it’s that RMOUG Training Days time of the year again!

As a techie, I didn’t put a lot of time into my slides when I first started presenting, thinking I’d simply dazzle them with my amazing knowledge.

What I found is that slide deck skills don’t just help your presentation, they help you tell the story better, which should be the goal of every presenter.  No matter how technical we are, we most likely haven’t had any formal Powerpoint training and as I’ve been upping my “PPT skills” this last year, I thought I’d share with the rest of the class…:)

Presenter View

Presenter View something that would have really helped me out when I first started.  To not have to remember every little detail and to have my notes displayed at the bottom of my screen would have been a stellar advancement.

Having notes at the bottom upped my presentation game, as I found I removed much of the text on my slides and instead used reference links to blog posts, articles and white papers to fully engage the attendee in my sessions vs. receiving minimal thoughts displayed on a slide.

I found that I went to more full screen graphics with a single phrase and was able to simply talk with the audience, having my notes keep me on topic, without having a slide full of discussion points that were more for me than those in attendance

Animations

 

A picture is worth a thousand words, but a moving picture?  That’s a whole story and the reason people go to movies, watch TV and why even adults like cartoons.  We can display an idea with movement more completely than a still image.

Powerpoint makes animations quite simple and it just takes the ability to group, highlight and animate with the animation toolbar:

All the basic and even a number of advanced animations are available to make your presentations pop and the ability to create a complex animation to get your point across!

Full Screen Graphics

Why is it great to have every other or every third slide a full screen graphic?  It gives the audience a rest between the data they’ve been taking in and with technical or complex topic presentations, your attendee is likely to appreciate the break.

No matter if you’ve just discussed the algorithm used for a specific data masking feature or like the slide above, discussing networking for geeky introverts, a slide like this makes an impact and the audience will more likely remember intricate details when combined with a humorous phrase and memorable picture.

Always use stock images and if you end up using an image from someone else’s site, etc. ask and give credit.

 

Layout

I used to update all my slides to a new template by copying them over in slide view, but now I know that I can switch them to a new slide template and switch out individual slides to new single template slides with the Layout menu in Powerpoint.

Having more control about the layout-  everything from text, to graphics, etc. saves me time from manually updating everything in a single slide.  If there is a common format that you use, you can make a copy of the template and save it off to use in the future, too.

Well, it’s time for me to get more tasks in preparation for the conference done!  We’re looking forward to seeing everyone at RMOUG Training Days 2017!

Conference Sessions:

Wednesday:  Introduction for Connor McDonald from Oracle, 2017 Keynote- 9:45am, Main Ballroom

  Lunch with the Experts- 12:30pm, Main Ballroom

                       Women in Technology Round Table with Komal Goyal and Rene Antunez– 1:15pm, room 1A

Social Media, the Next Generation with Rene Antunez– 4pm, room 1A

Welcome Reception Host- 5:30pm in the Main Ballroom

Thursday:      Delphix Hands on Lab- 9:00am in the OTN Area

 Lunch with the Experts– 12:30pm, Main Ballroom

Closing Session for the OTN Area– 2:00pm

Virtualization and the Cloud– 4:00pm- room 4F

I will be found at the Registration Desk, the OTN area doing interviews, taking pictures and videos throughout the other times of the conference!

Posted in DBA Life, Eventify Events Tagged with: , ,

February 1st, 2017 by dbakevlar

Delphix focuses on virtualizing non-production environments, easing the pressure on DBAs, resources and budget, but there is a second use case for product that we don’t discuss nearly enough.

Protection from data loss.

Jamie Pope, one of the great guys that works in our pre-sales engineering group, sent Adam and I an article on one of those situations that makes any DBA, (or an entire business, for that matters) cringe.  GitLab.com was performing some simple maintenance and someone deleted the wrong directory, removing over 300G of production data from their system.  It appears they were first going to use PostgreSQL “vacuum” feature to clean up the database, but decided they had extra time to clean up some directories and that’s where it all went wrong.  To complicate matters, the onsite backups had failed, so they had to go to offsite ones, (and every reader moans…)

Even this morning, you can view the tweets of the status for the database copy and feel the pain of this organization as they try to put right the simple mistake.

Users are down as they work to get the system back up.  Just getting the data copied before they’re able to perform the restore is painful and as a DBA, I feel for the folks involved:

How could Delphix have saved the day for GitLab?  Virtual databases, (VDBs) are read/write copies and derived from a recovered image that is compressed, duplicates removed and then kept in a state of perpetual recovery having the transactional data applied in a specific interval, (commonly once every 24 hrs) to the Delphix Engine source.  We support a large number of database platforms, (Oracle, SQL Server, Sybase, SAP, etc) and are able to virtualize the applications that are connected to them, too.  The interval of how often we update the Delphix Engine source is configurable, so depending on network and resources, this interval can be decreased to apply more often, depending on how up to date the VDBs need to be vs. production.

With this technology, we’ve come into a number of situations where customers suffered a cataclysmic failure situation in production.  While traditionally, they would be dependent upon a full recovery from a physical backup via tape, (which might be offsite) or scrambling to even find a backup that fit within a backup to tape window, they suddenly discovered that Delphix could spin up a brand new virtual database with the last refresh before the incident from the Delphix source and then use a number of viable options to get them up and running quickly.

  1. Switch the users and application to point to the new VDB that was recovered to the point in time, (PIT) before the incident occurred.  Meanwhile, IT is able to take their time recovering the production database with the physical backup, with little outage to the business.
  2. Create a VDB to the PIT before the failure and then create a connection between the production and the VDB, making a copy back to production of the data that was lost.
  3. If there was dire loss, (i.e. disk, etc.)  create a VDB to the PIT before the failure and perform what’s called a V2P, or virtual to physical, rehydrating the virtual data to become the new physical database.

This is the type of situation happens more often then we’d like to admit.  Many times resources have been working long shifts and make a mistake due to exhaustion, other times someone unfamiliar and with access to something they shouldn’t simply make a dire mistake, but these things happen and this is why DBAs are always requesting two or three methods of backups.  We learn quite quickly we’re only as good as our last backup and if we can’t protect the data, well, we won’t have a job for very long.

Interested in testing it out for yourself?  We have a really cool free Delphix trial via Amazon cloud that uses your AWS account.  There’s a source host and databases, along with a virtual host and databases, so you can create VDBs, blow away tables, recovery via a VDB, create a V2P, (virtual to physical) all on your own.

 

Posted in AWS Trial, cloning, Delphix Tagged with: , ,

January 31st, 2017 by dbakevlar

I’ve been at Delphix for just over six months now.  In that time, I was working with a number of great people on a number of initiatives surrounding competitive, the company roadmap and some new initiatives.  With the introduction of our CEO, Chris Cook, new CMO, Michelle Kerr and other pivotal positions within this growing company, it became apparent that we’d be redirecting our focus on Delphix’s message and connections within the community.

I was still quite involved in the community, even though my speaking had been trimmed down considerably with the other demands at Delphix.  Even though I wasn’t submitting abstracts to many of the big events I’d done so in previous years, I still spoke at 2-3 events each month during the fall and made clear introductions into the Test Data Management, Agile and re-introduction into the SQL Server communities.

As of yesterday, my role was enhanced so that evangelism, which was previously 10% of my allocation, is now going to be upwards of 80% as the Technical Evangelist for the Office of the CTO at Delphix.  I’m thrilled that I’m going to be speaking, engaging and blogging with the community at a level I’ve never done before.  I’ll be joined by the AWESOME Adam Bowen, (@CloudSurgeon on Twitter) in his role as Strategic Advisor and as the first members of this new group at Delphix.  I would like to thank all those that supported me to gain this position and the vision of the management to see the value of those in the community that make technology successful day in and day out.

I’ve always been impressed with the organizations who recognize the power of grassroots evangelism and the power it has in the industry.  What will I and Adam be doing?  Our CEO, Chris Cook said it best in his announcement:

As members of the [Office of CTO], Adam and Kellyn will function as executives with our customers, prospects and at market facing events.  They will evangelize the direction and values of Delphix; old, current, and new industry trends; and act as a customer advocate/sponsor, when needed.  They will connect identified trends back into Marketing and Engineering to help shape our message and product direction.  In this role, Adam and Kellyn will drive thought leadership and market awareness of Delphix by representing the company at high leverage, high impact events and meetings. []

As many of you know, I’m persistent, but rarely patient, so I’ve already started to fulfill my role and be prepared for some awesome new content, events that I’ll be speaking at and new initiatives.  The first on our list was releasing the new Delphix Trial via the Amazon Cloud.  You’ll have the opportunity to read a number of great posts to help you feel like an Amazon guru, even if you’re brand new to the cloud.  In the upcoming months, watch for new features, stories and platforms that we’ll introduce you to. This delivery system, using Terraform, (thanks to Adam) is the coolest and easiest way for anyone to try out Delphix, with their own AWS account and start to learn the power of Delphix with use case studies that are directed to their role in the IT organization.

Posted in AWS Trial, DBA Life, Delphix, Oracle Tagged with: ,

January 30th, 2017 by dbakevlar

I don’t want to alarm you, but there’s a new Delphix trial on AWS!  It uses your own AWS account and with a simple set up, allows you to deploy a trial Delphix environment.  Yes, you hear me right-  just with a couple steps, you could have your own setup to work with Delphix!

There’s documentation to make it simple to deploy, simple to understand and then use cases for individuals determined by their focus, (IT Architect, Developer, Database Administrator, etc.)

This was a huge undertaking and I’m incredibly proud of Delphix to be offering this to the community!

So get out there and check this trial out!  All you need is an AWS account on Amazon and if you don’t have one, it only takes a few minutes to create one and set it up, just waiting for a final verification before you can get started!  If you have any questions or feedback about the trial, don’t hesitate to email me at dbakevlar at gmail.

Posted in AWS Trial, Cloud, Delphix Tagged with: , ,

January 24th, 2017 by dbakevlar

So you’ve deployed targets with Delphix on AWS and you receive the following error:

It’s only a warning, but it states that you’re default of 87380 is below the recommended second value for the ipv4.tcp.rmem property.  Is this really an issue and do you need to resolve it?  As usual, the answer is “it depends” and its all about on how important performance is to you.

What is net.ipv4.tcp.rmem?

To answer this question, we need to understand network performance.  I’m no network admin, so I am far from an expert on this topic, but as I’ve worked more often in the cloud, it’s become evident to me that the network is the new bottleneck for many organizations.  Amazon has even build a transport, (the Snowmobile) to bypass this challenge.

The network parameter settings in question have to do with network window sizes for the cloud host in question surrounding TCP window reacts and WAN links.  We’re on AWS for this environment and the Delphix Admin Console was only the messenger to let us know that our setting currently provided for this target are less than optimal.

Each time the sender hits this limit, they must wait for a window update before they can continue and you can see how this could hinder optimal performance for the network.

Validation First

To investigate this, we’re going to log into our Linux target and SU over as root, which is the only user who has the privileges to edit this important file.:

$ ssh delphix@<IP Address for Target>
$ su root

As root, let’s first confirm what the Delphix Admin Console has informed us of by running the following command:

$ sysctl -a | grep net.ipv4.tcp_rmem 

net.ipv4.tcp_rmem = 4096 87380 4194304

There are three values displayed in the results:

  • The first value is the minimum amount of receive window that will be set to each TCP connection, even when the system is overwhelmed.
  • The default value allocated to each tcp connection,
  • The third is the maximum that can be allocated to any TCP connection.

To translate what this second value corresponds to-  this is the size of data in flight any sender can communicate via TCP to the cloud host before having to receive a window update.

So why are faster networks better?  Literally, the faster the network, the closer the bits and the more data that can be transferred.  If there’s a significant delay, due to a low setting on the default of how much data can be placed on the “wire”, then the receive window won’t be used optimally.

This will require us to update our parameter file and either edit or add the following lines:

net.ipv4.tcp_window_scaling = 1

net.core.rmem_max = 16777216

net.ipv4.tcp_rmem = 4096 12582912 16777216
I’m using the value as recommended by Brendan Gregg’s blog post on tuning EC2 instances.  This leaves a pretty narrow difference between the minimum and maximum for the window receive, but it is now within the recommended range for enhanced performance.
After you’ve updated the sysctl.conf file, you’ll need to reload it with the following command:
$ sysctl -p /etc/sysctl.conf
$ sysctl -a | grep net.ipv4.tcp_rmem 

net.ipv4.tcp_rmem = 4096 12582912 16777216

Ahhh, that looks much better… 🙂

Posted in AWS Trial, Delphix Tagged with: , ,

January 18th, 2017 by dbakevlar

So Brent Ozar’s group of geeks did something that I highly support-  a survey of data professional’s salaries.  Anyone who knows me, knows I live by data and I’m all about transparency.  The data from the survey is available for download from the site and they’re promoting app developers to download the Excel spreadsheet of the raw data and work with it.

Now I’m a bit busy with work as the Technical Intelligence Manager at Delphix and a little conference that I’m the director for, called RMOUG Training Days, which is less than a month from now, but I couldn’t resist the temptation to load the data into one of my XE databases on a local VM and play with it a bit...just a bit.

It was easy to save the data as a CSV and use SQL Loader to dump it into Oracle XE.  I could have used BCP and loaded it into SQL Server, too, (I know, I’m old school) but I had a quick VM with XE on it, so I just grabbed that quick to give me a database to query from.  I did edit the CSV and removed both the “looking” column and took out the headers.  If you choose to keep them, make sure you add the column back into the control file and update the “options ( skip=0)” to be “options ( skip=1)” to not load the column headers as a row in the table.

The control file to load the data has the following syntax:

--Control file for data --
options ( skip=0 )
load data
 infile 'salary.csv'
 into table salary_base
fields terminated by ','
optionally enclosed by '"'
 (TIMEDT DATE "MM-DD-YYYY HH24:MI:SS"
 , SALARYUSD
 , PRIMARYDB
 , YEARSWDB
 , OTHERDB
 , EMPSTATUS
 , JOBTITLE
 , SUPERVISE
 , YEARSONJOB
 , TEAMCNT
 , DBSERVERS
 , EDUCATION
 , TECHDEGREE
 , CERTIFICATIONS
 , HOURSWEEKLY
 , DAYSTELECOMMUTE
 , EMPLOYMENTSECTOR)

and the table creation is the following:

create table SALARY_BSE(TIMEDT TIMESTAMP not null,
SALARYUSD NUMBER not null,
COUNTRY VARCHAR(40),
PRIMARYDB VARCHAR(35),
YEARSWDB NUMBER,
OTHERDB VARCHAR(150),
EMPSTATUS VARCHAR(100),
JOBTITLE VARCHAR(70),
SUPERVISE VARCHAR(80),
YEARSONJOB NUMBER,
TEAMCNT VARCHAR(15),
DBSERVERS VARCHAR(50),
EDUCATION VARCHAR(50),
TECHDEGREE VARCHAR(75),
CERTIFICATIONS VARCHAR(40),
HOURSWEEKLY NUMBER,
DAYSTELECOMMUTE VARCHAR(40),
EMPLOYMENTSECTOR VARCHAR(35));

I used Excel to create some simple graphs from my results and queried the data from SQL Developer, (Jeff would be so proud of me for not using the command line… :))

Here’s what I queried and found interesting in the results.

We Are What We Eat, err Work On

The database flavors we work on may be a bit more diverse than most assume.  Now this one was actually difficult, as the field could be freely typed into and there were some mispellings, combinations of capital and small letters, etc.  The person who wrote “postgress”, yeah, we’ll talk… 🙂

The data was still heavily askew towards the MSSQL crowd. Over 2700 respondents were SQL Server and only 169 were listed their primary database platform as others, but Oracle was the majority:

You Get What you Pay For

Now the important stuff for a lot of people is the actual salary.  Many folks think that Oracle DBAs make a lot more than those that specialize in SQL Server, but I haven’t found that and as this survey demonstrated, the averages were pretty close here, too. No matter if you’re Oracle or SQL Server, we ain’t making as much as that Amazon DBA…:)

Newbies, Unite

Many of those who filled out the survey haven’t been in the field that long, (less than five years).  There’s still a considerable amount of folks who’ve been in the industry since it’s inception.

We’ve Found Our Place

Of the 30% of us that don’t have degrees in our chosen field, most of us stopped after getting a bachelors to find our path in life:

There’s still a few of us, (just under 200) out there who had to accumulate a lot of school loans  getting a masters or a Doctorate/PHD, before we figured out that tech was the place to be…:)

Location, Location, Location

The last quick look I did was to see by country, what were the top and bottom average salaries for DBAs-

 

Not too bad, Switzerland and Denmark… 🙂

Data Is King

I wish there’d been more respondents to the survey, but very happy with the data that was provided.  I’m considering doing one of my own, just to get more people from the Oracle side, but until then, here’s a little something to think about as we prep for the new year and another awesome year in the database industry!

 

 

Posted in DBA Life, Oracle Tagged with: , ,

December 16th, 2016 by dbakevlar

screen-shot-2016-12-16-at-12-05-22-pm

On the first day with Delphix, I provisioned with glee, an IT Manager Happy.

On the second day with Delphix, I provisioned with glee, two SAP ASE and an IT Manager Happy.

On the third day with Delphix, I provisioned with glee, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the fourth day with Delphix, I provisioned with glee, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the fifth day with Delphix, I provisioned with glee…

Five Cloud Migrations!

Four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the sixth day with Delphix, I provisioned with glee,

Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the seventh day with Delphix, I provisioned with glee,

Seven developers coding, Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the eighth day with Delphix, I provisioned with glee,

Eight testers testing, Seven developers coding, Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the ninth day with Delphix, I provisioned with glee,

Nine applications applying, Eight testers testing, Seven developers coding, Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the tenth day with Delphix, I provisioned with glee,

Ten DevOps leading, Nine applications applying, Eight testers testing, Seven developers coding, Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the eleventh day with Delphix, I provisioned with glee,

Eleven DB2 humming, Ten DevOps leading, Nine applications applying, Eight testers testing, Seven developers coding, Six SQL Servers running, Five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and an IT Manager Happy.

On the twelfth day with Delphix, I provisioned with glee,

Twelve databases masking, Eleven DB2 humming, Ten DevOps leading, nine applications applying, eight testers testing, seven developers coding, six SQL Servers running, five Cloud Migrations, four EBS Clones, three Oracle Databases, two SAP ASE and as IT Manager Happy.

happy_h

Posted in DBA Life, Delphix Tagged with: ,

  • Facebook
  • Google+
  • LinkedIn
  • Twitter