Category: AWS Trial

March 24th, 2017 by dbakevlar

“Enough already with the DHCP complaints!”  It’s Friday, so it means soon, we can stop worrying about such stuff and can get onto better things…

Until then, I’m sure other folks will be working with an AWS instance, go to log in after setting it up the initial time and want to know why the ssh connection that worked fine the day before is now hanging.  In fact, if anything has been done to renew their DHCP release and change their IP Address, this could occur.

There’s always more than one way to skin a cat and along with setting aliases for your favored IP address, the second option is to follow the next steps.

What’s Your Number?

So you can get to your instances via the EC2 console and use the Delphix admin console without any issue, but if you try to ssh into a source or target, (aka instances) the prompt just hangs.  The problem is that all that lovely automation that built out this incredible environment for the Delphix AWS Trial, also built out the security group setting for inbound traffic for you, tying the access to your IP address.  This information was entered in the terraform.tfvars file and the IP in this file no longer matches your new IP Address.

  • Get the new IP Address.  Number of methods- but I’m lazy and just type in “What is my ip address” in Google.

Update Your Security Group Rule

  1. In the the AWS EC2 Console, click on the instance you’re trying to log into.
  2. Click on the Description tab in the lower part of the screen.

3. In this tab, you’ll see a listing for Security Groups.  Click on the group name to the right of Security Groups.

4. Click on Edit

5. You’ll see what was your previous IP address, (if you haven’t updating your terraform.tfvars since the build, you can verify this.)

6. Update the IP address with your new one and click on SAVE.

Now try to SSH into the box.  You’ll need to do this for any and all instances that you want to connect to via a terminal.

Hint hint:  This is also the location to add a new workstation that needs access to work on the instances.

Happy Friday!

Posted in AWS Trial

March 20th, 2017 by dbakevlar

Now that I’ve loaded a ton of transactions and did a bunch of work load on my source database with the SH sample schema and Swingbench, I’ve noted how little impact to the databases using different cloud tools, (which will come in a few later posts) now I’m going to show you how easy it is to create a new VDB from all of this, WITH the new SH data included.  During all of this time, the primary users of my Delphix VDB, (virtualized databases) would have been working in the previous iage, but someone wants that new SH schema now that my testing has completed.

To do this, I open up my Delphix admin console, (using the IP address for the Delphix Engine from the AWS Trail build output), log in as delphix_admin and open up the Source group to access the Employee Oracle 11g database, (aka ORCL.)

I know my new load is complete on the ORCL database and need to take a new snapshot to update the Delphix Engine outside of the standard refresh interval, (I’m set at the default of every 24 hrs.)  Access this by clicking on the Configuration tab and to take a snapshot, I simply click on the camera icon.

A snapshot will take a couple seconds, as this is a very, very small database, (2.3G) and then you can click on Timeflow to view the new snapshot available for use.  Ensure the new snapshot is chosen by moving the slider all the way to the right and look at the timestamp, ensuring it’s the latest, matching your recent one.

Click on Provision and it will default to the Source host, change to the target, update to a new, preferred database name, (if you don’t like the default) and then you may have to scroll down to see the Next button to go through the subsequent steps in the wizard.  I know my Macbook has a smaller screen and I do have to scroll to see the Next button.  After you’ve made any other changes, click on Finish and let the job run.  Don’t be surprised by the speed that a VDB is provisioned-  I know it’s really fast, but it really did create a new VDB!

Now that we have it, let’s connect to it from SQL*Plus and check prove that we got the new SH schema over.

Using the IP Address for the Linux Target that was given to use in our AWS Trial build, let’s connect:

ssh delphix@<linuxtarget IP Address>

Did you really just create a whole new VDB?

[delphix@linuxtarget ~]$ ps -ef | grep pmon
delphix   1148  1131  0 18:57 pts/0    00:00:00 grep pmon
delphix  16825     1  0 Mar09 ?        00:00:06 ora_pmon_devdb
delphix  16848     1  0 Mar09 ?        00:00:06 ora_pmon_qadb
delphix  31479     1  0 18:30 ?        00:00:00 ora_pmon_VEmp6C0

Yep, there it is…

Now let’s connect to it.

Set our environment:

. 11g.env

Set the ORACLE_SID to the new VDB

export ORACLE_SID=VEmp6C0

Connect to SQL*Plus as our SH user using the password used in our creation on the source database, ORCL:

$ sqlplus sh

Enter password: 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select object_type, count(*) from user_objects
  2  group by object_type;

OBJECT_TYPE       COUNT(*)

------------------- ----------
INDEX PARTITION     196
TABLE PARTITION     56
LOB                 2
DIMENSION           5
MATERIALIZED VIEW   2
INDEX               30
VIEW                1
TABLE               16

8 rows selected.

SQL> select table_name, sum(num_rows) from user_tab_statistics
  2  where table_name not like 'DR$%'
  3  group by table_name
  4  order by table_name;

TABLE_NAME       SUM(NUM_ROWS)

------------------------------ -------------
CAL_MONTH_SALES_MV             48
CHANNELS                       5
COSTS                          164224
COUNTRIES                      23
CUSTOMERS                      55500
FWEEK_PSCAT_SALES_MV           11266
PRODUCTS                       72
PROMOTIONS                     503
SALES                          1837686
SALES_TRANSACTIONS_EXT         916039
SUPPLEMENTARY_DEMOGRAPHICS     4500
TIMES                          1826

12 rows selected.

Well, lookie there, the same as the source database we loaded earlier... 🙂  Next, we’ll go into the stuff that always gets my interest- performance data, the cloud and visualization tools.

 

Posted in AWS Trial, Oracle Tagged with: ,

March 16th, 2017 by dbakevlar

I love valuable metric data and I wanted to see what’s offered from different cloud providers.  Metrics are data and data ensures that when performance is impacted, we have the information we need to answer questions and resolve issues.

For my first adventure, we’ll start with a high level review of Amazon.  Cloudwatch is one of the main tools to inspect the workload on instances in EC2, using the AWS Trial while Swingbench runs the Sales History workload.

There is additional charges for this feature, but it’s pretty minimal per instance vs. other tools, so I’ll talk about the basics here and you can make up your own mind.

Once you enter this console, the graph area will be empty and you’ll need to add metrics to the graph.  Since I’m using EC2, I click on EC2 in the Metrics area below the empty graph and each of the metrics available for each of my instances will be shown in a list.

Click on the instance and metric you’re interested and then a list of options will show.  Click on Add to graph to add it to the graph above the list and have it start collecting metric data.

You can also click on the check box at the top of All Metrics to add them all.

Once you’ve completed adding the metrics for all the instances you’re interested in, you’re graph will look something like this:

As you can see, you can highlight the line on the graph to match it to the metric/instance name below the graph or vice-versa.

As most of the work for Swingbench is going to the source and then to the Delphix Engine to track the changes to be used by the targets, those first two instances and network activity will be the highest usage.  I haven’t played with the Graphed Options tab, yet and will leave that for another day.

If there’s a metric for an instance you’ve added that isn’t providing any value to the charge, it’s simplest to click on the Graphed Metrics tab, then click on the checkmark in the list and it will stop displaying it in the graph.

If you haven’t had an opportunity to play with the free AWS Trial of Delphix, check it out!  It’s a great way to become familiar with Delphix and AWS-  all in one shot!

 

Posted in AWS Trial, Cloud Tagged with: ,

March 13th, 2017 by dbakevlar

Swingbench is a one of the best choices for easy loads on a database.  I wanted to use it against the SH sample schema I loaded into my Oracle Source database and I haven’t used Swingbench outside of the command line quite a while back, (my databases seem to always come with a load on them!)  so it was time to update my Swingbench skills and catch up with the user interface.  Thanks to Dominic Giles for keeping the download, features and documentation so well maintained.

After adding the application rights to run on my Macbook Pro, I was impressed by the clean and complete interface.  I wanted to connect it to my AWS instance and as we talk about, the cloud is a lot simpler a change than most DBAs first consider.

When first accessing, Swingbench will prompt you to choose what pre-configured workload you’d like to utilize.  I had already set up the Sales History schema in my AWS Trial source database, so I chose Sales History and then had to perform a few simple configurations to get it to run.

Username: sh

Password: <password for your sh user>

Connect String: <IP Address for AWS Instance>:<DB Port>:<services name>

Proceed down to the tab for Environment Variables and add the following:

ORACLE_HOME  <Oracle Home>

I chose the default 16 connections to start out, but you can add more if you’d like. You can also configure stats collection, snapshot collection before and after the workload.

I set my autoconnect to true, but the default is to not start the load until you hit the green arrow button.  The load will then execute the workload with the amount of connections requested until you hit the red stop button.  You should see the users logged in at the bottom right and in the events window:

Next post we’ll discuss what you’ll see when running a Swingbench on a source database, the Delphix Engine host and subsequently refreshes to a VDB, (virtual database.)  We’ll also discuss other tools that can grant you visibility to optimization opportunities in the cloud.

 

 

Posted in AWS Trial, Cloud, Oracle Tagged with: , ,

February 23rd, 2017 by dbakevlar

This was received by one of our Delphix AWS Trial customers and he wasn’t sure how to address it.  If any others experience it, this is the why it occurs and how you can correct it.

You’re logged into your Delphix Administration Console and you note there is a fault displayed in the upper right hand console.  Upon expanding, you see the following warning for the databases from the Linux Target to the Sources they pull updates from:

 

Its only a warning and the reason it’s only a warning is that it doesn’t stop the user from performing a snapshot and provisioning, but it does impact the timeflow, which could be a consideration if a developer were working and needed to go back in time during this lost time.

How do you address the error?  Its one that’s well documented by Delphix, so simply proceed to the following link, which will describe the issue, how to investigate and resolve.

Straightforward, right?  Not so much in my case.  When the user attempted to ssh into the linux target, he received an IO error:

$ ssh delphix@xx.xxx.xxx.x2

$ ssh: connect to host xx.xxx.xxx.x2 port 22: Operation timed out

I asked the user to then log into Amazon EC2 dashboard and click on Instances.  The following displayed:

Oh-oh….

By highlighting the instance, the following is then displayed at the lower part of the dashboard, displaying that their is an issue with this instance:

Amazon is quickly on top of this once I refresh the instances and all is running once more.  Once this is corrected and all instances show a green check mark for the status, then I’m able to SSH into the console with out issue:

$ ssh delphix@xx.xxx.xxx.x2

Last login: Tue Feb 21 18:05:28 2017 from landsharkengine.delphix.local

[delphix@linuxtarget ~]$

Does this resolve the issue in the Delphix Admin Console?  Depends… and the documentation linked states that the problem is commonly due to a problem with network or source system, but in this case, it was the target system that suffered the issue in AWS.

As this is a trial and not a non-production system that is currently being used, we will skip recovering the logs to the target system and proceed with taking a new snapshot.  The document also goes into excellent ways to deter from experiencing this type of outage in the future.  Again, this is just a trial, so we won’t put these into practice for a trial environment that we can easily drop and recreate in less than an hour.

Tips For Real Delphix Environments

There are a few things to remember when you’re working with the CLI:

-If you want to inspect any admin information, (such as these logs, as shown in the document) you’ll need to be logged in as the delphix_admin@DOMAIN@<DE IP Address>.

So if you don’t see the “paths” as you work through the commands, it’s because you may have logged in as sysadmin instead of the delphix_admin.

-If you’ve marked the fault as “Resolved”, there’s no way to resolve the timeflow issue, so you’ll receive the following:

ip-10-0-1-10 timeflow oracle log> list timeflow='TimeFlow' missing=true

No such Timeflow ''TimeFlow''.

-If the databases are down, it’s going to be difficult for Delphix to do anything with the target database.  Consider updating the target to auto restart on an outage.  To do so, click on the database in question, click on Configuration and change it to “On” for the Auto VDB Restart.

Here’s a few more tidbits to make you more of an expert with Delphix.  Want to try out the free trail with AWS?  All you need is a free Amazon account and it’s only about 38 cents an hour to play around with a great copy of our Delphix Engine, including a deployed Source and Target.

Just click on this link to get started!

Posted in AWS Trial, Delphix

February 20th, 2017 by dbakevlar

We’ve been working hard to create an incredible new trial version of Delphix that uses AWS, which is built with the open source product Terraform.  Terraform is a tool that anyone can use to build, version and manage a product effectively and seamlessly in a number of clouds.  We are currently using it to implement to AWS, but there is a bright future for these types of open source products and I’m really impressed with how easy its made it to deploy compute instances, the Delphix Engine and supporting architecture on AWS EC2.  If you’d like to read up on Terraform, check out their website.

The Delphix Admin Console and Faults

After building out the Delphix environment with the Engine and the a Linux source/target, the first step for many is to log into the Delphix Admin console.  You can view any faults during the build at the upper right corner under Faults. One error that I’ve noticed comes up in after a successful build is the following:

AWS Console to the Rescue

By logging into your AWS EC2 console, you can view the instances that are being used.  As you’ll note, the error says that the Delphix Engine is using an unsupported  instance type m4.large.  Yes in our EC2 console, we can see the Delphix Engine, (last in the list and with the name ending in “DE”) that no, actually it isn’t.

It’s actually a m4.xlarge instance type.  What’s even more interesting, is that the Linux Target, (LT) and Linux Source, (LS) are both m4.large instance types, yet no warning was issued for either of these instances as unsupported.

AWS EC2 Supported Instance Types

You can locate what types of instance types are supported for AWS EC2 with the following link.  At this page, we can also see that both the m4.large and the m4.xlarge instance type IS SUPPORTED.

Knowing that we’ve validated that the instance type is supported means that I can safely ignore it and proceed to work through the trial without worry.

If you’re planning on deploying a production Delphix Engine on AWS, inspect the following document to ensure you build it with the proper configuration.

Nothing to see here and thought I better let everyone know before someone lumps Amazon with CNN… 🙂

Posted in AWS Trial, Oracle Tagged with: ,

February 13th, 2017 by dbakevlar

When tearing down an AWS Delphix Trial, we run the following command with Terraform:

>terraform destroy

I’ve mentioned before that every time I execute this command, I suddenly feel like I’m in control of the Death Star in Star Wars:

As this runs outside of the AWS EC2 web interface, you may see some odd information in your dashboard.  In our example, we’ve run “terraform destroy” and the tear down was successful:

So you may go to your volumes and after verifying that yes, no volumes exist:

The instances may still show the three instances that were created as part of the trial, (delphix engine, source and target.)

These are simply “ghosts of instances past.”  The tear down was completely successful and there’s simply a delay before the instance names are removed from the dashboard.  Notice that they no longer are listed with a public DNS or IP address.  This is a clear indication that these aren’t currently running, exist or more importantly, being charged for.

Just one more [little] thing to be aware of… 🙂

Posted in AWS Trial, Delphix Tagged with: ,

February 13th, 2017 by dbakevlar

Now, for most of us, we’re living in a mobile world, which means as our laptop travels, our office moves and our IP address changes.  This can be a bit troubling for those that are working in the cloud and our configuration to our cloud relies on locating us via our IP Address being the same as it was in our previous location.

What happens if you’re IP Address changes from what you have in your configuration file, (in Terraform’s case, your terraform.tfvars file) for the Delphix AWS Trial?  I set my IP Address purposefully incorrect in the file to demonstrate what would happen after I run the terraform apply command:

It’s not the most descriptive error, but that I/O timeout should tell you right away that terraform can’t connect back to your machine.

Addressing the IP Address Issue

Now, we’ll tell you to capture your current IP address and update the IP address in the TFVARS file that resides in the Delphix_demo folder, but I know some of you are wondering why we didn’t just build out the product to adjust for an IP address change.

The truth is, you can set a static IP Address for your laptop OR just alias your laptop with the IP Address you wish to have.  There are a number of different ways to address this, but looking into the most common, let’s dig into how we would update the IP address vs. updating the file.

Static IP Address

You can go to System Preferences or Control Panel, (depending on which OS you’re on) and click on Network and configure your TCP/IP setting to manual and type in an IP Address there.  The preference is commonly to choose a non-competitive IP address, (often the one that was dynamically set will do as your manual one to retain) and choose to save the settings.  Restart the PC and you can then add that to your configuration files.  Is that faster than just updating the TFVARS file-  nope.

Setting the IP Address- Mac/Linux

The second way to do this is to create an Alias IP address to deter from the challenge of each location/WiFi move having it automatically assigned.

Just as above, we often will use the IP address that was given dynamically and just choose to keep this as the one you’ll keep each time.  If you’re unsure of what your IP is, there are a couple ways to collect this information:

Open up a browser and type in “What is my IP Address

or from a command prompt, with “en0” being your WiFi connection, gather your IP Address one of two ways:

$ dig +short myip.opendns.com @resolver1.opendns.com
$ ipconfig getifaddr en0

Then set the IP address and cycle the your WiFi connection: 


$ sudo ipconfig set en0 INFORM <IP Address>
$ sudo ifconfig en0 down 
$ sudo ifconfig en0 up

You can also click on your WiFi icon and reset it as well.  Don’t be surprised if this takes a while to reconnect.  Renewing and releasing of IP addresses can take a bit across the network and the time required must be recognized.

An Alias on Mac/Linux

Depending on which OS you’re on.  Using the IP Address from your tfvars file, set it as an alias with the following command:

$ sudo ifconfig en0 alias <IP Address> 255.255.255.0

Password: <admin password for workstation>

If you need to unset it later on:

sudo ifconfig en0 -alias <IP Address>

I found this to be an adequate option-  the alias was always there, (like any other alias, it just forwards everything onto the address that you’re recognized at in the file.) but it may add time to the build, (still gathering data to confirm this.)  With this addition, I shouldn’t have to update my configuration file, (for the AWS Trial, that means setting it in our terraform.tfvars in the YOUR_IP parameter.)

Setting your IP Address on Windows

The browser commands to gather your IP Address work the same way, but if you want to change it via the command line, the commands are different for Windows PC’s:

netsh interface ipv4 show config

You’ll see your IP Address in the configuration.  If you want to change it, then you need to run the following:

netsh interface ipv4 set address name="Wi-Fi" static <IP Address> 255.255.255.0 <Gateway>
netsh interface ipv4 show config

You’ll see that the IP Address for your Wi-Fi has updated to the new address.  If you want to set it to DHCP, (dynamic) again, run the following:

netsh interface ipv4 set address name="Wi-Fi" source=dhcp

Now you can go wherever you darn well please, set an alias and run whatever terraform commands you wish.  All communication will just complete without any error due to a challenging new IP address.

Ain’t that just jiffy? OK, it may be quicker to just gather the IP address and update the tfvars file, but just in case you wanted to know what could be done and why we may not have built it into the AWS Trial, here it is! 🙂

Posted in AWS Trial, Delphix Tagged with: , , ,

February 10th, 2017 by dbakevlar

I ran across an article from 2013 from Straight Talk on Agile Development by Alex Kuznetsov and it reminded me how long we’ve been battling for easier ways of doing agile in a RDBMS environments.

Getting comfortable with a virtualized environment can be an odd experience for most DBAs, but as soon as you recognize how similar it is to a standard environment, we stop over-thinking it and it makes it quite simple to then implement agile with even petabytes of data in an relational environment without using slow and archaic processes.

The second effect of this is to realize that we may start to acquire secondary responsibilities and take on ensuring that all tiers of the existing environment are consistently developed and tested, not just the database.

A Virtual Can Be Virtualized

Don’t worry-  I’m going to show you that its not that difficult and virtualization makes it really easy to do all of this, especially when you have products like Delphix to support your IT environment. For our example, we’re going to use our trusty AWS Trial environment and we have already provisioned a virtual QA database.  We want to create a copy of our development virtualized web application to test some changes we’ve made and connect it to this new QA VDB.

From the Delphix Admin Console, go to Dev Copies and expand to view those available.  Click on Employee Web Application, Dev VFiles Running.  Under TimeFlow, you will see a number of snapshots that have been taken on a regular interval.  Click on one and click on Provision.

Now this is where you need the information about your virtual database that you wish to connect to:

  1.  You will want to switch from provisioning to the source to the Linux Target.

Currently the default is to connect to the existing development database, but we want to connect to the new QA we wish to test on.  You can ssh as delphix@<ipaddress for linuxtarget> to connect to and gather this information.

2.  Gathering Information When You Didn’t Beforehand

I’ve created a new VDB to test against, with the idea, that I wouldn’t want to confiscate an existing VDB from any of my developers or testers.  The new VDB is called EmpQAV1.  Now, if you’re like me, you’re not going to have remembered to grab the info about this new database before you went into the wizard to begin the provisioning.  No big deal, we’ll just log into the target and get it:

[delphix@linuxtarget ~]$ . 11g.env
[delphix@linuxtarget ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0.4/db_1
[delphix@linuxtarget ~]$ ps -ef | grep pmon
delphix  14816     1  0 17:23 ?        00:00:00 ora_pmon_devdb
delphix  14817     1  0 17:23 ?        00:00:00 ora_pmon_qadb
delphix  17832     1  0 17:32 ?        00:00:00 ora_pmon_EmpQAV1
delphix  19935 19888  0 18:02 pts/0    00:00:00 grep pmon

I can now set my ORACLE_SID:

[delphix@linuxtarget ~]$ export ORACLE_SID=EmpQAV1

Now, let’s gather the rest of the information we’ll need to connect to the new database by connecting to the database and gathering what we need.

[delphix@linuxtarget ~]$ lsnrctl services
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-FEB-2017 18:13:06
Copyright (c) 1991, 2013, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
Services Summary...
Service "EmpQAV1" has 1 instance(s).
  Instance "EmpQAV1", status READY, has 1 handler(s) for this service...

Provision Your VFile

Fill in all the values required in the next section of the provisioning setup:

Click Next and add any requirements to match your vfile configuration that you had for the existing environment.  For this one, there aren’t any, (additional NFS Mount points, etc.)  Then click Next and Finish.

The VFile creation should take a couple minutes max and you should now see an environment that looks similar to the following:

This is a fully functional copy of your web application, created from another virtual copy that can test against a virtual database, ensuring that all aspects of a development project are tested thoroughly before releasing to production!

Why would you choose to do anything less?

 

 

 

Posted in AWS Trial, Delphix, Oracle Tagged with: ,

February 10th, 2017 by dbakevlar

So you thought you were finished configuring your AWS target, eh?  I already posted a previous time on how to address a fault with the RMEM, but now we’re onto the WMEM.  Wait, WM-what?

No, I fear a DBAs work is never over and when it comes to the cloud, our skill set has just expanded from what it was when we worked on-premise!

Our trusty Delphix Admin Console keeps track of settings on all our sources and targets, informing us when the settings aren’t set to what is recommended, so that we’ll be aware of any less than optimal parameters that could effect performance.

As we address latency in cloud environments, network settings become more important.

How WMEM Differs from RMEM

RMEM= receive

WMEM=send

Where RMEM is quite easy to remember as receive settings, we get to thank

 

As root, we’ll add another line to the sysctl.conf file to reflect values other than defaults:

$ echo 'net.ipv4.tcp_wmem= 102404194304 12582912' >> /etc/sysctl.conf

Reload the values into the system:

$ sysctl -p /etc/sysctl.conf

Verify the settings are now active:

$ sysctl -a | grep net.ipv4.tcp_wmem

net.ipv4.tcp_wmem = 10240 4194304 12582912

That’s all there is to it.  Now you can mark the fault as resolved in the Delphix Admin Console.

Posted in AWS Trial, Delphix Tagged with: , ,

February 8th, 2017 by dbakevlar

There are more configurations for AWS than there are fish in the sea, but as the rush of folks arrive to test out the incredibly cool AWS Trial for Delphix, I’ll add my rendition of what to look for to know you’re AWS setup is prepped to successfully deploy.

The EC2 Dashboard View

After you’ve selected your location, set up your security user/group and key pairs, there’s a quick way to see, (at least high level) if you’re ready to deploy the AWS Trial to the zone in question.

Go to your EC2 Dashboard and to the location, (Zone) that you plan to deploy your trial to and you should see the following:

Notice in the dashboard, you can see that the key pairs, (1) and the expected Security Groups, (3) are displayed, which tells us that we’re ready to deploy to this zone.  If we double click on the Key Pair, we’ll see that its match to the one we downloaded locally and will use in our configuration with Terraform:

How Terraform Communicates with AWS

These are essential to deploying in an AWS zone that’s configured as part of your .tfvars file for terraform.  You’ll note in the example below, we have both designated the correct zone and the key pair that is part of the zone we’ll be using to authenticate:


#VERSION=004

#this file should be named terraform.tfvars

# ENTER INPUTS BELOW

access_key="XXXXXXX"

secret_key="XXXXXXXXXX"

aws_region="us-east-1"

your_ip="xxx.xx.xxx.xxx"

key_name="Delphix_east1" #don't include .pem in the key name 

instance_name="Delphix_AWS"

community_username="xxx@delphix.com"

community_password="password"

Hopefully this is a helpful first step in understanding how zones, key pairs and security groups interact to support the configuration file, (tfvars) file that we use with the Delphix deployment via Terraform into AWS.

 

Posted in AWS Trial, Delphix, Oracle Tagged with: ,

February 1st, 2017 by dbakevlar

Delphix focuses on virtualizing non-production environments, easing the pressure on DBAs, resources and budget, but there is a second use case for product that we don’t discuss nearly enough.

Protection from data loss.

Jamie Pope, one of the great guys that works in our pre-sales engineering group, sent Adam and I an article on one of those situations that makes any DBA, (or an entire business, for that matters) cringe.  GitLab.com was performing some simple maintenance and someone deleted the wrong directory, removing over 300G of production data from their system.  It appears they were first going to use PostgreSQL “vacuum” feature to clean up the database, but decided they had extra time to clean up some directories and that’s where it all went wrong.  To complicate matters, the onsite backups had failed, so they had to go to offsite ones, (and every reader moans…)

Even this morning, you can view the tweets of the status for the database copy and feel the pain of this organization as they try to put right the simple mistake.

Users are down as they work to get the system back up.  Just getting the data copied before they’re able to perform the restore is painful and as a DBA, I feel for the folks involved:

How could Delphix have saved the day for GitLab?  Virtual databases, (VDBs) are read/write copies and derived from a recovered image that is compressed, duplicates removed and then kept in a state of perpetual recovery having the transactional data applied in a specific interval, (commonly once every 24 hrs) to the Delphix Engine source.  We support a large number of database platforms, (Oracle, SQL Server, Sybase, SAP, etc) and are able to virtualize the applications that are connected to them, too.  The interval of how often we update the Delphix Engine source is configurable, so depending on network and resources, this interval can be decreased to apply more often, depending on how up to date the VDBs need to be vs. production.

With this technology, we’ve come into a number of situations where customers suffered a cataclysmic failure situation in production.  While traditionally, they would be dependent upon a full recovery from a physical backup via tape, (which might be offsite) or scrambling to even find a backup that fit within a backup to tape window, they suddenly discovered that Delphix could spin up a brand new virtual database with the last refresh before the incident from the Delphix source and then use a number of viable options to get them up and running quickly.

  1. Switch the users and application to point to the new VDB that was recovered to the point in time, (PIT) before the incident occurred.  Meanwhile, IT is able to take their time recovering the production database with the physical backup, with little outage to the business.
  2. Create a VDB to the PIT before the failure and then create a connection between the production and the VDB, making a copy back to production of the data that was lost.
  3. If there was dire loss, (i.e. disk, etc.)  create a VDB to the PIT before the failure and perform what’s called a V2P, or virtual to physical, rehydrating the virtual data to become the new physical database.

This is the type of situation happens more often then we’d like to admit.  Many times resources have been working long shifts and make a mistake due to exhaustion, other times someone unfamiliar and with access to something they shouldn’t simply make a dire mistake, but these things happen and this is why DBAs are always requesting two or three methods of backups.  We learn quite quickly we’re only as good as our last backup and if we can’t protect the data, well, we won’t have a job for very long.

Interested in testing it out for yourself?  We have a really cool free Delphix trial via Amazon cloud that uses your AWS account.  There’s a source host and databases, along with a virtual host and databases, so you can create VDBs, blow away tables, recovery via a VDB, create a V2P, (virtual to physical) all on your own.

 

Posted in AWS Trial, cloning, Delphix Tagged with: , ,

January 31st, 2017 by dbakevlar

I’ve been at Delphix for just over six months now.  In that time, I was working with a number of great people on a number of initiatives surrounding competitive, the company roadmap and some new initiatives.  With the introduction of our CEO, Chris Cook, new CMO, Michelle Kerr and other pivotal positions within this growing company, it became apparent that we’d be redirecting our focus on Delphix’s message and connections within the community.

I was still quite involved in the community, even though my speaking had been trimmed down considerably with the other demands at Delphix.  Even though I wasn’t submitting abstracts to many of the big events I’d done so in previous years, I still spoke at 2-3 events each month during the fall and made clear introductions into the Test Data Management, Agile and re-introduction into the SQL Server communities.

As of yesterday, my role was enhanced so that evangelism, which was previously 10% of my allocation, is now going to be upwards of 80% as the Technical Evangelist for the Office of the CTO at Delphix.  I’m thrilled that I’m going to be speaking, engaging and blogging with the community at a level I’ve never done before.  I’ll be joined by the AWESOME Adam Bowen, (@CloudSurgeon on Twitter) in his role as Strategic Advisor and as the first members of this new group at Delphix.  I would like to thank all those that supported me to gain this position and the vision of the management to see the value of those in the community that make technology successful day in and day out.

I’ve always been impressed with the organizations who recognize the power of grassroots evangelism and the power it has in the industry.  What will I and Adam be doing?  Our CEO, Chris Cook said it best in his announcement:

As members of the [Office of CTO], Adam and Kellyn will function as executives with our customers, prospects and at market facing events.  They will evangelize the direction and values of Delphix; old, current, and new industry trends; and act as a customer advocate/sponsor, when needed.  They will connect identified trends back into Marketing and Engineering to help shape our message and product direction.  In this role, Adam and Kellyn will drive thought leadership and market awareness of Delphix by representing the company at high leverage, high impact events and meetings. []

As many of you know, I’m persistent, but rarely patient, so I’ve already started to fulfill my role and be prepared for some awesome new content, events that I’ll be speaking at and new initiatives.  The first on our list was releasing the new Delphix Trial via the Amazon Cloud.  You’ll have the opportunity to read a number of great posts to help you feel like an Amazon guru, even if you’re brand new to the cloud.  In the upcoming months, watch for new features, stories and platforms that we’ll introduce you to. This delivery system, using Terraform, (thanks to Adam) is the coolest and easiest way for anyone to try out Delphix, with their own AWS account and start to learn the power of Delphix with use case studies that are directed to their role in the IT organization.

Posted in AWS Trial, DBA Life, Delphix, Oracle Tagged with: ,

January 30th, 2017 by dbakevlar

I don’t want to alarm you, but there’s a new Delphix trial on AWS!  It uses your own AWS account and with a simple set up, allows you to deploy a trial Delphix environment.  Yes, you hear me right-  just with a couple steps, you could have your own setup to work with Delphix!

There’s documentation to make it simple to deploy, simple to understand and then use cases for individuals determined by their focus, (IT Architect, Developer, Database Administrator, etc.)

This was a huge undertaking and I’m incredibly proud of Delphix to be offering this to the community!

So get out there and check this trial out!  All you need is an AWS account on Amazon and if you don’t have one, it only takes a few minutes to create one and set it up, just waiting for a final verification before you can get started!  If you have any questions or feedback about the trial, don’t hesitate to email me at dbakevlar at gmail.

Posted in AWS Trial, Cloud, Delphix Tagged with: , ,

January 24th, 2017 by dbakevlar

So you’ve deployed targets with Delphix on AWS and you receive the following error:

It’s only a warning, but it states that you’re default of 87380 is below the recommended second value for the ipv4.tcp.rmem property.  Is this really an issue and do you need to resolve it?  As usual, the answer is “it depends” and its all about on how important performance is to you.

What is net.ipv4.tcp.rmem?

To answer this question, we need to understand network performance.  I’m no network admin, so I am far from an expert on this topic, but as I’ve worked more often in the cloud, it’s become evident to me that the network is the new bottleneck for many organizations.  Amazon has even build a transport, (the Snowmobile) to bypass this challenge.

The network parameter settings in question have to do with network window sizes for the cloud host in question surrounding TCP window reacts and WAN links.  We’re on AWS for this environment and the Delphix Admin Console was only the messenger to let us know that our setting currently provided for this target are less than optimal.

Each time the sender hits this limit, they must wait for a window update before they can continue and you can see how this could hinder optimal performance for the network.

Validation First

To investigate this, we’re going to log into our Linux target and SU over as root, which is the only user who has the privileges to edit this important file.:

$ ssh delphix@<IP Address for Target>
$ su root

As root, let’s first confirm what the Delphix Admin Console has informed us of by running the following command:

$ sysctl -a | grep net.ipv4.tcp_rmem 

net.ipv4.tcp_rmem = 4096 87380 4194304

There are three values displayed in the results:

  • The first value is the minimum amount of receive window that will be set to each TCP connection, even when the system is overwhelmed.
  • The default value allocated to each tcp connection,
  • The third is the maximum that can be allocated to any TCP connection.

To translate what this second value corresponds to-  this is the size of data in flight any sender can communicate via TCP to the cloud host before having to receive a window update.

So why are faster networks better?  Literally, the faster the network, the closer the bits and the more data that can be transferred.  If there’s a significant delay, due to a low setting on the default of how much data can be placed on the “wire”, then the receive window won’t be used optimally.

This will require us to update our parameter file and either edit or add the following lines:

net.ipv4.tcp_window_scaling = 1

net.core.rmem_max = 16777216

net.ipv4.tcp_rmem = 4096 12582912 16777216
I’m using the value as recommended by Brendan Gregg’s blog post on tuning EC2 instances.  This leaves a pretty narrow difference between the minimum and maximum for the window receive, but it is now within the recommended range for enhanced performance.
After you’ve updated the sysctl.conf file, you’ll need to reload it with the following command:
$ sysctl -p /etc/sysctl.conf
$ sysctl -a | grep net.ipv4.tcp_rmem 

net.ipv4.tcp_rmem = 4096 12582912 16777216

Ahhh, that looks much better… 🙂

Posted in AWS Trial, Delphix Tagged with: , ,

  • Facebook
  • Google+
  • LinkedIn
  • Twitter