Subscribe to Blog via Email
So you thought you were finished configuring your AWS target, eh? I already posted a previous time on how to address a fault with the RMEM, but now we’re onto the WMEM. Wait, WM-what?
No, I fear a DBAs work is never over and when it comes to the cloud, our skill set has just expanded from what it was when we worked on-premise!
Our trusty Delphix Admin Console keeps track of settings on all our sources and targets, informing us when the settings aren’t set to what is recommended, so that we’ll be aware of any less than optimal parameters that could effect performance.
As we address latency in cloud environments, network settings become more important.
Where RMEM is quite easy to remember as receive settings, we get to thank
As root, we’ll add another line to the sysctl.conf file to reflect values other than defaults:
$ echo 'net.ipv4.tcp_wmem= 102404194304 12582912' >> /etc/sysctl.conf
Reload the values into the system:
$ sysctl -p /etc/sysctl.conf
Verify the settings are now active:
$ sysctl -a | grep net.ipv4.tcp_wmem net.ipv4.tcp_wmem = 10240 4194304 12582912
That’s all there is to it. Now you can mark the fault as resolved in the Delphix Admin Console.
So you’ve deployed targets with Delphix on AWS and you receive the following error:
It’s only a warning, but it states that you’re default of 87380 is below the recommended second value for the ipv4.tcp.rmem property. Is this really an issue and do you need to resolve it? As usual, the answer is “it depends” and its all about on how important performance is to you.
To answer this question, we need to understand network performance. I’m no network admin, so I am far from an expert on this topic, but as I’ve worked more often in the cloud, it’s become evident to me that the network is the new bottleneck for many organizations. Amazon has even build a transport, (the Snowmobile) to bypass this challenge.
The network parameter settings in question have to do with network window sizes for the cloud host in question surrounding TCP window reacts and WAN links. We’re on AWS for this environment and the Delphix Admin Console was only the messenger to let us know that our setting currently provided for this target are less than optimal.
Each time the sender hits this limit, they must wait for a window update before they can continue and you can see how this could hinder optimal performance for the network.
To investigate this, we’re going to log into our Linux target and SU over as root, which is the only user who has the privileges to edit this important file.:
$ ssh delphix@<IP Address for Target>
$ su root
As root, let’s first confirm what the Delphix Admin Console has informed us of by running the following command:
$ sysctl -a | grep net.ipv4.tcp_rmem net.ipv4.tcp_rmem = 4096 87380 4194304
There are three values displayed in the results:
To translate what this second value corresponds to- this is the size of data in flight any sender can communicate via TCP to the cloud host before having to receive a window update.
So why are faster networks better? Literally, the faster the network, the closer the bits and the more data that can be transferred. If there’s a significant delay, due to a low setting on the default of how much data can be placed on the “wire”, then the receive window won’t be used optimally.
This will require us to update our parameter file and either edit or add the following lines:
net.ipv4.tcp_rmem = 4096 12582912 16777216
$ sysctl -p /etc/sysctl.conf
$ sysctl -a | grep net.ipv4.tcp_rmem net.ipv4.tcp_rmem = 4096 12582912 16777216
Ahhh, that looks much better… 🙂
I’ve been discussing for years about the importance of network to database performance, especially once I started working on VLDBs, (Very Large Databases) but its a topic that often is disregarded. Now that I’m working more and more in the cloud, it’s become more evident the importance of the network to our survival.
For each and every cloud project I’ve been involved in, there is evidently going to be multiple challenges that turn to the network administrator for a solution. I don’t blame the administrator in any way when he becomes exasperated by our requests. As it is my solemn duty to protect the database, the network administrator is the sole protector of the network. You’ll hear a frustrated DBA say, “just open the &^$# network up! Let’s just get this connected to our cloud provider!” I have to admit that this request must be akin to someone asking a DBA to provide SYSDBA to a developer in production.
So yes, there are a lot of moving parts in a cloud environment. No, not all of them are at the database level, but many of them could be at the network level. This means that your new cloud environment must connect past firewalls, proxies, blocked ports and authentication steps that may not have been required back in the sole on-premise days.
Yeah, there’s a bit more to the network than demonstrated in the picture above.
The database connection needs a secure connection past the firewall and may require proxy configurations to access via a web browser. The application interface to manage them may require proxy settings in browsers that may have automated processes to manage outside a manual proxy setting. You may have network configurations that are different from one local office to another. We’ve only discussed configuration and haven’t even considered speed, packet size and bandwidth.
So here is my recommendation- make friends with your network administrator. In fact, take the ol’ chap out for a beer or two. Learn about what it takes to master, protect and ensure the company’s network from the threats outside. Learning about the network will provide you with incredible value as a cloud administrator and you may get a great friend out of the venture, too. For those of you that don’t make friends with your network admin, I don’t want to be hearing about any mishaps with phenobarbital to get the information, OK? 🙂