Solaris Cluster and Decreasing Resource Needs After Upgrade
Delphix Engineering and Support are pretty amazing folks. They continue to pursue for solutions, no matter how much time it takes and the complex challenges they’re faced with supporting heterogenous environments, hardware configurations and customer needs.
This post is in support of the effort from our team that resulted in stability to a previously impacted Solaris 11.2 cluster configuration. The research, patching, testing and then resulting certification from Oracle was a massive undertaking from our team and I hope this information serves the community, but in no way is recommended by Delphix. It’s just what was done to resolve the problem, after logical decisions for the use of the system by our team.
Challenge
Research
$ echo “::memstat” | mdb -k
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 151528 3183 24%
Anon 185037 1623 12%
...
We can also look at it a second way, breaking down the kernel memory areas with kmsastat:
::kmsastat cache buf buf buf memory alloc alloc name size in use total in use succeed fail ------------------------- ------ ------ ------ --------- --------- ----- kmem_magazine_1 16 3371 3556 57344 3371 0 kmem_magazine_3 32 16055 16256 524288 16055 0 kmem_magazine_7 64 29166 29210 1884160 29166 0 kmem_magazine_15 128 6711 6741 876544 6711 0 ...
Oracle ZFS ARC Cache
Next- Oracle ZFS has a very smart cache layer, also referred to as ARC (Adaptive replacement cache). Both a blessing and a curse, ARC consumes as much memory that is available, but is supposed to free up memory to other applications if it’s needed. This memory is used to supplement any slow disk I/O. When inspecting our environment, a significant amount was being over-allocated to ARC. This may be due to the newness of Oracle 12.2, but in a cluster, memory starvation can be a common cause of node eviction.
We can inspect the size stats for the ARC in the following file:
view /proc/spl/kstat/zfs/arcstatsThis assumes ZFS is mounted on /proc, so your actual arcstats file may reside in a different path location than shown above. Inside the file, review the following information:
cis the target size of the ARC in bytesc_maxis the maximum size of the ARC in bytessizeis the current size of the ARC in bytes
Ours was eating up everything left, taking 100% of memory left, as we’ll discuss in the next section of this post.
Oracle Clusterware Memory
The Oracle clusterware is a third area that was investigated for frivolous memory usage that could be trimmed down. There’s some clear documented steps to investigate issues with misconfigurations and feature issues from Oracle that can assist in identifying many of these.
So, post upgrade and patching, what can you do to trim down memory usage to avoid memory upgrades to support the cluster upgrade?
Changes
- Added set user_reserve_hint_pct=80
- This change was made to limit the ZFS on how much memory for the ARC cache. There was a significant issue for the customer when CRS processes weren’t able to allocate memory. 80% was the highest percentage this could be set without a node reboot being experienced, something we all prefer not to happen.
- Stopped the Cluster Health Monitor, (CHM) process. This is a brand new background process in 12c Clusterware and collects workload data, which is significantly more valuable in a production environment, but in development and test? It can easily be a subsequent drain on CPU and memory that could be better put to use for more virtual databases.
- To perform this, the following commands were used as the root user:
$ crsctl stop res ora.crf -init
$ crsctl delete res ora.crf -init
- Removed the Trace File Analyzer Collector (tfactl). This background process collects the many trace files Oracle generates into a single location. Handy for troubleshooting, but it’s Java-based and has a significant memory footprint and subject to java heap issues.
- It was uninstalled with the following command as the $ORACLE_HOME owner on each node of the cluster:
$ tfactl uninstall
- Engineering stopped and disabled the Cluster Verification Utility, (CVU). In previous version this was a utility that could be manually added to the installation or performed post to troubleshoot issues via an Admin. This is another feature that simply eats up resources that could be reallocated to dev and test environments, so it was time to stop and disable it with the following:
$ srvctl cvu stop $ srvctl cvu disable
Additional Changes
- Reduced memory allocation for the ASM instance.
- The ASM instance in 12.2 is now using 1Gb of memory, where previous 256Mb. That’s a huge change that can impact other features dependent on that memory.
- Upon research, it was found that 750Mb was adequate, so if more memory reallocation is required, consider lowering the memory on each node to 750Mb.
- To perform this set of instance level parameter change, run the following on any of the nodes and then restart each node until the cluster has been cycled to put the change into effect:
$ export ORACLE_HOME=<Grid Home> $ export ORACLE_SID=<Local ASM SID> $ sqlplus / as sysasm alter system set "_asm_allow_small_memory_target"=true scope=spfile; alter system set memory_target=750m scope=spfile; alter system set memory_max_target=750m scope=spfile;
High CPU usage features can be troubling for most DBAs, but when it’s experienced on development and test databases that are often granted less resources to begin with vs. production, a change can often enhance the stability and longevity of these environments.
- Disabled high-res time ticks in all databases, including ASM DBs, regular DBs, and the Grid Infrastructure Management Repository DB (GIMR, SID is -MGMTDB). High-res ticks are a new feature in 12c, and they seem to cause a lot of CPU usage from cluster time-keeping background processes like VKTM. Here’s the SQL to disable high-res ticks (must be run once in each DB):
alter system set "_disable_highres_ticks"=TRUE scope=spfile;
- Solaris Kernel: 1GB of RAM
- ARC Cache: between 1-2GB
- Oracle Clusterware: 3Gb
