Recommendations for Oracle 19c Patches in Azure

*Previously posted on the Microsoft Data Architecture Blog for the Tech community.

Oracle 19c is the terminal release for Oracle 12c.  If you aren’t familiar with that term, a terminal release is the last point release of the product.  There were terminal releases for previous Oracle versions (10.2.0.4, 11.2.0.7.0) and after 19c, the next terminal release will be 23c.  Therefore, you don’t see many 18c, 20c or 21c databases.  We’ve gone to yearly release numbers, but the fact remains that 19c is going to receive all major updates and continue to be supported unlike the non-terminal releases.

Oracle will tell you for support, you should be upgrading to 19c.  Premier Oracle Support ended for December 1^st, 2020 and as we discussed, not many are going to choose or stay on non-terminal releases, so 19c is it.

With that said, we must offer guidance on recommended practices for Oracle versioning and patching in Azure.  Although I will list any bugs and document IDs that back up the recommendations I’m making, be aware that many of these will be behind Oracle Support’s paywall, so you’ll only be able to access them with an Oracle Support CSI.  Let’s talk about the things not to do first-

Don’t Upgrade DURING Your Migration

I know it sounds like an awesome idea to upgrade to the latest database version while you are migrating to the cloud, but please, don’t do these two things- migrating to the cloud and upgrading the database/app at the same time.  It’s a common scenario that I’m brought in after the Azure specialists are left scratching their head or scrambling to explain what has changed and then I come in to tell them to stand down because it’s the DATABASE THAT’S CHANGED.

Do Patch to the LATEST Patchset for Oracle

I am part of the crowd that often did the latest patchset -1 approach.  We would always be one patchset behind and let others figure out how many bugs might be introduced by the patch that had sneaked through testing.

Not anymore… I have a few customers on 19.14, which should be safe, considering the previous practice I mentioned, but the sheer number of bugs and serious bugs that were experienced has changed my thinking to recommend going to the latest patchset.

I think it’s easy to think, “Oh, it’s just a small bug” but I’m in agreement with you, if it’s a small impact and it has an easy work around, that’s one thing, but these bugs I’m referring to are quite impactful and here’s how:

High CPU Usage

• 19.14 release, there were 11 bugs that caused high CPU usage for Oracle.
• High CPU usage to the point of doubling the core count for the VM the database ran on in Azure.
• Doubling the need for Oracle licenses for the database, even though it was a bug that was causing all the additional CPU usage.
• At $47500 list price per processor license, this isn’t something I’d recommend letting go on.

For one customer that I was deeply involved in, the VM sizing required 20 vCPU to run the workload.  I sized up to 32 vCPU for peak workloads and yet they were at 97.6% CPU busy with a 64-core machine.  The workload hadn’t changed, and the CPU usage traced was out of control!

I would start here: After Upgrade to 19c, One or More of the Following Issues Occur on Non-Linux Platforms: High Paging/Swapping, High CPU, Poor Performance, ORA-27nnn Errors, ORA-00379 Errors, ORA-04036 Errors (Doc ID 2762216.1)

Bug examples for high CPU usage in 19.14:

NB	Prob	Bug	Fixed	Description
	II	31050103	19.15, 23.1.0.0.0	fbda: slow sql performance when running in pluggable database
	–	32869560	19.15, 21.6	HIGH CPU ON KXSGETRUNTIMELOCK AND SSKGSLCAS
	I	29446010	20.1	Query Using LIKE Predicate Spins Using NLS_SORT=’japanese_m’ NLS_COMP=’linguistic’
	–	32431067	23.1.0.0.0	Data Pump Export is Slow When Exporting Scheduler Jobs Due to Query Against SYS.KU$_PROCOBJ_VIEW
	–	33380871	19.15, 21.6	High CPU on KSLWT_UPDATE_STATS_ELEM
	–	33921441	19.15	Slow performance in AQ dequeue processing
*	II	32075777		Performance degradation by Wnnn processes after applying july 2020 DBRU
	III	32164034		Database Hang Updating USER$ When LSLT (LAST SUCCESSFUL LOGIN TIME) Is Enabled
	III	30664385		High count of repetitive executions for sql_id 35c8afbgfm40c during incremental statistics gathering
	II	29559415		DMLs on FDA enabled tables are slow, or potential deadlocks on recursive DML on SYS_FBA_* tables
	II	29448426	20.1	Killing Sessions in PDB Eventually Results in Poor Buffer Cache Performance Due To Miscalculating Free Buffer Count

 

Time Slip

This issue will often display an ORA-00800 error and you will need to check the extended trace file for details.  It will include the VKTM in the error arguments.

…/trace/xxxxx_vktm_xxxx.trc

ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM], [Check traces and OS configuration], [Check Oracle document and MOS notes]

The trace file will include additional information about the error, including:

Kstmmainvktm: failed in setting elevated priority

Verify: SETUID is set on ORADISM and restart the instance highres_enabled

This refers to a bug and has two documents around time drift and how to address it-

ORA-00800: soft external error, arguments: [Set Priority Failed], [VKTM] (Doc ID 2718971.1)

I’d also refer to this doc, even though you aren’t running AIX:

Bug 28831618 : FAILED TO ELEVATE VKTM’S PRIORITY IN AIX WITH EVENT 10795 SET

Network Connection Timeouts

Incident alerting will occur in the alert log, and it will require viewing the corresponding trace file for the incident.

ORA-03137: malformed TTC packet from client rejected.

ORA-03137: Malformed TTC Packet From Client Rejected: [12569] (Doc ID 2498924.1)

Potential Tracing to gather more data:

Getting ORA-12569: TNS:Packet Checksum Failure While Trying To Connect Through Client (Doc ID 257793.1)

Block Corruption

Thanks to Jeff Steiner from the NetApp team who advised on this one.

Bug 32931941 – Fractured block Corruption Found while Using DirectNFS (Doc ID 32931941.8)

• This can result in 100’s to 1000’s of corrupted blocks in an Oracle database.
• All customers using dNFS with 19c should run 19.14 or higher to avoid being vulnerable to this bug.

Also follow the Recommended Patches for Direct NFS Client (Doc ID 1495104.1)

Summary

If you’re considering an upgrade to Oracle 19c, please review the following Oracle Doc:

Things to Consider to Avoid Database Performance Problems on 19c (Doc ID 2773012.1)

It really is worth your time and can save you a lot of time and headache.