Restarting a Duplicate Process From a VERY Failed State
As part of an 11g Upgrade, it was found that a database environment could be built from one of the upgraded databases through a duplicate. As this process had never been performed before in this fashion, a test was in order. The test was an excellent chance to discover that the OSUser that performs the duplicate process was the proud owner of a .kshrc file with hard coded Oracle variables set which was an excellent choice if you want to really screw up a new duplicate database… 
Scenario:
1. Duplicate has started with correct variables set.
2. Subsequent shell scripts then “upset” the environment mid-process leaving a failed duplicate with ASM files for the controlfiles created, but the DBID and dbname still set to the target database, not the auxillary.
3. Post the failure, the auxillary, (duplicate) database can only be mounted, not opened.
| After setting environment to building auxillary database environment, after duplicate failure and you see in the spfile, name of db is no longer the auxilary database. |
| 1. Shutdown the auxilary database: |
| [oracledbs]$ sqlplus ‘/as sysdba’ |
| SQL> shutdown; |
| ORA-01109: database not open |
| Database dismounted. |
| ORACLE instance shut down.@ |
| In a second screen set to +ASM instance for auxillary, remove controlfiles for target db that have been set to that name through the asmcmd command console: |
| [oracle@dbs]$ asmcmd |
| ASMCMD> ls |
| DATA_H/ |
| DATA_RAM_H/ |
| ASMCMD> cd DATA_H |
| ASMCMD> ls |
| DB_H/ |
| ASMCMD> cd DB_H |
| ASMCMD> ls |
| CONTROLFILE/ |
| ASMCMD> cd CONTROLFILE |
| ASMCMD> ls |
| control1.ctl <–these are control files for auxillary, stuck with dbname of target, can’t be renamed, can’t mount db! |
| control2.ctl |
| control3.ctl |
| current.389.766743537 |
| current.445.766743537 |
| current.500.766743537 |
| ASMCMD> rm control* |
| You may delete multiple files and/or directories. |
| Are you sure? (y/n) y |
| ASMCMD> quit |
| Back on your original screen now, restart the auxillary with pfile set to corect dbname and start with this pfile- |
| SQL> startup nomount pfile=’/u01/app/oracle/product/11.2.0/dbhome_2/dbs/initdb_h.ora’ |
| ORACLE instance started. |
| Total System Global Area 7.6964E+10 bytes |
| Fixed Size 2215704 bytes |
| Variable Size 3.0065E+10 bytes |
| Database Buffers 4.6708E+10 bytes |
| Redo Buffers 189513728 bytes |
| SQL> create spfile from pfile; |
| File created. |
| SQL> shutdown; |
| ORACLE instance shut down. |
| SQL> startup nomount; |
| ORACLE instance started. |
| Total System Global Area 7.6964E+10 bytes |
| Fixed Size 2215704 bytes |
| Variable Size 3.0065E+10 bytes |
| Database Buffers 4.6708E+10 bytes |
| Redo Buffers 189513728 bytes |
| Now you can restart the duplicate process and the database will again be recognized correctly. |
How to Break and EM12c and Other Hobbies
As many know I’ve been busy trying to bring our many-times patched 10g Enterprise Manager migrated to a new server with EM12c. I thought it would be interesting to see how fast I could break it, considering my skills, I knew it might not be a challenge.
Scenario
1. Bug with listener refusing to connect dynamically to EM12c repository database.
2. After releasing code to the database for advanced notification, the SYSMAN.MGMT_ADMIN_DATA package has gone invalid, which is used to manage much of the repository at the command line.
3. The repository owner password has become corrupted.
These were my high level notes as I went through the troubleshooting:
| Reconfigure Repository: |
| $AGENT_HOME/bin/emctl config oms -store_repos_details -repos_port 1521 -repos_sid emrep12c -repos_host host_nm -repos_user SYSMAN -repos_pwd password |
| Stopping Old Oracle OMS and configuring: |
| export ORACLE_HOME=/opt/oracle/app/OracleHomes/oms10g |
| cd $ORACLE_HOME/opmn/bin |
| ./opmnctl stopall |
| cd $ORACLE_HOME/bin |
| ./emctl config oms -change_repos_pwd |
| cd $ORACLE_HOME/opmn/bin |
| ./opmnctl startall |
| Invalid package after loop from listener bug, corrupt SYSMAN password and invalid MGMT_XXX pkgs! |
| Can’t reset password, pkg used for it is invalid! |
| ORA-04063: package body “SYSMAN.MGMT_ADMIN_DATA” has errors |
| ORA-06508: PL/SQL: could not find program unit being called: “SYSMAN.MGMT_ADMIN_DATA” |
| ORA-06512: at line 1 |
| Can’t start repository, password error in logs: |
| Error occurred. Check the log /local/u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/secure.log |
| [oracle@vwgrid01 bin]$ ./emctl start oms |
| Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 |
| Soooo close…but did I get far enough?, (Nope, failure…have to remove everything!!) |
| Can’t drop repository: |
| <Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD sys_password -SYSMAN_PWD password |
| Can’t reconfigure through the system, either! |
| <Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -cluster -SYS_PWD sys_password -SYSMAN_PWD password |
| No chance either…I’m stuck in a terrible loop! |
| Solution- |
| backup and then edit out the 12c environment from the inventory.xml file in the ContentsXML dir of the Lsinventory |
| kill any process that are still running out the the EM12c home. |
| rm -rf the 12c home directory |
| Uninstall the emrep12 database in the 11g home |
| Create a new emrep12 database in the 11g home |
| recreate a new inventory file: |
| $ORACLE_HOME/oui/bin/runInstaller -silent -invPtrLoc “/u01/app/oraInventory/oraInst.loc” -attachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0″ ORACLE_HOME_NAME=”OraDb11g_home2″ |
| Install the EM12c once again, as now the installation appears to have never existed… |
| If a restart of the installation needs to be done, you need to look in the home it fails on, (*saying that it’s already installed there) and remove the *_temp file from the dir, then try again. |
