As many know I’ve been busy trying to bring our many-times patched 10g Enterprise Manager migrated to a new server with EM12c.  I thought it would be interesting to see how fast I could break it, considering my skills, I knew it might not be a challenge.

Scenario

1.  Bug with listener refusing to connect dynamically to EM12c repository database.

2.  After releasing code to the database for advanced notification,  the SYSMAN.MGMT_ADMIN_DATA package has gone invalid, which is used to manage much of the repository at the command line.

3.  The repository owner password has become corrupted.

These were my high level notes as I went through the troubleshooting:

Reconfigure Repository:
$AGENT_HOME/bin/emctl config oms -store_repos_details -repos_port 1521 -repos_sid emrep12c -repos_host host_nm -repos_user SYSMAN -repos_pwd password
Stopping Old Oracle OMS and configuring:
export ORACLE_HOME=/opt/oracle/app/OracleHomes/oms10g
cd $ORACLE_HOME/opmn/bin
./opmnctl stopall
cd $ORACLE_HOME/bin
./emctl config oms -change_repos_pwd
cd $ORACLE_HOME/opmn/bin
./opmnctl startall
Invalid package after loop from listener bug, corrupt SYSMAN password and invalid MGMT_XXX pkgs!
Can’t reset password, pkg used for it is invalid!
ORA-04063: package body “SYSMAN.MGMT_ADMIN_DATA” has errors
ORA-06508: PL/SQL: could not find program unit being called: “SYSMAN.MGMT_ADMIN_DATA”
ORA-06512: at line 1
Can’t start repository, password error in logs:
Error occurred. Check the log /local/u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
[oracle@vwgrid01 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Soooo close…but did I get far enough?, (Nope, failure…have to remove everything!!)
Can’t drop repository:
<Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD sys_password -SYSMAN_PWD password
Can’t reconfigure through the system, either!
<Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -cluster -SYS_PWD sys_password -SYSMAN_PWD password
No chance either…I’m stuck in a terrible loop!
Solution-
backup and then edit out the 12c environment from the inventory.xml file in the ContentsXML dir of the Lsinventory
kill any process that are still running out the the EM12c home.
rm -rf the 12c home directory
Uninstall the emrep12 database in the 11g home
Create a new emrep12 database in the 11g home
recreate a new inventory file:
$ORACLE_HOME/oui/bin/runInstaller -silent -invPtrLoc “/u01/app/oraInventory/oraInst.loc” -attachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0″ ORACLE_HOME_NAME=”OraDb11g_home2″
Install the EM12c once again, as now the installation appears to have never existed…
If a restart of the installation needs to be done, you need to look in the home it fails on, (*saying that it’s already installed there) and remove the *_temp file from the dir, then try again.

« »