How to Break and EM12c and Other Hobbies
As many know I’ve been busy trying to bring our many-times patched 10g Enterprise Manager migrated to a new server with EM12c. I thought it would be interesting to see how fast I could break it, considering my skills, I knew it might not be a challenge.
Scenario
1. Bug with listener refusing to connect dynamically to EM12c repository database.
2. After releasing code to the database for advanced notification, the SYSMAN.MGMT_ADMIN_DATA package has gone invalid, which is used to manage much of the repository at the command line.
3. The repository owner password has become corrupted.
These were my high level notes as I went through the troubleshooting:
| Reconfigure Repository: |
| $AGENT_HOME/bin/emctl config oms -store_repos_details -repos_port 1521 -repos_sid emrep12c -repos_host host_nm -repos_user SYSMAN -repos_pwd password |
| Stopping Old Oracle OMS and configuring: |
| export ORACLE_HOME=/opt/oracle/app/OracleHomes/oms10g |
| cd $ORACLE_HOME/opmn/bin |
| ./opmnctl stopall |
| cd $ORACLE_HOME/bin |
| ./emctl config oms -change_repos_pwd |
| cd $ORACLE_HOME/opmn/bin |
| ./opmnctl startall |
| Invalid package after loop from listener bug, corrupt SYSMAN password and invalid MGMT_XXX pkgs! |
| Can’t reset password, pkg used for it is invalid! |
| ORA-04063: package body “SYSMAN.MGMT_ADMIN_DATA” has errors |
| ORA-06508: PL/SQL: could not find program unit being called: “SYSMAN.MGMT_ADMIN_DATA” |
| ORA-06512: at line 1 |
| Can’t start repository, password error in logs: |
| Error occurred. Check the log /local/u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/secure.log |
| [oracle@vwgrid01 bin]$ ./emctl start oms |
| Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 |
| Soooo close…but did I get far enough?, (Nope, failure…have to remove everything!!) |
| Can’t drop repository: |
| <Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD sys_password -SYSMAN_PWD password |
| Can’t reconfigure through the system, either! |
| <Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -cluster -SYS_PWD sys_password -SYSMAN_PWD password |
| No chance either…I’m stuck in a terrible loop! |
| Solution- |
| backup and then edit out the 12c environment from the inventory.xml file in the ContentsXML dir of the Lsinventory |
| kill any process that are still running out the the EM12c home. |
| rm -rf the 12c home directory |
| Uninstall the emrep12 database in the 11g home |
| Create a new emrep12 database in the 11g home |
| recreate a new inventory file: |
| $ORACLE_HOME/oui/bin/runInstaller -silent -invPtrLoc “/u01/app/oraInventory/oraInst.loc” -attachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0″ ORACLE_HOME_NAME=”OraDb11g_home2″ |
| Install the EM12c once again, as now the installation appears to have never existed… |
| If a restart of the installation needs to be done, you need to look in the home it fails on, (*saying that it’s already installed there) and remove the *_temp file from the dir, then try again. |
Lovely or sad to see that I’m not the only one struggling with what seems like randomness in OEM 🙂 keep up the good work with EM12 and your blog. Always a pleasure reading it 🙂
Regards
André