Monthly Archives: November 2011

Restarting a Duplicate Process From a VERY Failed State

Posted on by
Reply

As part of an 11g Upgrade, it was found that a database environment could be built from one of the upgraded databases through a duplicate.  As this process had never been performed before in this fashion, a test was in order.  The test was an excellent chance to discover that the OSUser that performs the duplicate process was the proud owner of a .kshrc file with hard coded Oracle variables set which was an excellent choice if you want to really screw up a new duplicate database… :)

payday loans lenders online

Scenario:

1. Duplicate has started with correct variables set.

2.  Subsequent shell scripts then “upset” the environment mid-process leaving a failed duplicate with ASM files for the controlfiles created, but the DBID and dbname still set to the target database, not the auxillary.

3.  Post the failure, the auxillary, (duplicate) database can only be mounted, not opened.

After setting environment to building auxillary database environment, after duplicate failure and you see in the spfile, name of db is no longer the auxilary database.
1.  Shutdown the auxilary database:
 [oracledbs]$ sqlplus ‘/as sysdba’
SQL> shutdown;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.@
In a second screen set to +ASM instance for auxillary, remove controlfiles for target db that have been set to that name through the asmcmd command console:
[oracle@dbs]$ asmcmd
ASMCMD> ls
DATA_H/
DATA_RAM_H/
ASMCMD> cd DATA_H
ASMCMD> ls
DB_H/
ASMCMD> cd DB_H
ASMCMD> ls
CONTROLFILE/
ASMCMD> cd CONTROLFILE
ASMCMD> ls
control1.ctl <–these are control files for auxillary, stuck with dbname of target, can’t be renamed, can’t mount db!
control2.ctl
control3.ctl
current.389.766743537
current.445.766743537
current.500.766743537
ASMCMD> rm control*
You may delete multiple files and/or directories.
Are you sure? (y/n) y
ASMCMD> quit
Back on your original screen now, restart the auxillary with pfile set to corect dbname and start with this pfile-
SQL>  startup nomount pfile=’/u01/app/oracle/product/11.2.0/dbhome_2/dbs/initdb_h.ora’
ORACLE instance started.
Total System Global Area 7.6964E+10 bytes
Fixed Size                  2215704 bytes
Variable Size            3.0065E+10 bytes
Database Buffers         4.6708E+10 bytes
Redo Buffers              189513728 bytes
SQL> create spfile from pfile;
File created.
SQL> shutdown;
ORACLE instance shut down.
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 7.6964E+10 bytes
Fixed Size                  2215704 bytes
Variable Size            3.0065E+10 bytes
Database Buffers         4.6708E+10 bytes
Redo Buffers              189513728 bytes
Now you can restart the duplicate process and the database will again be recognized correctly.

How to Break and EM12c and Other Hobbies

Posted on by
Reply

As many know I’ve been busy trying to bring our many-times patched 10g Enterprise Manager migrated to a new server with EM12c.  I thought it would be interesting to see how fast I could break it, considering my skills, I knew it might not be a challenge.

Scenario

1.  Bug with listener refusing to connect dynamically to EM12c repository database.

2.  After releasing code to the database for advanced notification,  the SYSMAN.MGMT_ADMIN_DATA package has gone invalid, which is used to manage much of the repository at the command line.

3.  The repository owner password has become corrupted.

These were my high level notes as I went through the troubleshooting:

Reconfigure Repository:
$AGENT_HOME/bin/emctl config oms -store_repos_details -repos_port 1521 -repos_sid emrep12c -repos_host host_nm -repos_user SYSMAN -repos_pwd password
Stopping Old Oracle OMS and configuring:
export ORACLE_HOME=/opt/oracle/app/OracleHomes/oms10g
cd $ORACLE_HOME/opmn/bin
./opmnctl stopall
cd $ORACLE_HOME/bin
./emctl config oms -change_repos_pwd
cd $ORACLE_HOME/opmn/bin
./opmnctl startall
Invalid package after loop from listener bug, corrupt SYSMAN password and invalid MGMT_XXX pkgs!
Can’t reset password, pkg used for it is invalid!
ORA-04063: package body “SYSMAN.MGMT_ADMIN_DATA” has errors
ORA-06508: PL/SQL: could not find program unit being called: “SYSMAN.MGMT_ADMIN_DATA”
ORA-06512: at line 1
Can’t start repository, password error in logs:
Error occurred. Check the log /local/u01/app/oracle/product/12.1.0/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
[oracle@vwgrid01 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Soooo close…but did I get far enough?, (Nope, failure…have to remove everything!!)
Can’t drop repository:
<Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD sys_password -SYSMAN_PWD password
Can’t reconfigure through the system, either!
<Database ORACLE HOME>/bin/emca -deconfig dbcontrol db -repos drop -cluster -SYS_PWD sys_password -SYSMAN_PWD password
No chance either…I’m stuck in a terrible loop!
Solution-
backup and then edit out the 12c environment from the inventory.xml file in the ContentsXML dir of the Lsinventory
kill any process that are still running out the the EM12c home.
rm -rf the 12c home directory
Uninstall the emrep12 database in the 11g home
Create a new emrep12 database in the 11g home
recreate a new inventory file:
$ORACLE_HOME/oui/bin/runInstaller -silent -invPtrLoc “/u01/app/oraInventory/oraInst.loc” -attachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0″ ORACLE_HOME_NAME=”OraDb11g_home2″
Install the EM12c once again, as now the installation appears to have never existed…
If a restart of the installation needs to be done, you need to look in the home it fails on, (*saying that it’s already installed there) and remove the *_temp file from the dir, then try again.