Error starting managed server from Console :Unable to create a server socket for listening on channel "Default"

While starting the managed server from the Admin console, after nodemanager configuration, managed server was failing with status as "FAILED_to_START"

Unable to create a server socket for listening on channel "Default". The address 151.49.38.9 might be incorrect or another process is using port 18101: java.net.BindException: Cannot assign requested address.

Solution:Setup was two node cluster. I configured nodemanager and started it. but in console node manager configuration for managed server opam_server2 was showing localhost and hence it was trying to start from first node(where domain was configured) and it was failing. So update localhost with node2 hostname as below.


In Weblogic console, go to Environment -> Machines -> Configuration -> Node  Manager.

Update opam_server2 manager server with remote server host name(test2.opam.com).

Failed to start RAC 11gr2 database instance after applying one off patch

When I tried to start the database after applying patch, I got the error as below.

[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.124.148 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
[oracle@test4201 ~]$ ps -ef|grep pmon
oracle    7221  6673  0 09:38 pts/1    00:00:00 grep --color=auto pmon
[oracle@test4201 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 09:39:04 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
SQL> exit

Solution:

I tried by setting cluster_interconnect parameter in init file, but it didn't help.

testms1.cluster_interconnects=10.243.51.15
testms2.cluster_interconnects=10.243.51.34



[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.15 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.34 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
[oracle@test4201 ~]$ vi /u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log



What mistake I had done is, I had missed post steps which does relinking of oracle binaries. I relinked the binaries on both the nodes with "oracle" user as below.

Note: If you are using rds protocol, command will be make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle . Check the below link on how to check protocol being used.
http://jongsma.wordpress.com/2012/08/08/mixing-udp-and-rds-on-an-exadata/

Node1 :

[oracle@test4201 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'

Node2:

[oracle@test4202 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'


Now started the database and it came up.

[oracle@test4201 dbs]$ . oraenv
ORACLE_SID = [testms] ? testms1
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test4201 dbs]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:00 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.


ORACLE_SID = [oracle] ? testms2
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base has been set to /u01/app/oracle
[oracle@test4202 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:34 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.
SQL> 

Oracle Home inventory is corrupted in 11gr2 RAC

While checking oracle inventory with opatch, I got error as below.

[oracle@test4201 logs]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/opatch2013-10-29_12-47-27PM_1.log

List of Homes on this system:

  Home name= Ora11g_gridinfrahome1, Location= "/u01/app/11203/grid"
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73
[oracle@test4201 logs]$


Solution:

Detach the oracle home and attach it again as below.

[oracle@test4201 logs]$ cd $ORACLE_HOME/oui/bin
[oracle@test4201 bin]$ ./runInstaller -detachHome ORACLE_HOME="/u01/app/oracle/product/11203/testms"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16378 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'DetachHome' failed.
[oracle@test4201 bin]$ ./runInstaller -silent -invPtrLoc /u01/app/oraInventory -attachHome ORACLE_HOME="/u01/app/oracle/product/11203/testms" ORACLE_HOME_NAME="OraDb11g_home1"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16378 MB    Passed
The inventory pointer is located at /u01/app/oraInventory
The inventory is located at /u01/app/oraInventory
'AttachHome' was successful.

Check now

[oracle@test4201 bin]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/opatch2013-10-29_12-52-50PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/lsinv/lsinventory2013-10-29_12-52-50PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.3.0
There are 1 products installed in this Oracle Home.

OPatch failed with error code 41 while applying the database patch in 11gR2 RAC

OPatch failed with error code 41 while applying the database patch in 11gR2 RAC


oracle@test4201 12646746]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log

Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'
Verifying environment and performing prerequisite checks...
Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:


Following executables are active :
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log

Recommended actions: OPatch needs to modify files which are being used by some processes.

OPatch failed with error code 41


In log it's showing that

36 [Oct 29, 2013 5:27:27 AM]    Start fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
 37 [Oct 29, 2013 5:27:27 AM]    Finish fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
 38 [Oct 29, 2013 5:27:27 AM]    Following executables are active :
 39                              /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
 40 [Oct 29, 2013 5:27:27 AM]    Prerequisite check "CheckActiveFilesAndExecutables" failed.
 41                              The details are:
 42
 43
 44                              Following executables are active :
 45                              /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
 46 [Oct 29, 2013 5:27:27 AM]    OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
 47                              You will be still able to rollback patches after this cleanup.
 48                              Do you want to proceed? [y|n]
 49 [Oct 29, 2013 5:27:30 AM]    Y (auto-answered by -silent)
 50 [Oct 29, 2013 5:27:30 AM]    User Responded with: Y
 51 [Oct 29, 2013 5:27:30 AM]    Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" before cleanup is 282613767 bytes.
 52 [Oct 29, 2013 5:27:30 AM]    Deleting the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
 53 [Oct 29, 2013 5:27:30 AM]    Deleted the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
 54 [Oct 29, 2013 5:27:30 AM]    Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" after cleanup is 282613767 bytes.
 55 [Oct 29, 2013 5:27:30 AM]    UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
 56                              deleted, Please refer log file.
 57 [Oct 29, 2013 5:27:30 AM]    [ Error during Prerequisite for apply phase ] Detail:                              OUI-67074:ApplySession failed during prerequisite checks: Prerequisite check "Che
 58 [Oct 29, 2013 5:27:30 AM]    OUI-67035:System is intact, OPatch will not restore the system
 59 [Oct 29, 2013 5:27:30 AM]    Finishing ApplySession at Tue Oct 29 05:27:30 UTC 2013

Solution:

1.Identify the process which is accessing /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 executable.



[oracle@test4201 bp17]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1

                     USER        PID ACCESS COMMAND
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1:
                     oracle    23612 ....m emagent

[oracle@test4201 bp17]$ ps -ef|grep 23612
oracle   22753 21386  0 22:31 pts/6    00:00:00 grep --color=auto 23612
oracle   23612 12053  0 10:13 pts/11   00:00:39 /u01/app/oracle/product/11203/testms/bin/emagent

In my case, it's found that EM agent was running so I killed that process as it was test environment. In production environment, check the status and stop the EM agent.

[oracle@test4201 bin]$ kill -9 23612
[oracle@test4201 bin]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
[oracle@test4201 bin]$ pwd

2. Now try to apply the patch.


[oracle@test4201 12646746]$ opatch apply                                                                                                                                                            Oracle Interim Patch Installer version 11.2.0.3.5                                                                                                                                                    Copyright (c) 2013, Oracle Corporation.  All rights reserved.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Oracle Home       : /u01/app/oracle/product/11203/testms                                                                                                                                            Central Inventory : /u01/app/oraInventory                                                                                                                                                               from           : /u01/app/oracle/product/11203/testms/oraInst.loc                                                                                                                                OPatch version    : 11.2.0.3.5                                                                                                                                                                       OUI version       : 11.2.0.3.0                                                                                                                                                                       Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log                                                                                                                                                                                                                                                               Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'                                                                                                                      Verifying environment and performing prerequisite checks...                                                                                                                                          Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]                                                                                                                  All checks passed.                                                                                                                                                                                   Provide your email address to be informed of security issues, install and                                                                                                                            initiate Oracle Configuration Manager. Easier for you if you use your My                                                                                                                             Oracle Support Email address/User Name.                                                                                                                                                              Visit http://www.oracle.com/support/policies.html for details.                                                                                                                                       Email address/User Name:                                                                                                                                                                                                                                                                                                                                                                                  You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y



This node is part of an Oracle Real Application Cluster.
Remote nodes: 'test4202'
Local node: 'test4201'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11203/testms')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...

Patching component oracle.network.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Verifying the update...

The local system has been patched.  You can restart Oracle instances on it.


Patching in rolling mode.


The node 'test4202' will be patched next.


Please shutdown Oracle instances running out of this ORACLE_HOME on 'test4202'.
(Oracle Home = '/u01/app/oracle/product/11203/testms')

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'test4202'
   Apply-related files are:
     FP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt"
     DP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt"
     MP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt"
     RC = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/remote_cmds.txt"

Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt" with actual path.
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2


The node 'test4202' has been patched.  You can restart Oracle instances on it.

There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'test4202' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
'

Patch 12646746 successfully applied
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log

OPatch succeeded.
[oracle@test4201 12646746]$ cd ..

OPatch failed with error code 73

I tried to apply the patch to database instance and got the error as below.

[oracle@test01db 13936066]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/13936066_Oct_29_2013_09_24_25/apply2013-10-29_09-24-25AM_1.log

Applying interim patch '13936066' to OH '/u01/app/oracle/product/11203/testdbms'
Verifying environment and performing prerequisite checks...
[ Error during Oracle Home discovery Phase]. Detail: OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/11203/testdbms. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

[ Error during Oracle Home discovery Phase]. Detail: OPatch failed: ApplySession failed to prepare the system. Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/16311483 is corrupted
   No read permission to above directory
Please remove one-off entry [ 16311483 ] from /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml and retry if above directory corrupted, or Please check the contents of the directory ORACLE_HOME/inventory/oneoffs/16311483 for read permission
Log file location: /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/13936066_Oct_29_2013_09_24_25/apply2013-10-29_09-24-25AM_1.log

Recommended actions: Please make sure no other OPatch or OUI processes is running. Try running $ORACLE_HOME/oui/bin/runInstsaller.

OPatch failed with error code 22


When I checked the inventory, got the error as below.

[oracle@test01db ContentsXML]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/opatch2013-10-29_09-50-44AM_1.log



Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/16311483 is corrupted
   No read permission to above directory
Please remove one-off entry [ 16311483 ] from /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml and retry if above directory corrupted, or Please check the contents of the directory ORACLE_HOME/inventory/oneoffs/16311483 for read permission

OPatch failed with error code 73



So when I checked the patch 16311483 at $ORACLE_HOME/inventory/oneoffs, couldn't find it.

[oracle@test01db ContentsXML]$ ls -lrt $ORACLE_HOME/inventory/oneoffs/16311483
ls: /u01/app/oracle/product/11203/testdbms/inventory/oneoffs/16311483: No such file or directory
[oracle@test01db ContentsXML]$


Solution:

Take the backup of /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml file and then remove  "16311483" corresponding entry(from ) from comps.xml. After that check the inventory as below.


[oracle@test01db ContentsXML]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/opatch2013-10-29_10-13-01AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/lsinv/lsinventory2013-10-29_10-13-01AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.3.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  12646746     : applied on Tue Oct 29 05:45:31 UTC 2013
Unique Patch ID:  14825629
   Created on 14 May 2012, 16:17:31 hrs PST8PDT
   Bugs fixed:
     12646746



--------------------------------------------------------------------------------

OPatch succeeded.

Configure SSH Keys for RAC on Linux to establish user equivalance

1.Configure SSH on each node in the cluster. Log in as the "oracle" user and perform the following tasks on each RAC node.

[oracle@test4202 ~]$ mkdir ~/.ssh
[oracle@test4202 ~]$ ls -lart
total 28
-rw-r--r--  1 oracle oinstall  124 Oct  8 12:41 .bashrc
-rw-r--r--  1 oracle oinstall  176 Oct  8 12:41 .bash_profile
-rw-r--r--  1 oracle oinstall   33 Oct  8 12:41 .bash_logout
drwxr-xr-x 14 root   root     4096 Oct 24 08:55 ..
-rw-------  1 oracle oinstall 1736 Oct 24 09:05 .bash_history
drwxr-xr-x  2 oracle oinstall 4096 Oct 24 10:12 .ssh
drwx------  3 oracle oinstall 4096 Oct 24 10:12 .
[oracle@test4202 ~]$ chmod 700 ~/.ssh
[oracle@test4202 ~]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
2e:83:65:b5:c6:0d:11:b1:db:a6:a5:98:bd:85:6d:25 oracle@test4202
The key's randomart image is:
+--[ RSA 2048]----+
|        +o       |
|         o       |
|        +        |
|       o *       |
|      o S E .    |
|     + * B o     |
|    . = * +      |
|       o +       |
|        .        |
+-----------------+
[oracle@test4202 ~]$ cd /home/oracle/.ssh/
[oracle@test4202 .ssh]$ ls -lrt
total 8
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:13 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:13 id_rsa


Note:The RSA public key is written to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file.


2.Generate authorized_keys

Log in as "oracle" user on node1, generate an "authorized_keys" file on node1 and then copy it to node2 as below.

cd /home/oracle/.ssh/

[oracle@test4201 .ssh]$ ls -lrt
total 8
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:10 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:10 id_rsa
[oracle@test4201 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@test4201 .ssh]$ ls -lrt
total 12
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:10 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:10 id_rsa
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:11 authorized_keys
[oracle@test4201 .ssh]$ scp authorized_keys oracle@test4202:/home/oracle/.ssh/
Password:
authorized_keys                                                                                                                                          100%  398     0.4KB/s   0.4KB/s   00:00

Next, log in as the "oracle" user on node2 and perform as below.

[oracle@test4202 tmp]$ cd /home/oracle/.ssh/
[oracle@test4202 .ssh]$ ls -rlt
total 12
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:13 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:13 id_rsa
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:14 authorized_keys
[oracle@test4202 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0iKAmCxhxPJg+VLCHZuSzvix74Dw79ze5UJdx54hTfQ70Z/orYrTrOimSf0INSqBmqpiofTqu8W0Wu9acGtunsdJe++f2Ew58slF8svHCXaKyIMxBQheg7uCKEoDaaXbiZAD/30fGKAO5nL5IVZd9Pc7TVufutZbwGknbZxKwIXoFOUFDW3D0EkMiuFFiItWNjLoQvAx84hW5wJ2ImWc0mOSt2NDTk2O4pZ7plxeEYqr2Gy/bTwNba40r1ig7fwsRHh1j1t/cRYxY2s9IqlZeISynb6NaE4PmNHINypfuXb8f8kPt/g/GU8hsW8gbTBsnZPKDr+tdEDQjbnFk1eYjQ== oracle@test4201
[oracle@test4202 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@test4202 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0iKAmCxhxPJg+VLCHZuSzvix74Dw79ze5UJdx54hTfQ70Z/orYrTrOimSf0INSqBmqpiofTqu8W0Wu9acGtunsdJe++f2Ew58slF8svHCXaKyIMxBQheg7uCKEoDaaXbiZAD/30fGKAO5nL5IVZd9Pc7TVufutZbwGknbZxKwIXoFOUFDW3D0EkMiuFFiItWNjLoQvAx84hW5wJ2ImWc0mOSt2NDTk2O4pZ7plxeEYqr2Gy/bTwNba40r1ig7fwsRHh1j1t/cRYxY2s9IqlZeISynb6NaE4PmNHINypfuXb8f8kPt/g/GU8hsW8gbTBsnZPKDr+tdEDQjbnFk1eYjQ== oracle@test4201
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtdMGC7W1aGAyKuxR5HO2JsTprL3INqp4AGw5uBS3OyeA0pyrfqA+zB7MTxBTi7apJYqaGdVpTsmdawGQ46JQbfXu6Ob78KsQDHdJupzeSw5iBmCcsnzZ54LE/8Qv5FIh9R8HQH8Z/Yx69CfeRK8jPBPOPqdw9rHUhbpoy6u5PZhFgApEZyKM3IYqAU+ty/4lzeDAf/y+uBwK5kSRWzs5fAoSrWjzhyrzNERqN22pN8+OyYm295IPi2cDRb7NnInZdqfrTzWHpwzqrJL/88GTFWkgpC8oJp9sVl/v5KAJtnQCaXpyOsZraNwcZxyO07tIJLFcW0PPNMyk46vegO4tmw== oracle@test4202



[oracle@test4202 .ssh]$ scp authorized_keys oracle@test4201:/home/oracle/.ssh/
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Password:
authorized_keys                                                                                                                                          100%  796     0.8KB/s   0.8KB/s   00:00


3.Test the setup as below

Node1
[oracle@test4201 .ssh]$ ssh test4201 date
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Thu Oct 24 10:17:15 PDT 2013
[oracle@test4201 .ssh]$ ssh test4202 date
Thu Oct 24 10:17:29 PDT 2013
[oracle@test4201 .ssh]$


Node2
[oracle@test4201 .ssh]$ ssh test4201 date
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Thu Oct 24 10:17:15 PDT 2013
[oracle@test4201 .ssh]$ ssh test4202 date
Thu Oct 24 10:17:29 PDT 2013
[oracle@test4201 .ssh]$

Note:  You can also run the sshUserSetup.sh script available at /grid/sshsetup to setup user equivalance

RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

I got error as below while doing resync of the database with catalog.

connected to target database: TESTDB (DBID=2269473865)

connected to recovery catalog database

PL/SQL package RMAN_UINLUPP.DBMS_RCVCAT version 11.02.00.02 in RCVCAT database is not current

PL/SQL package RMAN_UINLUPP.DBMS_RCVMAN version 11.02.00.02 in RCVCAT database is not current

RMAN> RUN

2> {

3> RESYNC CATALOG;

4> REPORT SCHEMA;

5> }

6> LIST BACKUP SUMMARY;

7>

8>

DBGSQL:     RCVCAT> begin dbms_rcvman.dumpPkgState('RCVMAN after sqlerror'); end;

DBGSQL:        sqlcode = 6550

DBGSQL:     RCVCAT> begin dbms_rcvcat.dumpPkgState('RCVCAT after sqlerror');end;

DBGSQL:        sqlcode = 6550

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of resync command at 10/18/2013 02:01:30

RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

Recovery Manager complete.

Reason:Database was not registered in the catalog. Register the target database and check it.

[oracle@uiddx4101 log]$ rman target / catalog rman_cat/rman_cat@rman

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Oct 18 13:25:06 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TESTDB (DBID=2269473865)

connected to recovery catalog database

RMAN> RESYNC CATALOG;

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of resync command at 10/18/2013 13:25:12

RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

RMAN> register database;

database registered in recovery catalog

starting full resync of recovery catalog

full resync complete

RMAN>

RMAN>

RMAN> RESYNC CATALOG;

starting full resync of recovery catalog

full resync complete

RMAN> exit

ORA-14223: Deferred segment creation is not supported for this table

While creating table, we got error as below.

SQL> create table test3(id number, name varchar2(200)) SEGMENT CREATION DEFERRED;
create table test3(id number, name varchar2(200)) SEGMENT CREATION DEFERRED
*
ERROR at line 1:
ORA-14223: Deferred segment creation is not supported for this table

Reason:You are trying to create table in SYSTEM tablespace

Eg:

SQL> sho parameter DEFERRED_SEGMENT_CREATION;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

deferred_segment_creation            boolean     TRUE

SQL> sho user;

USER is "SYS"

SQL> create table test1(id number, name varchar2(200)) SEGMENT CREATION DEFERRED;

create table test1(id number, name varchar2(200)) SEGMENT CREATION DEFERRED

*

ERROR at line 1:

ORA-14223: Deferred segment creation is not supported for this table

table create with deferred segment failed because it's being created in system tablespace. Now we will create as below.

SQL> create table test1(id number, name varchar2(200));

Table created.

SQL> select segment_created from dba_tables where table_name='TEST1';

SEG

---

YES

SQL> select count(*) from user_segments where segment_name='TEST1';

  COUNT(*)

----------

        1

Now table got created and also initial segment got allocated though deferred_segment_creation is set to TRUE(Default)

Now Let us try to create table table in another user who belongs USERS tablespace.

SQL> conn a/a

Connected.

SQL>  create table test2(id number, name varchar2(200));

Table created.

SQL> select count(*) from user_segments where segment_name='TEST2';

  COUNT(*)

----------

         0

SQL> select segment_created from user_tables where table_name='TEST2';

SEG

---

NO

SQL> select count(*) from user_extents where segment_name='TEST2';

  COUNT(*)

----------

         0

SQL>

So table got created with deferred segment.

Restrictions on Deferred Segment Creation:

You cannot defer segment creation for the following types of tables: index-organized tables, clustered tables, global temporary tables, session-specific temporary tables, internal tables, typed tables, AQ tables, external tables, and tables owned by SYS, SYSTEM, PUBLIC, OUTLN, or XDB.

Deferred segment creation is supported on partitions and subpartitions beginning with Oracle Database 11g Release 2 (11.2.0.2).

Deferred segment creation is not supported for bitmap join indexes and domain indexes.

Deferred segment creation is not supported in dictionary-managed tablespaces.

Deferred segment creation is not supported in the SYSTEM tablespace.

Serializable transactions do not work with deferred segment creation. Trying to insert data into an empty table with no segment created causes an error.

;

How to modify SCAN name in RAC 11gR2.

Recently we encountered an issue where we were unable to make connection to database and after analysis found that SCAN name has got changed in the database configuration.  To resolve the issue I had to update with correct SCAN name and below is the procedure that was followed.

1. Login with GRID owner(oracle) and check current status as below.
[root@acldx0041 ~]# su - oracle
[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl config scan
SCAN name: scan-idxa-test-stg, Network: 1/144.23.173.128/255.255.255.224/bondeth0
SCAN VIP name: scan1, IP: /scan-idxa-test-stg/140.21.173.148
SCAN VIP name: scan2, IP: /scan-idxa-test-stg/140.21.173.146
SCAN VIP name: scan3, IP: /scan-idxa-test-stg/140.21.173.147
[oracle@acldx0041 ~]$ nslookup scan-idxa-test-stg
Server:         10.221.45.71
Address:        10.221.45.71#53

** server can't find scan-idxa-test-stg: NXDOMAIN

What I find is SCAN name scan-idxa-test-stg is not resolving to any VIP in DNS. When I checked the SCAN VIP(Eg: 140.21.173.148), it was resolving to hostname scan-idxa-test-ha.dba.com which is correct SCAN Name.

oracle@acldx0041 ~]$ nslookup 140.21.173.148
Server:         10.221.45.71
Address:        10.221.45.71#53

Non-authoritative answer:
148.173.20.144.in-addr.arpa     name = scan-idxa-test-ha.dba.com.

Authoritative answers can be found from:
173.20.144.in-addr.arpa nameserver = ns1.dba.com.
173.20.144.in-addr.arpa nameserver = ns4.dba.com.
ns1.dba.com  internet address = 148.80.1.20
ns4.dba.com  internet address = 148.80.112.100

[oracle@acldx0041 ~]$ nslookup scan-idxa-test-ha.dba.com.
Server:         10.221.45.71
Address:        10.221.45.71#53

Non-authoritative answer:
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.148
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.146
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.147

2. So now I have to update with correct SCAN name in OCR and is done as below.

Stop scan and scan listeners with GRID owner(oracle) as below.
[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node acldx0041
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node acldx0042
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node acldx0041
[oracle@acldx0041 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node acldx0041
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node acldx0042
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node acldx0041
[oracle@acldx0041 ~]$ srvctl stop scan_listener
[oracle@acldx0041 ~]$ srvctl stop scan
[oracle@acldx0041 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is not running
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running
[oracle@acldx0041 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is not running
SCAN VIP scan2 is enabled
SCAN VIP scan2 is not running
SCAN VIP scan3 is enabled
SCAN VIP scan3 is not running

3.Once this is done, now we will modify configurations to point to correct scan name. This has to be done with root user.

[root@acldx0041 ~]# export GRID_HOME=/u01/app/11.2.0.3/grid
[root@acldx0041 ~]# export PATH=$GRID_HOME/bin:$PATH
[root@acldx0041 ~]# srvctl modify scan -n scan-idxa-test-ha.dba.com


4.Now connect with GRID owner r and start SCAN and SCAN listener.

[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl modify scan_listener -u
[oracle@acldx0041 ~]$ srvctl start scan_listener
[oracle@acldx0041 ~]$ srvctl config scan
SCAN name: scan-idxa-test-ha.dba.com, Network: 1/144.23.173.128/255.255.255.224/bondeth0
SCAN VIP name: scan1, IP: /scan-idxa-test-ha.dba.com/140.21.173.146
SCAN VIP name: scan2, IP: /scan-idxa-test-ha.dba.com/140.21.173.147
SCAN VIP name: scan3, IP: /scan-idxa-test-ha.dba.com/140.21.173.148
[oracle@acldx0041 ~]$ tnsping scan-idxa-test-ha.dba.com

TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 21-AUG-2013 07:25:46

Copyright (c) 1997, 2011, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0.3/dbhome_1/network/admin/sqlnet.ora

Used EZCONNECT adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.148)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.146)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.147)(PORT=1521)))
OK (0 msec)

and update remote_listener to point to correct SCAN name.

SQL> sho parameter remote_listener;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      scan-idxa-test-ha:1521



Please refer Doc ID 952903.1 for more details.

RMAN RESTORE DATABASE command fails with RMAN-06023

RMAN  RESTORE DATABASE command failed with RMAN-06023 as below.

RMAN> run {
2> SET NEWNAME FOR DATABASE   TO  '/fsnadmin/sangamesh/spshidms/database/%b';
3> SET NEWNAME FOR tempfile  1 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
4> SET NEWNAME FOR tempfile  2 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
5> SET NEWNAME FOR tempfile  3 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
6> restore database from tag 'ROLLF_COPY';
7> switch datafile all;
8> switch tempfile all;
9> }


executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 14-AUG-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=883 device type=DISK

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/14/2013 07:36:42
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 12 found to restore
RMAN-06023: no backup or copy of datafile 11 found to restore
RMAN-06023: no backup or copy of datafile 10 found to restore
RMAN-06023: no backup or copy of datafile 9 found to restore
RMAN-06023: no backup or copy of datafile 8 found to restore
RMAN-06023: no backup or copy of datafile 7 found to restore
RMAN-06023: no backup or copy of datafile 6 found to restore
RMAN-06023: no backup or copy of datafile 5 found to restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 3 found to restore
RMAN-06023: no backup or copy of datafile 2 found to restore.
RMAN-06023: no backup or copy of datafile 1 found to restore

RMAN>

Solution:
Disable FRA during recovery process commenting db_recovery_file_dest and db_recovery_file_dest_size parameters in parameter file. Refer Doc ID 965122.1 for more details.

Error while performing RMAN restoration

We got below error while restoring database from backup(ASM) to file system

RMAN> run {
2> SET NEWNAME FOR DATABASE   TO  '/fsnadmin/sangamesh/spshidms/database/%b';
3> SET NEWNAME FOR tempfile  1 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
4> SET NEWNAME FOR tempfile  2 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
5> SET NEWNAME FOR tempfile  3 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
6> restore database;
7> switch datafile all;
8> switch tempfile all;
9> }

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 14-AUG-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=758 device type=DISK

channel ORA_DISK_1: restoring datafile 00001
input datafile copy RECID=1349 STAMP=823381234 file name=+RECO_ACLO1/spshidmp/datafile/system.2090.792669571
destination for restore of datafile 00001: /fsnadmin/sangamesh/spshidms/database/system.307.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/system.2090.792669571"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/system.2090.792669571
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1349 (+RECO_ACLO1/spshidmp/datafile/system.2090.792669571
channel ORA_DISK_1: restoring datafile 00002
input datafile copy RECID=1355 STAMP=823381244 file name=+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
destination for restore of datafile 00002: /fsnadmin/sangamesh/spshidms/database/sysaux.306.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1355 (+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
channel ORA_DISK_1: restoring datafile 00003
input datafile copy RECID=1354 STAMP=823381243 file name=+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575
destination for restore of datafile 00003: /fsnadmin/sangamesh/spshidms/database/undotbs1.305.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1354 (+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575

Reason:
Parameters db_file_name_convert and db_create_online_log_dest_1 are not set. So set these parameters and try.
Eg:
*.db_file_name_convert='+RECO_ACLO1/prod/datafile/','/fsnadmin/sangamesh/test/database/'

*.db_create_online_log_dest_1=’/fsnadmin/sangamesh/test/database/

Taking cold backup for database running on ASM using RMAN in EXADATA

Follow the below steps to take cold back of database running on ASM.

1.Start the the instances in all the nodes in mount stage.


SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.7504E+10 bytes
Fixed Size                  2235448 bytes
Variable Size            4429185992 bytes
Database Buffers         1.3019E+10 bytes
Redo Buffers               53477376 bytes
Database mounted.
SQL>


2.Connect rman from any node and take the backup as below.

[oracle@acldx0041 12Aug]$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Aug 14 05:53:43 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: SPSHIDMP (DBID=515650406, not open)

RMAN> backup device type disk format '/fsnadmin/test/rman_backup_43/backup/%U' database plus archivelog;


Starting backup at 14-AUG-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1522 instance=spshidmp1 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=55 RECID=107 STAMP=797410838
input archived log thread=1 sequence=58 RECID=106 STAMP=797407220
input archived log thread=1 sequence=59 RECID=108 STAMP=797439625
input archived log thread=2 sequence=56 RECID=109 STAMP=797455353
input archived log thread=1 sequence=60 RECID=112 STAMP=797464816

3.Once done, start the instances on all the nodes.

SQL> alter database open;

Database altered.

Authentication failed with error as "Invalid sign in" when tried to login oim console

When we tried to login to oimconsole with xelsysadm user, it failed and gave error as "Invalid sign in".

When we checked in oim managed server log, we found below message.
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: oracle.ucp.UniversalConnectionPoolException: Error during pool creation in Universal Connection Pool Manager MBean: oracle.ucp.UniversalConnectionPoolException: Error during pool creation in Universal Connection Pool Manager: oracle.ucp.UniversalConnectionPoolException: Universal Connection Pool already exists in the Universal Connection Pool Manager. Universal Connection Pool cannot be added to the Universal Connection Pool Manager
        at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:368)
        at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:49)
        at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:80)
        at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.createConnectionPool(UniversalConnectionPoolManagerMBeanImpl.java:316)
        at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:631)
        ... 117 more
Exception encountered when trying to login as admin {0}

javax.security.auth.login.LoginException: javax.security.auth.login.LoginException: java.lang.SecurityException: [Security:090304]Authentication Failed: User oiminternal javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User oiminternal denied
        at weblogic.security.auth.login.UsernamePasswordLoginModule.login(UsernamePasswordLoginModule.java:199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:684)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at Thor.API.Security.LoginHandler.weblogicLoginHandler.login(weblogicLoginHandler.java:62)
        at oracle.iam.platform.OIMClient.login(OIMClient.java:134)
        at oracle.iam.platform.OIMClient.login(OIMClient.java:114)
        at oracle.iam.platform.OIMInternalClient.loginAsOIMInternal(OIMInternalClient.java:102)
        at oracle.iam.scheduler.impl.util.SchedulerUtil.getSchedulerService(SchedulerUtil.java:841)


Reason:It seems that it's unable to get password for xelsysadm user from database.

Solution: Go to weblogic console and then

1.  myrealm->providers-> OIMAuthenticationProvider-> provider specific
2. Click lock and edit and update the correct OIM schema password and save it.
3. Once done, restart all Admin and managed servers

Error while starting Admin server using node manager

We got below error while starting Admin server using node manager

wls:/nm/base_domain>
wls:/nm/base_domain>
wls:/nm/base_domain> nmStart('AdminServer')
Starting server AdminServer ...
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server 'AdminServer'
wls:/nm/base_domain>



Reason:
Domain details are not updated in $MW_HOME/wlserver_10.3/common/nodemanager/nodemanager.domains

So update domain name in nodemanager.domains file and start it using nodemanager.

Format => DOMAIN_NAME = DOMAIN_PATH

Eg:
OAMDomain=/u01/app/domains/OAMDomain

OWB Control Center Service Fails to Start after OWB cloning.

We cloned 10.2.0.2 database and OWB. When we tried to start the control center service after host name change in WB_RT_SERVICE_NODES(present in  repository owner schema) table, we got error as below

SQL> @start_service.sql
Not available
Diagnostics:
service startup failure using command "/orasw/app/oracle/product/owb10201/owb/b
in/unix/run_service.sh -manual 1 /orasw/app/oracle/product/owb10201 BIREPOWN sl
c04kvm.us.oracle.com 1521 BI221SF1" reason ORA-29532: Java call terminated by u
ncaught Java exception: java.io.IOException: service early exit: code=0 : err=
: out=2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - OCI fix performed

2013/03/24-23:30:44-PDT [1AC1FE4] Thin driver connection time - 126 millisecond
(s)
2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - property user.timezo
ne value is PST8PDT
2013/03/24-23:30:44-PDT [1AC1FE4] Connection Manager - conn
ection.timezone not set. Defaulting to value SERVICE
2013/03/24-23:30:44-PDT [1

PL/SQL procedure successfully completed.

Reason:

The OS open file limit for the Oracle user is set either set to unlimited or less value. In our case it was

-bash-3.2$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 124064
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 124064
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Solution:

1.Change open files limit to 65536 in /etc/security/limits.conf for oracle user  with root as below

oracle     soft   nofile    131072
oracle    hard   nofile    65536

2. Login with new session and start the service.

-bash-3.2$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Mon Mar 25 00:48:26 2013

Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> conn birepown/birepown
Connected.
SQL> @stop_service.sql
Not Available

PL/SQL procedure successfully completed.

SQL> @start_service.sql
Available

PL/SQL procedure successfully completed.

SQL> @show_service.sql
Available

PL/SQL procedure successfully completed.

SQL> exit

Using network link to import data in Oracle

Using network link to import data in Oracle

1.Create database link ,in destination database, pointing to source database from where you want import the data as below.

In our case:
Source Schema =dwadm
Destination Schema=zz_dwadm

SQL> conn zz_dwadm/zz_dwadm
Connected.
SQL> create database link QA1DB_TO_DEVDB connect to dwadm identified by dwadm using 'QA1DB';

Database link created.

SQL> exit
Disconnected from Oracle

2.Import the table TEST_TABLE from dwadm into zz_dwadm using below command

impdp directory=dump_dir network_link=QA1DB_TO_DEVDB tables=dwadm.TEST_TABLE logfile=impdp.log remap_schema=dwadm:zz_dwadm


Import: Release 11.2.0.1.0 - Production on Wed Mar 13 09:40:30 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Username: zz_dwadm/zz_dwadm

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Error while connecting to OWB control center in RAC or EXADATA

We got below error while connecting to OWB repository  in EXADATA, from any node other than node from which we installed repository

2013/02/06:02:46    OMB01118: Could not connect to repository! ORA-29540: class oracle/wh/runtime/server/KeyManagement does not exist
ORA-06512: at "OWBSYS.GETROLESTARTED", line 31
ORA-06512: at line 1

2013/02/06:02:46    Unable to connect to repository.

Reason: 

You OWB repository is not configured properly. Probably, you might have missed to register other nodes in OWB while configuring repository. 

Solution: 

Refer the Doc ID 459961.1 for further details.

To cleanup OWB repository and creating OWBSYS Schema with Repository Objects

To cleanup OWB repository

Go to $ORACLE_HOME/owb/UnifiedRepos


cd /u01/app/oracle/product/11.2.0.3/dbhome_1/owb/UnifiedRepos

[oracle@slcm02db01 UnifiedRepos]$ ls -lrt clean_owbsys.sql
-rw-r--r-- 1 oracle oinstall 163 Sep 26  2008 clean_owbsys.sql

2.Connect to owbsys and Stop Oracle Warehouse Builder Control Center service.
SQL> conn owbsys/owbsys
connected
SQL> @?/owb/rtp/sql/stop_service.sql


3.Connect to database and execute the clean_owbsys.sql script to drop all the objects in the existing OWBSYSas

[oracle@slcm02db01 UnifiedRepos]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Mon Feb 4 00:36:03 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @clean_owbsys.sql
User dropped.


User dropped.


Role dropped.


Role dropped.


Role dropped.

SQL>

Creating an OWBSYS Schema with Repository Objects

1.connect to the database as sysdba and run the cat_owb.sql script to set up OWBSYS in a tablespace in your database.
For example, this command sets up OWBSYS in the USERS tablespace

SQL> @?/owb/UnifiedRepos/cat_owb.sql USERS
Enter Tablespace Name for OWBSYS user:

Package created.


Package body created.

Create user OWBSYS with default tablespace USERS ...
old   1: CREATE USER OWBSYS IDENTIFIED BY OWBSYS DEFAULT TABLESPACE &owbsys_tbs
new   1: CREATE USER OWBSYS IDENTIFIED BY OWBSYS DEFAULT TABLESPACE USERS

User created.

User OWBSYS has been created.

PL/SQL procedure successfully completed.


Grant succeeded.

Configuring the Control Center Service

If you are installing OWB 11.2 on an older Oracle Database release, or if you
integrating OWB with Oracle Business Intelligence Standard Edition , you must run the reset_owbcc_home.sql script to ensure that the
Control Center Service functions correctly. Steps are as below.

1.Connect to the database as sysdba and run the reset_owbcc_home.sql script to force the Control Center Service to
run from Oracle Warehouse Builder 11.2 installation.

SQL> @?/owb/UnifiedRepos/reset_owbcc_home.sql
Enter the full path of the Oracle home for the OWB Control Center install.
If you are installing in a Windows environment, please ensure that the case of the path exactly matches the Oracle install path including the drive letter.

OWB Control Center Home:
/u01/app/oracle/product/11.2.0.3/dbhome_1/owb

PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.

SQL>

Unlocking the OWBSYS and OWBSYS_AUDIT Accounts

connect to the database as sysdba and execute below commands.

alter user owbsys identified by password account unlock;
alter user owbsys_audit identified by password account unlock;

Setting the OWB Home for Remote Installations
==============================================
You must create a database directory OWB_REMOTE_ADMIN that is accessed by remote
installations, so they can read the server's OWB_HOME/owb/bin/admin directory.

Connect to the database as sysdba and set the OWB_REMOTE_HOME variable using script remote_owb_install.sql as below

SQL> @?/owb/UnifiedRepos/remote_owb_install.sql
Oracle Home for the Remote OWB software installation
/u01/app/oracle/product/11.2.0.3/dbhome_1

Directory created.


Grant succeeded.


Grant succeeded.

SQL>

Creating the First Workspace in the Repository

To use a repository, you must define an initial workspace, and the owner of that
workspace. The following instructions use the Repository Assistant.

To create a workspace:
1. At the command prompt, navigate to the UNIX bin directory:
cd $ORACLE_HOME/owb/bin/unix

2. Start the repository assistant.
 ./reposinst.sh

3. On the Welcome screen of the Repository assistant, click Next and follow the instructions.

OBIEE 11g startup issue :fails to start system components.

While starting system components using opmnctl, it fails with below error.

$ ./opmnctl startall
opmnctl startall: starting opmn and all managed processes...
/orasw/MW_HOME/Oracle_BI1/opmn/bin/opmn: unexpected exit: code 512
opmnctl startall: opmn failed to start.

Reason

Check OPMN logs to find the issue.

cd $INSTANCE_HOME/diagnostics/logs/OPMN/opmn

$ ls -lrt

total 10072

-rw-r----- 1 oracle oinstall       0 Sep 27  2011 debug.log

-rw-r----- 1 oracle oinstall     432 Jan 13 07:06 opmn.out

-rw-r----- 1 oracle oinstall  257891 Jan 22 00:32 logquery~5.log

-rw-r----- 1 oracle oinstall  269575 Jan 22 00:32 logquery~4.log

-rw-r----- 1 oracle oinstall  273531 Jan 22 00:32 logquery~3.log

-rw-r----- 1 oracle oinstall  279896 Jan 22 00:32 logquery~2.log

-rw-r----- 1 oracle oinstall  290644 Jan 22 00:32 logquery~1.log

-rw-r----- 1 oracle oinstall 8894259 Feb  5 00:41 opmn.log

In the log we found that OPMN Local Port and Remote ports are already in use by some other process.

[2013-02-04T04:48:36][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-04T04:49:50][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-04T04:49:50][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][90][OPMN][code:ons-internal]ONS server initiated

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][522][OPMN][code:pm-internal]PM state directory exists: /orasw/MW_HOME/instances/instance1/config/OPMN/opmn/states

[2013-02-05T00:41:07][opmn][NOTIFICATION:1][675][OPMN][code:pm-internal]OPMN server ready.  Request handling enabled.

[2013-02-05T00:41:07][opmn][ERROR:1][17][OPMN][code:ons-listener]any: BIND (Address already in use)

Solution:

1. Go to $INSTANCE_HOME/config/OPMN//opmn.xml and check the ports.

In our case it was 9500 and 9501 ports

2. Check who is using these ports using below command.

$ netstat -an |grep 9500

tcp        0      0 ::ffff:10.241.37.9:9500     :::*                        LISTEN

tcp        0      0 ::ffff:10.241.37.9:9500     ::ffff:10.241.37.9:60741    ESTABLISHED

tcp        0      0 ::ffff:10.241.37.9:60741    ::ffff:10.241.37.9:9500     ESTABLISHED

$ netstat -an |grep 9501

tcp        0      0 :::9501                     :::*                        LISTEN

So both ports are being used.

3. Either terminate processes using these ports or change ports in opmn.xml with ports not in use.

In our case, 9502 and 9503 weren't in use so replaced with these ports in opmn.xml.

4.Start system components now.

cd $INSTANCE_HOME/bin

$ ./opmnctl startall

opmnctl startall: starting opmn and all managed processes...

$ ./opmnctl status

Processes in Instance: instance1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

coreapplication_obiccs1          | OracleBIClusterCo~ |   31392 | Alive

coreapplication_obisch1          | OracleBIScheduler~ |   31394 | Alive

coreapplication_obijh1           | OracleBIJavaHostC~ |   31393 | Alive

coreapplication_obips1           | OracleBIPresentat~ |   31391 | Alive

coreapplication_obis1            | OracleBIServerCom~ |   31395 | Alive