How to modify SCAN name in RAC 11gR2.

Recently we encountered an issue where we were unable to make connection to database and after analysis found that SCAN name has got changed in the database configuration.  To resolve the issue I had to update with correct SCAN name and below is the procedure that was followed.

1. Login with GRID owner(oracle) and check current status as below.
[root@acldx0041 ~]# su - oracle
[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl config scan
SCAN name: scan-idxa-test-stg, Network: 1/144.23.173.128/255.255.255.224/bondeth0
SCAN VIP name: scan1, IP: /scan-idxa-test-stg/140.21.173.148
SCAN VIP name: scan2, IP: /scan-idxa-test-stg/140.21.173.146
SCAN VIP name: scan3, IP: /scan-idxa-test-stg/140.21.173.147
[oracle@acldx0041 ~]$ nslookup scan-idxa-test-stg
Server:         10.221.45.71
Address:        10.221.45.71#53

** server can't find scan-idxa-test-stg: NXDOMAIN

What I find is SCAN name scan-idxa-test-stg is not resolving to any VIP in DNS. When I checked the SCAN VIP(Eg: 140.21.173.148), it was resolving to hostname scan-idxa-test-ha.dba.com which is correct SCAN Name.

oracle@acldx0041 ~]$ nslookup 140.21.173.148
Server:         10.221.45.71
Address:        10.221.45.71#53

Non-authoritative answer:
148.173.20.144.in-addr.arpa     name = scan-idxa-test-ha.dba.com.

Authoritative answers can be found from:
173.20.144.in-addr.arpa nameserver = ns1.dba.com.
173.20.144.in-addr.arpa nameserver = ns4.dba.com.
ns1.dba.com  internet address = 148.80.1.20
ns4.dba.com  internet address = 148.80.112.100

[oracle@acldx0041 ~]$ nslookup scan-idxa-test-ha.dba.com.
Server:         10.221.45.71
Address:        10.221.45.71#53

Non-authoritative answer:
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.148
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.146
Name:   scan-idxa-test-ha.dba.com
Address: 140.21.173.147

2. So now I have to update with correct SCAN name in OCR and is done as below.

Stop scan and scan listeners with GRID owner(oracle) as below.
[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node acldx0041
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node acldx0042
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node acldx0041
[oracle@acldx0041 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node acldx0041
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node acldx0042
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node acldx0041
[oracle@acldx0041 ~]$ srvctl stop scan_listener
[oracle@acldx0041 ~]$ srvctl stop scan
[oracle@acldx0041 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is not running
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running
[oracle@acldx0041 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is not running
SCAN VIP scan2 is enabled
SCAN VIP scan2 is not running
SCAN VIP scan3 is enabled
SCAN VIP scan3 is not running

3.Once this is done, now we will modify configurations to point to correct scan name. This has to be done with root user.

[root@acldx0041 ~]# export GRID_HOME=/u01/app/11.2.0.3/grid
[root@acldx0041 ~]# export PATH=$GRID_HOME/bin:$PATH
[root@acldx0041 ~]# srvctl modify scan -n scan-idxa-test-ha.dba.com


4.Now connect with GRID owner r and start SCAN and SCAN listener.

[oracle@acldx0041 ~]$ export GRID_HOME=/u01/app/11.2.0.3/grid
[oracle@acldx0041 ~]$ export PATH=$GRID_HOME/bin:$PATH
[oracle@acldx0041 ~]$ srvctl modify scan_listener -u
[oracle@acldx0041 ~]$ srvctl start scan_listener
[oracle@acldx0041 ~]$ srvctl config scan
SCAN name: scan-idxa-test-ha.dba.com, Network: 1/144.23.173.128/255.255.255.224/bondeth0
SCAN VIP name: scan1, IP: /scan-idxa-test-ha.dba.com/140.21.173.146
SCAN VIP name: scan2, IP: /scan-idxa-test-ha.dba.com/140.21.173.147
SCAN VIP name: scan3, IP: /scan-idxa-test-ha.dba.com/140.21.173.148
[oracle@acldx0041 ~]$ tnsping scan-idxa-test-ha.dba.com

TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 21-AUG-2013 07:25:46

Copyright (c) 1997, 2011, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0.3/dbhome_1/network/admin/sqlnet.ora

Used EZCONNECT adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.148)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.146)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=140.21.173.147)(PORT=1521)))
OK (0 msec)

and update remote_listener to point to correct SCAN name.

SQL> sho parameter remote_listener;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      scan-idxa-test-ha:1521



Please refer Doc ID 952903.1 for more details.

RMAN RESTORE DATABASE command fails with RMAN-06023

RMAN  RESTORE DATABASE command failed with RMAN-06023 as below.

RMAN> run {
2> SET NEWNAME FOR DATABASE   TO  '/fsnadmin/sangamesh/spshidms/database/%b';
3> SET NEWNAME FOR tempfile  1 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
4> SET NEWNAME FOR tempfile  2 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
5> SET NEWNAME FOR tempfile  3 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
6> restore database from tag 'ROLLF_COPY';
7> switch datafile all;
8> switch tempfile all;
9> }


executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 14-AUG-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=883 device type=DISK

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/14/2013 07:36:42
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 12 found to restore
RMAN-06023: no backup or copy of datafile 11 found to restore
RMAN-06023: no backup or copy of datafile 10 found to restore
RMAN-06023: no backup or copy of datafile 9 found to restore
RMAN-06023: no backup or copy of datafile 8 found to restore
RMAN-06023: no backup or copy of datafile 7 found to restore
RMAN-06023: no backup or copy of datafile 6 found to restore
RMAN-06023: no backup or copy of datafile 5 found to restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 3 found to restore
RMAN-06023: no backup or copy of datafile 2 found to restore.
RMAN-06023: no backup or copy of datafile 1 found to restore

RMAN>

Solution:
Disable FRA during recovery process commenting db_recovery_file_dest and db_recovery_file_dest_size parameters in parameter file. Refer Doc ID 965122.1 for more details.

Error while performing RMAN restoration

We got below error while restoring database from backup(ASM) to file system

RMAN> run {
2> SET NEWNAME FOR DATABASE   TO  '/fsnadmin/sangamesh/spshidms/database/%b';
3> SET NEWNAME FOR tempfile  1 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
4> SET NEWNAME FOR tempfile  2 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
5> SET NEWNAME FOR tempfile  3 TO  '/fsnadmin/sangamesh/spshidms/database/%b';
6> restore database;
7> switch datafile all;
8> switch tempfile all;
9> }

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 14-AUG-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=758 device type=DISK

channel ORA_DISK_1: restoring datafile 00001
input datafile copy RECID=1349 STAMP=823381234 file name=+RECO_ACLO1/spshidmp/datafile/system.2090.792669571
destination for restore of datafile 00001: /fsnadmin/sangamesh/spshidms/database/system.307.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/system.2090.792669571"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/system.2090.792669571
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1349 (+RECO_ACLO1/spshidmp/datafile/system.2090.792669571
channel ORA_DISK_1: restoring datafile 00002
input datafile copy RECID=1355 STAMP=823381244 file name=+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
destination for restore of datafile 00002: /fsnadmin/sangamesh/spshidms/database/sysaux.306.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1355 (+RECO_ACLO1/spshidmp/datafile/sysaux.2091.792669571
channel ORA_DISK_1: restoring datafile 00003
input datafile copy RECID=1354 STAMP=823381243 file name=+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575
destination for restore of datafile 00003: /fsnadmin/sangamesh/spshidms/database/undotbs1.305.790709109
ORA-19505: failed to identify file "+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575"
ORA-17503: ksfdopn:2 Failed to open file +RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575
ORA-15001: diskgroup "RECO_ACLO1" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-19600: input file is datafile-copy 1354 (+RECO_ACLO1/spshidmp/datafile/undotbs1.2036.792669575

Reason:
Parameters db_file_name_convert and db_create_online_log_dest_1 are not set. So set these parameters and try.
Eg:
*.db_file_name_convert='+RECO_ACLO1/prod/datafile/','/fsnadmin/sangamesh/test/database/'

*.db_create_online_log_dest_1=’/fsnadmin/sangamesh/test/database/

Taking cold backup for database running on ASM using RMAN in EXADATA

Follow the below steps to take cold back of database running on ASM.

1.Start the the instances in all the nodes in mount stage.


SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.7504E+10 bytes
Fixed Size                  2235448 bytes
Variable Size            4429185992 bytes
Database Buffers         1.3019E+10 bytes
Redo Buffers               53477376 bytes
Database mounted.
SQL>


2.Connect rman from any node and take the backup as below.

[oracle@acldx0041 12Aug]$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Aug 14 05:53:43 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: SPSHIDMP (DBID=515650406, not open)

RMAN> backup device type disk format '/fsnadmin/test/rman_backup_43/backup/%U' database plus archivelog;


Starting backup at 14-AUG-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1522 instance=spshidmp1 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=55 RECID=107 STAMP=797410838
input archived log thread=1 sequence=58 RECID=106 STAMP=797407220
input archived log thread=1 sequence=59 RECID=108 STAMP=797439625
input archived log thread=2 sequence=56 RECID=109 STAMP=797455353
input archived log thread=1 sequence=60 RECID=112 STAMP=797464816

3.Once done, start the instances on all the nodes.

SQL> alter database open;

Database altered.

Authentication failed with error as "Invalid sign in" when tried to login oim console

When we tried to login to oimconsole with xelsysadm user, it failed and gave error as "Invalid sign in".

When we checked in oim managed server log, we found below message.
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: oracle.ucp.UniversalConnectionPoolException: Error during pool creation in Universal Connection Pool Manager MBean: oracle.ucp.UniversalConnectionPoolException: Error during pool creation in Universal Connection Pool Manager: oracle.ucp.UniversalConnectionPoolException: Universal Connection Pool already exists in the Universal Connection Pool Manager. Universal Connection Pool cannot be added to the Universal Connection Pool Manager
        at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:368)
        at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:49)
        at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:80)
        at oracle.ucp.admin.UniversalConnectionPoolManagerMBeanImpl.createConnectionPool(UniversalConnectionPoolManagerMBeanImpl.java:316)
        at oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:631)
        ... 117 more
Exception encountered when trying to login as admin {0}

javax.security.auth.login.LoginException: javax.security.auth.login.LoginException: java.lang.SecurityException: [Security:090304]Authentication Failed: User oiminternal javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User oiminternal denied
        at weblogic.security.auth.login.UsernamePasswordLoginModule.login(UsernamePasswordLoginModule.java:199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:684)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at Thor.API.Security.LoginHandler.weblogicLoginHandler.login(weblogicLoginHandler.java:62)
        at oracle.iam.platform.OIMClient.login(OIMClient.java:134)
        at oracle.iam.platform.OIMClient.login(OIMClient.java:114)
        at oracle.iam.platform.OIMInternalClient.loginAsOIMInternal(OIMInternalClient.java:102)
        at oracle.iam.scheduler.impl.util.SchedulerUtil.getSchedulerService(SchedulerUtil.java:841)


Reason:It seems that it's unable to get password for xelsysadm user from database.

Solution: Go to weblogic console and then

1.  myrealm->providers-> OIMAuthenticationProvider-> provider specific
2. Click lock and edit and update the correct OIM schema password and save it.
3. Once done, restart all Admin and managed servers

Error while starting Admin server using node manager

We got below error while starting Admin server using node manager

wls:/nm/base_domain>
wls:/nm/base_domain>
wls:/nm/base_domain> nmStart('AdminServer')
Starting server AdminServer ...
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server 'AdminServer'
wls:/nm/base_domain>



Reason:
Domain details are not updated in $MW_HOME/wlserver_10.3/common/nodemanager/nodemanager.domains

So update domain name in nodemanager.domains file and start it using nodemanager.

Format => DOMAIN_NAME = DOMAIN_PATH

Eg:
OAMDomain=/u01/app/domains/OAMDomain