CRS-0215: Could not start resource 'ora..vip' [ID 356535.1]

Oracle Server - Enterprise Edition - Version: 10.1.0.4 to 11.1.0.7
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 10.1.0.4 to 11.1.0.7

Symptoms

The command : crs_stat -t output shows VIP is offline and trying to start it gives error :
CRS-0215: Could not start resource 'ora.dbtest2.vip'.

Example: crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....st2.gsd application ONLINE ONLINE dbtest2
ora....st2.ons application ONLINE ONLINE dbtest2
ora....st2.vip application ONLINE OFFLINE

# ./srvctl start nodeapps -n dbtest2
dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2)
dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2)
dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2)
dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2)
CRS-1006: No more members to consider
CRS-0215: Could not start resource 'ora.dbtest2.vip'.
CRS-0210: Could not find resource ora.dbtest2.LISTENER_DBTEST2.lsnr.


To know what is the nodeapps configuration in particular for VIP,
issue the command: srvctl config nodeapps -n <nodename> -a -g -s -l
Example:
/u01/crs/bin/srvctl config nodeapps -n dbtest2 -a -g -s -l
VIP exists.: /dbtest2-vip/10.11.11.198/255.255.255.0/eri0
GSD exists.
ONS daemon exists.
Listener does not exist.


Further debug it by uncommenting the environment variable _USR_ORA_DEBUG=1 in the script $ORA_CRS_HOME/bin/racgvip
OR simply as root user, issue the command : crsctl debug log res "ora.dbtest2.vip:5"
You may turn off debugging with command : crsctl debug log res "ora.dbtest2.vip:0"
Start the VIP using srvctl start nodeapps again. This will create a log for VIP starting problem for 10.2 and above version
in directory $ORA_CRS_HOME/log/<nodename>/racg/*vip.log

 

Example: last lines of the *.vip.log shows -

2005-02-09 20:38:06.711: [    RACG][1] [5602][1][ora.dbtest2.vip]: 203800 [ 5604 ] Checking interface existance
203800 [ 5604 ] Calling getifbyip
203800 [ 5604 ] getifbyip:  started for 10.11.11.198
203800 [ 5604 ] getifbyip: returning IF eri0:1
203800 [ 5604 ] Completed getifbyip eri0:1
203801 [ 5604 ] Completed with in
2005-02-09 20:38:06.711: [    RACG][1] [5602][1][ora.dbtest2.vip]: itial interface test
203801 [ 5604 ] checkIf: start for if=eri0
203801 [ 5604 ] checkIf: -z defaultgw
203801 [ 5604 ] defaultgw:  started
203801 [ 5604 ] defaultgw:  completed with 10.11.11.1
203801 [ 5604 ] checkIf: -n defaultgw
203804 [ 5604 ] checkIf:
2005-02-09 20:38:06.711: [    RACG][1] [5602][1][ora.dbtest2.vip]: in while, before sleep
203805 [ 5604 ] checkIf: in while, before sleep
203806 [ 5604 ] checkIf: checked if=eri0 failed
Interface eri0 checked failed (host=dbtest2)
203806 [ 5604 ] checkIf: end for if=eri0
203806 [ 5604 ] Performing CRS_STAT testing
203806
2005-02-09 20:38:06.711: [    RACG][1] [5602][1][ora.dbtest2.vip]:  [ 5604 ] Completed CRS_STAT testing
203806 [ 5604 ] Completed second gateway test
203806 [ 5604 ] Interface tests

    Failed to start VIP 10.11.11.198 (host=dbtest2)                 

From above we see the VIP:10.11.11.198 is correct however oracle function checkIf
is failing when trying to reach the default gateway IP:10.11.11.1

Changes

• Either the default gateway has got changed to someother IP -or-
• It is on a different network from the client (the network where the VIP is configured on) -or-
• It is not being used anymore.
  

Cause

By default, the server's default gateway is used as a ping target during the Oracle RAC 10g VIP status check action.
Upon a ping failure, Oracle will decide that the current interface where the VIP is running has failed, and will initiate
an interface / internode VIP failover.

In above case, we used just one node for CRS installation, so the VIP coudn't failover to other nodes and thus reported additional error like :
CRS-1006: No more members to consider
CRS-0215: Could not start resource 'ora.dbtest2.vip'.

Solution

10.1.0.4 and above introduced a parameter FAIL_WHEN_DEFAULTGW_NOT_FOUND in the file
$ORA_CRS_HOME/bin/racvip to address this problem.

The following steps will fix the VIP starting problem for above mentioned scenario.

1- stop nodeapps
2- As root,
vi the script $ORA_CRS_HOME/bin/racgvip and change the value of
variable FAIL_WHEN_DEFAULTGW_NOT_FOUND=0 .
3- start nodeapps and you should see the resources ONLINE

You may proceed with netca and dbca to create a RAC database after this.

---