Moving OCR and Voting Disk to Oracle ASM Online

Prior Oracle 11gR2, it was only possible to the OCR and Voting Disk files in multiple raw devices, shared between the nodes of a cluster.
From this version on, it’s very interesting to take the advantages and capabilities of ASM (striping and mirror) to store those files.
To store the OCR and Voting Disk files on a diskgroup, you must obey the following rules:
    If your diskgroup has “external” redundancy, it must have at least one disk associated.
    If your diskgroup has “normal” redundancy, it must have at least 3 disks associated, even though it can be create diskgroup with only 2 disks.
    If your diskgroup has “high” redundancy, it must have at least 5 disks associated, even though it can be create diskgroup with only 3 disks.
In this article, I will use the second option, normal redundancy. As in my server there are only 2 disks available for the diskgroup, I asked to the storage team to add another 1 GB disk:
[root@orarac01 ~]# /etc/init.d/oracleasm listdisks
DATA01
OCRVOTE01
OCRVOTE02
The disk has already been available:
[root@orarac01 ~]# multipath -ll
DATA01 (36000c29ab0accd959da8f1f78dd2de2f) dm-4 VMware,Virtual disk
size=300G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:2:0 sde 8:64 active ready running
OCRVOTE03 (36000c299550e110b041c9ab26acd33ac) dm-5 VMware,Virtual disk
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:3:0 sdf 8:80 active ready running
OCRVOTE02 (36000c29b67329567f6a6547d6fd3c769) dm-2 VMware,Virtual disk
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:0 sdc 8:32 active ready running
OCRVOTE01 (36000c29c1a57ee047d443991ef48894a) dm-3 VMware,Virtual disk
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:1:0 sdd 8:48 active ready running
Adding a new disk to oracleasm:
[root@orarac01 ~]# /etc/init.d/oracleasm createdisk OCRVOTE03 /dev/dm-5
Marking disk "OCRVOTE03" as an ASM disk:                   [  OK  ]
[root@orarac01 ~]# /etc/init.d/oracleasm listdisks
DATA01
OCRVOTE01
OCRVOTE02
OCRVOTE03
Enabling disk on another RAC node:
[root@orarac02 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@orarac02 ~]# /etc/init.d/oracleasm listdisks
DATA01
OCRVOTE01
OCRVOTE02
OCRVOTE03
[root@orarac02 ~]#
The next step is to create the new ASM diskgroup
First, let’s try to put them as “High Redundancy” and see what happens.
CREATE DISKGROUP OCRVOTE HIGH REDUNDANCY
FAILGROUP OCRVOTE_FG01 DISK '/dev/oracleasm/disks/OCRVOTE01' NAME OCRVOTE01  SIZE 1024M
FAILGROUP OCRVOTE_FG02 DISK '/dev/oracleasm/disks/OCRVOTE02' NAME OCRVOTE02  SIZE 1024M
FAILGROUP OCRVOTE_FG03 DISK '/dev/oracleasm/disks/OCRVOTE03' NAME OCRVOTE03  SIZE 1024M
ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M';
[root@orarac01 ~]# crsctl replace votedisk +OCRVOTE
Failed to create voting files on disk group OCRVOTE.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.
[root@orarac01 ~]# tail -n 200 $GRID_HOME/log/orarac01/client/crsctl_root.log
...
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: === clsssConfigLock ===
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: lock(0x2838a50), version(1), size(1088)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: offsets(24), activever(186647296)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: id(15), instantiation(5), incarn(1)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: mapoff(28), configoff(548), mapsize(512)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: state(10), holders(1), waiters(0)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: committimestamp(0), commitstate(0)
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: === Map (of first 7 entries) ===
2013-06-18 11:46:13.220: [ CSSCLNT][278546160]clsssConfigLockTrace: 010 000 000 000 000 000 000
2013-06-18 11:46:13.221: [  CRSCTL][278546160]crsctl_replace_votedisk: calling kgfdvfPushDiskString
2013-06-18 11:46:13.397: [  CRSCTL][278546160]function crsctl_replace_votedisk, error 1 [clsuSlosFormatDiag called with non-error slos.]
.
 
2013-06-18 11:46:13.397: [  CRSCTL][278546160]crsctl_replace_votedisk: pushed the ASM DS to the profile 1
[   CLWAL][278546160]clsw_Initialize: OLR initlevel [30000]
2013-06-18 11:46:13.432: [  CRSCTL][278546160]crsctl_format_diskgroup: creating diskgroup OCRVOTE
2013-06-18 11:46:16.555: [  CRSCTL][278546160]function crsctl_format_diskgroup, error 1 SLOS: cat=-1, opn=(none), dep=(none), loc=ORA-01405: f
.
 
2013-06-18 11:46:16.555: [  CRSCTL][278546160]crsctl_format_diskgroup: diskgroup OCRVOTE creation with status 1. Please check the alert log file for ASM
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: === clsssConfigUnlock ===
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: lock(0x2837b50), version(1), size(1088)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: offsets(24), activever(186647296)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: id(15), instantiation(5), incarn(3)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: mapoff(28), configoff(548), mapsize(512)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: state(0), holders(0), waiters(0)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: committimestamp(0), commitstate(0)
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: === Map (of first 7 entries) ===
2013-06-18 11:46:16.568: [ CSSCLNT][278546160]clsssConfigLockTrace: 000 000 000 000 000 000 000
...
[root@orarac01 ~]# tail -n 500 /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
...
Tue Jun 18 11:46:13 2013
NOTE: Creating voting files in diskgroup OCRVOTE
Tue Jun 18 11:46:13 2013
NOTE: Voting File refresh pending for group 2/0x7f386577 (OCRVOTE)
NOTE: Attempting voting file creation in diskgroup OCRVOTE
NOTE: voting file allocation on grp 2 disk OCRVOTE_0000
NOTE: voting file allocation on grp 2 disk OCRVOTE_0001
NOTE: voting file allocation on grp 2 disk OCRVOTE_0002
ERROR: Voting file allocation failed for group OCRVOTE
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_6301.trc:
ORA-15274: Not enough failgroups (5) to create voting files
NOTE: Attempting voting file refresh on diskgroup OCRVOTE
NOTE: Voting file relocation is required in diskgroup OCRVOTE
NOTE: Attempting voting file relocation on diskgroup OCRVOTE
NOTE: voting file deletion on grp 2 disk OCRVOTE_0000
NOTE: voting file deletion on grp 2 disk OCRVOTE_0001
NOTE: voting file deletion on grp 2 disk OCRVOTE_0002
...
The error is that it is necessary at least 5 failgroups to create the Voting Disk.
We will then recreate them with the correct redundancy:
CREATE DISKGROUP OCRVOTE NORMAL REDUNDANCY
FAILGROUP OCRVOTE_FG01 DISK '/dev/oracleasm/disks/OCRVOTE01' NAME OCRVOTE01  SIZE 1024M
FAILGROUP OCRVOTE_FG02 DISK '/dev/oracleasm/disks/OCRVOTE02' NAME OCRVOTE02  SIZE 1024M
FAILGROUP OCRVOTE_FG03 DISK '/dev/oracleasm/disks/OCRVOTE03' NAME OCRVOTE03  SIZE 1024M
ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M';
   
[root@orarac01 ~]# crsctl replace votedisk +OCRVOTE
Successful addition of voting disk 05ca1320e3244f70bf45d1fc37d325d5.
Successful addition of voting disk 1affc0b6da314fe8bfd8dec18af4923b.
Successful addition of voting disk 3d87bf2793e04fe4bfd6764f24ffafb1.
Successful deletion of voting disk 4c9ddcb1acd14f27bf77652370a1a5b1.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4266: Voting file(s) successfully replaced
[root@orarac01 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   05ca1320e3244f70bf45d1fc37d325d5 (/dev/oracleasm/disks/OCRVOTE01) [OCRVOTE]
2. ONLINE   1affc0b6da314fe8bfd8dec18af4923b (/dev/oracleasm/disks/OCRVOTE02) [OCRVOTE]
3. ONLINE   3d87bf2793e04fe4bfd6764f24ffafb1 (/dev/oracleasm/disks/OCRVOTE03) [OCRVOTE]
Located 3 voting disk(s).
The Voting Disk step is done.
The next step is the CRS. Note that we are transferring him from another diskgroup (+DATA), which is used to store datafiles, to this another diskgroup recently created:
[root@orarac01 ~]# ocrconfig -replace +DATA -replacement +OCRVOTE
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location
As we have only one OCR location configured, the “replace” command doesn’t work. We’ll need to add another one and delete the old:
[root@orarac01 ~]# ocrconfig -add +OCRVOTE
[root@orarac01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2836
         Available space (kbytes) :     259284
         ID                       : 1089167638
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
Now erase the old:
   
[root@orarac01 ~]# ocrconfig -delete +DATA
[root@orarac01 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2836
         Available space (kbytes) :     259284
         ID                       : 1089167638
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
                                    Device/File not configured
 
         Cluster registry integrity check succeeded
 
         Logical corruption check succeeded
 
[root@orarac01 ~]# cat /etc/oracle/ocr.loc
#Device/file +DATA getting replaced by device +OCRVOTE
ocrconfig_loc=+OCRVOTE
local_only=false

Done!

参考至：http://www.dbarj.com.br/en/2013/06/moving-ocr-voting-disk-to-oracle-asm-online/
如有错误，欢迎指正
邮箱:[e-mail]