成功安装Oracle10g的RAC

安装虚拟机，我使用的是VMware-server-2.0.2-203138.exe
B/S方式管理，有点不如WorkStation用着方便。
硬件配置：
网卡1：bridge模式
网卡2：host-only
硬盘1：本地20G，预先分配空间，Indepentant，位置SCSI 0:0
硬盘2：共享,200M，预先分配空间，Indepentant，位置SCSI 1:0，用于安装OCR
硬盘3：共享,200M，预先分配空间，Indepentant，位置SCSI 1:1，用于安装Voting Disk
硬盘4：共享,4G，预先分配空间，Indepentant，位置SCSI 1:2，
硬盘5：共享,4G，预先分配空间，Indepentant，位置SCSI 1:3，用于安装Oracle数据库
硬盘6：共享,2G，预先分配空间，Indepentant，位置SCSI 1:4，
硬盘7：共享,2G，预先分配空间，Indepentant，位置SCSI 1:5，用于做闪回空间，如果磁盘空间有限可以不创建。

装Linux，我使用的AS4，
创建dba,oinstall组
创建oracle用户

修改环境变量
[oracle@rac1 ~]$ vi ~/.bash_profile
export EDITOR=vi
export ORACLE_SID=prod1
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/u01/crs
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
PATH=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH

修改内核参数：
sysctl.conf
kernel.shmall=2097152
kernel.shmmax=536970912
kernel.shmmni=4096
kernel.sem=250 32000 100 128
fs.file-max=65536
net.ipv4.ip_local_port_range=1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

# /etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16484
oracle soft nofile 1024
oracle hard nofile 65536

分区，/dev/sdb等，200M/200M/4G/4G/2G/2G
fdisk /dev/sdb
每个只分一个区，主分区，不要格式化

安装ASM软件
[root@rac1 sysconfig]#  rpm -ivh oracleasm*

[root@rac1 sysconfig]#  vi /etc/sysconfig/rawdevices
增加设置映射
/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1
/dev/raw/raw3 /dev/sdd1
/dev/raw/raw4 /dev/sde1
/dev/raw/raw5 /dev/sdf1
/dev/raw/raw6 /dev/sdg1

重启服务
service rawdevices restart

查看结果：
[root@rac1 sysconfig]#  ls -l /dev/raw
total 0
crw-rw----  1 root disk 162, 1 Jan 12 14:47 raw1
crw-rw----  1 root disk 162, 2 Jan 12 14:47 raw2
crw-rw----  1 root disk 162, 3 Jan 12 14:47 raw3
crw-rw----  1 root disk 162, 4 Jan 12 14:47 raw4
crw-rw----  1 root disk 162, 5 Jan 12 14:47 raw5
crw-rw----  1 root disk 162, 6 Jan 12 14:47 raw6

修改所有权
[root@rac1 sysconfig]#   vi /etc/udev/permissions.d/50-udev.permissions
修改raw的所有权信息
raw/*:oracle:dba:0660
保存后，重启服务，再用ls检查
[root@rac1 sysconfig]#  service rawdevices restart
[root@rac1 sysconfig]# ls -l /dev/raw
total 0
crw-rw----  1 oracle dba 162, 1 Jan 12 14:58 raw1
crw-rw----  1 oracle dba 162, 2 Jan 12 14:58 raw2
crw-rw----  1 oracle dba 162, 3 Jan 12 14:58 raw3
crw-rw----  1 oracle dba 162, 4 Jan 12 14:58 raw4
crw-rw----  1 oracle dba 162, 5 Jan 12 14:58 raw5
crw-rw----  1 oracle dba 162, 6 Jan 12 14:58 raw6

设置网卡
[root@rac1 ~]$ vi /etc/sysconfig/network-scripts/ifcfg-eth0
设置IP为:21.156.173.240，掩码：255.255.255.0
[root@rac1 ~]$ vi /etc/sysconfig/network-scripts/ifcfg-eth1
设置IP为:10.1.1.1，掩码：255.255.255.0
重启网卡
[root@rac1 ~]service network restart

修改hosts文件
[oracle@rac1 ~]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               www.wallomn.com localhost
21.156.173.240          rac1
21.156.173.241          rac1-vip
10.1.1.1                rac1-prv

21.156.173.242          rac2
21.156.173.243          rac2-vip
10.1.1.2                rac2-prv

注：其中21.156.173.241          rac1-vip，是安装好cluster后，起作用的IP，好像叫飘移IP

找开虚拟机目录，打开rac1.vmx，添加以下内容，不然后复制虚拟机文件后，无法启动。
disk.locking="false"
diskLib.dataCacheMaxSize="0"
diskLib.dataCacheMaxReadAheadSize="0"
diskLib.dataCacheMinReadAheadSize="0"
diskLib.dataCachePageSize="4096"
diskLib.maxUnsyncedWrites="0"

scsi1:1.deviceType="disk"
scsi1:2.deviceType="disk"
scsi1:3.deviceType="disk"
scsi1:4.deviceType="disk"
scsi1:5.deviceType="disk"
scsi1:6.deviceType="disk"

用拷贝的虚拟机文件的方式，生成rac2节点，启动时告诉VMWare是拷贝的。
然后Linux会提示些东西，选择"do nothing"。
rac2顺利启动后，首先修改网上配置。
[root@rac2 network-scripts]vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改IP为21.156.173.242
[root@rac2 network-scripts]vi /etc/sysconfig/network-scripts/ifcfg-eth1
修改IP为10.1.1.2
重启网卡
[root@rac2 network-scripts]service network restart

建立ssh互信关系
首先确保两个机器能够ping通。
在rac1上执行下面操作
[oracle@rac1 ~]$ mkdir ~/.ssh
[oracle@rac1 ~]$ chmod 700 ~/.ssh

[oracle@rac1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
1c:f4:bb:9f:eb:09:4b:f4:57:ac:e9:11:af:32:9f:8f oracle@rac1

[oracle@rac1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
14:55:6e:8c:a3:82:26:fe:11:c6:58:b1:d5:8f:16:f7 oracle@rac1

上述操作在rac2上同样进行一次。

回到rac1，操作中，要输入rac2的oracle用户的密码
[oracle@rac1 ~]$ ssh rac2 cat ~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys
The authenticity of host 'rac2 (21.156.173.242)' can't be established.
RSA key fingerprint is c6:e3:d1:23:49:1a:a4:ff:53:59:1e:48:3a:51:a1:6f.
Are you sure you want to continue connecting (yes/no)? yes  
Warning: Permanently added 'rac2,21.156.173.242' (RSA) to the list of known hosts.
oracle@rac2's password:
[oracle@rac1 ~]$ ssh rac2 cat ~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys
oracle@rac2's password:

上传文件到rac2，看到提示信息表明成功
[oracle@rac1 .ssh]$ scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys
oracle@rac2's password:
authorized_keys                                       100% 3199     3.1KB/s   00:00

两个机器互相测试一下
[oracle@rac2 .ssh]$ ssh rac1 date
The authenticity of host 'rac1 (21.156.173.240)' can't be established.
RSA key fingerprint is c6:e3:d1:23:49:1a:a4:ff:53:59:1e:48:3a:51:a1:6f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac1,21.156.173.240' (RSA) to the list of known hosts.
Thu Jan 12 20:43:44 EST 2012
[oracle@rac2 .ssh]$ ssh rac1-prv date
The authenticity of host 'rac1-prv (10.1.1.1)' can't be established.
RSA key fingerprint is c6:e3:d1:23:49:1a:a4:ff:53:59:1e:48:3a:51:a1:6f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac1-prv,10.1.1.1' (RSA) to the list of known hosts.
Thu Jan 12 20:43:54 EST 2012

[oracle@rac1 .ssh]$ ssh rac2 date
[oracle@rac1 .ssh]$ ssh rac2-prv date

如果正确显示信息，说明设置正确

时间同步
rac1做服务器。
[root@rac1 ~]# service xinetd start
设置开机望到时间服务
[root@rac1 ~]# chkconfig time on
[root@rac1 ~]# service ntpd start
[root@rac1 ~]# chkconfig ntpd on
[root@rac1 ~]# chkconfig xinetd on

编辑任务表，设置定时同步
[root@rac2 etc]# crontab -e
0-59/1 * * * * /user/bin/rdate -s 10.1.1.1 >/dev/null 2>&1
0-59/1 * * * * /user/sbin/ntpdate 10.1.1.1 >/dev/null 2>&1

配置asm，两个节点都执行
[root@rac1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]: oracle
Default group to own the driver interface [dba]: dba
Start Oracle ASM library driver on boot (y/n) [y]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [  OK  ]
Scanning the system for Oracle ASMLib disks: [  OK  ]

配置ASM，仅rac1需要执行，rac2不需要。
[root@rac1 ~]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdd1
Marking disk "VOL1" as an ASM disk: [  OK  ]
[root@rac1 ~]# /etc/init.d/oracleasm createdisk VOL2 /dev/sde1
Marking disk "VOL2" as an ASM disk: [  OK  ]
[root@rac1 ~]# /etc/init.d/oracleasm createdisk VOL3 /dev/sdf1
Marking disk "VOL3" as an ASM disk: [  OK  ]
[root@rac1 ~]# /etc/init.d/oracleasm createdisk VOL4 /dev/sdg1
Marking disk "VOL4" as an ASM disk: [  OK  ]
[root@rac1 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

rac2节点执行配置，
[root@rac2 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]: oracle
Default group to own the driver interface [dba]: dba
Start Oracle ASM library driver on boot (y/n) [y]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [  OK  ]
Scanning the system for Oracle ASMLib disks: [  OK  ]
查看一下asm配置，应该同样看到rac1的配置，因为存储是共享的。
[root@rac2 ~]# /etc/init.d/oracleasm listdisks
也可以使用下面的命令
[root@rac2 ~]# service oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

在Linux系统上准备好cluster软件，然后检查一下安装环境，如无特别说明，都是基于Oracle用户的。
[oracle@rac1 cluvfy]$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose
有一堆提示，这里不写了，反正注意下一ERROR、failed字样就是了。有些可以忽略，是Oracle的BUG。

启动安装（在图形界面里进行，如不行先执行一下xhost +）
[oracle@rac1 cluvfy]$ pwd
/share/clusterware/cluvfy
执行安装
./runInstaller

将软件安装到目录
/u01/crs
添加上rac2节点
设置eth0为public,eth1为private(如果ssh没有配好，到这一步之前会有错)
设置OCR Configure冗余方式为：External Redundancy，位置设置为：/dev/raw/raw2（前面配置的裸设备）
设置Voting Disk Configure冗余方式为：External Redunancy，位置设置为：/dev/raw/raw1（前面配置的裸设备）

执行两个脚本，脚本1：orainstRoot.sh
[root@rac1 /]# /u01/app/oracle/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to oinstall.
The execution of the script is complete

脚本2：root.sh，执行时间有点长，耐心。
[root@rac1 /]# /u01/crs/root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-prv rac1
node 2: rac2 rac2-prv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw1
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        rac1
CSS is inactive on these nodes.
        rac2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.

再在rac2节点上执行上面两个脚本
如果报内网IP不是public错误,以root身份执行脚本/u01/crs/bin/vipca，设置一下。

执行完毕后，在rac1上，切换回oracle用户，执行命令检查一下。
[oracle@rac1 u01]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    OFFLINE              
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    OFFLINE              

两个东西没有online，检查了一下屏幕提示的安装日志，说我没有配默认网关，哎，这也要管。
配上网关，重启了一下network，就好了。如果再不行，就重启一下Linux。

在图形界面安装Oracle软件
[root@rac1 u01]# xhost +
我的Oracle安装文件，放在/u01/orasoft目录中，执行安装脚本
[oracle@rac1 u01]$ /u01/orasoft/runInstaller
Cluster Install界面上，两个节点：rac1、rac2都选上。
仅安装软件，先不安装数据库。
然后执行root.sh脚本，以root用户的身份执行。两个节点上都执行。
[root@rac1 db_1]# pwd
/u01/app/oracle/product/10.2.0/db_1
[root@rac1 db_1]# ./root.sh
Running Oracle10 root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.

两个节点都运行完毕后，点击图形界面的OK按钮。

然后使用netca建立监听。建立cluster configuration，选两个节点，其它默认。
建完后，监听会自动启动。

查看一下监听的状态
[oracle@rac1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 13-JAN-2012 15:37:29

Copyright (c) 1991, 2005, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_RAC1
Version                   TNSLSNR for Linux: Version 10.2.0.1.0 - Production
Start Date                13-JAN-2012 15:29:42
Uptime                    0 days 0 hr. 7 min. 47 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/10.2.0/db_1/network/log/listener_rac1.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=21.156.173.241)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=21.156.173.240)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

也可以检查一下配置文件
[oracle@rac1 admin]$ pwd
/u01/app/oracle/product/10.2.0/db_1/network/admin
[oracle@rac1 admin]$ cat listener.ora
# listener.ora.rac1 Network Configuration File: /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.ora.rac1
# Generated by Oracle configuration tools.

LISTENER_RAC1 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 21.156.173.240)(PORT = 1521)(IP = FIRST))
    )
  )

SID_LIST_LISTENER_RAC1 =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
      (PROGRAM = extproc)
    )
  )

然后就是建库了。使用dbca就可以了，可建单实例的，也可建集群的库。我们当然选择集群了。
一定记得两个节点rac1、rac2都选上。
库类型选通用(general)
全局数据库名、SID名：prod（前面设置环境变量的时候，两个节点的实例名一个为prod1，一个为prod2）
存储机制选ASM（Cluster File System好像不太容易成功）
ASM实例使用pfile方式，
创建磁盘组，DG1，冗余方式选择Normal(需要两块盘)，选/dev/raw/raw3、/dev/raw/raw4
创建磁盘组，RECOVERY，冗余方式选择Normal(需要两块盘)，选/dev/raw/raw5、/dev/raw/raw6
Database Area就选刚创建的+DG1
启动flash recovery，放在RECOVERY磁盘组上。
其它的选默认就可以了。当然，如果你愿意，也可以选一下字符集等参数。


测试一下集群的效果
用宿主机建立两个本地服务名，检测一下数据的效果。
SQL>  conn system/oracle@rac2
已连接。
SQL> create table scott.test_rac as (select * from scott.dept);

表已创建。

SQL> conn system/oracle@rac1
已连接。
SQL> select count(*) from scott.test_rac;

  COUNT(*)
----------
         4