Installing Oracle10g R2 RAC on vmware suse

Installing Oracle10g R2 RAC
Part1
- Prerequisites
- Installing/Preparing Guest machine (SuSE10.1)
Prerequisites
This example shows how to install Oracle10g R2 RAC with raw devices (block devices), without using ocfs2 or asmlib, so no other modules need to be downloaded or installed before installing Oracle 10g R2 RAC.
--------------------------------------------------------------------
Hardware requirements£º
--------------------------------------------------------------------
Dell PE2850 one
CPU£º Intel Xeon  3.6GHz ¡Á2
Memory £º 4GB
Hard Drive : 40 -50 GB needed for vitural guests
---------------------------------------------------------------------
Host Operating system and software£º
---------------------------------------------------------------------
OS£º Windows Server 2003  Enterprise 5.2.3790
VM£º VMWare GSX Server 3.2.1 build-19281 for windows
--------------------------------------------------------------------
Software requirements£º
--------------------------------------------------------------------
Guest OS£º SuSE Linux Enterprise 10.1 (4cd or 1DVD)
Download Here: http://en.opensuse.org/Released_Version

$ uname -a
Linux sles101 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006 i686 i686 i386 GNU/Linux
DB£º Oracle 10gR2 for x86 linux
clusterware£º
http://download.oracle.com/otn/linux/oracle10g/10201/10201_clusterware_linux32.zip
database£º
http://download.oracle.com/otn/linux/oracle10g/10201/10201_database_linux32.zip
--------------------------------------------------------------
Configuring VMWare virtual Networks:
--------------------------------------------------------------  
Start -->  Programs  --> VMware --> VMware GSX Server --> Manage Virtual Networks
From Virtual Network Editor , Select Host Virtual Network Mapping£º
The public address is using "Bridged" mode to utilize the physical subnets (172.16.2.0/24, on same subnet with the host machine.
VMnet1's subnet address is 192.168.235.0/24 (use "host-only" mode) - as  private address for the 2 nodes.
Installing/Preparing Guest machine (SuSE10.1)
-----------------------------------------------------------------------------
 Creating Guest machine (SuSE10.1) in VMWare
-----------------------------------------------------------------------------
1. Open VMware Virtual Machine Console
2.  Click New Virtual Machine
3.  Select the Appropriate Configuration£º
 Select Custom£¬ click £ÛNext£Ý
4.Select a Guest Operating System£º
 Select  Linux£¬Version SuSE Enterprise Linux£¬ click £ÛNext£Ý
5.Name the Virtual Machine£º
 SLES101, location d:\sles101 £¬ click £ÛNext£Ý
6.Set Access Rights£º
 Use default £¬ click £ÛNext£Ý
7.Startup / Shutdown Options£º
 Use default, User that powers on the virtual machine£¬ click £ÛNext£Ý
8.Memory for the Virtual Machine£º
1024MB£¬ click £ÛNext£Ý
9.Network Type£º
 select  Do not use a network connection£¬ click £ÛNext£Ý( configure later )
10.Select I/O Adapter Types£º
 select  LSI Logic£¬ Click £ÛNext£Ý
11.Select a Disk£º
 select  Create a new virtual disk£¬ click £ÛNext£Ý
12.Select a Disk Type£º
 select  SCSI£¬ click £ÛNext£Ý
13.Specify Disk Capacity£º
 Enter   8GB£¬ select Allocate all disk space now£¬ deselect Split disk into 2GB files£¬
 click £ÛNext£Ý
14.Specify Disk File£º
 d:\sles101\system.vmdk£¬ click £ÛFinish£Ý
15.Info£º
Creating pre-allocated virtual disk...
16  .Hardware Type£º
 select Ethernet Adapter£¬ click£ÛNext£Ý
17 .Network Type£º
 select Custom£¬ select  "Bridged" £¬Connect at power on£¬ click £ÛFinish£Ý
18. add one more NIC£º
 click £ÛAdd£Ý
19 .Hardware Type£º
 select  Ethernet Adapter£¬ click £ÛNext£Ý
20 .Network Type£º
 select Custom£¬ "Host-only" £¬Connect at power on£¬ click £ÛFinish£Ý
22.  Exit VMware Virtual Machine Console£º
I will use the first node to clone the second node.
------------------------------------------
Create additional disks £º
------------------------------------------
1. On Window Host machine, Open Command Prompt¡£
2. Change to GSX  Installation directory,  c:\program files\vmware\vmware gsx server£º
3. Run DOS command to create OCR + CRS voting disks,
vmware-vdiskmanager -c -s  500Mb -a lsilogic -t 2 d:\sles101\ocr.vmdk
vmware-vdiskmanager -c -s 500Mb -a lsilogic -t 2 d:\sles101\css.vmdk
[OCR and CSS partition could be on the same virtual disk, and you can also create mirror disk (1 for OCR, 2 for CSS) if neccessary, each OCR or CSS partition need >256MB disk space.]
 
4. Create 1 Oracle product disk >4GB, used for local disk on both nodes
vmware-vdiskmanager -c -s 4Gb -a lsilogic -t 2 d:\sles101\u01.vmdk
5. Create 4 ASM disks for Oracle data & flash recovery area£º
vmware-vdiskmanager -c -s 2Gb -a lsilogic -t 2 d:\sles101\asm1.vmdk
vmware-vdiskmanager -c -s 2Gb -a lsilogic -t 2 d:\sles101\asm2.vmdk
vmware-vdiskmanager -c -s 2Gb -a lsilogic -t 2 d:\sles101\asm3.vmdk
vmware-vdiskmanager -c -s 2Gb -a lsilogic -t 2 d:\sles101\asm4.vmdk
6. Edit (do not use notepad) d:\sles101\system.vmx and d:\sles102\system.vmx (after cloned from sles101), add the following lines:
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.filename = "d:\sles101\ocr.vmdk"
scsi1:1.deviceType = "plainDisk"
scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.filename = "d:\sles101\css.vmdk"
scsi1:2.deviceType = "plainDisk"
scsi1:3 .present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.filename = "d:\sles101\asm1.vmdk"
scsi1:3.deviceType = "plainDisk"
scsi1:4 .present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.filename = "d:\sles101\asm2.vmdk"
scsi1:4.deviceType = "plainDisk"
scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.filename = "d:\sles101\asm3.vmdk"
scsi1:5.deviceType = "plainDisk"
scsi1:6.present = "TRUE"
scsi1:6.mode = "independent-persistent"
scsi1:6.filename = "d:\sles101\asm4.vmdk"
scsi1:6.deviceType = "plainDisk"
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
save and exit.

[If the vitual machines will be installed on VMWare ESX server, use "LSI Logic" for the SCSI Controller type. If you use "Bus Logic", the drivers could not be loaded and no hard drive will be located for installation.]
 
--------------------------------------------------------------------
Install SuSE Linux 10.1 (sles) on first node
--------------------------------------------------------------------
Please refer to the following Novell official website for the installation guide:
Installing Oracle10g R2 Database on SLES10 for i386 - step by step
Partition Settings
For "partitioning" of the Installation Settings screen, create LVM partitions for "/", "/tmp", swap,"/u01"(the oracle product location). Create with this sequence: PV->VG->LV
Create Physical Volume from raw partition (primary partition) first, then create new Volume Group and add Physical Volumes to the group, at last mount the Volume Group to the mounting points with format option.
The reason to use LVM is that it is easy to extend the size in case not sufficient later. For example, the following results showed that the "swap" space was extended from 1GB up to 1.89GB by adding 2 partitions (/dev/sdc1 and /dev/sdc2) to the first partition (/dev/sda3) which are all in the VG "swapvg".

$ /sbin/pvscan
  PV /dev/sdb1   VG u01vg    lvm2 [3.99 GB / 88.00 MB free]
  PV /dev/sda5   VG tmpvg    lvm2 [704.00 MB / 0    free]
  PV /dev/sda3   VG swapvg   lvm2 [1.00 GB / 0    free]
  PV /dev/sdc1   VG swapvg   lvm2 [500.00 MB / 0    free]
  PV /dev/sdc2   VG swapvg   lvm2 [500.00 MB / 88.00 MB free]
  PV /dev/sda2   VG rootvg   lvm2 [4.00 GB / 4.00 MB free]
  Total: 6 [10.66 GB] / in use: 6 [10.66 GB] / in no VG: 0 [0   ]
$ /sbin/lvscan
  ACTIVE            '/dev/u01vg/u01' [3.90 GB] inherit
  ACTIVE            '/dev/tmpvg/temp' [704.00 MB] inherit
  ACTIVE            '/dev/swapvg/swap' [1.89 GB] inherit
  ACTIVE            '/dev/rootvg/root' [4.00 GB] inherit
"/boot" partition doesn't have to be use LVM because 100MB is enough for it.
But do not use LVM for any shared disks like CSS, OCR or ASM disks.
On x86 and Itanium systems, although Red Hat Enterprise Linux 3 and SUSE Linux Enterprise Server provide a Logical Volume Manager (LVM), this LVM is not cluster-aware. For this reason, Oracle does not support the use of logical volumes with RAC on x86 and Itanium systems for either Oracle Clusterware or database files.
http://download-west.oracle.com/docs/cd/B19306_01/install.102/b14203/storage.htm#sthref729
Software Settings:
When the Software Selection and System Tasks screen appear, select the C/C++ Compiler and Tools (must have) and Oracle Server Base (not mandatory).
Use the "Search" function, the following software/package are also needed to be installed:
Openmotif
mc (optional)
readline-devel (required if you want to use rlwrap)
sysstat
setarch

In this example, "orarun" package is not installed.

--------------------------------------------------------------------------------------------
Add configurations to the first SuSE linux node (sles101)
before cloning the Second node
--------------------------------------------------------------------------------------------
1. Edit /etc/hosts, and comment out or remove the line that maps the IP address 127.0.0.2 to the systems fully qualified hostname.
127.0.0.1       localhost
172.16.2.145    sles101.test.com sles101
172.16.2.146    sles102.test.com sles102
192.168.235.145 sles101-priv.test.com sles101-priv
192.168.235.146 sles102-priv.test.com sles102-priv
172.16.2.147    sles101-vip.test.com sles101-vip
172.16.2.148    sles102-vip.test.com sles102-vip

2. Edit /etc/SuSE-Release, change version number from 10 to 9.
$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 9 (i586)
VERSION = 9
[It can be reverted back after Oracle 10g RAC has been installed.]

3. Edit /etc/sysctl.conf
$ cat /etc/sysctl.conf
# Disable response to broadcasts.
# You don't want yourself becoming a Smurf amplifier.
net.ipv4.icmp_echo_ignore_broadcasts = 1
IXDBA.NET¼¼ÊõÉçÇø
# enable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 1
# enable ipV6 forwarding
#net.ipv6.conf.all.forwarding = 1
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 1
#controls whether core dumps will append the PID to the core filename.
#Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Oracle recommended kernel parameters
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 256 32000 100 128
kernel.msgmnb = 65535
kernel.msgmni = 2878
fs.file-max = 131072
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_max = 262144
net.core.wmem_max = 262144
net.core.rmem_default = 262144
net.core.wmem_default = 262144
4. Edit /etc/pam.d/login, and add the following line (will be added automatically if "orarun" rmp package is installed)
session     required       pam_limits.so
 
5. Add "oracle" user and "dba" group
#mkdir -p /u01/app
#groupadd -g 115 dba
#useradd -u 175 -g 115 -d /home/oracle -s /bin/bash -c "Oracle Software Owner" -p oracle oracle
#chown -R oracle:dba /u01
 
6. Create partitions on shared disks and configuring the raw devices
Use "fdisk" to create primary partitions on all shared disks (2x 500MB partitions "sdd1" and "sde1" for OCR and CSS, 4x 2GB for ASM disks - "sdf1-sdi1")
$ cat /etc/raw
# /etc/raw
# The format of this file is:
# raw<N>:<blockdev>
# this means: bind /dev/raw/raw1 to /dev/hdb1
raw1:sdd1    #this will be the css space
raw2:sde1    #this will be used as ocr space
raw3:sdf1    #for ASM disk member
raw4:sdg1
raw5:sdh1
raw6:sdi1
Run the following command to make sure all raw devices started.
$/usr/sbin/raw -qa
/dev/raw/raw1:  bound to major 8, minor 49
/dev/raw/raw2:  bound to major 8, minor 65
/dev/raw/raw3:  bound to major 8, minor 81
/dev/raw/raw4:  bound to major 8, minor 97
/dev/raw/raw5:  bound to major 8, minor 113
/dev/raw/raw6:  bound to major 8, minor 129
In case of any raw partition could not be bound, please check if there is the correct devices being created under "/dev/" directory, run "partprobe" to refresh. This could be happened when you created a primary partition on a shared disk of node1, but the new device is not displayed on the other node.
 
7. Create raw devices for OCR and CSS shared disks, and mounting points for ASM disks.
#mkdir -p /u02/oradata/asm
#mknod /u02/oradata/css c 162 1
#mknod /u02/oradata/ocr c 162 2

这两句一定要...
#mknod /u02/oradata/asm/lun1 c 162 3
#mknod /u02/oradata/asm/lun2 c 162 4
#mknod /u02/oradata/asm/lun3 c 162 5
#mknod /u02/oradata/asm/lun4 c 162 6
#chmod 660 /u02/oradata/css
#chmod 660 /u02/oradata/ocr
#chmod 660 /u02/oradata/asm/*
#chown oracle.dba /u02/oradata/css
#chown oracle.dba /u02/oradata/ocr
#chown oracle.dba /u02/oradata/asm/*
 
lun1-4 will be used as ASM disks for oracle data, after CRS and Database software is installed, run DBCA to configure ASM storage and add lun1-4 as members for the ASM disk groups. (If EMC storage will be used to provide the share disks, we can also create same character devices to link them.)
 
8. Login as "oracle" user and modify the enviroment file /home/oracle/.profile, see following example with some of useful aliases.
$ cat .profile
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=$ORACLE_BASE/product/10.2/db_1
ORACLE_SID=sles
ORA_CRS_HOME=$ORACLE_BASE/product/10.2/crs_1
export ORACLE_DB=sles
export ORACLE_BASE ORACLE_HOME ORACLE_SID ORA_CRS_HOME
export NLS_DATE_FORMAT='MM/DD/YYYY HH24:MI:SS'
export PS1='[`whoami`@${HOSTNAME} ${PWD}]\n$ '
PATH=$PATH:$ORACLE_HOME/bin
LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}$ORACLE_HOME/lib:/lib:$ORACLE_HOME/ctx/lib
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export AGENT_HOME PATH LD_LIBRARY_PATH CLASSPATH TNS_ADMIN
alias ASM='export ORACLE_SID=+ASM1'
alias dbn='export ORACLE_SID=sles1'
alias envora='env | grep ORA'
alias ssh2='ssh sles102'
alias crs='cd $ORA_CRS_HOME/bin'
alias cddb='cd $ORACLE_HOME/bin'
alias sysdba='sqlplus / as sysdba'
alias envora='env | grep ORA'
alias sqlplus='rlwrap sqlplus'

9. And don't forget to install vmware-tools (run from non-X Windows logon session)
# ./vmware-install.pl
-------------------------------------------------------

Clone the second node (sles102)
-------------------------------------------------------
1. After above configuration finished, shutdown the first node sles101, and copy all the vmware machine files to folder d:\sles102. Edit sles.vmx and make sure all shared disks are configured with the correct disk path. 
 
2. power on the second node sles102 and update the NIC IP addresses.
During the boot, you will find the NICs eth0 and eth1on sles102 were renamed to eth2 and eth3. You need edit the "/etc/udev/rules.d/30_net_persistent_names.rules" file to change them back to device name eth0 and eth1.
 
sles102:/etc/udev/rules.d # cat 30-net_persistent_names.rules
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:0c:29:6a:97:0d", IMPORT="/lib/udev/rename_netiface %k eth1"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:0c:29:6a:97:03", IMPORT="/lib/udev/rename_netiface %k eth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:92:7f:09", IMPORT="/lib/udev/rename_netiface %k eth2"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:92:64:de", IMPORT="/lib/udev/rename_netiface %k eth3"
The NICs with MAC address "00:0c:29:6a:97:0d" and "00:0c:29:6a:97:03" are invalid here and no longer exist in this linux server, therefore, should be renamed with new MAC addresses that eth2 and eth3 are holding. The modified contents of rule file should looks like:
sles102:/etc/udev/rules.d # cat 30-net_persistent_names.rules
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:92:7f:09", IMPORT="/lib/udev/rename_netiface %k eth1"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:92:64:de", IMPORT="/lib/udev/rename_netiface %k eth0"
Then, restart the udev and network service, you will get your eth0 and eth1 back.
 
--------------------------------------------------------------------------------------
Create user equivelence for user "root" and "oracle"
--------------------------------------------------------------------------------------
User equivalance should be configured for both "root" and "oracle" user, so that no password is required for authentication of copying/accessing between all RAC nodes.
Public keys (RSA and DSA) are required to be used with ssh login sessions. In this example, "ssh-keygen" will be used to generate the keys.
 
Creating User Equivalance Procedures are as below:
a. Generate public and private keys by "ssh-keygen" for user 'root', and copy all public keys to the "/root/.ssh/authorized_keys" on node1
b. Repeat above on node2
c. Merge the 2x authorized_keys contents on both nodes
d. Running "ssh" session for all public and private hostnames of both RAC nodes (generating the "known_hosts" file)
e. Repeat step a-d to generate authorized_keys file for user 'oracle'
a. On sles101, create RSA and DSA keys for user "root"
[root@sles101 .ssh]# ssh-keygen -b 1024 -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
33:2e:a1:20:09:6c:5d:d5:9f:1c:79:d4:09:b6:40:41 [e-mail]
IXDBA.NET¼¼ÊõÉçÇø
[root@sles101 .ssh]# ssh-keygen -b 1024 -t rsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
33:2e:a1:20:09:6c:5d:d5:9f:1c:79:d4:09:b6:40:41 [e-mail]
[root@sles101 .ssh]# ls -al
total 20
drwx------   2 root root 4096 Oct 14 16:38 .
drwxr-x---  14 root root 4096 Oct 14 10:55 ..
-rw-------   1 root root  668 Oct 14 16:38 id_dsa
-rw-r--r--   1 root root  619 Oct 14 16:38 id_dsa.pub
-rw-------   1 root root  668 Oct 14 16:38 id_rsa
-rw-r--r--   1 root root  619 Oct 14 16:38 id_rsa.pub
[root@sles101 .ssh]# cat id_dsa.pub >>authorized_keys
[root@sles101 .ssh]# cat id_rsa.pub >>authorized_keys
copy authorized_keys to "sles102"
[root@sles101 .ssh]# scp authorized_keys root@sles102:/root/.ssh/authorized_keys
The authenticity of host 'sles102 (172.16.2.146)' can't be established.
RSA key fingerprint is a8:ec:93:18:1c:e4:66:67:76:e0:73:ab:5e:90:e8:4b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sles102,172.16.2.146' (RSA) to the list of known hosts.
root@sles102's password:
authorized_keys                                                                            100%  619     0.6KB/s   00:00
[root@sles101 .ssh]#
b. on sles102 , generate the DSA and RSA keys for "sles102" using same command lines.
c. Then merge the contents of "authorized_keys" and copied it to both nodes.
Now the "authorized_keys" file on both nodes contains the public keys for each other.
d. Running "ssh" session for all public and private hostnames of both RAC nodes (generating the "known_hosts" file)
[root@sles101 .ssh]# ssh sles101 date
[root@sles101 .ssh]# ssh sles102 date
[root@sles101 .ssh]# ssh sles101-priv date
[root@sles101 .ssh]# ssh sles102-priv date
[root@sles102 .ssh]# ssh sles101 date
[root@sles102 .ssh]# ssh sles102 date
[root@sles102 .ssh]# ssh sles101-priv date
[root@sles102 .ssh]# ssh sles102-priv date
e. Repeat step a-d to generate authorized_keys file for user 'oracle'
(Tips: running "ssh sles101 date" instead of "ssh sles101" so that it will not run into the ssh session)
Part2
- Installing CRS
- Configuring VIP (vipca)
-----------------------------------------------------------
Installing Oracle Clusterware (CRS)
-----------------------------------------------------------
1. copy the source zip file onto the source location or on an NFS share.
2. Before installing CRS (clusterware), it is better to run the prerequisites scr¨©pts:
[oracle@sles101 /mnt/oracle/clusterware/cluvfy]
$ ./runcluvfy.sh stage -pre crsinst -n sles101,sles102 -verbose
Performing pre-checks for cluster services setup
....
Checking node reachability...
Check: Node reachability from node "sles101"
  Destination Node                      Reachable?            
  ------------------------------------  ------------------------
  sles101                               yes                   
  sles102                               yes       
 
Result: Node reachability check passed from node "sles101".
Checking user equivalence...
Check: User equivalence for user "oracle"
  Node Name                             Comment               
  ------------------------------------  ------------------------
  sles102                               passed                
  sles101                               passed                
Result: User equivalence check passed for user "oracle".
Checking administrative privileges...
Check: Existence of user "oracle"
  Node Name     User Exists               Comment               
  ------------  ------------------------  ------------------------
  sles102       yes                       passed                
  sles101       yes                       passed                
Result: User existence check passed for "oracle".
Check: Existence of group "oinstall"
  Node Name     Status                    Group ID              
  ------------  ------------------------  ------------------------
  sles102       exists                    106                   
  sles101       exists                    106                   
Result: Group existence check passed for "oinstall".
Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Comment   
  ----------------  ------------  ------------  ------------  ------------  ------------
  sles102           yes           yes           yes           yes           passed    

www.ixdba.net

  sles101           yes           yes           yes           yes           passed    
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity...
Interface information for node "sles102"
  Interface Name                  IP Address                      Subnet        
  ------------------------------  ------------------------------  ----------------
  eth0                            172.16.2.146                    172.16.2.0    
  eth1                            192.168.235.146                 192.168.235.0 
Interface information for node "sles101"
  Interface Name                  IP Address                      Subnet        
  ------------------------------  ------------------------------  ----------------
  eth0                            172.16.2.145                    172.16.2.0    
  eth1                            192.168.235.145                 192.168.235.0 
Check: Node connectivity of subnet "172.16.2.0"
  Source                          Destination                     Connected?    
  ------------------------------  ------------------------------  ----------------
  sles102:eth0                    sles102:eth0                    yes           
  sles102:eth0                    sles101:eth0                    yes           
  sles102:eth0                    sles101:eth0                    yes           
  sles101:eth0                    sles101:eth0                    yes           
Result: Node connectivity check passed for subnet "172.16.2.0" with node(s) sles102,sles101.
Check: Node connectivity of subnet "192.168.235.0"
  Source                          Destination                     Connected?    
  ------------------------------  ------------------------------  ----------------
  sles102:eth1                    sles101:eth1                    yes           
Result: Node connectivity check passed for subnet "192.168.235.0" with node(s) sles102,sles101.
Suitable interfaces for the private interconnect on subnet "172.16.2.0":
sles102 eth0:172.16.2.146
sles101 eth0:172.16.2.145
Suitable interfaces for the private interconnect on subnet "192.168.235.0":
sles102 eth1:192.168.235.146
sles101 eth1:192.168.235.145
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.
Checking system requirements for 'crs'...
Check: Total memory
  Node Name     Available                 Required                  Comment 
  ------------  ------------------------  ------------------------  ----------
  sles102       1011.04MB (1035300KB)     512MB (524288KB)          passed  
  sles101       1011.04MB (1035300KB)     512MB (524288KB)          passed  
Result: Total memory check passed.
Check: Free disk space in "/tmp" dir
  Node Name     Available                 Required                  Comment 
  ------------  ------------------------  ------------------------  ----------
  sles102       437.9MB (448412KB)        400MB (409600KB)          passed  
  sles101       400.22MB (409824KB)       400MB (409600KB)          passed  
Result: Free disk space check passed.
Check: Swap space
  Node Name     Available                 Required                  Comment 

www.ixdba.net

  ------------  ------------------------  ------------------------  ----------
  sles102       1.89GB (1982456KB)        1GB (1048576KB)           passed  
  sles101       1.89GB (1982456KB)        1GB (1048576KB)           passed  
Result: Swap space check passed.
Check: System architecture
  Node Name     Available                 Required                  Comment 
  ------------  ------------------------  ------------------------  ----------
  sles102       i686                      i686                      passed  
  sles101       i686                      i686                      passed  
Result: System architecture check passed.
Check: Kernel version
  Node Name     Available                 Required                  Comment 
  ------------  ------------------------  ------------------------  ----------
  sles102       2.6.16.21-0.8-smp         2.4.21-15EL               passed  
  sles101       2.6.16.21-0.8-smp         2.4.21-15EL               passed  
Result: Kernel version check passed.
Check: Package existence for "make-3.79"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         make-3.80-202.2                 passed        
  sles101                         make-3.80-202.2                 passed        
Result: Package existence check passed for "make-3.79".
Check: Package existence for "binutils-2.14"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         binutils-2.16.91.0.5-23.4       passed        
  sles101                         binutils-2.16.91.0.5-23.4       passed        
Result: Package existence check passed for "binutils-2.14".
Check: Package existence for "gcc-3.2"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         gcc-4.1.0-28.4                  passed        
  sles101                         gcc-4.1.0-28.4                  passed        
Result: Package existence check passed for "gcc-3.2".
Check: Package existence for "glibc-2.3.2-95.27"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         glibc-2.4-31.2                  passed        
  sles101                         glibc-2.4-31.2                  passed        
Result: Package existence check passed for "glibc-2.3.2-95.27".
Check: Package existence for "compat-db-4.0.14-5"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         missing                         failed        
IXDBA.NETÉçÇøÂÛ̳
  sles101                         missing                         failed        
Result: Package existence check failed for "compat-db-4.0.14-5".
Check: Package existence for "compat-gcc-7.3-2.96.128"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         compat-gcc-7.3-2.96.128         passed        
  sles101                         compat-gcc-7.3-2.96.128         passed        
Result: Package existence check passed for "compat-gcc-7.3-2.96.128".
Check: Package existence for "compat-gcc-c++-7.3-2.96.128"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         compat-gcc-c++-7.3-2.96.128     passed        
  sles101                         compat-gcc-c++-7.3-2.96.128     passed        
Result: Package existence check passed for "compat-gcc-c++-7.3-2.96.128".
Check: Package existence for "compat-libstdc++-7.3-2.96.128"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         compat-libstdc++-5.0.7-20compat-libstdc++-7.3-2.96.128  failed        
  sles101                         compat-libstdc++-5.0.7-20compat-libstdc++-7.3-2.96.128  failed        
Result: Package existence check failed for "compat-libstdc++-7.3-2.96.128".
Check: Package existence for "compat-libstdc++-devel-7.3-2.96.128"
  Node Name                       Status                          Comment    
  ------------------------------  ------------------------------  ----------------
  sles102                         compat-libstdc++-devel-7.3-2.96.128  passed        
  sles101                         compat-libstdc++-devel-7.3-2.96.128  passed        
Result: Package existence check passed for "compat-libstdc++-devel-7.3-2.96.128".
Check: Package existence for "openmotif-2.2.3"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         openmotif-2.2.4-21.2            passed        
  sles101                         openmotif-2.2.4-21.2            passed        
Result: Package existence check passed for "openmotif-2.2.3".
Check: Package existence for "setarch-1.3-1"
  Node Name                       Status                          Comment       
  ------------------------------  ------------------------------  ----------------
  sles102                         setarch-1.3-1                   passed        
  sles101                         setarch-1.3-1                   passed        
Result: Package existence check passed for "setarch-1.3-1".
Check: Group existence for "dba"
  Node Name     Status                    Comment               
  ------------  ------------------------  ------------------------
  sles102       exists                    passed                
IXDBA.NET¼¼ÊõÉçÇø
  sles101       exists                    passed                
Result: Group existence check passed for "dba".
Check: Group existence for "oinstall"
  Node Name     Status                    Comment               
  ------------  ------------------------  ------------------------
  sles102       exists                    passed                
  sles101       exists                    passed                
Result: Group existence check passed for "oinstall".
Check: User existence for "nobody"
  Node Name     Status                    Comment               
  ------------  ------------------------  ------------------------
  sles102       exists                    passed                
  sles101       exists                    passed                
Result: User existence check passed for "nobody".
System requirement failed for 'crs'
Pre-check for cluster services setup was unsuccessful on all the nodes.
There are 3 errors in this verification:
- Error on VIP: this error could be ignored becuase VIPs have not been configured at this time, and also becuase the public IPs are located in the provate subnets (172.16.2.0/24)
- package compat-db-4.0.14-5 was not installed, but this doesn't matter.
- package compat-libstdc++ has 2 versions, becuase different libaray files are required.

1.Unzip 10201_clusterware_linux32.zip and 10201_database_linux32.zip
2.Install cluster software£º
Logon with user "oracle"
3. Unset variables
$unset ORA_CRS_HOME
$unset ORACLE_HOME
$unset ORA_NLS10
$unset TNS_ADMIN
4.Start installation GUI on sles101
$ /mnt/oracle/clusterware/runInstaller [-ignoreSysPrereqs] &

[Note:] Before running the installer, it is better to sychronize the date and time on all RAC nodes. Setup NTP client and query time from a local Winodows domain controller, or change the time manually.
If you want to redirect the X-windows GUI to Microsoft Windows lient machine, install X-Deep32 or Xmanager2.0 on your windows desktop, and use "SSH Secure Shell" client to start installation.
¡ñ Welcome£º
Click £ÛNext£Ý
¡ñ Specify Inventory directory and credentials£º
Inventory directory: /u01/app/oracle/oraInventory
Operating System group name: dba
Click £ÛNext£Ý
¡ñ Specify Home Details£º
Name: OraCrs10g_home
Location: /u01/app/oracle/product/10.2/crs_1
Click £ÛNext£Ý
¡ñ Product Specific Prerequisite Checks£º
Click £ÛNext£Ý
¡ñ Specify Cluster Configuration£º
Cluster Name: crs
--------------------------------------------------------
|Public Node Name |Private Node Name |Virtual Node Name|
--------------------------------------------------------
|  sles101  |  sles101-priv  |  sles101-vip  |  
--------------------------------------------------------
|  sles102  |  sles102-priv  |  sles102-vip  |
--------------------------------------------------------
Click £ÛNext£Ý
¡ñ Specify Network Interface Usage£º
--------------------------------------------------
| Interface Name | Subnet | Interface Type |
--------------------------------------------------
| eth0 | 172.16.2.0 | Public |
--------------------------------------------------
| eth1 | 192.168.235.0 | Private |
--------------------------------------------------
Click £ÛNext£Ý
¡ñ Specify OCR Location£º[Oracle Cluster Registry]
Click External Redundacy
Specify OCR Location: /u02/oradata/ocr
Click£ÛNext£Ý
¡ñ Specify Voting Disk Location£º
Click External Redundacy
Voting Disk Location: /u02/oradata/css
Click £ÛNext£Ý
¡ñ Summary£º
Click £ÛInstall£Ý
¡ñ Execute Configuration scr¨©pts£º Do not lick "OK" and close this windoe until you finish running the 2 scr¨©pts.
scr¨©pts to be executed on each node by "root" user:
a. /u01/app/oracle/oraInventory/orainstRoot.sh
[root@sles101 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to dba.
The execution of the scr¨©pt is complete
b. /u01/app/oracle/product/crs/root.sh
[root@sles101 crs]# ./root.sh
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
assigning default hostname sles101 for node 1.
assigning default hostname sles102 for node 2.
Successfull