Public, VIPs and SCAN VIPs are resolved by DNS. The
private IPs for Cluster Interconnects are resolved through /etc/hosts. The hostname along with public/private and NAS network
is configured at the time of OEL network installations. The final Network
Configurations files are listed here.

This package is
located in the rpm directory on Clusterware Media and needs to be installed
after the group oinstall is created. In my case, as this was a fresh install of
10g R2 on a new machines, old versions of cvuqdisk was not present. If it is,
then the older version needs to be removed first.

export CVUQDISK_GRP=oinstall

echo $CVUQDISK

rpm –ivh cvuqdisk-1.0.1-1.rpm

[root@node1 rpm]# pwd

/home/oracle/10gr2/clusterware/rpm

[root@node1 rpm]# export CVUQDISK_GRP=oinstall

[root@node1 rpm]# echo $CVUQDISK_GRP

oinstall

[root@node1 rpm]# rpm -ivh rpm –ivh cvuqdisk-1.0.1-1.rpm

Preparing...########################################### [100%]

1:cvuqdisk########################################### [100%]

[root@node1 rpm]# rpm -qa | grep cvuqdisk

cvuqdisk-1.0.1-1

[root@node1 rpm]#

Configure Oracle RAC Owner’s User Environment (oracle):

(a)Set the umask
to 022 by putting below line into these users’ (oracle) .bash_profile files:

umask 022

Then, executed the .bash_profile and verified
that the correct value of umask is displayed.

[oracle@node1 ~]$ . .bash_profile

[oracle@node1 ~]$ umask

(b)Setting up X11
forwarding:

Created the file ~/.ssh/config to
disable the X11Forwadding by placing below line in it.

Host *

ForwardX11 no

(c)Suppressed
the Terminal output on STDOUT and STDERR to prevent Installation errors:

Modified the file ~/.bashrc (or .cshrc for C shell) with below entry.

Bourne, Bash, or Korn
shell:

if [ -t 0 ]; then

stty intr ^C

fi

C shell:

test -t 0

if ($status == 0) then

stty intr ^C

endif

(d)Increased the
Shell Limits:

Recommended:

Resource

Soft Limit

Hard Limit

Processes

2047

16384

Open File
Descriptors

1024

65536

Stack

10240

10240- 32768

Set:

Resource

Soft Limit

Hard Limit

Processes

131072

131072

Open File
Descriptors

131072

131072

Stack

32768

32768

Added the following
lines to the /etc/security/limits.conf file:

oraclesoftnofile131072

oraclehardnofile131072

oraclesoftnproc131072

oraclehardnproc131072

oraclesoftcoreunlimited

oraclehardcoreunlimited

oraclesoftmemlock3500000

oraclehardmemlock3500000

#
Recommended stack hard limit 32MB for oracle installations

#
oraclehardstack32768

Added the following
line in the /etc/pam.d/login file, if it does not already exist:

session
required /lib/security/pam_limits.so

For the Bourne, Bash,
or Korn shell,add the following lines
to the /etc/profile:

if [ $USER = "oracle" ];
then

if [ $SHELL = "/bin/ksh"
]; then

ulimit -p 131072

ulimit -n 131072

else

ulimit -u 131072 -n 131072

fi

fi

For the C shell (csh or
tcsh), add the following lines to the /etc/csh.login.

if ( $USER == "oracle" )
then

limit maxproc 131072

limit descriptors 131072

endif

(e)Set the below
Kernel Parameters with recommended range in /etc/sysctl.conf. This
was already set with the installation of oracle-validated package. I had to uncomment
the lines related to 10g parameters and comment the same parameters for 11g.
After the modification, ran the sysctl –p to enable the changes..

/etc/sysctl.conf

# Kernel sysctl configuration file for Oracle
Enterprise Linux

#

# For binary values, 0 is disabled, 1 is
enabled.See sysctl(8) and

# sysctl.conf(5) for more details.

# Controls IP packet forwarding

net.ipv4.ip_forward = 0

# Controls source route verification

net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing

net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging
functionality of the kernel

kernel.sysrq = 1

# Controls whether core dumps will append the PID to
the core filename

# Useful for debugging multi-threaded applications

kernel.core_uses_pid = 1

# Controls the use of TCP syncookies

net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes

kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage
queue

kernel.msgmax = 8192

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 4294967295

# Controls the maximum number of shared memory
segments, in pages

kernel.shmall = 1073741824

# For 11g, recommended value for file-max is 6815744

#fs.file-max = 6815744

# For 10g, uncomment 'fs.file-max 327679', comment
other entries for this parameter and re-run sysctl -p

fs.file-max =
327679

kernel.msgmni = 2878

kernel.sem = 250 32000 100 142

kernel.shmmni = 4096

net.core.rmem_default = 262144

# For 11g, recommended value for net.core.rmem_max
is 4194304

#net.core.rmem_max = 4194304

# For 10g, uncomment 'net.core.rmem_max 2097152',
comment other entries for this parameter and re-run sysctl -p

net.core.rmem_max=2097152

net.core.wmem_default = 262144

# For 11g, recommended value for wmem_max is 1048576

#net.core.wmem_max = 1048576

# For 10g, uncomment 'net.core.wmem_max 262144',
comment other entries for this parameter and re-run sysctl -p

(e)Reloaded
the udev rules and started the udev services to make the changes effective.

udevcontrol reload_rules

start_udev

(f)Verified
that these disks now labeled appropriately

(g)Created
the Raw Bindings for OCR and Voting Devices (On All the RAC Nodes).

Placed
the below lines into the /etc/sysconfig/rawdevices

# raw device bindings

# format:<rawdev> <major> <minor>

#<rawdev> <blockdev>

# example: /dev/raw/raw1 /dev/sda1

#/dev/raw/raw2 8 5

/dev/raw/raw1 /dev/ocrdisk01

/dev/raw/raw2 /dev/ocrdisk02

/dev/raw/raw3 /dev/votingdisk01

/dev/raw/raw4 /dev/votingdisk02

/dev/raw/raw5 /dev/votingdisk03

Change
the permissions and ownership of the raw devices as shown below.

chown root:oinstall /dev/raw/raw1

chown root:oinstall /dev/raw/raw2

chown oracle:oinstall /dev/raw/raw3

chown oracle:oinstall /dev/raw/raw4

chown oracle:oinstall /dev/raw/raw5

chmod 640 /dev/raw/raw1

chmod 640 /dev/raw/raw2

chmod 640 /dev/raw/raw3

chmod 640 /dev/raw/raw4

chmod 640 /dev/raw/raw5

Restart
the service rawdevices on all the nodes.

service rawdevices restart

Because
the ownerships and permissions on the raw devices get reset to root on every
reboot, the above lines were placed into /etc/rc.local
to set the permissions back to oracle user after the reboot. (ON all the RAC
Nodes):

/etc/rc.local:

[root@node1 sysconfig]# cat /etc/rc.local

#!/bin/sh

#

# This script will be executed *after* all the
other init scripts.

# You can put your own initialization stuff in
here if you don't

# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

chown root:oinstall /dev/raw/raw1

chown root:oinstall /dev/raw/raw2

chown oracle:oinstall /dev/raw/raw3

chown oracle:oinstall /dev/raw/raw4

chown oracle:oinstall /dev/raw/raw5

chmod 640 /dev/raw/raw1

chmod 640 /dev/raw/raw2

chmod 640 /dev/raw/raw3

chmod 640 /dev/raw/raw4

chmod 640 /dev/raw/raw5

Configure the iSCSI
disk Devices for Oracle ASM with ASMLib:

(a)Verified that the below RPMs are installed before
configuring the ASM driver.

oracleasm-2.6.18-194.el5-2.0.5-1.el5

oracleasm-support-2.1.3-1.el5

oracleasm-2.6.18-194.el5xen-2.0.5-1.el5

oracleasm-2.6.18-194.el5PAE-2.0.5-1.el5

oracleasm-2.6.18-194.el5debug-2.0.5-1.el5

oracleasmlib-2.0.4-1.el5

(a)Configured the
ASMLib (All the RAC nodes):

Provide the oracleasm owner “oracle” and group “oinstall”
while configuring the oracleasm.

oracleasm configure –i

(e)Loaded the ASMLib Module (All the RAC nodes):

[root@node1 ~]# oracleasm init

(f)Created the ASM disks using oracleasm: (ONLY
from one of the RAC nodes)

oracleasm
createdisk DATADSK01 /dev/sdd1

oracleasm
createdisk FRADSK02 /dev/sde1

oracleasm
scandisks

oracleasm
listdisks

(g)On the Remaining RAC nodes, simply scanned the ASM disks
to instantiate these newly created disks

oracleasm
scandisks

oracleasm
listdisks

(h)Verified that these ASM
disk can be discovered by ASM Libraries (oracleasmlib) on all the RAC
Nodes during Creation of ASM Disks.