Each
node in the cluster must have 3 network adapter (eth0, eth1and eth2) one for
the public, second one for the private network interface (internodes
communication, interconnect) and third one for the Network Storage System
(Private). You make sure that if you configure eth1 as the private interface
for node1 then, eth1 must be configured as private interface for the node2.

Oracle recommends that you set shared memory segment
attributes as well as semaphores to the following values.. I added the
following lines to /etc/sysctl.conf file. Every OS process needs semaphore
where It waits on for the resources. If the current value for any parameter is
higher than the value listed in this table, then do not change the value of
that parameter.

To see the current setting in the kernel, please use
the below command.

Add or edit the following line in
the /etc/pam.d/login file, if it does not already exist:

session required
/lib/security/pam_limits.so

For the Bourne, Bash, or Korn
shell, add the following lines to the /etc/profile:

if [ $USER =
"oracle" ]; then

if [ $SHELL =
"/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384
-n 65536

fi

fi

For the C shell (csh or tcsh),
add the following lines to the /etc/csh.login.

if ( $USER ==
"oracle" ) then

limit maxproc
16384

limit descriptors
65536

endif

Enable
SSH oracle user Equivalency on Both the Cluster Nodes:

To
configure SSH user equivalency, you must create RSA and DSA keys on each
cluster node and copy these keys
from all the cluster node members into an authorized key file on each node.
Follow the below steps to achieve this task.su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh(A)
Generate the RSA and DSA keys on Each NODE:

/usr/bin/ssh-keygen
-t rsa
/usr/bin/ssh-keygen -t dsa

(B)
on node1-pub:Add
keys to the authorized key file and then send the same file to every nodes in
cluster:

During
executing step B - (1), you may be prompted as show below. Enter
"yes" and continue.

Now,
try executing the date (or any other command) on remote node to make sure that
oracle is not asked for the password. Do the same on both the nodes in cluster.
You should not receive any error message while you execute these commands on
all the nodes.

To fix this, create the /home/oracle/.ssh/config file with the below
contents.

Host *
Forwardx11 no

You
may be prompted to enter 'yes' or 'no' for the first time performing ssh
commands. Simply enter 'yes' and continue. Afterwards, when oracle connects to
the remote node, it should not be asked for the password.

[oracle@node2-pub oracle]$ ssh node2-prv date
Warning: Permanently added the RSA host key for IP address '192.168.203.2' to
the list of known hosts.
Sun Dec 18 20:14:16 CST 2005
[oracle@node2-pub oracle]$ ssh node2-pub date
Sun Dec 18 20:15:05 CST 2005You
may be prompted to enter 'yes' or 'no' for the first time performing ssh
commands. Simply ente'yes' and continue. Afterwards, when oracle connects to
the remote node, it should not be asked for the password.

If
you get then below error message when try to connect to remote node, please
make sure that the firewall is disabled on the remote node.

Every node in the cluster must have access to the shared
disk. So the shared disk must support the concurrent access to all nodes in
cluster in order to successfully build 11g R1 RAC. There are different types of
Storage Management softwares out there that allow you to build NAS/SAN. I have
chosen 'openfiler' because it is Linux 2.6 kernel based Storage Management OS
and it is powered by open source applications such as iSCSI.You can attach any type of external disks to
this server. You can then create volume groups on this device(s) and later
these volume groups are discovered as a new scsi devices on the RAC nodes.

I have followed the above guide to configure the openfiler system and create
the volumes for ASM disks and OCFS disks.
Below are the sample screen shots for my openfiler setup.

The external disk is presented to the server as SCSI disks as shown below. In
my case it is /dev/sda. I have created physical volume on
this device and then created volume group rac11g on entire physical volume.

The below 5 volumes are created under the rac11g volume group.

Also
make sure that each volume allows the shared access to all the nodes in the
cluster. You can do that by clicking the 'Edit' link on the above screen
for each volume name as shown below.

The
below screen shows that both the nodes in the cluster has shared access on the
storage. Click on the General tab to add/modify the rac nodes info. The network
information provided in this table are private network for the shared storage.

At
the End, make sure that iSCSI protocol is enabled in openfiler. You can enable
by clicking on the services tab.

(4) Discovering the volumes to the RAC nodes as scsi devices

NOTE:
make sure that SELinux and firewall has been disabled on all the RAC nodes. If
not, then disable by "Administration Sections"
--> "system settings" --> "security Level"Make Sure that the service iscsi-target is running on nas-server:

[root@nas-server ~]#
service iscsi-target status

ietd (pid 10688) is running...

[root@nas-server ~]#

on
all the RAC nodes:

Make
sure that the iscsi-initiator is installed on RAC nodes. If not, then download
the specified RPM and install them.

[root@node1-pub rpms]#
rpm -qa | grep iscsi

iscsi-initiator-utils-4.0.3.0-5

[root@node1-pub rpms]# service iscsi start -- if not already running

[root@node1-pub rpms]# service iscsi status

iscsid (pid 3256 3253) is running...

Configure
nodes for the Disk Discovery:

Edit
the /etc/iscsi.conf and assign the address of the
iscsi target to the below parameter:

# --------------------------

# Discovery Address Category

# --------------------------

# To localize parameters to targets found on a particular discovery address

# (i.e., to override the global settings), you need to use the

# "DiscoveryAddress" setting. The format for the
"DiscoveryAddress" setting is

# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instance name. An

# example of these settings would be:

#

#DiscoveryAddress=10.4.100.0

# or

#DiscoveryAddress=10.4.100.1:3260

# or

DiscoveryAddress=192.168.1.33 # <-- This is the address of the nas-server
#

(b)

Reboot All the Nodes and run the iscsi-ls command to see if the volumes have
been discovered on the RAC nodes as scsi devices.

(4) Map the Volumes on the iscsi-target (nas-server) to the Disks disceverd on
the local RAC nodes.

Host IDTarget IDdiscovered as

0
iqn.2006-01.com.openfiler:rac11g.ocfs-dsk
---

1
iqn.2006-01.com.openfiler:rac11g.asm-dsk4
---

2
iqn.2006-01.com.openfiler:rac11g.asm-dsk3
---

3
iqn.2006-01.com.openfiler:rac11g.asm-dsk2
---

4
iqn.2006-01.com.openfiler:rac11g.asm-dsk1
---

Now, run the below command to find out the "Attached" devices to the
Host IDs. The scsi Id in this output maps to the Host ID on the
"iscsi-ls" output.

[root@node2-pub rpms]# dmesg | grep Attached

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0

Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0

Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0

Attached scsi disk sde at scsi4, channel 0, id 0, lun 0

In first line, scsi0 (Host Id 0) has device "sda" attached to it. So,
by filling the above table

with this information gives the mapping of discovered disks at client to
its actual Volumes on the iscsi-target.

Host IDTarget IDVolume Namediscovered as

0 iqn.2006-01.com.openfiler:rac11g.ocfs-dsk
ocfs-dsk sda

1
iqn.2006-01.com.openfiler:rac11g.asm-dsk4 asm-dsk4
sdb

2
iqn.2006-01.com.openfiler:rac11g.asm-dsk3 asm-dks3
sdc

3 iqn.2006-01.com.openfiler:rac11g.asm-dsk2
asm-dsk2 sdd

4
iqn.2006-01.com.openfiler:rac11g.asm-dsk1
asm-dsk1 sde

Partitioning the Shared disk:

I
am going to have 4 disks assigned to ASM and one for OCFS. So, I have created a
single partition on these 5 disks. Create partitions from ONLY one of the
available RAC nodes. This can be any node in cluster.

I
have seen that the iscsi devices no longer available on rac nodes after reboot
of nas-server (Openfiler). This is most likely because the volume groups are
not scanned after the reboot on Openfiler server. I have included the necessary
commands to scan and activate these volume groups after every reboot of
openfiler server into the startup script (/etc/rc.local).

Also make sure that firewall and SELinux is disabled on all the machines in
configurations.

SOLUTION: I have Followed the below steps resolves the mentioned issue in my
case.

stop the
iscsi-target on nas-server as root -- service iscsi-target stop

Scan the
systems for the volume groups as root -- vgscan

Activate
the volumes as root -- vgchange -ay

start
the iscsi-target on nas-server as root -- service iscsi-target start

confirm
that volumegroups are available as root -- lvdisplay

On the
client machines (rac nodes), restart the iscsi service as root -- service
iscsi restart

Confirm
that the iscsi devices are available as root -- iscsi-ls

In
/etc/rc.local on nas-server (openfiler) add
below lines:

vgscan
vgchange -ay
service iscsi-target restart

In
/etc/rc.local on each node (rac nodes) add
below line:

service iscsi restart

2. Device name not persistent after reboot of RAC nodes (node1-pub,
node2-pub):

I
have seen that the iscsi device names are changed after reboot of rac nodes.
For e.g, device /dev/sda1 now becomes /dev/sdb1
after reboot. This behavior has caused very serious issues for the OCR and Vote
disks as well as the disks formatted with ocfs2 devices. They don't get mounted
automatically because they are not persistent after reboot. While
goggling on how to overcome this issue on CentOS 4, I came across an utility
called "devlabel" that are developed by Dell Inc and it is available
to download free from Dell's Official Website. This utility creates the symlink
to the device names by considering physical devices' UUID. So, even after the
reboot, the UUID for any devices stays at it is and so the symlink that you
create using devlabel always points to the UUID of the device.

NOTE: You do not need to create symlink using devlabel for the devices that you
are going to use for ASM disks.

(1)Download
the latest rpm of devlabel from Dell's
website and Install it on all the RAC nodes.

I have seen that after reboot of RAC nodes, the devlabel does not get started
automatically and so the ocfs2 mountpoint gets failed because there is no
SYMLINK named /dev/ocfs2 in the kernel that points to ocfs2 disk. So, I have to
reload the SYMLINKs
and mount the ocfs2 device manually after the reboot. as shown below:

To
avoid manual intervention, I added the below entries into start up scripts /etc/rc.local file.

The below is the sample output of my /etc/rc.local

# Affirms the mappings of symlinks
to storage devices. If a
# device cannot be found by its UUID, the existing symlink will
# be deleted. If the device name has changed, it will update the
# symlink. Both start and restart are synonyms for this command.

The
configuration of the ocfs2 resides in the /etc/ocfs2/cluster.conf file that is
generated by running ocfs2console command and then is propagated to
different nodes in the cluster.

Running ocfs2console command line utility brings this screen.

You
can close this box as you can enable the o2cb service later.

Click
Cluster --> Configure Nodes. Add node names for each node one-by-one.
Make sure to add exact same node name as it has been returned by the `hostname`
command.

My
host name names are node1-pub.hingu.net and node2-pub.hingu.net for node 1 and
node 2 respectively.

Propagate
the files to all the nodes in cluster:

So,
the above settings of "Name" (node1-nas and node2-nas) in the ocfs2
configuration will lead to below error when I try to enable o2cb service.

[root@node1-pub
rpms]# /etc/init.d/o2cb enable

Writing O2CB configuration: OK

Starting O2CB cluster ocfs2: Failed

Cluster ocfs2 created

Node node1-nas added

Node node2-nas added

o2cb_ctl: Configuration error discovered while populating cluster ocfs2.
None of its nodes were considered local. A node is considered local when
its node name in the configuration matches this machine's host name.

Stopping O2CB cluster ocfs2: OK

[root@node1-pub rpms]#

So,
stop o2cb service, open the /etc/ocfs2/cluster.conf file and update the
hostname value to the one that is returned by `hostname` command.
Do not update the IP. Start the service and load it again and the error
should go away.