In this phase, the 3-node 11gR2 Grid Infrastructure was installed
with the below information by following my note on “Installing 11gR2 RAC on Linux”.

There, I had created ASM Diskgroups DATA and FRA with the size of 60G and 10G
respectively.

Grid Infrastructure Software (Clusterware + ASM
11.2.0.1):

Server: All the RAC Nodes

ORACLE_BASE: /u01/app/oracle

ORACLE_HOME:/u01/app/grid11201

Owner: oracle (Primary Group: oinstall, Secondary Group:
dba)

Permissions: 755

OCR/Voting Disk Storage Type: ASM

Oracle Inventory Location: /u01/app/oraInventory

Install
11gR1 RAC HOME:

The EBS R 12.1.1 software comes with the default database
version 11.1.0.7. So, I had to create the RAC HOME with the same patch level as
the seeded Home. The below database software were downloaded from Oracle’s
Website and installed them in the order listed by creating the new 11gR1 RAC
(11.1.0.7) HOME /u01/app/oracle/EBSR12.

This step
is not required but I would always set up the ssh user equivalency for ease of
maintenance.

On All
the Apps Nodes:

su - applmgr

mkdir ~/.ssh

chmod 700 ~/.ssh

Generate the RSA and DSA keys:

/usr/bin/ssh-keygen
-t rsa

/usr/bin/ssh-keygen
-t dsa

On appsnode1:

touch
~/.ssh/authorized_keys

cd ~/.ssh

(a)Add these
Keys to the Authorized_keys file.

cat
id_rsa.pub >> authorized_keys

cat
id_dsa.pub >> authorized_keys

(b)Send this file
to appsnode2.

scp
authorized_keys appsnode2:.ssh/

On appsnode2:

(a)Add these
Keys to the Authorized_keys file.

cd
~/.ssh

cat
id_rsa.pub >> authorized_keys

cat
id_dsa.pub >> authorized_keys

(b)Send this file
to appsnode1.

scp
authorized_keys appsnode1:.ssh/

On both the Apps Nodes:

chmod
600 ~/.ssh/authorized_keys

ssh
appsnode1 date

ssh
appsnode2 date

ssh
appsnode1.hingu.net date

ssh
appsnode2.hingu.net date

Entered
'yes' and continued when prompted

Download and Stage the R12 Software on one of
APPS as well as DB Node:

(1)Downloaded the
E-Business Suite software Release 12.1.1 from otn.oracle.com and copied them to
the /tmp directory as shown here.

(2)Created the Staging
Directory /u01/StageR12
to stage the R12 Software.

(3)Created the script
called /u01/StageR12/unzip.sh to unzip the downloaded software to the Stage Directory.

mkdir /u01/StageR12

cd /u01/StageR12

for i in `ls -1 /tmp/R12_for_Linux/*/*.zip`; do echo
unzip $i; done > unzip.sh

for i in `ls -1 /tmp/R12_for_Linux/*/*/*.zip`; do
echo unzip $i; done >> unzip.sh

chmod 755 unzip.sh

[applmgr@appsnode1 StageR12]$ cat unzip.sh

unzip /tmp/R12_for_Linux/Start/B53824-01_1of4.zip

unzip /tmp/R12_for_Linux/Start/B53824-01_2of4.zip

unzip /tmp/R12_for_Linux/Start/B53824-01_3of4.zip

unzip /tmp/R12_for_Linux/Start/B53824-01_4of4.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_1of3.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_2of3.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD1/V15573-01_3of3.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_1of3.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_2of3.zip

unzip
/tmp/R12_for_Linux/APPL_TOP/DVD2/V15574-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD1/V15564-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD1/V15564-01_2of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD1/V15564-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD2/V15565-01_1of2.zip

unzip
/tmp/R12_for_Linux/Databases/DVD2/V15565-01_2of2.zip

unzip
/tmp/R12_for_Linux/Databases/DVD3/V15566-01_1of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD3/V15566-01_2of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD3/V15566-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD4/V15567-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD4/V15567-01_2of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD4/V15567-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD5/V15568-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD5/V15568-01_2of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD5/V15568-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD6/V15569-01_1of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD6/V15569-01_2of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD6/V15569-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD7/V15570-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD7/V15570-01_2of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD7/V15570-01_3of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD8/V15571-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD8/V15571-01_2of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD8/V15571-01_3of3.zip

unzip /tmp/R12_for_Linux/Databases/DVD9/V15572-01_1of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD9/V15572-01_2of3.zip

unzip
/tmp/R12_for_Linux/Databases/DVD9/V15572-01_3of3.zip

unzip
/tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_1of3.zip

unzip
/tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_2of3.zip

unzip /tmp/R12_for_Linux/RDBMS/DVD1/V15576-01_3of3.zip

unzip /tmp/R12_for_Linux/Tools/DVD1/V15575-01.zip

[applmgr@appsnode1 StageR12]$

(4)Ran the unzip.sh
script from the Stage directory (/u01/StageR12)

cd /u01/StageR12

./unzip.sh

During the unzipping
process, when prompted, the dvd.label and component.label files were replaced.
You can choose not to overwrite them as well because the dvd.label is same for
all the DVDs belonging to the same category. For e.g, Databases has 9 DVDs and
each of these DVDs dvd.label file is same.

cd /u01/StageR12

rm ora*/component.label

rm unzip.sh

After
unzipping all the DVDs by running the unzip.sh script, the
following directory structure was created.

With this, the pre-Installation steps are completed
successfully and are ready to proceed with the Fresh Install of EBS R12 Next.

The Fresh Installation
of R12 Database using Rapid Install cannot be directly installed on RAC HOME.
It comes with non-RAC single instance binaries. So, I connected to one of the
RAC Nodes (node1.hingu.net) and started the Rapid Install which installed the
local non-RAC 11gR1 (11.1.0.7) Database. During the later stage, this database
was then converted to RAC.

Using the X
Terminal, Invoked the /u01/StageR12/startCD/Disk1/rapidwiz/rapidwiz.

This
patch needs to be applied to use the named listener on the DB tier. Without
this patch, the listener name has to be LISTENER and I wanted to use the name LABDB_LISTENER for
the application database listener so I decided to apply this patch.

a)Downloaded
the patch 8919489.

b)Downloaded
the Patch 9583541 (post requirement for the patch 8919489).

c)Stopped
the Application on both the nodes.

d)Applied
the patch 8919489 using adpatch by following its README on appsnode1

e)Applied
the patch 8919489 using adpatch by following its README on appsnode2

f)Ran
the AutoConfig again on Admin Node (appsnode1.hingu.net) after applying the patch on appsnode2.

g)Tried
to start the Application on appsnode1.hingu.net and it got warning message saying it could not
deploy the latest EAR file.

The
Forms were manually deployed by following the Metalink note “Deploying a New
forms.ear File in Oracle Applications Release 12 [ID 397174.1]”. While running
the script to deploy the forms, it got errored out with the below error message
complaining it could not find the opmnctl executable.

TXK::Process::run('TXK::Process=HASH(0xb7cad114)','HASH(0x9756fe4)')
called at /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl
line 1587

TXK::RunScript::execOPMNControl('HASH(0x9a3b224)')
called at
/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl
line 599

require
/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl
called at
/u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/RunScript.pm line
105

TXK::RunScript::require('TXK::RunScript','/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115...')
called at /u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm
line 177

eval
{...} called at
/u01/app/oracle/ebsR12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177

TXK::Script::run('TXK::Script=HASH(0x9b1c980)','/u01/app/oracle/ebsR12/inst/apps/LABDB_appsnode1/logs/appl/rg...','/u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/patch/115...')
called at /u01/app/oracle/ebsR12/apps/apps_st/appl/fnd/12.0.0/bin/txkrun.pl
line 174

While
further debugging, it was found out that the ORACLE_HOME and TNS_ADMIN
variables in $IAS_ORACLE_HOME/opmn/bin/opmnctl
were pointing to /nfs/bld/d26/PRDXBLD9/apps/tech_st/10.1.3 instead of /u01/app/oracle/ebsR12/apps/tech_st/10.1.3.
These variables were correctly modified and then continued with the Note
397174.1 to deploy the forms.ear and after that, it got
deployed successfully.

After
that, the Application services were started on both the application nodes using
$INST_TOP/admin/scripts/adstrtal.sh and
tested logging in to the application and forms and the results were all
successful.

Added
the TNS_ADMIN entry in the /u01/app/oracle/EBSR12/bin/racgwrap
(NEW_11GR1_RAC_HOME) file right under the ORACLE_HOME entry.

node1:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export
TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1

node2:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export
TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB2_node2

node3:/u01/app/oracle/EBSR12/bin/racgwrap

ORACLE_HOME=/u01/app/oracle/EBSR12

export ORACLE_HOME

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB3_node3

From
RAC Node 1 (node1.hingu.net):

Stopped
the listener LABDB created during the default installation on port 1522 from
the OLD_11GR1_HOME.

Set
the TNS_ADMIN for node1 as
shown below and invoked the netca from NEW_11GR1_RAC_HOME to add the listener LABDB_LISTENER on
RAC node node1

export
TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB1_node1

/u01/app/oracle/EBSR12/bin/netca

Select
the “Cluster Configuration”

Select
node1

Select
“Listener Configuration”

Select
“Add”

Listener
Name: LABDB_LISTENER

Selected
Protocol: TCP

Port:
1522

Finish

Now,
set the TNS_ADMIN for node 2 as shown below and invoked the same netca again
add the listener LABDB_LISTENER on
RAC node node2.

export TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB2_node2

/u01/app/oracle/EBSR12/bin/netca

Select
the “Cluster Configuration”

Select
node2

Select
“Listener Configuration”

Select
“Add”

Listener
Name: LABDB_LISTENER

Selected
Protocol: TCP

Port:
1522

Finish

Now,
set the TNS_ADMIN for node 3 as shown below and invoked the same netca again to
add the listener LABDB_LISTENER on
RAC node node3.

export
TNS_ADMIN=/u01/app/oracle/EBSR12/network/admin/LABDB3_node3

/u01/app/oracle/EBSR12/bin/netca

Select
the “Cluster Configuration”

Select
node3

Select
“Listener Configuration”

Select
“Add”

Listener
Name: LABDB_LISTENER

Selected
Protocol: TCP

Port:
1522

Finish

Created the tnsnames.ora under
the NEW_11GR1_RAC_HOME:

node1:/u01/app/oracle/EBSR12/network/admin/LABDB1_node1/tnsnames.ora

node2:/u01/app/oracle/EBSR12/network/admin/LABDB2_node2/tnsnames.ora

node3:/u01/app/oracle/EBSR12/network/admin/LABDB3_node3/tnsnames.ora

The
above mentioned tnsnames.ora files were created with the below contents on each
of the database nodes. The IFILE entries were the only entry that was node
specific and were adjusted appropriately on each node. The
$TNS_ADMIN/<SID>_<nodename>_ifile.ora was created on each node with
the same below information.

The Instance
Connectivity from each of the RAC instances was verified using SQLPLUS from RAC
nodes individually. I would not run adconfig.pl without having successful
connection to each of the RAC instances.

RAC Instance Connections
Verification:

The
Instance Connectivity from each of the RAC instances was verified using SQLPLUS
from RAC nodes individually. The connectivity was confirmed before running the
adconfig.pl on each of the RAC instances.

On all
the RAC Nodes, ran the /u01/app/oracle/EBSR12/nls/data/old/cr9idata.pl
script to create the /u01/app/oracle/EBSR12/nls/data/9idata
directory. Without this directory, the adconfig.sh on DB tier may fail on this.

Ran
the /u01/app/oracle/EBSR12/appsutil/bin/adbldxml.pl
to build the XML on each of the RAC Nodes and provided the appropriate
values of DB Host Name, Database Port, SID, and Service Name. This will
generate the context file <SID>_hostname.xml under /u01/app/oracle/EBSR12/appsutil.

[oracle@node1 bin]$ ./adbldxml.pl

Starting context file generation for db tier..

Using JVM from
/u01/app/oracle/EBSR12/appsutil/jre/bin/java to execute java programs..

APPS Password: apps

The log file for this adbldxml session is
located at:

/u01/app/oracle/EBSR12/appsutil/log/adbldxml_10311041.log

UnsatisfiedLinkError exception loading native
library: njni11

Could not Connect to the Database with the
above parameters, Please answer the Questions below

Enter Hostname of Database server: node1

Enter Port of Database server: 1522

Enter SID of Database server: LABDB1

Enter Database Service Name: LABDB

Enter the value for Display Variable: 1

The context file has been created at:

/u01/app/oracle/EBSR12/appsutil/LABDB1_node1.xml

Once
the context file got generated, the below variables were modified (in RED) to
their appropriate values.

The
AutoConfig was again ran on all the RAC Nodes using below command in the
reverse order so that all the nodes’ tnsnames.ora file contains the same
information.

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB3_node3/adautocfg.shß for node3

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB2_node2/adautocfg.shß for node2

/u01/app/oracle/EBSR12/appsutil/scripts/LABDB1_node1/adautocfg.shß for node1

Modify the
$ORACLE_HOME/dbs/<SID>_APPS_BASE.ora on all the RAC Nodes:

The
<SID>_APPS_BASE.ora file under the /u01/app/oracle/EBSR12/dbs contains
the wrong values for the control_files parameter. So, this file was modified on
all the RAC nodes with the correct value of control_files parameter.

At this
stage, the FND Nodes contained the RAC nodes information in the application
database.

Run
the AutoConfig on the Application nodes.

After
enabling and running the AutoConfig successfully on the DB tier RAC nodes, it
is required to run the autoconfig on all the Application nodes so that they can
generate the new RAC specific TNS and jdbc_url entry. I followed the below
steps to accomplish this requirement.

·Modify $TNS_ADMIN/tnsnames.ora file and jdbc_url in the CONTEXT_FILE.

·Check the connection to the database after the above
modification.

·Run the AutoConfig

·Repeat the above 3 steps on remaining application
node appsnode2.

·At last, run the AutoConfig on the Admin Node (appsnode1).

Modify the tnanames.ora and
jdbc_url:

The
CONNECT_DATA entry in the $TNS_ADMIN/tnsnames.ora file was modified to have SERVICE_NAME and
INSTANCE_NAME value for all the TNS Alias belonging to Database instance. (On
both the Application Nodes appsnode1 and appsnode2)

LABDB=

(DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=node1.hingu.net)(PORT=1522))

(CONNECT_DATA=

(SERVICE_NAME=LABDB)

(INSTANCE_NAME=LABDB1)

)

LABDB_BALANCE=

(DESCRIPTION=

(ADDRESS_LIST=

(LOAD_BALANCE=YES)

(FAILOVER=YES)

(ADDRESS=(PROTOCOL=tcp)(HOST=node1.hingu.net)(PORT=1522))

)

(CONNECT_DATA=

(SERVICE_NAME=LABDB)

)

)

Jdbc_url
value in the below context file was modified with the same above information as
shown on both the application nodes appsnode1 and appsnode2.

After
successful completion of autoconfig and re-sourcing the environment, the $TNS_ADMIN/tnsnames.ora file
was containing the tns aliases for all the RAC instances individually and also
had a BALANCE entry containing all the DB VIPs as shown below:

LABDB_BALANCE=

(DESCRIPTION=

(ADDRESS_LIST=

(LOAD_BALANCE=YES)

(FAILOVER=YES)

(ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip.hingu.net)(PORT=1522))

(ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip.hingu.net)(PORT=1522))

(ADDRESS=(PROTOCOL=tcp)(HOST=node3-vip.hingu.net)(PORT=1522))

)

(CONNECT_DATA=

(SERVICE_NAME=LABDB)

)

)

The
new jdbc_url in the CONTEXT_FILE on appsnode1 was reflecting all the 3 VIPs of
database RAC nodes at this stage.