Monday, February 16, 2009

Today morning, we (the DBAs) were faced with a strange issue. On Saturday (the 14th of February - (St.) Valentine's day!!), the UNIX boxes hosting our critical test instance went down for scheduled maintenance without any prior information. So, we could not bring down our databases and applications.

As soon as the unix box came up, we encountered this issue. Let me briefly describe the environment, the issue, the analysis performed and of course, the solution to this most vexing problem.

Environment:

Apps Version: 11.5.10.2 with ATG RUP 5

DB Version: 10.2.0.3

Architecture: Two Node instance with database + admin on one tier and web server on another.

PS: Another instance (development) exists on the same server with the same configuration, which was working fine.

Issue:

When trying to access the login page of Oracle Apps, received the below error. Even OAM login page was throwing the same error.

Request URI: /OA_HTML/AppsLocalLogin.jspException:

java.lang.ExceptionInInitializerError

Request URI:/OA_HTML/AppsLocalLogin.jspException:

java.lang.NoClassDefFoundError

As soon as the services were bounced, we were receiving the first error. After some time, we received the second error (yes, truth is indeed stranger than fiction!!).

Analysis:

Since, we were receiving "NoClassDefFoundError", performed the following:

1. Checked the CLASSPATH and AF_CLASSPATH variable and ensured that every path is valid.

2. Checked all the class paths defined by wrapper.classpath in jserv.properties using the command below:

8. After exhausting all the options, I decided to take a break and promptly headed for lunch!

9. Post lunch, I tried to re-examine the issue from a different perspective. As point 3 above states, I was receiving an error while trying to check the Guest username using aoljtest.jsp. Again checked the guest username and password in the palces mentioned in point 4 above and found them to be consistent.

10. Finally (actually, I had found this note earlier, but ignored it because I found the guest username and password to be correct), I found Note: 458064.1, which I found relevant on three counts:

It mentions about the "java.lang.ExceptionInInitializerError" error.

we were also on ATG RUP 5 (5473858), as the note mentions.

It also mentions that "Verify Guest user" ends with an error (though different from the one that I was receiving).

Thursday, February 5, 2009

When passing a variable in unix, the general convention is to use $variable. Although this does suffice in most cases but in some rare cases, it gives connotations totally different than what was originally intended, as shown in the below example.

The requirement

Copy all files ending with ver to ver_old. For example, suppose I have a file called formver, I would like to copy it to formver_old.

The code

below is the code that I wrote to achieve the above requirement.

for i in `ls *ver`

do

cp -p $i $i_old

done

The pitfall

A cursory glance at the code indicates that it should run smoothly like a river. But it doent' and herein lies the catch. Let's have a look at the error and figure out what exactly is wrong with the above code.__________________________________________________________________

$ for i in `ls *ver`

> do

> cp -p $i $i_old

> done

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1 f2

cp [-f] [-i] [-p] [-@] f1 ... fn d1

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1 f2

cp [-f] [-i] [-p] [-@] f1 ... fn d1

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1

cp [-f] [-i] [-p] [-@] f1 ... fn

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1

cp [-f] [-i] [-p] [-@] f1 ... fn

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1

cp [-f] [-i] [-p] [-@] f1 ... fn

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

cp: Insufficient arguments (1)

Usage: cp [-f] [-i] [-p] [-@] f1

cp [-f] [-i] [-p] [-@] f1 ... fn

cp -r-R [-H-L-P] [-f] [-i] [-p] [-@] d1 ... dn-1 dn

__________________________________________________________________

Can you guess or pinpoint as to what exactly is wrong with the above code, which makes it throw this error?

It took me sometime but I finally figured it out. $i_old was being considered as a variable i.e., the shell was reading the variable as $i_old and not as the variable $i appended with an underscore(_) and 'old'.

The solution

The solution to this obviously is to use the flower brackets (braces/parantheses) for the variable $i so as to make it ubiquitous. Now the code looks like:

for i in `ls *ver`

do

cp -p ${i} ${i}_old

done

The conclusion

To conclude, I would like to say to my fellow DBAs who are also into shell scripting, watch out for small things like these, which might make or break your day!

Wednesday, February 4, 2009

Recently (last week, to be precise), I was researching an "Internal Server error" while opening Installed Base.

The error log had error "client denied by server configuration" (not very helpful, I would say). The mod_jserv.log was much more informative and in fact, quite specific. Refer the note marked in red below.__________________________________________________________________

/H3><TABLE BORDER=1 WIDTH=100%><TR><TH ALIGN=CENTER>Line #</TH><TH ALIGN=CENTER>Error</TH></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>990</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) Enumeration enum = request.getParameterNames(); </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>991</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) while (enum.hasMoreElements()) </TD></TR><TR><TD WIDTH=7% VALIGN=TOP><P ALIGN=CENTER>993</TD><TD> as of release 1.5, 'enum' is a keyword, and may not be used as an identifier (try -source 1.4 or lower to use 'enum' as an identifier) String param = (String)enum.nextElement();__________________________________________________________________

Research

Now, the above error gave me an idea. I decided to check the Metalink Note 300482.1 - Overview of Using Java with Oracle E-Business Suite Release 11i.

This note clearly states that for 11.5.9, the JDK (Java SE) version supported is 1.4.2.

Does that mean that JDK 5 and JDK 6 are not supported for 11.5.9? Not quite, as I found out.

The Note 304099.1 (Using J2SE Version 5.0 with Oracle E-Business Suite 11i) does NOT mention 11.5.9 anywhere, implying that JDK 5 is indeed not supported with 11.5.9.

However, the Note 401561.1 (Using J2SE Version 6 with Oracle E-Business Suite 11i) does mention that JDK 6 can be used with 11.5.9 provided (yes, as the ads say - conditions apply!) that you "ensure that your E-Business Suite Release 11i system has been upgraded to Maintenance Pack 11.5.9 CU2 or higher with ATG_PF.H RUP5 or higher on the 11.5.9 stream."

Conclusion

This means that 11.5.9 is indeed supported with JDK 6 but NOT JDK 5 (of course, provided that the above conditions are satisfied).

Because, 11.5.9 is NOT supported with JDK 5, I believe we are hitting the above issue (internal server error in installed base). As of now, there is no solution to this issue apart from

1. downgrading JDK from 5 to 1.4.22. modifying the xml to point to 1.4.23. running autoconfig to ensure the changes are reflected.

I was trying to find something in the CONTEXT_FILE when I happened to notice the AF_CLASSPATH env variable (not that I haven't noticed it earlier). But this time, it set me thinking as to what might its purpose be since we already had a CLASSPATH variable and its value seemed to be the same as the AF_CLASSPATH variable.

The AF_CLASSPATH variable is used by JAVA concurrent programs and must use literal (full) path values.

By combining the above two notes, we can perhaps safely say that the AF variables are set for letting programs know the location of java (AFJVAPRG), JRE (AF_JRE_TOP) and the classpath (AF_CLASSPATH) on the concurrent node.

Also, note that these variables are also set on the web node (and must be set to valid values too).

Visitors!

The following text will not be seen after you upload your website,
please keep it in order to retain your counter functionality
Insurance is one of the bare necessities of life. simply by putting attention to website offerring auto insurance quotes the purchace process can be optimized. Pity you won’t be around to collect your life insurance. counter