So we compared the size of the file /etc/ssl/certs/ca-certificates.crt on a working server and this server. Sure enough the file size was different. On further investigation we found that there was a backup called ca-bundle.crt.newrpm that had the correct size. So we took a backup of ca-bundle.crt and replaced it with the newer file. The error stopped coming. Even though the ca-certificates rpm version was same on both servers, someone had replaced the ca-bundle.crt file with the older version. We would have to investigate, why that was done, before this solution can be implemented, as it is possible that the new certificates.crt file broke something, because of which it was replaced by the older version. We should always have the latest ca-bundle.crt, so that we have the latest root certificates from the Certificate Authorities.

As per https://access.redhat.com/errata/RHSA-2013:1402 : Adobe Reader (acroread) allows users to view and print documents in Portable Document Format (PDF). Adobe Reader 9 reached the end of its support cycle on June 26, 2013, and will not receive any more security updates. Future versions of Adobe Acrobat Reader will not be available with Red Hat Enterprise Linux.

Some of our ERPs still use acroread for pasta printing. When we did the upgrade of OS to OEL 6, we had to reinstall adobe by getting the rpm from adobe site ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/ that I got from https://www.reddit.com/r/linux/comments/2hsgq6/linux_version_of_adobe_reader_no_longer/ Fortunately, adobe still hosts it. It is also available from Red Hat if you have a subscription:

After installing the rpm with rpm -ivh AdbeRdr9.5.5-1_i486linux_enu.rpm command, we got dependency error for libxml2.so. On installing libxml2.so, we got error for other missing dependencies. So eventually we did yum install AdbeRdr9.5.5-1_i486linux_enu.rpm which installed about 80 rpms that acroread needs to function.

After installation, we kept getting this error:

dirname: missing operand

The post on http://www.linuxquestions.org/questions/slackware-14/error-message-dirname-missing-operand-when-starting-acroread-827012/ recommended a fix:

The issue was fixed by commenting line 529 in the bash script /opt/Adobe/Reader/bin/acrobat

I have asked the DBA team to reach out to the Developers to stop using acroread and migrate to an alternative that is natively supported in Linux. Here's more from Red Hat on their security advisory https://access.redhat.com/errata/RHSA-2013:1402 :

Red Hat advises users to reconsider further use of Adobe Reader for Linux,as it may contain known, unpatched security issues. Alternative PDFrendering software, such as Evince and KPDF (part of the kdegraphicspackage) in Red Hat Enterprise Linux 5, or Evince and Okular (part of thekdegraphics package) in Red Hat Enterprise Linux 6, should beconsidered. These packages will continue to receive security fixes.

Red Hat will no longer provide security updates to these packages andrecommends that customers not use this application on Red Hat EnterpriseLinux effective immediately.

Last evening, Mukhtiar called me to assist in Demantra installation on Windows. By the time we got on screen share, he had already downloaded patch 2195481 and installed JDK 1.8.0_77. However, when he clicked setup.exe, nothing happened. We tried these :

1. Right click on setup.exe > Click Properties > Compatibility Tab > Run this program in compatibility mode for Windows XP2. Run the program as Administrator3. Tried changing registy settings as per some microsoft technotes

However nothing worked.

We located a file called setup.lax that was a configuration file for the installer and switched on stderr=console

If the client machine does not have JDK version 1.6 or 1.7 installed, the Installer prompts you to download and install it. After installation, set JAVA_HOME to the JDK installation directory (for example, C:\Program Files\Java\jdk1.6.0_16).

Nothing worked.

Mukhtiar located a blog that advised setting java heap size to 512m by setting this environment variable:

_JAVA_OPTIONS = -Xmx512M

The Java Virtual Machine takes two command line arguments which set the initial and maximum heap sizes: -Xms and -Xmx. You can add a system environment variable named _JAVA_OPTIONS, and set the heap size values there

As soon as this environment variable was set, setup.exe finally launched. All this time it was erroring out as the default 256MB was insufficient for starting the installer.

There was information about Out of Memory error on the installation guide too, but we had missed that:

Out of Memory Error Message

Verify memory settings:

From the Windows Start menu, choose Settings, Control Panel, and then Java. The Java Control Panel dialog box appears.

Click the Java Tab.

From the Java Applet Runtime Settings section, click View.

Verify that the value of the field Java Runtime Parameters is at least -Xmx256M and is based on the client machine hardware/RAM capacity. Oracle recommends value -Xmx512M for heavy-duty client side use.

For an old 11.5.10.2 ERP, we are moving from the architecture of "EBS application server in DMZ" to the architecture of "Reverse Proxy in DMZ and EBS application server in intranet". After doing all configurations, we hit the classic issue where, you login through extranet url visible on public internet which redirects to intranet url.

So https://extranet.example.com asks for SSO details and after keying in SSO username and password goes to http://intranet.example.com.

The support.oracle.com article DMZ Configuration with Oracle E-Business Suite 11i (Doc ID 287176.1) has listed 4 checks which could be the reason for this issue:

H6: Redirection to an Incorrect Server During Login

If you are getting redirected to an incorrect server during the login process, check the following:

Whether the hirearchy type of the profile options mentioned in Section 5.1 is set to SERVRESP .

select PROFILE_OPTION_NAME,HIERARCHY_TYPE from fnd_profile_options where profile_option_name in ('APPS_WEB_AGENT','APPS_SERVLET_AGENT','APPS_JSP_AGENT','APPS_FRAMEWORK_AGENT' ,'ICX_FORMS_LAUNCHER','ICX_DISCOVERER_LAUNCHER','ICX_DISCOVERER_VIEWER_LAUNCHER','HELP_WEB_AGENT','APPS_PORTAL','CZ_UIMGR_URL','ASO_CONFIGURATOR_URL','QP_PRICING_ENGINE_URL','TCF:HOST');

Whether the profile option values for the fnd profile options (APPS_FRAMEWORK_AGENT, APPS_WEB_AGENT, APPS_JSP_AGENT, APPS_SERVLET_AGENT) are pointing to the correct node. Replace the node_id with the node_id of the external and internal web tier. For example:

Whether the dbc file pointed to by the JVM parameter (JTFDBCFILE) in jserv.properties exists.

wrapper.bin.parameters=-DJTFDBCFILE=

This was incorrect. It was pointing to the intranet jdbc file location.

Whether the value of the parameter APPL_SERVER_ID set in the dbc file for the node is the same as the value of the server_id in the fnd_nodes table.

select node_name,node_id,server_id from fnd_nodes;

This was overwritten in the dbc file, with appl_server_id of intranet when autoconfig was done on intranet and overwritten with appl_server_id of extranet when autoconfig was done on extranet, as the DBC file location and name were same for both intranet and extranet.

I asked the DBA team to manually correct the dbc file name inside $IAS_CONFIG_HOME/Apache/Apache/Jserv/etc/jserv.properties

and create a file of that name in $FND_SECURE/$CONTEXT_NAME.dbc on the extranet node and bounce services. Once that was done, we tested and it worked. No more redirection to intranet URL.

Then I asked them to correct the s_dbc_file_name variable in the context file of extranet node. Run autoconfig on extranet, verify the value of dbcfile in jserv.properties DJTFDBCFILE parameter, verify that the DBC file had the server_id of the extranet node. Restart all services.

Checked again, and it worked again.

So apart from checking the values of context file variables like s_webentryhost, s_webentrydomain, s_active_port, you also need to check the value of s_dbc_file while verifying the setups for extranet configuration. This can happen in 11i , R12.1 and R12.2 also.

At this time of the year during holidays, the Apps DBA community is busy doing upgrades as longer downtimes are possible. In case you are facing any issues, please feel free to write to me at my email: oracleappstechnology@gmail.com . I will be glad to hear from you and help you.

We are currently upgrading one of our ERP instances from 11.5.10.2 to R12.2.5. One of the pre-upgrade steps is to execute the data fix script ap_wrg_11i_chrg_alloc_fix.sql. However, this script has been running very very slow. After 4 weeks of monitoring, logging SRs with Oracle, escalating etc., we started a group chat today with our internal experts. We had Ali, Germaine, Aditya, Mukhtiar, Martha Gomez and Zoltan. I also invited our top notch EBS Techstack expert John Felix. After doing explain plan on the sql, Based on the updates being done by the query I predicted that it will take 65 days to complete.

John pointed out that the query was using the index AP_INVOICE_DISTRIBUTIONS_N4 that had a very high cost. We used an sql profile that replaced AP_INVOICE_DISTRIBUTIONS_N4 with AP_INVOICE_DISTRIBUTIONS_U1. The query started running faster and my new prediction was that it would complete in 5.45 days.

John mentioned that now another select statement was using the same index AP_INVOICE_DISTRIBUTIONS_N4 that had a very high cost.

After discussing among ourselves, we decided to drop the index, run the script and re-create the index. Aditya saved the definition of the index and dropped it.

We have used OEL 6.6 image in our latest build. When we cloned an EBS R12.2 instance that was on OEL 5.7 to this new server that has OEL 6.6, During the clone, adcfgclone.pl was failing. On further checks, we discovered that sqlplus is crashing with segmentation fault error whenever we tried to connect to database:

sqlplus /nologconn apps/appsSegmentation Fault

So, I suggested the DBAs to do strace sqlplus apps/apps. The strace revealed many missing libraries:

We had another working OEL 6.4 instance where we checked for these libraries, and all of them were present.

The locate command was used to locate the full directory paths of the missing libraries

Then rpm -qf command was used to find out the rpm that would have the library:

$ rpm -qf /lib/libnss_sss.so.2

sssd-client-1.11.6-30.el6_6.3.i686

$ rpm -qf /lib/libnss_files.so.2

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libociei.so

error: file /lib/libociei.so: No such file or directory

$ rpm -qf /lib/libc.so.6

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libgcc_s.so.1

libgcc-4.4.7-3.el6.i686

$ rpm -qf /lib/libnsl.so.1

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libpthread.so.0

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libm.so.6

glibc-2.12-1.149.el6_6.9.i686

$ rpm -qf /lib/libdl.so.2

glibc-2.12-1.149.el6_6.9.i686

Since 10.1.2 home is 32-bit in EBS R12.1 and 12.2, all the libraries needed to be 32-bit.

Except for sssd-client, the other rpms were present. 64-bit version of sssd-client was present and whenver we tried to install the 32-bit rpm it would give this error, as the operating system thinks that it is already installed:

# yum install sssd-client.i686

Loaded plugins: security

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package sssd-client.i686 0:1.12.4-47.el6 will be installed

--> Finished Dependency Resolution

Error: Multilib version problems found. This often means that the root

Today, during a cutover when we were moving one of our ERP instance on Cisco UCS VMware VMs to Exalogic and Exadata, I got a call from Bimal. The extranet iSupplier URL had been configured, but whenever any user logged in, they were seeing the following error instead of the iSupplier OAF Home page:

Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured

A search on support.oracle.com showed many hits. I went through a few of them and ruled out the solutions given. This article sounded promising: Oracle SSO Failure - Unable to process request Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured (Doc ID 1474474.1).

The solution suggested:

There is a hardware load-balancer for a multi-tier environment on place, as well as an SSL accelerator.

For R12, there is a context variable, s_enable_sslterminator, that was set to "#".

This should be null for e-Business R12 using specific hardwarementioned before.

1. Set context variable, s_enable_sslterminator to null,

2. Re-ran autoconfig,

3. Re-test Single sign-ons via IE and Firefox now works as expected.

I asked the DBAs to check the value of s_enable_sslterminator:

grep s_enable_sslterminator

and sure enough the value was #

As per article Enabling SSL or TLS in Oracle E-Business Suite Release 12 (Doc ID 376700.1), the value of s_enable_sslterminator should be made null if you are using an SSL accelerator. In our case we use SSL certificate on the Load Balancer and never on Web servers.

Dhananjay pinged me today and told me that for their Paypal integration, they had to upgrade to Verisign G5 root certificate. This was the message from Paypal:

Global security threats are constantly changing, and the security of our merchants continues to be our highest priority. To guard against current and future threats, we are encouraging our merchants to make the following upgrades to their integrations:

Update your integration to support certificates using the SHA-256 algorithm. PayPal is upgrading SSL certificates on all Live and Sandbox endpoints from SHA-1 to the stronger and more robust SHA-256 algorithm.

Discontinue use of the VeriSign G2 Root Certificate. In accordance with industry standards, PayPal will no longer honor secure connections that require the VeriSign G2 Root Certificate for trust validation. Only secure connection requests that are expecting our certificate/trust chain to be signed by the G5 Root Certificate will result in successful secure connections.

There is a support.oracle.com article published on October 16, 2015 which has detailed steps for 11i and R12.1: How To Install Latest Verisign Root Certificates For Use With Paypal SDK 4.3.X (Doc ID 874433.1)The Verisign G5 root certificate can be downloaded from:Paypal Microsite about this change: https://www.paypal-knowledge.com/infocenter/index?page=content&id=FAQ1766&expand=true&locale=en_USUseful Links

Recently I spent almost a full day struggling to make out, why an sftp connection would not work without password, after setting up ssh equivalence. The keys were correct, the permissions on the directories were correct. The authorized_keys file looked ok. I copied the authorized_keys file of another account that was working fine. When I replaced the authorized_keys after taking backup of original authorized_keys, it started working. So then I proceeded to check the contents in a hex editor

On the left side you have the authorized_keys file created in Windows.On the right side you have the same authorized_keys file created in Unix.

If you notice the ends of the lines in the Windows file it shows CR LF, where as unix shows LF.

The one mistake I had done this time was create the authorized_keys file in Windows notepad, as I was teaching a Developer how to create authorized_keys file. Once I used vi on unix to create the authorized_keys file and pasted the same ssh key, sftp started working without prompting for password. I know that Windows/DOS and Unix have different newline characters. However, I was not able to apply that knowledge, till I compared the files in hex editor.

Whenever, a techie is able to get to the root cause of a problem, a deep sense of satisfaction is experienced. I am glad I got the opportunity to troubleshoot and fix the issue by getting to the root cause of the issue.

While doing a google search today I noticed that there is another blog that has copied all content from my blog and posted it as their own content and even kept a similar sounding name: http://oracleapps-technology.blogspot.com . I have made a DMCA complaint to google about this. The google team asked me to provide a list of URLs. I had to go through the copycat's whole blog and create a spreadsheet with two columns. One column with URL of my original post and second column with the URL of the copycat's blog. There were 498 entries. I patiently did it and sent the spreadsheet to google team and got a reply within 2 hours:

Hello,

In accordance with the Digital Millennium Copyright Act, we have completed processing your infringement notice. We are in the process of disabling access to the content in question at the following URL(s):

It seems that the post patch steps for a PSU OJVM patch were not done. We followed the steps given in above note were note completed. We completed these and adgendbc.sh completed successfully after that.

1.set the following init parameters so that JIT and job process do not start.

If spfile is used:

SQL> alter system set java_jit_enabled = FALSE;SQL> alter system set "_system_trig_enabled"=FALSE;SQL> alter system set JOB_QUEUE_PROCESSES=0;

2. Startup instance in restricted mode and run postinstallation step.

SQL> startup restrict

3.Run the postinstallation steps of OJVM PSU(Step 3.3.2 from readme)

Postinstallation

The following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node.

Install the SQL portion of the patch by running the following command. For an Oracle RAC environment, reload the packages on one of the nodes.

2.cd $ORACLE_HOME/sqlpatch/19282015

3.sqlplus /nolog

4.SQL> CONNECT / AS SYSDBA

5.SQL> @postinstall.sql

After installing the SQL portion of the patch, some packages could become INVALID. This will get recompiled upon access or you can run utlrp.sql to get them back into a VALID state.

7.cd $ORACLE_HOME/rdbms/admin

8.sqlplus /nolog

9.SQL> CONNECT / AS SYSDBA

SQL> @utlrp.sql

4. Reset modified init parameters

SQL> alter system set java_jit_enabled = true;SQL> alter system set "_system_trig_enabled"=TRUE;SQL> alter system set JOB_QUEUE_PROCESSES=10; -- or original JOB_QUEUE_PROCESSES value

With the release of AD and TXK Delta 6, Oracle has provided the feature of single file system on development instances for R12.2. Here's what they have mentioned in support.oracle.com article: Oracle E-Business Suite Applications DBA and Technology Stack Release Notes for R12.AD.C.Delta.6 and R12.TXK.C.Delta.6 (Doc ID 1983782.1)

Enhancements in AD and TXK Delta 6

4. New and Changed Features

Oracle E-Business Suite Technology Stack and Oracle E-Business Suite Applications DBA contain the following new or changed features inR12.AD.C.Delta.6 and R12.TXK.C.Delta.6.

4.1 Support for single file system development environments

A normal Release 12.2 online patching environment requires one application tier file system for the run edition, and another for the patch edition. This dual file system architecture is fundamental to the patching of Oracle E-Business Suite Release 12.2 and is necessary for production environments and test environments that are meant to be representative of production. This enhancement makes it possible to have a development environment with a single file system, where custom code can be built and tested. A limited set of adop phases and modes are available to support downtime patching of such a development environment. Code should then be tested in standard dual file system test environments before being applied to production.

More details are provided in Oracle E-Business Suite Maintenance Guide, Chapter: Patching Procedures):http://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531065.htm#6169002Support for Single File System Development EnvironmentsA normal Release 12.2 online patching environment requires two application tier file systems, one for the run edition and another for the patch edition. This dual file system architecture is fundamental to patching of Oracle E-Business Suite Release 12.2, and is necessary both for production environments and test environments that are intended to be representative of production. This feature makes it possible to create a development environment with a single file system, where custom code can be built and tested. The code should then always be tested in a standard dual file system test environment before being applied to production.You can set up a single file system development environment by installing Oracle E-Business Suite Release 12.2 in the normal way, and then deleting the $PATCH_BASE directory with the command:

$ rm -rf $PATCH_BASE

A limited set of adop phases and modes are available to support patching of a single file system development environment. These are:

·apply phase in downtime mode

·cleanup phase

Specification of any other phase or mode will cause adop to exit with an error.The following restrictions apply to using a single file system environment:

·You can only use a single file system environment for development purposes.

·You cannot use online patching on a single file system environment.

·You can only convert an existing dual file system environment to a single file system: you cannot directly create a single file system environment via Rapid Install or cloning.

·There is no way to convert a single file system environment back into a dual file system.

Shahed pinged me about this error. It was coming after logging in. This R12.1.3 instance had just migrated from an old server to a new one. Once you logged in this error would be displayed:

You Are Trying To Access a Page That Is No Longer Active.The Referring Page May Have Come From a Previous Session. Please Select Home To ProceedThe hits on support.oracle.com were not helpful, but a gave a clue that it may have something to do with session cookie. So I used Firefox to check http headers. If you press Ctrl+Shift+K, you will get a panel at the bottom of the browser. Click on Network tab, click on the AppsLocalLogin.jsp and on the right side of the pane, you'll see a cookie tab.

The domain appearing in the cookie tab was from the old server. So I checked:

Dhananjay came to me today. He said that his users were complaining about forms not launching after upgrading to the latest version of Chrome. On launching forms they got this error:

/dev60cgi/oracle forms engine Main was not found on this server

I recalled that Google Chrome team had announced that they would not support java going forward. Googling with keywords chrome java brought this page:

https://java.com/en/download/faq/chrome.xml#npapichrome

It states that:

NPAPI support by Chrome

The Java plug-in for web browsers relies on the cross platform plugin architecture NPAPI, which has long been, and currently is, supported by all major web browsers. Google announced in September 2013 plans to remove NPAPI support from Chrome by "the end of 2014", thus effectively dropping support for Silverlight, Java, Facebook Video and other similar NPAPI based plugins. Recently, Google has revised their plans and now state that they plan to completely remove NPAPI by late 2015. As it is unclear if these dates will be further extended or not, we strongly recommend Java users consider alternatives to Chrome as soon as possible. Instead, we recommend Firefox, Internet Explorer and Safari as longer-term options. As of April 2015, starting with Chrome Version 42, Google has added an additional step to configuring NPAPI based plugins like Java to run — see the section Enabling NPAPI in Chrome Version 42 and later below.

As of Chrome Version 42, an additional configuration step is required to continue using NPAPI plugins.

In your URL bar, enter:chrome://flags/#enable-npapi

Click the Enable link for the Enable NPAPI configuration option.

Click the Relaunch button that now appears at the bottom of the configuration page.

Developers and System administrators looking for alternative ways to support users of Chrome should see this blog, in particular "Running Web Start applications outside of a browser" and "Additional Deployment Options" section.

Once Dhananjay did the above steps, Chrome started launching forms again. He quickly gave these steps to all his users who had upgraded to the latest version of Chrome (version 42) and it started working form them too.

Oracle doesn't certify E-Business Suite forms on Chrome. Only self service pages of E-Business Suite are certified on Google Chrome.