Great things always happen overnight. That’s probably the case for European Oracle Apps DBAs in the same situation, like myself.

This morning I read Steven Chan’s latest blog post about Java Web Start certification with Oracle E-Business Suite and its support release to the public. We heard about it during some informal discussions at Collaborate 17 conference. Everything is ready, the documentation is prepared, and they are just waiting for final bug related to Workflow Activity Monitor to be addressed…

…and about twenty days later, it was released.

This is a superb evolutionary event! And, of course, I made time to fit in some good introductory, hands-on experience.

Initial thoughts while going through Doc ID 2188898.1

1. Java Runtime version for client.

JRE 8 Update 131 b31 or JRE 8 Update 121 b33 are required (as a minimum).
This isn’t clearly visible – these are special update releases (look at b31 and b33), available only through My Oracle Support download via Patch 25767257 and 25450542. A public release containing the support is only scheduled for next quarter and is set to be released with Update 141.

JRE 6 and JRE 7 are out of scope! If you don’t have JRE 8 support there in your system, now is the right time to think about it.

2. Support for R12.2. Also there is a support for R12.1.3. No 11i.

Patching requirement is minimal (three small patches for AD/TXK/FND), but you need to be on latest AD/TXK.C.8 and ATG_PF.C.6, and new PSU overlay for 10.1.2 Oracle Home which nicely supersedes the previous overlay released.

There are eight functional module patches posted so far. There are still known issues mentioned for other modules. These are less popular modules, and are to be implemented and added to the list at some point, I believe.
I would include this list in your patch maintenance, especially Oracle Workflow related for Activity Monitor.

s_forms_launch_method new context variable is implemented which sets the required profile options at Site level. However, you can still configure exceptions for users via User Level profile option values. Both methods are available. You can still leave browser plugin method as default, and enable JWS only for particular users, let’s say, for testing purposes.

3. “Only one instance per Java applet can be running on the client PC.”

Yes, it’s only possible to have one environment opened at the same time. This is a JWS limitation. Even IE’s noframemerging or “hey, I can use my second browser” is not an option here.

4. Chrome full usage support is finally available there for Forms.

Basically, browser support is not dependent on browser plugin availability. There is also Microsoft Edge support to mention and we can finally start using 64-bit Java, as there is no longer browser plugin dependency.

5. Known issues section.

From management and housekeeping perspective, docs are mentioning that the process is self cleanable, like frmservlet.jnlp downloaded files should be automatically cleared from disk after launch. There is a known issues section mentioning some nuances, open issues and workarounds for specific browsers and functionality areas in EBS. As always, it will be kept updated and will be periodically reviewed by many of us Apps DBAs for news.

Now to talk about my hands-on…

I used my R12.2.6 Vision lab instance built on AWS. The patching exercise took something like 30 minutes in total (10.1.2 patch, ebs patches, JAR regeneration). But I was doing this in hotpatch mode and not through online patch cycle, (yes, not the right way, but my goal was to avoid ADOP time consuming tasks.
I also installed the required JRE 8 update 131 b31 64-bit version on my laptop.

I made a huge mistake starting this testing on my MacOS, which introduced a number of issues that almost led me to a huge “why Oracle?” facepalm result. And it’s not just Gatekeeper security feature known issue mentioned for Safari on MacOS. But I’ll talk about this later.

Brought up my Windows 7 VM.
All IE 11, Firefox’s latest public update, and Chrome just worked like a charm. I didn’t experience any issues like I faced previously on my Mac. “Save File” to Downloads folder and then a double-click, or “Open with” to open the applet immediately. And your Forms session is up. Look at the screenshot – CHROME!

Applet window is transparently opening.

The example below is from Workflow Activity Monitor using Firefox. No extra browser tab / window opened in my case, although there was a mention of known issue with Firefox.

Applet files downloaded are automatically cleared once the applet is loaded, you will not find them on the disk. Chrome, as an example, updates the status of each item under Downloads tab.

So again.. Windows platform is covered well. What about the rest?

Linux is not a certified platform for Oracle E-Business Suite end user desktops. Although I was successfully using it in my experience, and it should work, the Oracle team still is not testing and certifying it. You may use and play around it at your own risk, and it shouldn’t be the right production direction. I personally haven’t tested it yet for JWS, but my guess is that it will have the same problem as MacOS is having. Maybe a to-do for this blog post update at some point.

MacOS… Steven Chan’s blog post, as mentioned above, states that “Safari on MacOS” is not certified because of MacOS Gatekeeper security feature that is “making the Java Web Start user experience very challenging”. That’s fine. We know a workaround to go to System Preferences app and click on Open Anyway button, though it’s required every time we launch the Forms session.

But… we don’t even reach this point. In the example below, I will outline a sequence of “nuances” I faced. It will be based on Safari screenshots, as only Safari is officially certified on MacOS. Absolutely the same issues I faced in Firefox on MacOS, and in Chrome on same MacOS.

First – we are trying to open Forms, but just getting this familiar screen.

Why? Because the URL still goes to browser plugin mode – “/forms/frmservlet?config=browserMode&appletmode=…”

We can go and set the ICX_FORMS_LAUNCHER profile option to “/forms/frmservlet?config=jws” on Site level, as MOS note outlines.

This works, but it will be required every time we run AutoConfig, as the profile option is always being reset to just “/forms/frmservlet” on Site level, and Forms opening process is supposed to follow FND_ENABLE_JAVA_WEB_START value direction.

After the manual fix this is what’s happening. Safari is downloading the applet.

We launch it and face a known Gatekeeper issue (only now).

Applet is loaded, but not Forms.

I would expect Safari to cover this itself, but in my case I have Firefox configured as default Web browser in the system.
That caused Firefox tab to be opened (called by initial applet) and a second applet download to be requested.

Once the second applet is up, I finally get the Forms session running. Sort of, it’s a similar flow that is happening with a browser plugin, but it is just killing the user experience.
Initially second applet was blocked by same Gatekeeper, and I had to use the known workaround. But at a later testing MacOS is always blocking only the first applet while launching Forms, second – not anymore. Maybe, I suspect the issue is with a miss of jndi extension for the file.

I got the Forms running. But what a process it was…
And nothing is cleared. My Downloads folder is full of these fndservlet.jndi files downloaded.

To summarize

This is a great moment and long expected feature that got finally released by the Oracle EBS team. But is it ready?

I would say, if your company is following strict control standards what software is used, including the browsers, you are, most probably, using same IE on Windows – this feature will work well. Forms will work even with good old browser plugin mode, your current JRE 6 or JRE 7, and you shouldn’t care and rush with this implementation.
Same comments about Firefox ESR – it will work as before. But not for a long time period until next ESR release branch merge.

If your company is following BYOD approach, multiple platforms in use like MacOS, you seek for Chrome browser support – at this stage only Windows platform is covered well. Others – are not officially certified, including “Safari on MacOS”. You can still use it, but you need to accept the “head ache” process as described by the example above. And that’s not about Safari only, as other browsers, like Firefox and Chrome, behaves in the same way on MacOS.

And not all Apps DBAs will agree to manually fix ICX_FORMS_LAUNCHER every time AutoConfig is executed. Though there might be User level exceptions for a defined end user base list.

If you are in process of upgrading to Oracle E-Business Suite 12.2.4, you would have went though this critical phase in the upgrade which is to apply the Online Patching Enablement patch:

13543062:R12.AD.C.

It’s very common to run into errors with this patch in the first try and have to apply it couple of times, in order to get all issues fixed and get online patching enabled. The recommended command to apply this patch is:

adpatch options=hotpatch,forceapply

When the time comes to re-apply the patch to fix problems, if you use the same command to reapply the patch, you will notice that the patch completed normal with in no time and nothing happens in the back end. This is because of a specific feature from Adpatch. ADPATCH by default skips jobs that are marked as “run successfully” in previous runs or as part of another patch. So we have to force it re-run those jobs. This can be done by using command below:

adpatch options=hotpatch,forceapply,nocheckfile

Sometimes we run into cases where Online Patching Enablement patch completes as “normal” and the actual online patching feature gets enabled where we see that a schema or two have failed to enable the EBR feature. As soon as APPS schema gets EBR enabled by this patch, even though other custom schemas failed to get enabled, Adpatch gets disabled and we are forced to adop utility from then on. In this scenario, we can still re-apply the Online Patch Enablement using Adpatch after setting the environment variable below:

export ENABLE_ADPATCH=YES

I see that online patching enablement exercise for every customer is a unique experience. Do post your experiences with this online patching enablement patch in the comments section. I’d love to hear your story!

When you try to rollback a weblogic patch in Oracle EBS R12.2, you might run into issues similar to below:
$ ./bsu.sh -remove -patchlist=YIJF -prod_dir=/u01/install/VISION/fs2/FMW_Home/wlserver_10.3 -verbose
Checking for conflicts..
No conflict(s) detected

If you observe the errors, it’s trying to restore the jar file to a location that is not present in the current filesystem. This error was captured in a VISION instance created from Oracle VM Template. So you can see that the file path seems to be a path used by an Oracle internal system where the VM template is created. There is not much harm caused by the above issue to the instance.

Now consider this hypothetical situation, where you are trying to rollback a weblogic patch in patch fs ( fs2 ) that was previously applied to fs1 in Production Oracle E-Business Suite R12.2. So the patch history in the patch fs ( fs2) will still have file paths of fs1. So when you rollback a patch it will restore the files to RUN FS ( fs1). This can cause a havoc and a big outage to online users.

As of now bsu utility doesn’t seem to handle issues this kind of situation. So all Oracle EBS Apps DBAs out there, make sure to check the below file for correct paths, before you try to rollback any weblogic patch
$ vi $FMW_HOME/patch_wls1036/registry/patch-backup.xml

A permanent solution to this problem would be to update bsu patch utility to not restore or touch files outside the weblogic home. Hope Oracle Development notices this issue before it effects some customers EBS production instance.

This is my first post in Pythian blog, and I wanted to share interesting use-cases that we’ve dealt with recently. I believe someone may see similar cases in the future, so these are the troubleshooting steps to get to the source and apply workaround to fix it. So, the root problem was that concurrent processing doesn’t start up, and Internal Manger log file reports error (same error also throwing Oracle forms trying to access Concurrent -> Administer form)

ORA-01591 error clearly identifies lock held by in-doubt distributed transaction string; however, DBA views DBA_2PC_PENDING and DBA_2PC_NEIGHBORS return no rows (also sys.pending_trans$ and sys.pending_sessions$ have no rows). If any of these views return any row, then most probably these transactions can be committed, rolled back or purged and the problem would be resolved. However, this wasn’t the case – so this must be some kind of glitch.

The first document to review is Master Note for Troubleshooting Oracle Managed Distributed Transactions (Doc ID 100664.1), but this gives no valuable information, and neither does the much detailed Manually Resolving In-Doubt Transactions: Different Scenarios (Doc ID 126069.1). Next step is to find out where ORA-01591 is coming from. Here, a database trace is very useful tool. I was lucky this time as error is thrown in forms, so it’s 2 clicks away from getting trace. If it’s not so easy in your case, the SERV_MOD_ACT_TRACE_ENABLE procedure can be used to get trace. In my case it was:

So now I have some details on ORA-01591 error. I know which transaction is holding it – but it’s not displayed in any of DBA_2PC _ views. Also trying to purge, commit or rollback this transaction will end in error as there is no full data about this transaction:

Trying to dig deeper, MoS note DBMS_LOGSTDBY.BUILD Seems to Hang And Does Not Return To SQL prompt. (Doc ID 747495.1) got my attention. Standby database is not used in this configuration, but the fact that x$ktuxe internal view contains information about transactions and query against it soon gave some results:

This is exact transaction where we identified earlier with trace (ORA-01591: lock held by in-doubt distributed transaction 35.15.13775). Once transaction is identified there is a way to get rid of it, and there are already a few options:

Using the same technique, I inserted dummy data into pending_trans$ and pending_sessions$ tables. Then purge_lost_db_entry solved the case, and the dummy entry was removed, along with references in x$ktuxe table (see examples in above sources I mentioned). According to the oracle-L comments, this is Oracle Support provided workaround to deal with a particular SR case – so most probably this is not a supported way to deal with such cases.

In my case, we didn’t know what created the half-purged transaction, and this test system was needed ASAP for training, so this method really saved the day.

I always like to get to the location for a conference a day in advance so I can

Get accustomed to the time change

Get a feel for my way around the venue

Figure out where my room is

Establish a few landmarks so I do not wander aimlessly around the facility and hotel as though every voyage is a new life experience

COLLABORATE officially starts on Tuesday, though there are education sessions all day Monday facilitated by the three main groups responsible for the show – the IOUG, OAUG, and Quest International Users Group. So where did this animal called COLLABORATE come from one may wonder?

Rewind to about 2004. The three above-mentioned user groups each had their own show. Each reached out to Oracle for logistic and education support, something that the vendor was (and still is) happy to give. It was starting to become obvious that the marketplace upheaval was having a dramatic effect on user group conference attendance. At the same time Oracle expressed a desire to support fewer shows. You do the math – it only made sense. Why not have a 4-5 day mega conference and work with Oracle for many facets of support. Not only were the attendees of each show being asked to pick one or the other; Oracle was investing a massive number of personnel to support all three shows separately. It was a cumulative decision to amalgamate the shows and we wondered where it all would start.

With the blessing of the IOUG board I made one of those very first phone calls to one more people on the OAUG board and the rest is history. I do not remember who I spoke to first and there were probably a handful of feelers going out from other places in the IOUG infrastructure to OAUG bigwigs. I spoke to board member Donna Rosentrater (@DRosentrater) and we jammed on what could/should become of a co-operative effort. We chatted a few times and the interest amongst board members of the IOUG and OAUG reflected cautious optimism that we could pull if off. Each user group had its own revenue stream from separate shows. We needed to embark down a path that would not put these at risk. That is what the brunt of the negotiations centered on and the work we did together led to the very first COLLBORATE at the Gaylord in Nashville in 2006.

Once the initial framework was established, it was time to turn the discussions over to the professionals. Both groups’ professional resources collaborated (hence the name maybe) and this mega/co-operative show became a reality. COLLABORATE 14 is the 9th show put on by Quest, OAUG, and IOUG. I am not going to say “this year’s show is going to be the best yet” as I believe that implicitly belittles previous successful events. Suffice to say, for what the user community needs from an information-sharing perspective – COLLABORATE is just what the doctor ordered.

Tomorrow’s a day off; wander aimlessly through Las Vegas tempted by curios, shops, food emporiums, and just about every other possible temptation one could think of. Sunday starts with a helicopter trip to the Grand Canyon and I went all out and forked over the extra $50 to sit in the convex bubble beside the pilot. There’s a bazillion vendors poised to whisk ine away to the canyon with a fly over the Hoover dam there or on the way back. I chose Papillon and am looking forward to the excitement of the day which starts at 5:10am with a shuttle to the site where the whirlybird takes off. Talk about taking one’s breath away.

“Seriously John, do you blog about anything else?”

Yeah, well… Evidence is strongly against me so far. :)

One of the more common questions I’ve received as a followup to my Build an E-Business Suite 12.1.3 Sandbox In VirtualBox in One Hour post has been, “Can I do this in a single node instead of creating two VMs?” The answer of course, is yes, but it never seemed like a good topic for a full blog post. Given the number of requests, however (and the patience and persistence of one reader in particular — hi Sandip!), I’m using this post to present quick notes on how to create a single-node EBS 12.1.3 Vision instance from the OVM templates, instead of the two-node system for which they’re designed.

In addition to the normal complete-lack-of-support caveats listed in the original post, please be aware that this post contains pointers and rough notes, not detailed instructions. Basically, I’ve just wrapped some formatting around some notes from a presentation I gave on this topic last summer. If you don’t understand what’s happening in the original set of instructions, these notes will not be useful to you at all. Please read the original post carefully before asking questions about this one.

System specs

Since we’re running apps and the database in a single node we need to configure a slightly more powerful single VM. Here’s partial output from ‘vboxmanage showvminfo’ that illustrates the important points (more memory, more CPU, and an extra disk for the Apps software). Otherwise, the configuration (network interfaces, rescue boot image setup, etc) is the same as in the original post.

Still the same

The following steps are almost the same as in the original post:

Download the software

Extract the templates

Convert the disk images to .vdi format (though you can skip the Apps server System.img disk, you won’t need it, only ebs1211apps.img). Of course, you’ll only need to create 1 VM at this step, attaching the Apps vdi as the third disk.

Boot the database server VM in rescue mode from the install CD — the steps to install the new kernel and run mkinitrd remain the same

Things change a bit before moving on to step 5, “Reboot and prepare for next steps,” as described below.

What’s different?

Apart from the obvious “no second VM to create,” here are the essential changes I made to my build process for a single-node Vision instance:

Before rebooting, add another line to /etc/fstab to attach the apps software volume:/dev/sdc1 /u02 ext3 defaults 1 0

Before rebooting, do not edit the /etc/sysconfig/oraclevm-template script. I found it to be easier to just let the script execute at boot time, although it did require me to be a bit more careful about my inputs.

After rebooting, the template configuration script will guide you through the configuration of the network interfaces and the Vision database tier, as described in the original post

Once the database is started, you’ll need to make a few changes to the scripts that configure, start, and stop the applications tier. First, log in to the VM as root, and then adjust the scripts to account for the new mount point. To save your sanity, it’s also necessary to comment out ovm_configure_network from the ebiz_1211_reconfig.sh script:

After the scripts have been adjusted, you’re ready to configure the apps tier. Again, as root, run the /u02/ebiz_1211_reconfig.sh script, which will invoke AutoConfig and ask you all the necessary questions. Your answers will differ from the two-node process in two important ways:

There is only one hostname for the environment now

All references to the apps software locations will point to /u02, not /u01

Here’s an excerpt of the Autoconfig run, with only the important/changed bits included:

Cleanup items and other reminders

To prevent annoyances when starting/stopping services, and logging in as oracle:

touch /home/oracle/.passchanged

rm /u02/E-BIZ/apps/apps_st/appl/*mydb*

Also, since our root disk came from the database server VM template, only database services will stop and start automatically upon server shutdown and boot. You will need to use the startapps.sh and stopapps.sh scripts in /u02 to manage the applications tier services.

Here are the Top 5 things in Oracle E-Business Suite world that will have major impact in 2014 and beyond.

1. Oracle E-Business Suite 12.2 Now Available

2013 started on a low note in Oracle E-Business Suite (EBS) World. Many people were expecting some announcement related to upcoming EBS release during Openworld 2012. But all they got it is a extension of support deadline for existing 11i EBS customers. Oracle finally announced Oracle EBS R12.2 few days before Openworld 2013. This releases packs exciting features like Online Patching, which elevates Oracle E-Business Suite ranking in ERP systems domain. Online Patching will enable large multi-national customers consolidate their ERP systems in different Countries into one Single Global Oracle E-Business Suite instance, as it cuts down the downtime required for patching maintenance window to all most nil. This is a big plus point for clients who cannot afford downtime because their user base is spread all over the world. 2014 will be a years of upgrades to R12.2 for all clients.

2. 12.1.0.1 Database Certified with Oracle E-Business Suite

Around the same time as R12.2 announcement, Oracle certified 12c Database with Oracle EBS. The good news here is they certified Oracle 11i also with 12c Database. This should give EBS clients option to get onto newest version of Oracle Database and take advantage of the new features of 12c database. The effort involved in upgrading database is significantly less than upgrading to newer version of EBS. So i believe many customers will uptake 12c database upgrade before the R12.2 EBS upgrade. Also upgrading database to newer version earlier than EBS, will save some hours during future R12.2 upgrade downtime window.

3. E-Business Suite Support Timelines Updated at OpenWorld 2013

Oracle once again extended the support timelines for 11i customers. They named it as Exception support and it ends on December 2015. During this Exception support period, Oracle will primarily provide fixes for Sev 1 issues and Security patches. So this gives 2 years of additional time to Customers on 11i to migrate to latest R12.2. With typical R12 upgrades taking around 1 year time, The sooner you plan and start your R12.2 migration the better.

4. No to Third-Party Tools to Modify Your EBS Database

Oracle Development warned officially in their blog about use third party tools to modify, archive & purge data in Oracle E-Business suite. Managing data growth in Oracle EBS is a known problem. Now Oracle wants customers to use Oracle Database technologies like ILM, Advanced Compression and Partitioning, to archive the data instead of using third party utilities. Note that all these database features will cost customers additional money in licensing costs. So get your bargaining hat on with your Oracle Account Manager and score some discounts using this oracle Achilles heel namely EBS purging and archiving data.

5. Sign E-Business Suite JAR Files Now

Do you remember the days when Oracle EBS moved from Oracle Jinitiator to Sun JRE for oracle forms? Then be prepared for one more similar thing around oracle forms. With stream of viruses and malware that exploit bugs in Oracle/Sun JRE made Oracle to tighten security around Oracle JRE. Its now required to sign forms jar files with a real certificate. In future releases of Oracle JRE7, Unsigned Oracle forms will stop working completely. So customers caught unaware of this will be in for big trouble with user complaints.

Section 2 of the My Oracle Support note should have all of the information you need to import the appliances into Virtualbox. It’s all far less complicated: no rescue boot, no mkinitrd, no config file tweaking. Just download, unpack, import, and go!

Last year, I wrote a blog post about installing Oracle E-Business Suite 12.1.3 in Virtualbox using templates designed for Oracle VM Server. It was a surprisingly popular post, and I enjoyed engaging with a lot of enthusiastic readers in the comments. With minimal OS configuration, no installer to run, and automated scripts provided by Oracle to create the instance, the appeal of having an “out-of-the-box” EBS Vision instance was huge, especially for non-Apps DBAs. The ability to deploy these templates in Virtualbox increases their accessibility for EBS technology enthusiasts who lack access to OVM server.

Well, the EBS 12.2 OVM templates are out, so it’s time to get busy with the new stuff! With a few small tweaks, easily illustrated in this blog post, you can have a working E-Business Suite 12.2.2 sandbox running in Virtualbox, without building your own servers from ground up, running an installer, or applying any patches.

What’s different this time?

If you’re familiar with the 12.1.3 version of these instructions, you’ll notice a few differences in this post:

No need to download a new kernel to replace the Xen kernel used by OVM. The kernel you need is already included; you just need to tell the server how to load it. This simplifies our lives quite a bit.

We’re building a single-node Vision instance this time, instead of multi-node. This was a pretty common request from readers of the 12.1.3 post, and I wanted to reduce the amount of VM resources I was using on my personal lab machine.

The resource footprint of the VM has changed significantly, particularly the memory requirements. The introduction of WebLogic Server (WLS) to the E-Business Suite techstack, along with the duplicated apps tier filesystem required to support online patching, have bulked up an already-hefty Vision instance.

You’ll notice the “One hour of work!” tagline is missing from this post. It’s probably still pretty close, since the long-running steps are hands-off, but I didn’t use a stopwatch this time.

Finally, of course, you’ll find that in general, the EBS 12.2 tech stack is a much different animal than 12.1. :)

Caveats

You really should not follow this guide if…
… you are an Apps DBA. Release 12.2 is very new. If you take this “shortcut,” you are depriving yourself of a lot of learning opportunities: new install processes, patching practice, etc.
… you have an underpowered test system (< 2CPU, < 8GB RAM). The specs I’m using for this blog post are the bare minimum. You might be tempted to cut corners, but you’ll be setting yourself up for pain later on.
… you don’t know your license status. The OVM templates (and EBS software in general) are not “free to install, free to learn” content like you’ll find on OTN. Tread carefully, and know what your support contract permits you to do!
… you need support from anyone (not even me, I can only point the way). OVM templates were not designed for use in VirtualBox, and furthermore, these templates were designed to be used in a multi-node (1 db tier, 1 app tier). This single-node Vision instance in VirtualBox is a double-Frankenstein job. Set your expectations accordingly.
… you have an OVM server at your disposal. Seriously, why jump through these hoops if you don’t have to?

Ingredients

Lots of disk space. Final footprint will be about 325GB, but the intermediary file conversion steps require a lot of space. You should be fine with a 1 TB drive.

8GB of memory. Minimum. WLS is a beast, baby! Rawr!

The previous requirement means that your host system needs to be capable of addressing that much memory, so a 32-bit host OS will not work here.

2 CPU cores, minimum. This will be slow enough as it is, why add the additional delay of db and app tier fighting each other for a single CPU core?

OEL 5.x install media to act as a rescue boot disk. We don’t need to download a new kernel, but we still need to boot in rescue mode to tinker a little bit.

An understanding of some basic Linux systems administration tasks.

Familiarity with configuring storage and network options in Virtualbox.

Not strictly required, but a review of Note 1590941.1 would be a good idea. Most of the content is obviously set in an Oracle VM Server context, but there’s some useful information there about extracting the templates and using the configuration scripts.

Patience! Long downloads, file conversions…you’re looking at hours of waiting for things to complete.

Configuration steps

When following these instructions, note that some of the commands and output may appear to be truncated. Even if you don’t see a scrollbar, scroll right to get the rest of the content. Once I figure out a less awkward presentation method, I’ll update.

Extract the templatesNote: If you’re using Windows and don’t have a Unix-like shell environment like Cygwin available, you may have to translate some of these steps to their Windows-y equivalents. Since we’re just concatenating and uncompressing files, I will assume you can do that. :-)

Unzip all the files you just downloaded (unzip 'V41*.zip'). This will produce two sets of files: OVM_OL5U9_X86_64_EBIZ12.2_VIS_DB_PVM.tgz.* (database tier template) and OVM_OL5U9_X86_64_EBIZ12.2_APPS_PVM.tgz.* (apps tier template).

Convert the disk image files to VDI format
Use the vboxmanage command-line utility to convert the database System.img file and the EBS.img files from both tiers to VDI format. You’ll note that the final size of the apps tier VDI file is much smaller than its source. This is expected; the VDI files are dynamic, and only contain about 50GB of data despite having a max size of 250GB. Also note that we’re only converting the images we need; since this will be a single-node instance, we don’t need the apps tier System.img file.

Configure your VirtualBox VM
Since we’re doing a single-node Vision install, we only need one VM. Here are the specs:

OS: Oracle Linux, 64-bit

CPUs: 2

Memory: 8GB (more if you have it available)

Device boot order: CD-ROM, Hard Disk

Storage: Attach Linux installation ISO to CD/DVD drive on the IDE controller and attach the three vdi files to the SATA controller, in the following order: DBRoot.vdi, DBData.vdi, AppSoftware.vdi

Network: At least one network interface, either Host-only or Bridged. If you choose Host-only, I recommend adding a second interface that uses NAT, so you can reach external (non-host) networks from your VM. If you use Bridged networking you don’t need this second interface.

The screenshot below shows the final configuration of my VM:

Boot VM in rescue mode from the install CD (round 1)
Enter “linux rescue” at the the boot: prompt to enter rescue mode:
Select the keyboard and language preferences that suit you, and select “No” when asked about configuring network interfaces. We don’t need them at this stage.
Select “Continue” at the “Rescue” screen to look for a Linux installation.
If you attached your disks properly, you will get a “Duplicate Labels” warning. This is because both the Vision database and Apps software disks have the same LVM label (EBS). We will fix that in the next step. For now, keep going by selecting the “Reboot” option. DO NOT ACTUALLY RESTART YOUR VM FROM THE VIRTUALBOX MENU, JUST HIT ‘ENTER’. You’ll find that the “reboot button” does not in fact, reboot the VM. Take a moment to savor the irony, then move on.
Select OK when you get the sad news about not finding a Linux installation:
Now we can fix our pesky disk label problem, as shown below. Once complete, exit to reboot the VM so we can continue with the real work.

Boot VM in rescue mode from the install CD (round 2)
Follow the same steps as above to boot in rescue mode. This time, the rescue boot should find a Linux installation, and give you the option to mount it. This is where we’ll make some adjustments to configure the server to use the correct kernel and mount both disks.

First, switch the root volume from from the rescue system to the OVM template

Your system is mounted under the /mnt/sysimage directory.
When finished please exit from the shell and your system will reboot.
sh-3.2# chroot /mnt/sysimage

Then, adjust /etc/fstab to include a mount point for the Apps tier filesystem. Make sure you set the last field in the EBS and EBSAPPS lines to ‘0’ to avoid the possibility of a lengthy fsck of those volumes during boot.

Remove the rescue DVD from the VM (menu navigation: Devices->CD/DVD Devices->Remove disk from virtual drive) and exit to reboot the VM. Otherwise, it’ll boot from the CD again. If you get a warning about the the drive being in use, choose the “force unmount” option; this won’t hurt anything.

sh-3.2# exit
exit
sh-3.2# exit

Network configuration
When the VM reboots, it will run a script included with the OVM template that performs one-time configuration of the network and sets up the 12.2 Vision database. Make sure you choose an IP address which is compatible with your VM’s primary network interface. In my case, that’s the host-only network vboxnet0, 192.168.56.0/255.255.255.0.IMPORTANT: Before you enter any network configuration info, now would be an excellent time to take a snapshot of your VM (menu navigation: Machine->Take Snapshot). Once network configuration completes, the script will roll directly into configuring the database tier, and if anything breaks during that process, you might have to re-extract and re-convert the template files again. That’s not a recipe for fun times. No, I totally didn’t (re)-learn this the hard way, why do you ask? ;-)

Answer the questions presented by the network configuration script. Unless you’re doing something a bit weird, your answers should be similar to mine. Remember the IP, domain, and hostname you choose; you’ll need them later.

Configuration of db tier
After network configuration completes, the template deployment script will invoke Rapid Clone to configure the database tier. All you need to supply at the beginning is the desired SID for the database. Remember that for later, too. :-) When RapidClone completes, you will be prompted to select new passwords for the root and oracle users.

Prepare the server for apps configuration
Before we can run the configuration script to configure the apps tier of the instance, we need to make a few adjustments. Please note: most of these steps will not be necessary if you’re trying to create a two-node instance; they’re only required because we’re trying to cram two nodes into one server.

Log in as root and adjust permissions on the ping executable. Otherwise, you may encounter issues with concurrent manager startup. This would probably not be necessary if we were using the Apps tier template’s root disk.

Next, we need to change the configuration scripts and associated files to use the new mount point for the apps tier filesystems (/u01 is expected by default), and to disable the network configuration steps (since we aren’t running the script on a new server, we don’t need it):

Execute the apps tier config script
Now, we’re finally ready to go! Execute the config.sh script (still connected as root), and answer the prompts with the values from the configuration of the database tier. I’ve left the output from my configuration run completely intact as a reference, but the only user inputs required are at the beginning of the process.Please note: Execution of this script will take quite a long time. It took at least an hour on my modest test system. Rapid Clone has to configure both the Run and Patch filesystems, and starts up the application services at the end. There is significant overhead involved in the multiple stops and starts of the WLS components during this process. This would not be as painful on a properly-sized system of course.

Now what?

Now the real fun begins! If you have added your VM’s IP and hostname to the local hosts file on your host machine, you can access your Vision instance at the URL listed after “Oracle E-Business Suite Login Page” at the end of the Apps tier Rapic Clone run. The default password for the admin user (SYSADMIN) is the same as it always is.

A few more things to note:

Use the startapps.sh and stopapps.sh scripts in /u02/install/scripts/ to (wait for it…) start and stop the Apps. Note that these scripts expect to be run by user oracle, not root. Also, these scripts have default passwords for the apps and weblogic users hard-coded in them, so if you change those passwords, you’ll need to update the scripts. Or you can just use the normal service start/stop scripts, but these wrapper scripts sure are convenient. ;)

Similarly, you’ll find start/stop scripts for database services in /u01/scripts (startdb.sh and stopdb.sh). These also need to be run as oracle.

Since this is a single-node system with both tiers owned by oracle, keeping your OS environment straight might take some practice. Try to remember to do database and apps work in separate sessions. To set the environment for the apps tier, use the script /u02/install/EBSapps.env (in most cases, you will want to set the “run” environment, e.g. “. /u02/install/EBSapps.env run” ). To set the environment for the database tier, invoke the /u01/install/11.2.0/SID_hostname.env file (e.g. “. /u01/install/11.2.0/VIS122_coriana6.env“). If you’re still learning Linux/Unix, please remember that you need to invoke these scripts with a leading “dot+space” as in the preceding examples, or the environment variable values will not be exported to your session.

The first time you try to log into your instance after starting it up, it will be probably be painfully slow. Subsequent logins and similar operations will merely be slow.

The first time I visited the login page and the initial Navigator page for the Vision instance, I had a few “Unable to load” errors for some components on the page. I found that a refresh of the page resolved these issues. I had similar issues upon launching Forms. It takes time for some of these things to be cached, and the default timeout values on the server might not be configured for workstation-grade test systems.

The WLS EM Console, if you need it, will be on port 7001 of your instance (e.g. http://coriana6.local.org:7001/em). Log in with user ‘weblogic’, with the password in the startapps.sh script.

The WLS Admin console for your instance is also on port 7001, at the following URL: http://your_host.your_domain:7001/console (e.g. http://coriana6.local.org:7001/console). Log in to this interface with the ‘weblogic’ user, too.

In general, unless you find issues with services not starting up cleanly, you probably won’t need the WLS URLs very much. Unless you’re an Apps DBA, and hopefully you’re not, because I said at the very beginning that this post is not for Apps DBAs. ;-)

I want to do something slightly different!

Maybe all of this just isn’t weird enough, and you would like to either:

Install the Production OVM templates instead of the Vision templates (Hi John!), or

Deploy the Vision templates in a two-node configuration

In either case, you should be able to find your way based on the instructions in this post. The only “magic” happening here is rescue boot and mkinitrd command to allow the use of the non-Xen kernel. After that, most of the steps should be more or less the same. In the first case listed above (production database templates), the only thing that changes is which database templates you download: parts V41175-V41177 instead of parts V41171-V41173. While I haven’t checked, presumably the network and db tier configuration script is going to be the same. In the second case (two-node configuration), you won’t need to worry about changing disk labels or altering /etc/fstab, but you will have to convert the Apps root disk to a VDI file, and do the mkinitrd steps twice. Also, when allocating resources for the two-node instance, I recommend a minimum of 6GB of memory for the app tier and 2GB for the database tier. Please note that I haven’t tried either approach; these are just educated guesses based on doc review and looking at the configuration scripts. If you’re successful with either of these alternative deployments, I’d love to hear about it in the comments!

Ok, we’re done! You can now give yourself (or an ORACLENERD you care about) the gift of an EBS 12.2.2 Vision instance, just in time for the holidays! Start downloading now, though, if you want to be ready by Christmas. ;)

Shared appltop setup in Oracle E-Business Suite (EBS) is very common nowadays, for obvious reasons. It cuts down the patching downtime enormously on large multi middle tier environments, as we just have apply the patches on only one node instead of all nodes. Any apps patch or apps techstack patch need to be applied only once, Thus Downtime window for production multi middle tier setups with shared Appl_top is significantly less. In the current era of Global Single Instance Architecture any investment in cutting down downtime windows is a very good choice.

One of the requirement for Shared Appl_top setup is to have apps filesystem shared between all the nodes. This requires a Single LUN mounted to all nodes with cluster filesystem on it. Another alternative to cluster filesystem is NFS mount point from a NAS device.

NFS mount point from a NAS device is the most recommended for Apps Filesystem, as this setup doesn’t have any overhead or issues that Cluster filesystems have. NFS doesnt have any fencing issues unlike Cluster filesystems. Yes, it’s slower in IO throughput, but Apps Filesystem doesn’t do a kot of IO. NFS IO performance is more than adequate for Shared Appl_top Filesystem.

Customers who don’t have NAS based storage around in their data centers, traditionally SAN with FC or iSCSI, have to use Cluster Filesystems like Redhat GFS, IBM GPFS or Veritas VxFS. I have seen a lot of customers chose Redhat GFS for their systems, but my experience with Shared Appl_top on Redhat GFS is not good. GFS has difficulties in handling directories that have lot of small files like Concurrent Manager log and output directories. Simple ls command on this directory hangs forever and seems to effect other nodes response time as well. Note that Oracle doesn’t normally recommend or certify any of the above mentioned Cluster filesystems with EBS. But in recent years, Oracle caved in to customer pressure and certified OCFS2 cluster filesystem with EBS R12 shared appltop.

In initial years of this certification, I had second thoughts about using this for Shared Appl_top. OCFS did not really have a good reputation with Oracle RAC which seems to have affected its credibility in this area. But latest versions of OCFS improved a lot and OCFS2 1.6 performed really well in my recent tests with Oracle EBS R12. I highly recommend OCFS2 in addition to NFS for Oracle EBS Shared Appl_top. Whatever you choose, just make sure to mount them with noatime option for better performance.

It’s here! It’s here! It’s finally here!

(what, you don’t get excited when you get new toys?)

Now that version 12.2 of Oracle Applications has been released, it’s time for eager Apps DBAs and non-DBA EBS geeks to get their hands dirty with new features! This post will walk through the steps required to install a single-node “Vision” instance of E-Business Suite Release 12.2. Apps DBAs can certainly learn a lot about the technical pieces of EBS 12.2 just by installing an empty, unconfigured “Prod” instance. I’m covering a Vision install in this post because I prefer having real data to play with, and don’t have time to learn how to configure all the apps modules myself. :)

This exercise is not for the faint of heart, nor the impatient, but it’s not terribly complicated. If you’ve done Vision installs for the 11i or R12.0/12.1 versions of E-Business Suite, you’re already familiar with the basics. Unlike previous versions of EBS, though, installing a test instance 12.2 onto a laptop or mid-grade workstation is a very tight squeeze. If you have a lab machine with server-class resources, I strongly recommend starting there. Since “can I run this on my personal workstation?” is a common question about EBS Vision installs, however, I’m going to walk through what I’ve done to get 12.2 running on my laptop.

Still with me? Let’s get going.

References

These are early days for EBS 12.2, and while this guide should be sufficient to get you started, you should keep an eye out for additional patches, updates to instructions, etc. Here are the base notes, and a few others released shortly after the software went GA. These are also the references you should consult if you suspect that I missed something in this guide. Hey, it could happen.

Note 1588372.1 has important warnings about setting up your software staging area

Note 1583109.1 (Oracle E-Business Suite Release 12.2 Information Center) is another good starting point to catch recent announcements and updates about Release 12.2.

You will need…

At a minimum, your test server will need:

2 CPU cores, more if you have them.

8GB of RAM. No, really. You’ll need it all. If you skimp on memory, you’ll find that patching in particular will be intolerably long at best, and prone to failure at worst.

Lots of disk space. In addition to the usual 10-12 GB for root+swap, you need 40G for the software staging area, and 275-300G for the database and applications. If you want to split apps and database, allocate 200G for the database and the remainder for the apps. If possible, put your staging area and the install target on separate physical devices.

Access to the internet/web/cloud/whatever, for downloading patches and OS packages. Alternatively, you can transfer these things manually, but who needs that kind of inconvenience?

In addition, you will need a lot of patience, and probably some side projects that don’t involve using a keyboard. This little adventure is going to tie up your workstation for quite a while.

Install steps

Download the EBS install media.
I covered this in my earlier post about 12.2 prep, so I won’t duplicate it here. Pay close attention to which zip files are actually required!

Set up your Linux server.
My server is running 64-bit Oracle Enterprise Linux (OEL) 6.4. If you’re installing on a different platform, your experience may be different. A full walkthrough of how to install Linux is beyond the scope of this post. If you need guidance, I recommend this OEL 6 install guide by the inimitable Tim Hall.I’ve covered most of the setup requirements (packages, disk sizing, etc.) in my previous post. Here are a few more details:

Important warning: Keep your hostname short! There’s an item buried at the end of Note 1320300.1 that warns against using a SID and hostname combination that would make the instance name for the 10.1.2 ORACLE_HOME (EBS_web_SID_hostname) longer than 30 characters. If your hostname is too long, the installer will fail when attempting to create the EBS web tier.

Set up /etc/hosts and /etc/sysconfig/network as described in the “Verifying Host Names” section of Note 1330701.1.

Adjust /etc/sysctl.conf. Full requirements are in Note 1330701.1, though I found that after installing the oracle-rdbms-server-11gR2-preinstall package, I only needed to make one change:

kernel.sem = 250 32000 100 142

Create database and application software owners, oraInventory, and and /etc/oraInst.loc
It’s possible to have the entire instance owned by a single user, but splitting the ownership of the database and applications between two OS users makes management of the services a bit clearer, and more closely reflects the situation you’d expect to see in real life. You could also use the ‘oracle’ user created by the 11gr2 pre-install rpm as the database owner if you prefer, instead of ‘oravis’.

Stage the software (buildStage.sh)Unlike earlier installs of Oracle Applications, staging is not as simple as unzipping all of the install media yourself. Use of the Oracle-supplied script to build a staging area is required. As you might guess from the df output in the previous step, I put the zip files containing the EBS 12.2 software in /mnt/zips. This is where buildStage.sh will find and unpack the install media; the only thing we need to unzip manually is the Rapid Install software.When the buildStage.sh run is complete, look for a “Stage area verified” message. If directories are listed as invalid, or if you see a message indicating that the stage area is incomplete, you’ll need to identify the missing files and add them to your zip file location. Hint: if you need to run buildStage.sh again, move the already-unzipped files out of the zip file stage location, and only add the new files for the next buildStage run. The second run will move along more quickly. Also, be aware that stage area “verification” is very basic: “Do the directories exist? Are they non-empty?” There is still a risk that a corrupted zip file could sink the whole enterprise. Hopefully you validated the zips after download with md5sum. :)

Start the EBS Installer
I ran my installer in VNC, because I don’t have a graphical console configured on my server. If you opted to install Gnome when you installed OEL, then you can just log in as root and launch the installer from the desktop.

Answer all the installer questions (screenshots ahoy, cap’n!)
There are a handful of possible options, but for the most part I’m taking the defaults. Fewer changes means fewer opportunities for error. I’ll experiment more when I’m not writing docs for others. ;-)

A few boring first steps: Opening splash screen; Confirming a new, non-express install; Declining security emails (unless you’re into that sort of thing); Confirming a new configuration…

Take the default port settings (note the two sets of ports, for patch and run filesystems)

Database node configuration. In this case, two changes to make: selecting ‘oravis’ as the datbaase OS user, and changing the base dir to /u01/oracle/VIS. This will cascade through the rest of the config, no need to change elsewhere.

For the app node configuration, just change the Apps OS owner to ‘applvis’

On the next screen, set a password for the Weblogic admin user, and supply the password you’d previously set for the applvis user. I left the remaining passwords at the default values, in compliance with protocol El-Ay-Zed-Why.

Click Next on the node info screen, and the system checks will start. Assuming you’ve sized your disks correctly, and set up the target directories, everything should look great at the end. Otherwise, fix any issues (denoted by a red “X” or “?”) and retry.

When the validation checks are all green, you can launch the install and find something else to do for a few hours:

You’re going to be staring at that “Installing Oracle Applications…” window with its blue bars for quite a while. You can also watch for output from the terminal where you launched rapidwiz, and tail the log files there:

Confirm that you have all green checkmarks in the post-install verification window.
On a slower system, the first pass at checking some of the services (HTTP, WLS Admin, etc) may fail. Before digging too far into troubleshooting, click “Retry” to see if the validation steps pass on a second attempt.

Post-install steps

The following post-install steps are split between Notes 1320300.1 and 1330701.1. There’s patching involved, which is pretty exciting, because you get to use the online patching tool (adop) right away (yay)! They’re hot patches, though, so it’s not a full, true online patch cycle yet (boo). That comes when it’s time to update to EBS 12.2.2 (yay)! I’m presenting these steps in the order that I executed them, which in retrospect might not have been the most efficient, but it didn’t break anything. A few post-install steps are given as pre-requisites for the 12.2.2 upgrade, so in the interest of keeping this post short (hah!) I’m leaving them until a later blog post.

Since some of the steps required everything but the WLS admin server to be down, and one of the steps requires a database bounce, I just stopped all apps services before working on these steps:

You’ll need to supply the WLS admin password in addtion to the APPS username and password when running adstpall.sh. Get used to that.

When reviewing my adop output below, please don’t jump to any conclusions about “normal” patch timing. Remember, this output is generated on a severly underpowered system, and all of these steps were run before I figured out that my VM needed an extra 2GB of RAM. :-)

Patch 17064510 (12.2 Release notes, Note 1320300.1) to fix fs_clone
As promised, here’s our first run of adop, albeit in hotpatch mode, which doesn’t require prepare or cutover/finalize steps. Make sure the WLS admin server is up ($ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start), or you’ll get a message like this:

Maintain snapshots
Not convinced that this is 100% necessary for a “play around” system, but I’m being a good kid and following the docs. :) I’ll spare you the full output this time. Just run:

[applvis@breen scripts]$ adadmin menu_option=UPDATE_CURRENT_VIEW
(many lines of output later)
AD Administration is complete.
Errors and warnings are listed in the log file
/u01/oracle/VIS/fs_ne/inst/VIS_breen/logs/appl/conc/log/adadmin.log
and in other log files in the same directory.

Run utlirp/utlrp to fix timestamp issues with package dependencies
Again, for a test system, one might be tempted to ignore this, but the install docs say it’s mandatory for Vision installs. Remember, we’re going to be recompiling all the PL/SQL in the database. That’s 10’s of thousands of objects. This is going to take a long time. Go play outside or something.

Seed table update: Patch 16605855
One more hotpatch, and this phase of the install is complete. As instructed, stop all app services if they’re started (adstpall.sh). Then start the WLS admin server if it’s not already (adadminsrvctl.sh start)

That’s a wrap! (Sort of…)

At this point, you can start exploring your 12.2 instance a little bit. You’ll find that on the surface it looks a lot like a 12.0 or 12.1 instance. The login URL, if you’ve followed these instructions, should be http://breen.local.org:8000/OA_HTML/AppsLocalLogin.jsp (substitute your hostname for breen.local.org, of course). If you can’t guess the default password for SYSADMIN and other user accounts, you’ll find them in the Install/Upgrade Guide.

But wait, there’s more! In upcoming posts, I’ll cover some basic orientation to your new Vision instance, and then we’ll dive in to upgrading it to version 12.2.2, which is where you’d expect to be if you were deploying 12.2 in real life.

Pythian is a global leader in data consulting and managed services. We specialize in optimizing and managing mission-critical data systems, combining the world’s leading data experts with advanced, secure service delivery. Learn more about Pythian’s Oracle Applications expertise, or take a look at other E-Business Suite (EBS) blog posts.