The latest news directly from E-Business Suite Development

Which Are Better: Family Packs or Consolidated Updates?

Boy, is my inbox going to get hammered because of this article. But when have I shied from controversy? (Always, a cynical voice mutters). Nevertheless, into the breach I charge.

A recurring question raised at OpenWorld came from customers debating between applying individual Family Packs or the much larger Consolidated Updates to their E-Business Suite environments. Which is better?

If I Were An IT Manager

I understand your dilemma: maintenance windows grow tighter by the
day. Staffing levels don't keep pace with rising workloads. Risk-averse stakeholders pressure you to keep the system
unchanged -- all the while clamoring for their own
personally-reported bugs to be fixed immediately. Given this climate, anything that looks like it makes your life more complex is going to be summarily rejected.

If I had the choice of applying a random and arbitrary combination of individual family packs (e.g. FWK.H + AD.I + XDO.H) or applying a single Consolidated Update (e.g. 11.5.10.CU2), I'd choose the latter without hesitation. If I had had the choice, Consolidated Updates are unquestionably what I'd choose.

Pulling Back the Curtain

The recommendation to apply a much larger patchset with broader impact might seem irrational, especially given what I just acknowledged is the typical IT environment. But bear with me: this may make more sense if I give you a glimpse into our development and testing processes.

How Consolidated Updates are Tested

Consolidated Updates (CU) are tested by the entire E-Business Suite development division in a central testing environment. In fact, Consolidated Updates are repeatedly tested in multiple iterations of these centralized testing environments, some built for automated regression testing, others for cross-product integration flows (e.g. "Order to Cash"), and even others built for internationalization testing with NLS character sets and localizations.

We also test multiple migration paths to a given Consolidated Update. For example, the 11.5.10.CU2 Consolidated Update was tested on top of the 11.5.9 Rapid Install, 11.5.9.CU2, the 11.5.10 Rapid Install, and 11.5.10.CU1.

On top of all that, Consolidated Updates are tested in a variety of so-called "advanced architectural configurations" that include load-balancers, DMZs, Single Sign-On, Discoverer, SSL, RAC, different JInitiator versions, different native Sun Java (JRE) plug-ins, different desktop platforms (e.g. Windows and Mac OS X), and so on.

And then, just because we have too much free time on our hands, the Consolidated Updates are tested on various operating system platforms, including Solaris, HP-UX, IBM AIX, and so on.

How Individual Family Packs are Tested

Individual Family Packs are tested by their respective product teams. They receive product team manual testing and automated regression testing. Depending on the product, they may receive some additional central testing by multiple product teams or in with advanced configurations, but this is relatively rare.

You can see that there are fundamental and profound differences in the depth and range of Consolidated Updates vs. Family Pack testing. A Consolidated Update receives massive, intensive, coordinated testing across all E-Business Suite products.

A reasonable analogy might be: Family Packs are to Consolidated Updates as emergency one-time patches are to Family Packs.

The Sum of the Parts Does Not Equal the Whole

Here's something else to consider: let's say that FIN_PF.G is in 11.5.10.CU2 (this is just an example -- I don't know if this is actually true). Even if you install FIN_PF.G on top of 11.5.9, you aren't officially on 11.5.10.CU2 since you haven't applied the actual 11.5.10.CU2 patchset. If you haven't explicitly applied 11.5.10.CU2 itself, Oracle Support does not consider you to be on that code level, regardless of the sum of the individual Family Packs that you might have applied.

So, when 11.5.9 is desupported, your calls to Oracle Support will prompt the usual discussions about being on a desupported release, even though your individual family packs may be up-to-date to 11.5.10.CU2.

Sometimes "More" is Better

Knowing all that, I always recommend applying the latest Consolidated Update instead of individual Family Packs. This guarantees an end-run around the hassles of possible family pack incompatibilities and desupport issues. In my view, the initial larger overhead of applying a bigger patch is far outweighed by the benefits.

I know that this will be a controversial position for some of you, so I welcome your comments. Let the debate begin...

In general I agree with this article. But what about the poor fellows of us, which are already beyond 11.5.10.2? Once you've applied the CU2 patch (which is already more than 2 years old), you face an unstructured labyrinth of mini-packs, family-packs, "consolidated rollups" and others beyond - it seems that every product development team has its own strategy how to deploy patches beyond 11.5.10.2.
I plead for the next CU patch, which (hopefully) will help us to navigate through this labyrinth.

Agree with Thomas. CU2 is well over 2 years old. What about the 8 missing Family Packs (_FP)?

Also, I'm pretty sure that CU2 did not include a lot of the _FPs or mini-packs due to timings / release dates.

I tell everyone that cares that they *should* apply the CU2 AND all _FP (Family Packs as I thought they were called?) and then any additional module mini-packs from the fully installed and pseudo products list (from patchsets.sh).

This also goes back to asking about a 11.5.10.3 CU. How goes that battle?

Your comparison is right on the money. Patch application is much like using a band-aid. A band-aid benefits the healing process, much like a patch benefits my EBS environment. Ripping off the band-aid quickly and suffering the pain (overhead) over a short time period (as in applying a CU) is much better than incurring smaller increments of pain over a longer time period (applying individual family packs).

As Thomas and John have pointed out, there is an issue with the timeliness of CU patch releases...and therein lies the rub with putting your opinion into practice. CU2 is now so old that customers often have no choice but to begin applying individual FPs in order to resolve their issues (or just to stay current). So customers are ripping off their respective band-aids slowly because we have no other options. Ouch!

So, we're with you: better to apply CUs than FPs. Now, if those of us limping along on 11.5.10.2 just had a CU to apply...

I agree that CU's, when available (yeah, where is CU3??) are the way to go if you can. In some cases, you end up with a humongous, riskier patch set anyway by the time you put together all the pre-reqs. And even if you merge as many of the patches as possible, I would assume that the CU is optimized for reduced downtime to a much better extent than you could do by yourself on 2 FP's and 56 one-offs...

What would really help with those "risk-averse stakeholders" would be if you could be as transparent as possible with the test scenarios and scripts that you have already run in Oracle QA. Even better if they let customers submit scenarios to add to your testing (hmm... cross-post to Mix?)

Funny, I should have anticipated that this article would reignite the discussion about a third 11.5.10 Consolidated Update (aka 11.5.10 CU3).&nbsp; Chalk it up to my post-Thanksgiving haze as I plow through my enormous email backlog.Jim,Good suggestion about our test scenarios and scripts run by Apps QA.&nbsp; We make all of our automated Apps 11i &amp; 12 regression testing scripts and infrastructure available free-of-charge via Metalink.&nbsp; These scripts are built with HP Mercury's WinRunner, and are released for each major E-Business Suite release.&nbsp; You can find more details about them (and a pointer to the master Metalink documentation for them) in this article:Automated Testing for the E-Business Suite&nbsp; http://blogs.oracle.com/schan/2006/06/automated_testing_for_the_ebus.htmlAll,It's true that those of you already on 11.5.10.CU2 still face additional patching requirements.&nbsp; It's also true that the number of those patches continues to grow.In his executive Q&amp;A at OpenWorld 2007, Cliff Godwin noted that there are currently no plans for creating a new 11.5.10.CU3 release. &nbsp; As you can tell from my description of the activities associated with putting out Consolidated Updates, decisions to release these updates aren't made lightly.&nbsp; CUs require an enormous concerted mobilization of resources across the Apps Division. As time goes on, the gap between 11.5.10.CU2 and the latest released patches grows, so I continue to present your feedback to our management team about the need for a third CU (I suspect that people are starting to find my repeated carping about this a touch annoying, but I persist).&nbsp; You should know that a collective customer request carries more weight than individual ones.&nbsp; One possible avenue for ensuring that your voices are combined in chorus would be to work with OAUG on polling the membership on this issue.&nbsp; I would be curious to see what weight a petition would carry.Regards,Steven&nbsp;

OAUG poll - Done. I will be drafting up a note from the OAUG Upgrade SIG Chair (me :-)) and will submit it to the user base to see what their responses will be...

Whose email address would you like me to use? In all seriousness, I think the response would be overwhelming and my gmail account wouldn't cut it and I don't want to saturate your account any more than it is already.

Why not set up a EBS customer poll on the need for a CU3 on Mix as a question to be answered by EBS customers? It's an easy way to gather feedback and the Mix team would love to have such a large group "kicking the tires".

John, Floyd,I'm all in favor of multiple lines of convergence when it comes to assessing customer demand.&nbsp; It's likely that different groups of customers will respond to an OAUG survey vs. an Oracle Mix poll, so it's useful to have both avenues of communication.I'll monitor the Mix poll regularly.&nbsp; John, feel free to use me as your contact for the OAUG responses.&nbsp; Fear not for my inbox -- it's tough; it can take it.Regards,Steven

In the spirit of Web 2.0, I decided that action was better than just yaking about it. Those that are interest in expressing their support or disapproval for a CU3 can vote on the idea at https://mix.oracle.com/ideas/13656-cu3-patch-for-ebs-11510 (you need to join Oracle Mix in order to vote).

Steve - I looked at the test scripts you mentioned (thanks for the reminder!) but they are only for the CU level, and as many have pointed out, CU2 is a good baseline, but most people need CU2+ minipacks, etc.

I would assume that there are no test cases for the CU2+ known bugs in those test scripts, since if they were known before release they (probably) would have been fixed. So what we would really need is the test plans from the mini-packs as well, and ideally the one-offs also. Obviously these tests would all pass, but at least we could focus our test time on the stuff that you aren't already testing.

Are these other tests not done by WinRunner, or is there some other reason they are not released?

Floyd,John Stouffer organized an email-bombing campaign via the OAUG SIG mailing list in parallel with your Oracle Mix poll.&nbsp; My inbox cringed in fearful anticipation of the responses.&nbsp; However, his email campaign only resulted in 23 votes for 11.5.10.CU3, some of which I suspect overlap with the Oracle Mix respondents and commenters on these threads.&nbsp; You're right, I'm afraid.&nbsp; I don't believe that this comes close to the level of customer demand to revive interest in producing a third Consolidated Update.Regards,Steven&nbsp;