Sunday, March 29, 2009

Update Jan 2011: Since this post is still among the most popular ones of this blog although being almost two years old it is probably worth an update.

- The most important information that you need to be aware of if you plan to use SQL Profiles is that you need an Enterprise Edition + Diagnostic Pack + Tuning Pack license. If you don't have these licenses you are not allowed to use SQL Profiles, or the other way around Oracle can claim you need to pay these licenses if you're going to use SQL Profiles.

Therefore if you're already on Oracle 11g you might want to use SQL Baselines instead for the same purpose - they seem to be available in all Editions and don't require any further licenses.

Update August 2012: SQL Baselines are only available with Enterprise Edition.

Jonathan Lewis for example recently wrote a short note on how to apply a baseline from a hinted statement to a non-hinted, which is what you usually want to achieve when dealing with third-party applications where you can't modify the source code.

More details about SQL Baselines can be found in the documentation, Kerry Osborne for example has some more examples and quirks he found summarized in his post about SQL Baselines.

If you're not yet on 11g and therefore can't use SQL Baselines you can still use Stored Outlines instead of SQL Profiles if you don't have the licences mentioned - as shown below the DBMS_OUTLN.CREATE_OUTLINE procedure unfortunately doesn't always work as expected. Therefore you can try to "hack" a Stored Outline as for example demonstrated by Charles Hooper here. His post also contains references to other sources on My Oracle Support and by Jonathan Lewis that describe that technique in more detail.

- I'm a bit puzzled that this post is such popular. I spend a significant amount of my time on performance related issues but I rarely resort to the techniques described here and the other mentioned posts about "Plan Stability". So I'm a bit curious and would like to encourage readers to leave a comment here why they think they need to use "Plan Stability" - there are usually a lot of others options I would evaluate first before thinking about using some kind of Plan Stability.

Original post:
If you have the need for plan stability - that is telling the database to use a particular execution plan no matter what the optimizer thinks otherwise - then you might be in the situation that the "good" execution plan is already available in the shared pool or in the AWR, so it would be handy if you could simply tell Oracle to use that particular execution plan to create a Stored Outline.

Note that in 11g this is all possible using the new SQL Plan Management framework (SPM), but that is not available in 10g, so we need to think differently.

In 10g the DBMS_OUTLN package has been enhanced with the CREATE_OUTLINE procedure to create an outline from an existing child cursor in the shared pool.

Please note that in releases prior to 10.2.0.4 there was a severe bug that caused your session to crash when using DBMS_OUTLN.CREATE_OUTLINE (Bug 5454975 which has been fixed in 10.2.0.4). The workaround is to enable the creation of stored outlines by issuing "alter session set create_stored_outlines = true;" before using DBMS_OUTLN.CREATE_OUTLINE. For more information see the Metalink Notes 463288.1 and 445126.1.

Note that from 10g on the hints required to create an outline are stored as part of the plan table in the OTHER_XML column as part of the XML detail information.

You can use the ADVANCED or OUTLINE option of the DBMS_XPLAN.DISPLAY* functions to display that OUTLINE information. For more information see e.g. here.

So that didn't work as expected. Although we were able to create an outline from the child cursor, it obviously didn't use the plan associated with the child cursor. Tracing the session didn't reveal why the CREATE_OUTLINE didn't use the outline information available from the shared pool.

Running the same test case in a slightly different order so that the outline is created before the statistics change corroborates the theory that the DBMS_OUTLN.CREATE_OUTLINE procedure might take the SQL from the cursor and internally execute an CREATE OUTLINE ... ON ..., and for whatever reason doesn't use the already available outline information.

So that worked, but still the question remains why DBMS_OUTLN.CREATE_OUTLINE doesn't use the available outline information in the shared pool.

Now let's turn to a different approach to achieve the same. 10g introduced SQL profiles that are primarily used to amend information that is not available to the cost based optimizer, e.g. in case of correlated column values the SQL Tuning Advisor of 10g can suggest to accept a SQL profile that scales the cardinality estimate so that the cardinality estimate is in the right ballpark.

A good explanation of SQL profiles can be found in Christian Antognini's publications.

But since SQL profiles internally consist of a set of hints, it could be possible to use SQL profiles instead of Stored Outlines to achieve the same.

There are two interesting aspects regarding this approach:

- We could use different sources to get the outline, e.g. instead of the shared pool we could get the hints from the AWR tables.

- SQL profiles support a "FORCE_MATCH" option that works similar to the CURSOR_SHARING literal replacement logic, i.e. SQL profiles can be forced to apply to multiple SQL statements that differ only by the literals used (i.e. no usage of bind variables).

So we are faced with two challenges in this regard:

1. Get the outline information, i.e. the full set of hints to provide plan stability
2. Create a SQL profile that consists of these hints

Get the outline information

There are two ways how the outline information could be obtained:

a) Use the DBMS_XPLAN.DISPLAY* functions with the ADVANCED or OUTLINE option and parse the this output to get the set of hints

b) Directly query the underlying tables/views to get the XML stored in the OTHER_XML column and extract the hints from that XML

a) Use the DBMS_XPLAN.DISPLAY* functions

Let me digress a little bit. Looking at the (already parsed a bit) output we get from the official DBMS_XPLAN function:

You'll notice that the INDEX_RS_ASC hint is split across two lines, so we can't simply use that query output to construct the hints because these hints would be potentially illegal and therefore we need to merge/concatenate these split lines.

This is a variation of the well known "columns-to-rows" aka. STRAGG/CONCAT issue and there are multiple ways how to deal with that using plain SQL.

It simply takes a collection of varchar2(500) strings that make up the profile.

So we can combine the two things into a procedure that generates us a SQL profile from either the shared pool or the AWR. Here's one for the shared pool. It takes four parameters: The SQL_ID, the child_number, the SQL profile category and whether to force a match or not.

1. The SQL profile created forces the plan we wanted, so it seems to work as expected
2. The FORCE_MATCH option of the SQL profiles allows to use this profile even for SQLs that are not an exact text match of the original statement. This is something that is as far as I know not possible using Stored Outlines.

So if you have the need to fix the execution plan, and you have that plan already in the shared pool or the AWR, using above procedures allow you to generate a SQL profile which seems to do exactly what we want.

Given the fact that the SQL profile even allows to share the plan for SQLs that differ only by literals I definitely favor the SQL profiles over the Stored Outlines approach.

This is Awsome information. I appreciate your posts on OTN a couple of weeks back, and I am so glad you are sharing this information. I was just about to go through this same procedure myself. Just today, I used display_awr to get the old plan back, and then create a sql_profile from the hints.

The only additional information I would give is to utilize sys.WRH$_SQLSTAT. By querying this view, and joining to dba_hist_snaphot, you can view historic information about how the query performance for each plan used.. I usually select out the hash_plan_value,executions_delta,gets_delta/executions_delta,elapsed_time_delta/executions_deltafor a sql_id.I then look at the elapsed time, and the different hash_plan_values to find the correct plan to go back to.. You can then use display_awr to show the hints you want for that specific plan.

This makes it very useful to go back and force an old plan that was right quickly, and concretely.

I have been using your scripts to create profiles quite a bit lately. The ability to go back to a plan in AWR quickly and easily is really useful. I actually wrote a little wrapper for both of your scripts (documented here SQL Profiles as you know) I did find small problem with the one that creates the profile based on the text in the shared pool though. I got a message saying that the SQL statement already had a profile on it when trying to create one based on a SQL statement in the shared pool. Turns out your script had used sql_text (which is a clob with the complete SQL text in DBA_HIST_SQLTEXT but is a truncated (1000 characters) version in in v$sql. So I changed your script to use the sql_fulltext column, which is a clob and has the whole string.

Yeah, it's all on the same system actually. When I say a lot I mean like 10 of them. They are all pretty much variations of the same query. (that's why I noticed the truncated SQL Text - several of them have an exact match on the first 1000 characters). Anyway, it's a long story, but basically the Profiles are a bandaid until we can fix the underlying cause. Nevertheless, it's been very helpful because the swing between the plans is pretty huge - one plan does a few hundred lio's, the other does s few hundred thousand lio's.

Very useful post and I have a question related to it.Today I have tried to create a outline using a dbms_outln.create_outline and hit issue with a plan stability. When I have created outline using dbms_outln package for that same query it was different plan then I created outline using create_stored_outlines = true.

If you read through the post you'll notice that in the beginning of the post I describe my finding that DBMS_OUTLN.CREATE_OUTLINE obviously didn't use the plan associated with the referenced child cursor from the shared pool, but probably re-optimized the SQL statement (parent cursor) associated with the child cursor and therefore generated a different plan to use for the outline in my case, because the underlying statistics had changed in meanwhile.

May you've hit a similar issue, I don't know?

It would be interesting to know, because I haven't investigated further why exactly DBMS_OUTLN.CREATE_OUTLINE behaved so counter-intuitively and didn't use the most obvious information available: The hints associated with the child cursor in the shared pool.

Sorry for my not clear post.Exactly I miss this very important statement from you post about creating outline using dbms_outln. I hit this same issue - different outline is created in case of using of dbms_outln - it looks like dbms_outln generate a new plan for a query.

Hi, Can outline hints ( consisting of execution steps) be used directly as hints in SQL rather than generate profiles and use these ? This can be helpful in extracting RBO's outlines and then forcing execution steps without profiles or outlines .The Query optimizer should have some additional hint that instructs it to use following outline hints in exec path. ThxAnil Bishnoie

> Can outline hints ( consisting of > execution steps) be used directly > as hints in SQL rather than > generate profiles and use these ?

Anil,

as far as I understand an Outline consists exactly of those set of hints. You should be able to apply them directly and 1:1 to the corresponding SQL statement to get the desired execution plan if no structural changes to the database have been applied that make some of the hints invalid.

> The Query optimizer should have > some additional hint that > instructs it to use following > outline hints in exec path.

In principle the "normal" hints contained in the Outline should be sufficient to force the desired execution plan, however there are actually some "special" hints contained in the Outline hint set. Those are the OUTLINE and OUTLINE_LEAF hints. I think Jonathan Lewis in some of his notes/blog posts had a good explanation what he thinks they are used for.

Plan instability is the bane of my Oracle existence. Far too often I can look in sql history and find a better plan than the one Oracle is using now. Of course this is a very small percentage of overall queries, and I generally only work on queries where there are complaints about performance, but still I see too many of them. Now that I know how to apply a profile from one query to another, I can either apply a profile from the historically good plan, or else if there's no history of a better plan, rewrite the query so that it performs well, run the optimized version in the problem database, and create a profile for the production query from the tuned query. What could be simpler than that first case - take the historically good plan, create a profile from its outline hints, fix your performance problem? And in the second case, if it's a third party application, and I don't want to start a statistics research/gathering exercise that might need to be repeated as statistics age, why wouldn't I want to take the outline hints from an optimized query and create a profile from them? I'm seeing good results from both cases. Seems like a great technique to me. I have trouble understanding why someone wouldn't use these techniques. We're already paying for those packs, why not use them? It's just too bad that the profiles aren't really permanent in their effects - I've seen Oracle stop using profiles that don't include cardinality or scale hints. I think I understand why Oracle doesn't supply a 100% guaranteed technique for freezing an execution plan, even though there queries that will need only a single plan for the life of a system are a pretty common class of query - the reason is that their responsibility is to their shareholders, not to their customers. But maybe I'm wrong and 11g plan baselines are really guaranteed to stick. Anyway thanks for your articles about sql profiles and for the pointer to Jonathan Lewis's baseline transplant article, I'm sure I'll want to look into using that method in 11g. A recap of why use sql profiles: when Oracle shows you that it failed to apply its previous good solution to a problem, and when it's easy to remind Oracle about that solution via application of a sql profile from a historically good plan, or when you can easily transplant hints from an optimized query into a profile, and in both cases the performance issues are restricted to a single query, and if you're already paying for the licenses, why wouldn't you want to use profiles?

Slight correction - plan instability USED TO BE the bane of my Oracle existence, but since I learned to apply profiles from history or from optimized queries, plan instability is often more of an opportunity to provide a simple, quick, localized solution than it is "the bane."

Almost all of this work has been done in 10g. Often I've been busy to the point where I just go with the sql profile fix because it's a quick fix that allows me to get back to whatever else I was working on. If I'm busy and I see a plan regression that can be fixed with a sql profile, I'll do that every time because a) the company needs to have the ability to apply such fixes and won't learn them unless I use them and advertise them and b) it's fast and usually gets the job done. Root cause doesn't matter much if you're too busy to do the drill down.

That's true for a subset of problems, the plan regression problems, and sometimes for the queries that need to be rewritten to keep the optimizer from getting the plan wrong. In some the optimizer just flat gets it wrong - join order, or timing of predicate application , or both - and I often can rewrite the query so that it performs well. I call it "immunizing your query against the optimizer mistakes" by just being really explicit with ordering and by using no_merge inline views to keep Oracle from not using what I'm telling it about join orders and how the tables should be joined and filtered. Sometimes I've tried to use the outline hints for the rewritten queries to create profiles for other queries. If it works, good, I rush back to whatever else I was working on. Root cause? We can address that later. When you work for a company where for some applications there's not much thought given to application tuning until after rollout, well maybe you can imagine that I don't have time to go after root cause on every plan regression.

you haven't mentioned a version in your comment, but in 11g your problem should be picked up by Adaptive Cursor Sharing, however only after things went wrong at least once. In pre-11g there is probably not really a good way to handle this from the database side only, so "application awareness" and corresponding code changes are in order.

For more information about ACS and what you can do from application side, you might want to read this post

Hi Randolf,Thanks for this post.I have a situation where I have to bring a good plan from one database to another.So, I should be able to just export the good plan hints, import the table into the other database and use this table to bulk collect into ar_profile_hintsand import the profile correct?Or will there be a need of some additional steps?

As mentioned in the beginning of the updated post, I would recommend using SQL Plan Baselines instead of SQL Profiles for general plan stability.

Only use SQL Profiles for that purpose if you need special features like the "FORCE_MATCH" option that isn't available for SQL Plan Baselines - and note that you need a Diagnostics + Tuning Pack license to use SQL Profiles - which you don't need for SQL Plan Baselines.

If you want to use SQL Profiles somehow getting the list of required hints to your remote system should be sufficient to get the job done.

An official procedure for doing so was provided by Carlos Sierra some time ago:

Randolf , thank you very much for this very insightful article.. However I had a query regarding Baselines ... What about a query that runs for both a very small amount of data and a very large amount of data.If the baseline is set for large amount of data then when we run for a small amount of data it will take minutes instead of seconds. The other way around (set the baseline for small and then run large) could make the large query go from hours to days. Is this possible and if it is so , is there anyway that this can be avoided ?

Remember that one of the features of SQL Plan Baselines is that you can have more than one plan associated with a SQL statement. So nothing stops you from adding two enabled plans as Baselines. Provided the optimizer then chooses the right one automatically of course.

It also depends on your exact circumstances - are we talking about the same SQL statement using binds, are we talking about the same SQL statement and the underlying tables do have vastly differing data volumes and so on.

But as I said, a Baseline can have multiple plans associated, so maybe that is already what you're looking for.

I think you explain clearly why using an outline is a bad idea. Data changes, a schema will grow between now and 2 years. The plan should adapt.

I find dbms_outline to be useful tool for very specific situations when all hopes are lost.

This happens to be the unfortunate case for me at the moment:1) Migration from 11g to 12c.2) Discovery of an important sub 0 second query now performing at 23 seconds.3) Investigation shows that 12c struggles with subqueries using UNION all.4) Compounded problem when baselines appear to fail for the same reason (We are trying to "stop the bleeding").5) I'm left with producing an outline from 11c (ALTER SESSION SET optimizer_features_enable='11.2.0.4.0'; ) and now applying it.6) Now I'm working with oracle on a root cause and gaining directives on what must be avoided in the query and what we need to look into changing in our code.7) When changes are applied: Remove outline.

It feels a bit like a game of chess when my moves are dictated by my opponent and I'm not winning.

What I'm interested in and don't understand from your comment is why SQL Plan Baselines failed and you're left with the Outline - after all I can't see what functionality the Outline offers that the Baseline doesn't offer - hence I could only think of a bug, but I would like to hear more about this.

In principle you should be able to produce a Baseline from the plan you get with OPTIMIZER_FEATURES_ENABLE = 11.2.0.4 as for example described in the linked article by Jonathan Lewis.