Friday, June 27, 2008

I'm starting here a loose series of applying the test cases used in the book "Cost-Based Oracle - Fundamentals" by Jonathan Lewis to 10.2.0.4 and 11.1.0.6.

The book covers 8i (8.1.7.4), 9iR2 (9.2.0.6) and 10gR1 (10.1.0.4), so I was keen on running the same tests on 10gR2 and 11gR1.

In the course of doing this I'll attempt to point out if there are any results that are noteworthy.

Before I begin here are the preparation steps and configuration details required to reproduce the test cases.

The INIT.ORA parameters relevant to the optimizer used for 10.2.0.4 and 11.1.0.6 have been taken from the sample INIT.ORAs provided as part of test case code depot. The INIT.ORAs used look like the following:

So no surprises here, works as expected, and the results correspond to those of 9iR2 and 10gR1.

The next script comes from chapter 1 ("What do you mean by cost?") and is called "first_rows.sql". It shows potential problems that may arise from the approach the optimizer takes when using the (deprecated) FIRST_ROWS optimizer mode.

remremִScript:ִִִfirst_rows.sqlremִAuthor:ִִִJonathanִLewisremִDated:ִִִִJuneִ2002remִPurpose:ִִProblemsִwithִFIRST_ROWSִoptimisationrem Adapted to 10g and 11g by Randolf GeistremremִLastִtestedִremִִִ11.1.0.6remִִִ10.2.0.4remִִִ10.1.0.4remִִִִ9.2.0.6remִִִִ8.1.7.4remremִTheִFIRST_ROWSִoptimizerִdoesִnotִhaveִaִsensibleִstrategyremִforִdealingִwithִoptimisationִofִanִORDERִBYִ(andִpossiblyremִotherִvariantsִsuchִasִGROUPִBY)ִwhenִitִfindsִitִcanִuseremִanִindexִtoִdoִtheִORDERִBYִ"freeִofִcharge".remremִThisִexampleִdemonstratesִtheִpoint.ִWithoutִtheִin-lineremִview,ִOracleִdoesִaִfullִscanִonִtheִprimaryִkeyִindexִtoremִreturnִtheִ100ִrequiredִrowsִinִorder,ִatִaִtotalִcostremִandִtotalִexecutionִtimeִthatִisִmuchִhigherִthanִthatִofremִusingִtheִrequiredִindex.remremִOfִcourse,ִitִisִarguablyִtheִcaseִthatִthisִisִcorrectremִbehaviourִifִweִassumeִthatִtheִtimeִtoִtheִfirstִrowִisremִimportant,ִandִthereforeִweִavoidִcollectingִaִlargeִnumberremִofִrowsִandִsortingִthem.ִִInִpractice,ִthisִisִperhapsִnotremִreallyִlikelyִtoִbeִtheִcase.remremִBugִnumberִ670328ִappliesremremִParameterִ_sort_elimination_cost_ratioִaffectsִtheִbreakremִpointִbetweenִoptimisingִforִtheִWHEREִclauseִandִoptimisingremִforִtheִORDERִBYִclause.remremִIfִtheִparameterִisִzero,ִtheִORDERִBYִclauseִtakesִprecedenceremִIfִtheִparameterִisִnon-zero,ִthenִtheִindexִisִusedִifִtheִcostִremִofִusingִitִisִlessִthanִtheִcostִofִdoingִtheִsortִmultipliedִremִbyִtheִvalueִofִtheִparameter.remremִSpecialִnoteִforִ10g:remִTheִparameterִstillִhasִtheִsameִeffectִinִgeneral,ִbutִzeroremִnowִmeansִzero,ִnotִ(effectively)ִinfinity.ִִTheִdefaultִofremִzeroִwillִnowִeffectivelyִignoreִtheִindexִoptionִunlessִitremִisִactuallyִcheaperִthanִtheִnon-indexִoption.ִAִnon-zeroremִvalueִwillִbehaveִasִitִalwaysִusedִtorem

The results in general correspond to those of the previous versions, but one interesting oddity showed up when I did the first test runs. The cost reported by my 10gR2 and 11gR1 systems for the query variations performing an ORDER BY was significantly higher than what is shown in the results that have been provided by Jonathan as part of the code depot.

It looks like the difference comes from the higher cost estimated for the ORDER BY sort. Further investigations showed that Jonathan probably performed the test runs using manual workarea_size_policy with 1M sort_area_size whereas I did the first test runs using the instance default setting of an pga_aggregate_target of 200M (see INIT.ORA parameters above).

The odd thing is that when turning CPU costing off ("_optimizer_cost_model"="io") then for that particular test case performing that tiny sort operation the cost of the sort is actually increasing when the amount of memory available for sorting is increasing. That means in other words, the higher the amount memory available the higher the cost of the small sort operation, due to the fact that the "IO cost per pass" in the optimizer trace file (10053 event) is higher if more memory is available and therefore the final IO cost of the sort is, too.

I was able to reproduce this strange behaviour in 9.2.0.8, 10.2.0.4 and 11.1.0.6. Due to the higher total cost when running the test case with more memory available I had to adjust the "_sort_elimination_cost_ratio" parameter accordingly used in the "first_rows.sql" example to get the same results. See the script for further explanation of this parameter.

All other parameters that I tried (using manual/automatic pga sizing, "all_rows" optimizer_mode instead of "first_rows") didn't change the outcome. When I was using "io" cost model I got in this particular case a consistent increase in I/O cost per pass when increasing the memory available for sorting.

I won't put further effort on this issue for the time being as there is a whole chapter (chapter 13) coming up later that covers sorting related issues in depth, so I'll see if I'm able to dig further there.

The final script from chapter 1 is called "view_merge_01.sql" and demonstrates the complex view merging transformation that can happen to your statement as part of the optimisation steps.

Conclusion for part 1: For first three scripts I couldn't detect any significant differences to the previous versions. A small oddity showed up regarding the I/O cost of a tiny sort operation when disabling the cpu_costing, but that could also be reproduced in 9.2.0.8, so that's not an actual difference to previous versions.

Because I have to use this silly dot character instead of space in order to keep a reasonable formatting in this blogger environment, I'll provide an updated code depot that contains the scripts and results I used for this series.

You can download it from here: My homepage (SQLTools++, an open source lightweight SQL Oracle GUI for Windows)

I'll update it from time to time when I do significant changes, and each time I add a new part to the series you'll find the updated code depot there as well.

The original code depot (still maintained by Jonathan) can be found here.

Sunday, June 1, 2008

Aggregated statistics on partition or global level are by definition of worse quality than actually gathered statistics, mainly because you can not derive each and every statistic by aggregation (e.g. number of distinct column values), nevertheless there might be applications where they are sufficient and maintaining them is far less resource intensive than actually gathering them.

Aggregated statistics are only maintained when calling DBMS_STATS.GATHER_*_STATS if two conditions are met: For all underlying (sub-)partition objects statistics are available (either gathered or aggregated themselves in case of subpartitioning) and there are no actual statistics gathered on the upper level. In case actual statistics have been gathered, these take always precedence over aggregated statistics which means they are never overwritten by aggregation. You can distinguish between aggregated and gathered statistics by checking the column "GLOBAL_STATS" e.g. in the *_STATISTICS dictionary views. If it says "YES", the statistics are gathered and won't be overwritten by aggregation. In case of "NO" the statistics are based on aggregation and can be updated by aggregating the underlying statistics (provided that for all underlying objects statistics are available).

Up to 10gR2 the documentation of ALTER TABLE EXCHANGE PARTITION mentiones that "All statistics of the table and partition are exchanged, including table, column, index statistics, and histograms. Oracle Database recalculates the aggregate statistics of the table receiving the new partition."

This is only correct for the first part, the second part regarding the automatic aggregation of statistics is a documentation bug. In the beginning of partitioning support with Oracle8 according to Oracle this was true but due to customer complaints that this automatic re-aggregation took a significant amount of time (probably in case there were numerous partitions or subpartitions) it has been removed from the default code. There is a (more or less) documented (internal) parameter "_MINIMAL_STATS_AGGREGATION" that has been introduced in Oracle 8i to control the aggregation of statistics. According to the documentation available the following applies: "If the parameter is FALSE, the aggregate statistics will be recomputed. If the parameter is TRUE, the statistics will not be recomputed.". See here the documentation. So you have to be a bit cautious when using this parameter as the setting is counter-intuitive. You have to set it to the non-default value of FALSE to activate the aggregation when exchanging partitions.

While coming across this issue a documentation bug has been raised with Oracle that should correct the last part of the first quoted paragraph regarding the default aggregation rule (so that the documentation correctly mentions that no aggregation is performed by default).

Now looking at the latest available documentation of 11gR1, the particular paragraph reads like this:

"Statistics and histograms on the table or partition are not exchanged. Use the DBMS_STATS package to reaggregate statistics or create a histogram for the table receiving the new partition."

This is again quite interesting as now both parts have been modified. It says now not only that no aggregation takes place but the statistics are not exchanged at all. This would mean a significant change in functionality in 11gR1 compared to pre-11 versions.

I think it is a good idea at this point to come up with a simple test case to check what the actual behaviour is in the current releases 10gR2 Patch Set 3 (10.2.0.4) and 11gR1 (11.1.0.6).

SQL> SQL> -- The exchange table now holds twice the data of subpartition pkey_1_101SQL> SQL> -- Now we get rid of global stats and use instead aggregated statistics on partition and global levelSQL> -- Otherwise exchanging subpartitions will never affect the partition or global level statsSQL> -- Gathered statistics on partition or global level take precedence over aggregated statistics, check the column GLOBAL_STATSSQL> -- If it shows YES, then the statistics have been gathered, if it shows NO then these are aggregated statisticsSQL> -- Aggregated stats are created/updated whenever all partitions/subpartitions of the lower level have statisticsSQL> -- and one of the lower level partitions/subpartitions are analyzed AND the parent level does not have global statsSQL> SQL> exec dbms_stats.delete_table_stats(user, 'partition_test', cascade_parts=>false)

SQL> SQL> -- The exchange table now holds twice the data of subpartition pkey_1_101SQL> SQL> -- Now we get rid of global stats and use instead aggregated statistics on partition and global levelSQL> -- Otherwise exchanging subpartitions will never affect the partition or global level statsSQL> -- Gathered statistics on partition or global level take precedence over aggregated statistics, check the column GLOBAL_STATSSQL> -- If it shows YES, then the statistics have been gathered, if it shows NO then these are aggregated statisticsSQL> -- Aggregated stats are created/updated whenever all partitions/subpartitions of the lower level have statisticsSQL> -- and one of the lower level partitions/subpartitions are analyzed AND the parent level does not have global statsSQL> SQL> exec dbms_stats.delete_table_stats(user, 'partition_test', cascade_parts=>false)

This simple test shows that the documentation is unfortunately still or again wrong since it is obvious that 11g still behaves like 10gR2 including the option to aggregate the statistics when setting the "_minimal_stats_aggregation" parameter.

So in my opinion the mentioned paragraph in the official documentation could be written like the following:"All statistics of the table and partition are exchanged, including table, column, index statistics, and histograms. Oracle Database does not recalculate any aggregated statistics of the table receiving the new partition by default. If you want to have this aggregation to happen automatically each time you exchange a partition, set the parameter '_minimal_stats_aggregation' to FALSE. The default setting of the parameter is TRUE which means that it prevents the aggregation from happening."