db-derby-dev mailing list archives

[jira] [Commented] (DERBY-5680) indexStat daemon processing tables over an over even when there are no changes in the tables

Date

Mon, 02 Apr 2012 18:37:23 GMT

[ https://issues.apache.org/jira/browse/DERBY-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244421#comment-13244421
]
Mike Matrigali commented on DERBY-5680:
---------------------------------------
I reviewed the change, could use a few more comments for those of us not familar with data
design of the statistics catalog, but I would
have expected it fix the problem. Not sure what the issue with the assert is, there is a
comment about when we turn on write mode, but
not sure when it is expected to get turned off - maybe at commit?
I was hoping the change could be less invasive by just dropping all statistics even bad ones
upfront with no checking, ie. not have to reget all the valid rows and recheck them always.
But I see the current
code logic wants to drop statistics one at a time, and that were problems trying to invalidate
all at once which might be similar to problems with dropping all rows at once.
It is not part of this change, but is there a possible problem with when invalidateStatements
is called? Unless you need it for some
locking reason it would seem better to invalidate after you do the update than in the middle.
Can queries sneak in between when a statistic is dropped and when you insert the new updated
statistic? If so it would be another reason
not to drop them all up front as it would increase the window. brett was reporting some wierdness
with working queries getting recompiled
to bad plans. I think we can think of his system as always having concurrent processors availble
to run statements constantly, so they
can almost instantaneously start recompiling once you invalidate their plan. Now if they
can get in after the invalidate, after the drop but before
the insert then they probably have worse stats (ie. no stats), vs the old stats.
> indexStat daemon processing tables over an over even when there are no changes in the
tables
> --------------------------------------------------------------------------------------------
>
> Key: DERBY-5680
> URL: https://issues.apache.org/jira/browse/DERBY-5680
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.8.2.2
> Reporter: Brett Bergquist
> Attachments: derby-5680-1a-drop_orphaned_stats.diff
>
>
> I think there is something wrong with the indexStats.
> The problem happens on many tables in the database.
> None of these tables are changing however, no inserts or deletes or updates. They are
being queried, however.
> Here is one such table.
> Here is the statistics for this table:
> Table (Index) 2 3
> ACCOUNTTABLE_CONFIG_BUNDLE (SQL081029110443810) numunique= 38390 numrows= 38390
2012-03-30 13:00:26.84
> ACCOUNTTABLE_CONFIG_BUNDLE (SQL100922215819290) numunique= 38390 numrows= 38390
2012-03-30 13:00:26.917
> There are in fact 38390 rows in the table.
> Here is some of the indexStat trace:
> Fri Mar 30 12:47:12 EDT 2012 Thread[DRDAConnThread_43,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
update scheduled, reason=[t-est=38390, i-est=2355 => cmp=2.7912562815443245] (queueSize=12)
> Fri Mar 30 12:47:48 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL081029110443810 (fc33890d-011d-491f-3d8c-0000376d74d3): rows=38390,
card=[38390]
> Fri Mar 30 12:47:48 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL100922215819290 (75608675-012b-3c38-b55c-000043ea6398): rows=38390,
card=[38390]
> Fri Mar 30 12:47:48 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
scan durations (c30625=91ms,c30625=98ms)
> Fri Mar 30 12:47:48 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
generation complete (210 ms)
> Fri Mar 30 12:47:49 EDT 2012 Thread[DRDAConnThread_44,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
update scheduled, reason=[t-est=38390, i-est=2355 => cmp=2.7912562815443245] (queueSize=19)
> Fri Mar 30 12:48:25 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL081029110443810 (fc33890d-011d-491f-3d8c-0000376d74d3): rows=38390,
card=[38390]
> Fri Mar 30 12:48:25 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL100922215819290 (75608675-012b-3c38-b55c-000043ea6398): rows=38390,
card=[38390]
> Fri Mar 30 12:48:25 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
scan durations (c30625=93ms,c30625=95ms)
> Fri Mar 30 12:48:25 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
generation complete (211 ms)
> Fri Mar 30 12:48:25 EDT 2012 Thread[DRDAConnThread_50,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
update scheduled, reason=[t-est=38390, i-est=2355 => cmp=2.7912562815443245] (queueSize=18)
> Fri Mar 30 12:48:57 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL081029110443810 (fc33890d-011d-491f-3d8c-0000376d74d3): rows=38390,
card=[38390]
> Fri Mar 30 12:48:57 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL100922215819290 (75608675-012b-3c38-b55c-000043ea6398): rows=38390,
card=[38390]
> Fri Mar 30 12:48:57 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
generation complete (243 ms)
> Fri Mar 30 12:49:27 EDT 2012 Thread[DRDAConnThread_56,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
update scheduled, reason=[t-est=38390, i-est=2355 => cmp=2.7912562815443245] (queueSize=20)
> Fri Mar 30 12:49:36 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL081029110443810 (fc33890d-011d-491f-3d8c-0000376d74d3): rows=38390,
card=[38390]
> Fri Mar 30 12:49:37 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
wrote stats for index SQL100922215819290 (75608675-012b-3c38-b55c-000043ea6398): rows=38390,
card=[38390]
> Fri Mar 30 12:49:37 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
scan durations (c30625=111ms,c30625=108ms)
> Fri Mar 30 12:49:37 EDT 2012 Thread[index-stat-thread,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
generation complete (238 ms)
> Fri Mar 30 12:49:37 EDT 2012 Thread[DRDAConnThread_49,5,main] {istat} "PKG_9145E_V1"."ACCOUNTTABLE_CONFIG_BUNDLE":
update scheduled, reason=[t-est=38390, i-est=2355 => cmp=2.7912562815443245] (queueSize=18)
> As can be seen, the "i-est" appears to be wrong and is used over and over even though
the statistics for the indexes have been updated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira