Create an option that allows a query to be cached, but not used for warming

Details

Description

The application that uses my Solr install builds complex filter queries for employees because they have access to everything, whereas most users have access to a small subset.

Because of this, autowarming on the filterCache can take 30-60 seconds even though autoWarm is set to just 4 queries.

If we had a way (probably a localparam) to tell Solr to not use those filters when autowarming, but to go ahead and put them in the filterCache and use them until there's a new commit, that would eliminate this problem. Employees might have their queries take longer, but regular users would not be affected.

Erick Erickson
added a comment - 06/Apr/12 15:11 Are there other auto-warming queries you want to have done? Because it almost sounds like you just want to turn off autowarming in the filter cache.
Or if they're unlikely to be re-used anyway, would it work to set cache=false on the original fq?

I would like to have our application code tag those nasty employee filters with something that makes them ineligible for autowarming, but still eligible for caching, which would keep them around until the next commit. I am pretty sure our code is capable of knowing that the user is a special user, typically admin or system.

An update cycle runs once a minute for the index as a whole, but changes are tracked on a per-shard basis. Commits on each shard are only done if something on that particular shard actually changes. The large shards where this is a problem typically go several minutes between commits, and that might extend to an hour or more.

I will talk to our developers about using the cache=false localparam for now, but I am hoping for the ability to use the cache for those nasty filters but not include them for warming. Having recently toyed with the cache code (SOLR-2906), I know this may not be trivial.

Shawn Heisey
added a comment - 06/Apr/12 19:33 I would like to have our application code tag those nasty employee filters with something that makes them ineligible for autowarming, but still eligible for caching, which would keep them around until the next commit. I am pretty sure our code is capable of knowing that the user is a special user, typically admin or system.
An update cycle runs once a minute for the index as a whole, but changes are tracked on a per-shard basis. Commits on each shard are only done if something on that particular shard actually changes. The large shards where this is a problem typically go several minutes between commits, and that might extend to an hour or more.
I will talk to our developers about using the cache=false localparam for now, but I am hoping for the ability to use the cache for those nasty filters but not include them for warming. Having recently toyed with the cache code ( SOLR-2906 ), I know this may not be trivial.

I never actually answered your first question. Yes, I do want most entries in the filter cache to be usable for autowarming. Most users have relatively few boolean clauses in their filter queries. Employees are the common exception. We get a few hundred boolean clauses in ours. Plans are being discussed to greatly reduce that, but I'm not sure we'll ever get away from it entirely.

Shawn Heisey
added a comment - 06/Apr/12 20:02 I never actually answered your first question. Yes, I do want most entries in the filter cache to be usable for autowarming. Most users have relatively few boolean clauses in their filter queries. Employees are the common exception. We get a few hundred boolean clauses in ours. Plans are being discussed to greatly reduce that, but I'm not sure we'll ever get away from it entirely.

Shawn Heisey
added a comment - 19/Apr/13 18:14 I hope someone can fix this, but I know that at this time it's not something I can tackle without generous hand-holding. If there are no takers soon, I'll go ahead and close the issue.
This is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean