expose extra statistics like slow queries

Details

Description

Solr Objects may want to expose extra information for diagnostics etc. These information can be large and we may not wish to show it in the stats page which can make it unwieldly. I propose an extension of the stats itself which can be shown separately.

I wish to implement slow queries as a part of this. There is a new interface which can be implemented by components

publicinterface StatsDetails {
publicinterface StatsDetails {
/**The details info is a Map of details name versus description. The details name must
* return the details when the getDetails() method is invoked on the same object.
* @return a map of details name vs the description. the description may be used in a link text
*/
public Map<String ,String> getDetailsInfo();
/**The details key should be known forthisObject .
* This key will may be obj=tained by invoking getDetailsInfo()
*
* @param detailsCommand
* @return
*/
public NamedList getDetails(String detailsCommand);
}

Hoss Man
added a comment - 21/Mar/12 18:08 Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.
email notification suppressed to prevent mass-spam
psuedo-unique token identifying these issues: hoss20120321nofix36

Hoss Man
added a comment - 27/May/10 23:09 Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...
http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E
Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.
A unique token for finding these 240 issues in the future: hossversioncleanup20100527

publicinterface StatsDetails {
/**The details info is a Map of details name versus description. The details name must
* return the details when the getDetails() method is invoked on the same object.
* @return a map of details name vs the description. the description may be used in a link text
*/
public Map<String ,String> getDetailsInfo();
/**The details key should be known forthisObject .
* This key can be obtained by invoking getDetailsInfo()
*
* @param detailsCommand
* @param params Any extra parameters which the command may understand. can be null also.
* @return
*/
public NamedList getDetails(String detailsCommand, SolrParams params);
}

Noble Paul
added a comment - 08/Apr/09 07:20 - edited Hi Hoss ,
you are right. It has no means to drill down further
How about this.
public interface StatsDetails {
/**The details info is a Map of details name versus description. The details name must
* return the details when the getDetails() method is invoked on the same object.
* @ return a map of details name vs the description. the description may be used in a link text
*/
public Map< String , String > getDetailsInfo();
/**The details key should be known for this Object .
* This key can be obtained by invoking getDetailsInfo()
*
* @param detailsCommand
* @param params Any extra parameters which the command may understand. can be null also.
* @ return
*/
public NamedList getDetails( String detailsCommand, SolrParams params);
}

my point is that instead of a marker interface that the components implement (which would only allow one level of drill down) the marker interface could be on new objects put into the NamedList returned by the existing getStatistics() method – when that interface is detected, show getLabel(), otherwise show toString().

the extended display (extrastats.jsp if you want, or some new option on stats.jsp) would be able to getDetails() on the object ... and any of those details might themselves implement StatsDetails to allow further drilldown.

Hoss Man
added a comment - 08/Apr/09 06:53 A simple string may not be quite sufficient
sure ... no disagreement.
my point is that instead of a marker interface that the components implement (which would only allow one level of drill down) the marker interface could be on new objects put into the NamedList returned by the existing getStatistics() method – when that interface is detected, show getLabel(), otherwise show toString().
the extended display (extrastats.jsp if you want, or some new option on stats.jsp) would be able to getDetails() on the object ... and any of those details might themselves implement StatsDetails to allow further drilldown.

Yes. The understanding is correct. Just that the detailed stats may be rendered by a separate jsp say extrastats.jsp?obj=foo&detail=name

The XSL based rendering is a problem for the current stats.jsp . XSL is too cumbersome for any advanced UI

The default stats.jsp can provide links with this information so that users can navigate from there and the current stats.jsp can be a entry point for further drill down. But it should also be possible to hit the extrastats.jsp directly which can just provide the links to all the objects with extra statistics

Another thing to consider is that stats has always displayed the toString() of the values in the getStatistics() NamedList

The concern here is that, the extra stats will be like mini apps which will give options to view and mine tons of data,
eg:

contents of my queryResultCache

hits for some items etc

all the queries which took more than 100 ms

or the ones where this 'fq' is appplied
*etc etc

These things would need features like faceted drill down, pagination etc. A simple string may not be quite sufficient

Noble Paul
added a comment - 08/Apr/09 05:08 is that what you had in mind?
Yes. The understanding is correct. Just that the detailed stats may be rendered by a separate jsp say extrastats.jsp?obj=foo&detail=name
The XSL based rendering is a problem for the current stats.jsp . XSL is too cumbersome for any advanced UI
The default stats.jsp can provide links with this information so that users can navigate from there and the current stats.jsp can be a entry point for further drill down. But it should also be possible to hit the extrastats.jsp directly which can just provide the links to all the objects with extra statistics
Another thing to consider is that stats has always displayed the toString() of the values in the getStatistics() NamedList
The concern here is that, the extra stats will be like mini apps which will give options to view and mine tons of data,
eg:
contents of my queryResultCache
hits for some items etc
all the queries which took more than 100 ms
or the ones where this 'fq' is appplied
*etc etc
These things would need features like faceted drill down, pagination etc. A simple string may not be quite sufficient

Noble: I presume by "There is a new interface which can be implemented by components" you're suggesting that this StatsDetails could be a marker interface implemented by things that already implement SolrInfoMBean, and if stats.jsp noticed something implementing this interface, then it would call getDetailsInfo() and for each name returned include a link to a future call of something like "stats.jsp?obj=foo&detail=name" which would call getDetails(name)

is that what you had in mind?

Another thing to consider is that stats has always displayed the toString() of the values in the getStatistics() NamedList ... perhaps a marker interface could be added to those objects, indicating when the toString behavior should be overridden and replaced with the ability to drill down...

Hoss Man
added a comment - 08/Apr/09 01:45 - edited Noble: I presume by "There is a new interface which can be implemented by components" you're suggesting that this StatsDetails could be a marker interface implemented by things that already implement SolrInfoMBean, and if stats.jsp noticed something implementing this interface, then it would call getDetailsInfo() and for each name returned include a link to a future call of something like "stats.jsp?obj=foo&detail=name" which would call getDetails(name)
is that what you had in mind?
Another thing to consider is that stats has always displayed the toString() of the values in the getStatistics() NamedList ... perhaps a marker interface could be added to those objects, indicating when the toString behavior should be overridden and replaced with the ability to drill down...
public interface DetailedStats {
public String getSummary();
public NamedList getDetails();
}
public class DetailedStatsNamedList extends NamedList implements DetailedStats {
public DetailedStatsNamedList( String summary) { this .summary = summary; }
public NamedList getDetails() return this ;
// ... hashCode & equals & toString
}
...
// stats.jsp would print getSummary() as a link to unwrapping getDetails()
...
This would have the advantage of allowing deeper and deeper levels of details to be exposed if the stats.jsp code was written generally enough.