The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.

Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

Brian Whitman (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616903#action_12616903 ] Brian Whitman commented on SOLR-659: ------------------------------------ An example of a bulk query using this patch. Without this patch such bulk queries will eventually time out or cause exceptions in the server as too much data is passed back and forth. {code:java} public SolrDocumentList blockQuery(SolrQuery q, int blockSize, int

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

Mike Klaas (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Klaas updated SOLR-659: ---------------------------- Fix Version/s: (was: 1.3) IMO it is too late in the release process for new features. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

Otis Gospodnetic (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic updated SOLR-659: ---------------------------------- Fix Version/s: 1.4 This looks simple enough. I haven't tried it. Brian, do you have a unit test you could attach? Or would it make more sense to have a custom QueryComponent for something like this? (I don't know yet) -- This message is automatically generated by JIRA. - You can reply to this email to add a

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

Shalin Shekhar Mangar (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683803#action_12683803 ] Shalin Shekhar Mangar commented on SOLR-659: -------------------------------------------- If I understand this correctly, it makes bulk queries cheaper at the expense of less precise scoring. But if I'm paging through some results and you modify the shard.start and shard.rows then I'll get inconsistent results. Is that correct? bq.

If I understand this correctly, it makes bulk queries cheaper at the expense of less precise scoring. But if I'm paging through some results and you modify the shard.start and shard.rows then I'll get inconsistent results. Is that correct?

bq. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

I do not understand that. Why will the client get more than rows? Or by client, did you mean the solr server to which the initial request is sent?

Explicitly set start and rows per shard for more efficient bulk queries across distributed Solr-----------------------------------------------------------------------------------------------

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

Yonik Seeley (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748986#action_12748986 ] Yonik Seeley commented on SOLR-659: ----------------------------------- I agree this makes sense to enable efficient bulk operations, and also fits in with a past idea I had about mapping shards.param=foo to param=foo during a sub-request. I'll give it a couple of days and commit if there are no objections. -- This message is

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.

johnson.hong (JIRA)[ https://issues.apache.org/jira/browse/SOLR-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769114#action_12769114 ] johnson.hong commented on SOLR-659: ----------------------------------- This is really helpful to bulk queries ,but how to handle the pagination of query results. e.g.at the first query,I set shards.start to 0 and set shards.rows to 30,it may return 50 documents,and i get 30 documents to show ,the other 20 documents is discarded ;then

This is really helpful to bulk queries ,but how to handle the pagination of query results.e.g.at the first query,I set shards.start to 0 and set shards.rows to 30,it may return 50 documents,and i get 30 documents to show ,the other 20 documents is discarded ;then how to get the next 30 documents ?

Explicitly set start and rows per shard for more efficient bulk queries across distributed Solr-----------------------------------------------------------------------------------------------

The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

--This message is automatically generated by JIRA.-You can reply to this email to add a comment to the issue online.