Details

Description

With MultiCore functionality coming along, it looks like we'll need to be able to:
A) snapshot each core's index directory, and
B) replicate any and all cores' complete data directories, not just their index directories.

I think that makes sense - distribute everything for a given core, not just its index. And the spellchecker could then also have its data dir (and only index/ underneath really) and be replicated in the same fashion.

Right?

Ryan:

Yes, that was my thought. If an arbitrary directory could be distributed, then you could have

Hoss Man
added a comment - 14/Jan/08 07:31 as i recall, most of the scripts currently take a "-d data_dir" option ... and then use makethe following assumptions...
1) $
{data_dir}/index is what gets snapshooted/snapinstalled
2) ${data_dir}
/snapshot* is the pattern for naming snapshots.
a good way to evolve the scripts would probably be to have an alternate set of options ... maybe "-D dir" and "-S snapshots" that assumes $
{dir}
is where all the data to be snapshooted/snapinstalled lives, and $
{snapshots}
is where all snapshoots live.
(except i think some of the scripts already have a -D .. snappuller or snapcleaner maybe?)
alternate idea: we don't have to add a command line option at all ... support for something like this could require special scripts.conf options.

This may be better suited as another bug, or even posted in the lucene project, but I also made a small patch for the lucene spellchecker.

Hitting the spellcheck request handler with a reopen command wasn't working after we installed a new spell index snapshot. Searcher was being created for the new index, but not a reader.

Also, if you rebuild the spell index after it has already been built, it cleans out the index. You then have to send it a rebuild again to actually rebuild the index. Frequency of words in the spell index seemed to remain constant when rebuilding the spell index multiple times.

Doug Steigerwald
added a comment - 21/Feb/08 14:03 This may be better suited as another bug, or even posted in the lucene project, but I also made a small patch for the lucene spellchecker.
Hitting the spellcheck request handler with a reopen command wasn't working after we installed a new spell index snapshot. Searcher was being created for the new index, but not a reader.
Also, if you rebuild the spell index after it has already been built, it cleans out the index. You then have to send it a rebuild again to actually rebuild the index. Frequency of words in the spell index seemed to remain constant when rebuilding the spell index multiple times.

Jeremy Hinegardner
added a comment - 29/Jul/08 04:59 If this patch works for folks, I would like to see it committed and put in the nightly snapshot. Or at least the RunExecutableListener.patch and solr-433.patch.
If there is any more work here, I'd be happy to work on it.

Jeremy Hinegardner
added a comment - 04/Aug/08 00:27 - edited I've combined solr-433.patch and RunExecutableListener.patch into a single patch against svn trunk. This also fixes a syntax bug in RunExecutableListener where super(core); was not invoked.

I'm interested in using this patch to replicate the spell index, but I am not using multiple cores. The scripts in the patch do not work for single core setups since it makes an assumptions that the core name will appear in the rsync path.

I am attaching another version of the patch which makes a few changes:

snapcleaner, snapshooter, and snappuller are backwards compatible with the naming convention used by the current scripts

Fixed a few small bugs where static strings were used instead of the corresponding variables

Avoid assuming that an executable will accept a '-c' parameter to specify the core name. Instead, allow a RunExecutableListener to be conditionally executed for specified cores. RunExecutableListener now accepts a <arr name="cores"> parameter. You would need to add a listener for each core. For example:

(Another reasonable alternative to this might also be to accept variables like $

{SOLR_CORE}

in <args> and <env> which are resolved by RunExecutableListener.)

Removed the '-c' parameter from snappuller, replacing it instead with the '-r' option to specify an rsync module. This allows us to not assume the location of the core's data path. Instead we would add a new module to rsyncd.conf for each core. e.g.

Jonathan Lee
added a comment - 06/Aug/08 21:23 - edited I'm interested in using this patch to replicate the spell index, but I am not using multiple cores. The scripts in the patch do not work for single core setups since it makes an assumptions that the core name will appear in the rsync path.
I am attaching another version of the patch which makes a few changes:
snapcleaner, snapshooter, and snappuller are backwards compatible with the naming convention used by the current scripts
Fixed a few small bugs where static strings were used instead of the corresponding variables
Avoid assuming that an executable will accept a '-c' parameter to specify the core name. Instead, allow a RunExecutableListener to be conditionally executed for specified cores. RunExecutableListener now accepts a <arr name="cores"> parameter. You would need to add a listener for each core. For example:
<listener event= "postOptimize" class= "core.RunExecutableListener" >
<arr name= "cores" > <str>core0</str> </arr>
<str name= "exe" >solr/bin/snapshooter</str>
<str name= "dir" >.</str>
<arr name= "args" > <str>-d /usr/local/solr/core0/data</str> </arr>
</listener>
<listener event= "postOptimize" class= "core.RunExecutableListener" >
<arr name= "cores" > <str>core1</str> </arr>
<str name= "exe" >solr/bin/snapshooter</str>
<str name= "dir" >.</str>
<arr name= "args" > <str>-d /usr/local/solr/core1/data</str> </arr>
</listener>
(Another reasonable alternative to this might also be to accept variables like $
{SOLR_CORE}
in <args> and <env> which are resolved by RunExecutableListener.)
Removed the '-c' parameter from snappuller, replacing it instead with the '-r' option to specify an rsync module. This allows us to not assume the location of the core's data path. Instead we would add a new module to rsyncd.conf for each core. e.g.
...
[core0]
path = /usr/local/solr/core0/data
comment = core0
[core1]
path = /usr/local/solr/core1/data
comment = core1
...
then use:
./snappuller -r core0

Nicolas Lalevée
added a comment - 24/Sep/08 15:38 The patch that works the best for me is the last one, Jonathan's one, as I run Solr with only one core but with two spellchecker indexes.
I have fixed in that patch the snapcleaner when called to explicitely clean the main index ( ./snapcleaner -i index ).
I also fixed every "USAGE" printing.
and I have included the patch on RunExecutableListener and a little simplified it (it can now access to the solr core instance).

Note that on the last patches, the grep to retrieve the snapshot is incorrect:

ls ${data_dir}|grep "${snap_prefix}\."|grep -v wip|sort -r|head -1

would always retrieve the latest one on the ls, it needs to be with an anchor in the grep for the prefix otherwise it will never update the index snapshot (since 'snapshot' is present in every snapshot of index)

Stephane Bailliez
added a comment - 02/Oct/08 02:17 Note that on the last patches, the grep to retrieve the snapshot is incorrect:
ls ${data_dir}|grep "${snap_prefix}\."|grep -v wip|sort -r|head -1
would always retrieve the latest one on the ls, it needs to be with an anchor in the grep for the prefix otherwise it will never update the index snapshot (since 'snapshot' is present in every snapshot of index)
ls ${data_dir}|grep "^${snap_prefix}\."|grep -v wip|sort -r|head -1
should be changed in snappuller and snapinstaller

Chris Haggstrom
added a comment - 16/Nov/08 21:32 I've been using the patch submitted by Jonathan Lee on 10-02-08 for replicating a spelling directory in addition to the index, and it works very well for that purpose.
I'm attaching a slightly modified patch that allows the snapshooter "-c" option to work with an index or spelling directory that is not named "index".

Hoss Man
added a comment - 27/May/10 22:05 Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...
http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E
Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.
A unique token for finding these 240 issues in the future: hossversioncleanup20100527

Hoss Man
added a comment - 21/Mar/12 18:08 Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.
email notification suppressed to prevent mass-spam
psuedo-unique token identifying these issues: hoss20120321nofix36