Re: Stopwords when searching?

At 11:26 PM 12/2/2001 -0800, Malcolm Box wrote:
>Does swish-e use the configured list of stopwords when searching so as
>to filter out looking for words that don't exist in the index?
Yes. Stopwords are NOT in the index, so it seems reasonable to not search
for them.
> From a few experiments here it would appear that it does not, which
>leads to some interesting result. To whit, using the supplied english
>stopwords list for indexing, and then searching for "to" finds a few
>documents, noticably those with "to" in a word that is not also a
>stopword. This is somewhat confusing to a user of the search engine!
It's even more confusing to me ;) Please post real examples!
The simple solution is to not use stopwords.
>I'd suggest that if a stoplist is configured then swish-e should not
>search for words that are part of the stoplist, instead displaying a
>message (a la Google).
Here's what swish-e provides:
> cat c
ignorewords to
> cat 1.html
That is the stopword to right there.
> ./swish-e -c c -i 1.html -T indexed_words -v 0
Indexing Data Source: "File-System"
Adding:[swishdefault:1] 'that' Pos:1 Stuct:0x1 ( FILE )
Adding:[swishdefault:1] 'is' Pos:2 Stuct:0x1 ( FILE )
Adding:[swishdefault:1] 'the' Pos:3 Stuct:0x1 ( FILE )
Adding:[swishdefault:1] 'stopword' Pos:4 Stuct:0x1 ( FILE )
Adding:[swishdefault:1] 'right' Pos:5 Stuct:0x1 ( FILE )
Adding:[swishdefault:1] 'there' Pos:6 Stuct:0x1 ( FILE )
Indexing done!
> ./swish-e -w 'stopword to' -H 9
# SWISH format: 2.1-dev-24
# Search words: stopword to
#
# Index File: index.swish-e
# Removed stopword: to
Here's swish just told you that the word was removed from the query.
..
# StopWords: to
List of stopwords in this index file
# BuzzWords:
# Search Words: stopword to
# Parsed Words: stopword
The query swish used after adjusting for Wordchars, stopwords and other
settings that might effect the search (and indexed) words.
There's also this
> ./swish-e -w 'to' -H 9
# SWISH format: 2.1-dev-24
# Search words: to
#
# Index File: index.swish-e
# Removed stopword: to
err: all search words too common to be useful
.
Bill Moseley
mailto:moseley@hank.org