OVERVIEW

This document describes how to download, build,
and install Swish-e from source.
Also found below is a basic overview of using Swish-e to index documents,
with pointers to other, more advanced examples.

This document also provides instructions
on how to get help installing and using Swish-e
(and the important information you should provide when asking for help).
Please read these instructions before requesting help
on the Swish-e discussion list.
See "QUESTIONS AND TROUBLESHOOTING".

Although building from source is recommended,
some OS distributions (e.g., Debian) provide pre-compiled binaries.
Check with your distribution for available packages.
Build from source,
if your distribution does not offer the current version of Swish-e.

Also, please read the Swish-e FAQ (SWISH-FAQ),
as it answers many frequently-asked questions.

Swish-e knows how to index HTML, XML, and plain text documents.
Helper applications and other tools are used to convert documents
such as PDF or MS Word into a format that Swish-e can index.
These additional applications and tools (listed below)
must be installed separately.
The process of converting documents is called "filtering".

NOTE: Swish-e version 4.2.0 installs a lot more files
when running "make install".
Be aware that the Swish-e documentation may thus include errors
about where files are located.
Please notify the Swish-e discussion list of any documentation errors.

Upgrading from previous versions of Swish-e

If you are upgrading from a previous version of Swish-e,
read the CHANGES page first.
The Swish-e index format may have changed
and existing indexes may not work with the newer version of Swish-e.

If you have existing indexes,
you may need to re-index your data
before running the "make install" step described below.
Swish-e may be run from the build directory after compiling,
but before installation.

Windows Users

A Windows binary version is available
as a separate download from the Swish-e site (http://swish-e.org).
Many of the installation instructions below will not apply to Windows users;
the Windows version is pre-compiled
and includes libxml2, zlib, xpdf, and catdoc.

A number of Perl modules may also be needed.
These can be installed with ActiveState's PPM utility.

Building from CVS

Please refer to the README.cvs file found in the documentation directory
$prefix/share/doc/swish-e.

SYSTEM REQUIREMENTS

Swish-e makes use of a number of libraries and tools
that are not distributed with Swish-e.
Some libraries need to be installed before building Swish-e from source;
other tools can be installed at any time.
See below for details.

Software Requirements

Swish-e is written in C.
It has been tested on a number of platforms,
including Sun/Solaris, Dec Alpha, BSD, Linux, Mac OS X, and Open VMS.

The GNU C compiler (gcc) and GNU make are strongly recommended.
Repeat: you will find life easier if you use the GNU tools.

Optional but Recommended Packages

Most of the packages listed below are available
as easily installable packages.
Check with your operating system vendor or install them from source.
Most are very common packages that may already be installed on your computer.

As noted below,
some packages need to be installed before building Swish-e from source,
while others may be added after Swish-e is installed.

Libxml2

libxml2 is very strongly recommended.
It is used for parsing both HTML and XML files.
Swish-e can be built and installed without libxml2,
but the HTML parser that is built into Swish-e
is not as accurate as libxml2.

http://xmlsoft.org/

libxml2 must be installed before Swish-e is built,
or it will not be used.

If libxml2 is installed in a non-standard location
(e.g., libxml2 is built with --prefix $HOME/local),
make sure that you add the bin directory to your $PATH
before building Swish-e.
Swish-e's configure script uses a program
created by libxml2 (xml2-config)
to find the location of libxml2.
Use which xml2-config to verify
that the program can be found where expected.
If you are using a packaged libxml2 from your OS distribution
(e.g. RPM, DEB, etc) you likely need to install
libxml2-devel as well to include the xml2-config script.

Zlib Compression

The Zlib compression library is commonly installed on most systems
and is recommended for use with Swish-e.
Zlib is used for compressing text stored in the Swish-e index.

http://www.gzip.org/zlib/

Zlib must be installed before building Swish-e.

Perl Modules

Although Swish-e is a compiled C program,
many support features use Perl.
For example, both the web spiders
and modules to help with filtering documents are written in Perl.

The following Perl modules may be required.
Check your current Perl installation,
as many may already be installed.

LWP
URI
HTML::Parser
HTML::Tagset
MIME::Types (optional)

Note that installing Bundle::LWP with the CPAN module

perl -MCPAN -e 'install Bundle::LWP'

will install many of the above modules.

If you wish to use HTML-Template with swish.cgi to generate output,
install:

HTML::Template
HTML::FillInForm

If you wish to use Template-Toolkit with swish.cgi
to generate output, install:

Template

Questions about installing these modules
may be sent to the Swish-e discussion list.

The search.cgi example script
requires both Template-Toolkit and HTML::FillInForm.

Indexing PDF Documents

Indexing PDF files requires the xpdf package.
This is a common package,
available with most operating systems
and often provided as an add-on package.

http://www.foolabs.com/xpdf/

Xpdf may be added after Swish-e is installed.

Indexing MS Word Documents

Indexing MS Word documents requires the Catdoc program.

http://www.wagner.pp.ru/~vitus/software/catdoc/

Catdoc may be added after Swish-e is installed.

Indexing MP3 ID3 Tags

Indexing MP3 ID3 Tags requires the MP3::Tag Perl module.
See http://search.cpan.org.
MP3::Tag may be installed after Swish-e is installed.

INSTALLATION

Here are brief installation instructions that should work in most cases.
Following this section are more detailed instructions and examples.

Building Swish-e

Download Swish-e using your favorite web browser
or a utility such as wget, lynx, or lwp-download.
Unpack and build the distribution, using the following steps:

Note: "swish-e-2.4.0" is used as an example.
Download the most current available version
and adjust the commands below!
Also, if you are running Debian,
see the notes below on building a .deb package
from the Swish-e source package.

Pay careful attention to the "prompt" character used
on the following command lines.
A "$" prompt indicates steps run as an unprivileged user.
A "#" indicates steps run as the superuser (root).

IMPORTANT:
Once Swish-e is installed, do not run it as the superuser (root) --
root is only required during the installation step,
when installing into system directories.
Please do not break this rule.

NOTE:
If you are upgrading from an older version of Swish-e,
be sure and review the CHANGES file.
Old index files may not be compatible with newer versions of Swish-e.
After building Swish-e (but before running "make install"),
Swish-e can be run from the build directory:

$ src/swish-e -V

To minimize downtime,
create new index files before running "make install",
by using Swish-e from the build directory.
Then, copy the index files to the live location and run "make install":

$ src/swish-e -c /path/to/config -f index.new

Keep in mind that the location you index from
may affect the paths stored in the index file.

Installing without root access

Here's another installation example.
This might be used if you do not have root access
or you wish to install Swish-e someplace other than /usr/local.

This example also shows building Swish-e in a "build" directory
that is separate from where the source files are located.
This is the recommended way to build Swish-e,
but it requires GNU Make.
Without GNU Make,
you will likely need to build from within the source directory,
as shown in the previous example.

Swish-e uses a configure script.
configure has many options,
but it uses reasonable and standard defaults.
Running

$ ../swish-e-2.4.0/configure --help

will display the options.

Two options are of common interest:
--prefix sets the top-level installation directory;
--disable-shared will link Swish-e statically,
which may be needed on some platforms (Solaris 2.6, perhaps).

Platforms may require varying link instructions
when libraries are installed in non-standard locations.
Swish-e uses the GNU autoconf tools for building the package.
autoconf is good at building and testing,
but still requires you to provide information appropriate for your platform.
This may mean reading the manual page for your compiler and linker
to see how to specify non-standard file locations.

For most Unix-type platforms,
you can use LDFLAGS and CPPFLAGS environment variables
to specify paths to "include" (header) files
and to libraries that are not in standard locations.

In this example, we do not have root access.
We have installed libxml2 and libz in $HOME/local.
Swish-e will also be installed in $HOME/local
(by using the --prefix setting).

In this case,
you would need to add $HOME/local/bin
to the start of your shell's $PATH setting.
This is required because libxml2 installs a program
that is used when running the configure script.
Before running configure, type:

Note the use of double quotes in the LDFLAGS line above.
This allows $HOME to be expanded within the text string.

Run-time paths

The -R option says to add a specified path (or paths)
to those that are used to find shared libraries at run time.
These paths are stored in the Swish-e binary.
When Swish-e is run,
it will look in these directories for shared libraries.

Some platforms may not support the -R option.
In this event,
set the LD_RUN_PATH environment variable before running make.

Some systems, such as Redhat,
do not look in /usr/local/lib for libraries.
In these cases,
you can either use -R, as above, when building Swish-e
or add /usr/local/lib to /etc/ld.so.conf and run ldconfig as root.

If all else fails,
you may need to actually read the man pages for your platform.

Building a Debian Package

The Swish-e distribution includes the files required
to build a Debian package.

What's installed

Swish installs a number of files.
By default, all files are installed below /usr/local,
but this can be changed by setting --prefix
when running configure (as shown above).
Individual paths may also be set.
Run configure --help for details.

Note that the Perl modules are not installed in the system Perl library.
Swish-e and the Perl scripts that require the modules
know where to find the modules,
but the perldoc program (used for reading documentation) does not.
This can be corrected by adding $prefix/lib/swish-e
and $prefix/lib/swish-e/perl to the PERL5LIB environment variable.

Documentation

Documentation can be found in the $prefix/share/doc/swish-e directory.
Documentation is in html format at $prefix/share/doc/swish-e/html and
can also be read on-line at the Swish-e web site:

http://swish-e.org/

The Swish-e documentation as man(1) pages

Running "make install" installs some of the Swish-e documentation as man pages.
The following man pages are installed:

SWISH-FAQ(1)
SWISH-CONFIG(1)
SWISH-RUN(1)
SWISH-LIBRARY(1)

The man pages are installed, by default, in the system man directory.
This directory is determined when configure is run;
it can be set by passing a directory name to configure.

For example,

./configure --mandir=/usr/local/doc/man

The man directory is specified relative to the --prefix setting.
If you use --prefix,
you do not normally need to also specify --mandir.

Information on running configure can be found by typing:

./configure --help

Join the Swish-e discussion list

The final step, when installing Swish-e,
is to join the Swish-e discussion list.

The Swish-e discussion list is the place
to ask questions about installing and using Swish-e,
see or post bug fixes or security announcements,
and offer help to others.
Please do not contact the developers directly.

The list is typically very low traffic,
so it won't overload your inbox.
Please take the time to subscribe.
See http://Swish-e.org.

If you are using Swish-e on a public site,
please let the list know,
so that your URL can be added to the list of sites that use Swish-e!

Please review the next section
before posting questions to the Swish-e list.

QUESTIONS AND TROUBLESHOOTING

Support for installation, configuration, and usage
is available via the Swish-e discussion list.
Visit http://swish-e.org for information.
Do not contact developers directly for help --
always post your question to the list.

It's very important to provide the right information
when asking for help.

Please search the Swish-e list archive before posting a question.
Also, check the SWISH-FAQ
to see if your question has already been asked and answered.

Before posting, use the available tools to narrow down the problem.

Swish-e has several switches
(e.g., -T, -v, and -k)
that may help you resolve issues.
These switches are described on the SWISH-RUN page.
For example, if you cannot find a document by a keyword
that you believe should be indexed,
try indexing just that single file
and use the -T INDEXED_WORDS option
to see if the word is actually being indexed.
First, try it without any changes to default settings:

swish-e -i testdoc.html -T indexed_words | less

if that works, add in your configuration file:

swish-e -i testdoc.html -c swish.conf -T indexed_words | less

If it still isn't working as you expect,
try to reduce the test document to a very small example.
This will be very helpful to your readers,
when you are asking for help.

Another useful trick is to use -H9 when searching,
to display full headers in search results.
Look at the "Parsed Words" header
to see what words Swish-e is searching for.

When posting, please provide the following information:

Use these guidelines when asking for help.
The most important tip is to provide the least amount of information
that can be used to reproduce your problem.
Do not paraphrase output -- copy-and-paste --
but trim text that is not necessary.

The exact version of Swish-e that you are using.
Running Swish-e with the -V switch will print the version number.
Also, supply the output from uname -a or similar command
that identifies the operating system you are running on.
If you are running an old version of swish,
be prepared for a response of "upgrade" to your question.

A summary of the problem.
This should include the commands issued
(e.g. for indexing or searching) and their output,
along with an explanation of why you don't think it's working correctly.
Please copy-and-paste the exact commands and their output,
instead of retyping, to avoid errors.

Include a copy of the configuration file you are using, if any.
Swish-e has reasonable defaults,
so in many cases you can run it without using a configuration file.
But, if you need to use a configuration file,
reduce it down to the absolute minimum number of commands
that is required to demonstrate your problem.
Again, copy-and-paste.

A small copy of a source document that demonstrates the problem.

If you are having problems spidering a web server,
use lwp-download or wget to copy the file locally,
then make sure you can index the document using the file system method.
This will help you determine if the problem
is with spidering or indexing.

If you expect help with spidering,
don't post fake URLs, as it makes it impossible to test.
If you don't want to expose your web page to the people on the Swish-e list,
find some other site to test spidering on.
If that works, but you still cannot spider your own site,
you may need to request help from others.
If so, you must post your real URL
or make a test document available via some other source.

If you are having trouble building Swish-e,
please copy-and-paste the output from make
(or from ./configure, if that's where the problem is).

The key is to provide enough information
so that others may reproduce the problem.

ADDITIONAL INSTALLATION OPTIONS

These steps are not required for normal use of Swish-e.

The SWISH::API Perl Module

The Swish-e distribution includes a module
that provides a Perl interface to the Swish-e C library.
This module provides a way to search a Swish-e index
without running the Swish-e program.
Searching an index will be many times faster
when running under a persistent environment
such as Apache/mod_perl with the SWISH::API module.

See the perl/README file for information
on installing and using the SWISH::API Perl module.

GENERAL CONFIGURATION AND USAGE

This section should give you a basic overview
of indexing and searching with Swish-e.
Other examples can be found in the conf directory;
these will step you through a number of different configurations.
Also, please review the SWISH-FAQ.

Swish-e is a command-line program.
The program is controlled by passing switches on the command line.
A configuration file may be used,
but often is not required.
Swish-e does not include a graphical user interface.
Example CGI scripts are provided in the distribution,
but they require additional setup to use.

Introduction to Indexing and Searching

Swish-e can index files that are located on the local file system.
For example, running:

swish-e -i /var/www/htdocs

will index all files in the /var/www/htdocs directory.
You may specify one or more files or directories with the -i option.
By default, this will create an index called index.swish-e
in the current directory.

To search the resulting index for a given word, try:

swish-e -w apache

This will find the word "apache" in the body or title
of the indexed documents.

As mentioned above,
Swish-e will index all files in a directory,
unless instructed otherwise.
So, if /var/www/htdocs contains non-HTML files,
you will need a configuration file to limit the files that Swish-e indexes.
Create a file called swish.conf:

The Swish-e configuration settings are described
in the SWISH-CONFIG manual page.
The order of statements in the configuration file is typically not important,
although some statements depend on previously set statements.
There are many possible settings.
Good advice is to use as few settings as possible
when first starting out with Swish-e.

The runtime options (switches) are described
in the SWISH-RUN manual page.
You may also see a summary of options by running:

swish-e -h

Swish-e has two other methods for reading input files.
One method uses a Perl helper script and the LWP Perl library
to spider remote web sites:

swish-e -S http -i http://localhost/index.html -v2

This will spider the web server running on the local host.
The -S option defines the input source method to be "http",
-i specifies the URL to spider,
and -v sets the verbose level to two.
There are a number of configuration options
that are specific to the -S http input source.
See SWISH-CONFIG.
Note that only files of Content-Type text/* will be indexed.

The -S http method is deprecated, however,
in favor of a variation on the following input method.

There is a general-purpose input method
wherein Swish-e reads input from a program
that produces documents in a special format.
The program might read and format data stored in a database,
or parse and format messages in a mailing list archive,
or run a program that spiders web sites (like the previous method).

The Swish-e distribution includes a spider program
that uses this method of input.
This spider program is much more configurable and feature-rich
than the previous (-S http) method.

To duplicate the previous example,
create a configuration file called swish2.conf:

# Example for spidering
# Use the "spider.pl" program included with Swish-e
IndexDir spider.pl
# Define what site to index
SwishProgParameters default http://localhost/index.html

Then, create the index using the command:

swish-e -S prog -c swish2.conf

This says to use the -S prog input source method.
Note that, in this case,
the IndexDir setting does not specify a file or directory to index,
but a program name to be run.
This program, spider.pl,
does the work of fetching the documents from the web server
and passing them to Swish-e for indexing.

The SwishProgParameters option is a special feature
that allows passing command-line parameters
to the program specified with IndexDir.
In this case, we are passing the word default
(which tells spider.pl to use default settings)
and the URL to spider.

Running a script under Windows requires specifying the interpreter
(e.g., perl.exe)
and then using SwishPropParameters
to specify the script and the script's parameters.
See Notes when using -S prog on MS Windows
on the SWISH-RUN page.

The advantage of the -S prog method of spidering
(over the previous -S http method)
is that the Perl code is only compiled once
instead of once for every document fetched from the web server.
In addition, it is a much more advanced spider with many, many features.
Still, as used here,
spider.pl will automatically index PDF or MS Word documents
if (when) Xpdf and Catdoc are installed.

A special form of the -S prog input source method is:

./myprog --option | swish-e -S prog -i stdin -c config

This allows running Swish-e from a program
(instead of running the external program from Swish-e).
So, this also can be done as:

One final note about the -S prog input source method.
The program specified with -i or IndexDir needs to be an absolute path.
The exception is when the program is installed in the libexecdir directory.
Then, a plain program name may be specified
(as in the example showing spider.pl, above).

All three input source methods are described in more detail
on the SWISH-RUN page.

Metanames and Properties

There are two key Swish-e concepts
that you need to be familiar with:
Metanames and Properties.

Metanames

Swish-e creates a reverse (i.e., inverted) index.
Just like an index in a book,
you look up a word and it lists the pages (or documents)
where that word can be found.

Swish-e can create multiple index tables within the same index file.
For example,
you might want to create an index that only contains words in HTML titles,
so that searches can be limited to title text.
Or, you might have descriptive words
that you would like to search,
stored in a meta tag called "keywords".

Some database systems might call these different "fields" or "columns",
but Swish-e calls them MetaNames
(as a result of its first indexing HTML "meta" tags).

To find documents containing "foo" in their titles, you might run:

swish-e -w swishtitle=foo

or, a more advanced example:

swish-e -w swishtitle=(foo or bar) or swishdefault=(baz)

The Metaname "swishdefault" is the name that is used by Swish-e
if no other name is specified.
The following two searches are thus equivalent:

swish-e -w foo
swish-e -w swishdefault=foo

When indexing HTML documents,
Swish-e indexes words in the body and title
under the Metaname "swishdefault".

Properties

Swish-e's search result is a list of files --
actually, Swish-e uses file numbers internally.
Data can be associated with each file number when indexing.
For example, by default Swish-e associates the file's name, title,
last modified date, and size with the file number.
These items can be printed in search results.

In Swish-e, this associated data is called a file's Properties.
Properties can be any data you wish to associated with a document --
in fact, the entire text of the document can be stored in the index.
What data is stored as a Property is controlled by the PropertyNames
(and other) configuration directives.

What properties are printed with search results
depends on the -x or -p switches.
By default,
Swish-e returns the rank, path/URL, title, and file size in bytes
for each result.

Getting Started With Swish-e

Swish-e reads a configuration file (see SWISH-CONFIG)
for directives that control whether and how Swish-e indexes files.
Swish-e is also controlled by command-line arguments
(see SWISH-RUN).
Many of the command-line arguments
have equivalent configuration directives (e.g., -i and IndexDir).

Swish-e does not require a configuration file,
but most people change its default behavior
by placing settings in a configuration file.

To try the examples below,
go to the tests subdirectory of the distribution.
The tests will use the *.html files in this directory
when creating the test index.
You may wish to review these *.html files
to get an idea of the various native file formats that Swish-e supports.

You may also use your own test documents.
It's recommended to use small test documents when first using Swish-e.

Step 1: Create a Configuration File

The configuration file controls what and how Swish-e indexes. The
configuration file consists of directives, comments, and blank lines.
The configuration file can be any name you like.

This example will work with the documents in the tests directory.
You may wish to review the tests/test.config configuration file used
for the make test tests.

For example, a simple configuration file (swish-e.conf):

# Example Swish-e Configuration file
# Define *what* to index
# IndexDir can point to a directories and/or a files
# Here it's pointing to the current directory
# Swish-e will also recurse into sub-directories.
IndexDir .
# But only index the .html files
IndexOnly .html
# Show basic info while indexing
IndexReport 1

And that's a simple configuration file.
It says to index all the .html files
in the current directory and sub-directories, if any,
and provide some basic output while indexing.

As mentioned above,
the complete list of all configuration file directives
is detailed in SWISH-CONFIG.

Step 2: Index your Files

Run Swish-e,
using the -c switch to specify the name of the configuration file.

So, the word sample was found in two documents.
The first number shown is the relevance (or rank) of the search term,
followed by the file containing the search term,
the title of the document,
and finally, the length of the document (in bytes).

The period ("."), sitting alone at the end,
marks the end of the search results.

Much more information may be retrieved while searching,
by using the -x and -H switches (see SWISH-RUN)
and by using Document Properties (see SWISH-CONFIG).

Phrase Searching

To search for a phrase in a document,
use double-quotes to delimit your search terms.
(The default phrase delimiter is set in src/swish.h.)

You must protect the quotes from the shell.

For example, under Unix:

swish-e -w '"this is a phrase" or (this and that)'
swish-e -w 'meta1=("this is a phrase") or (this and that)'

Or under the Windows command.com shell.

swish-e -w \"this is a phrase\" or (this and that)

The phrase delimiter can be set with the -P switch.

Boolean Searching

You can use the Boolean operators and, or, or not in searching.
Without these Boolean operatots,
Swish-e will assume you're anding the words together.

Context Searching

The -t option in the search command line
allows you to search for words that exist only in specific HTML tags.
This option takes a string of characters as its argument.
Each character represents a different tag in which the word is searched;
that is, you can use any combinations of the following characters:

META Tags

As mentioned above,
Metanames are a way to define "fields" in your documents.
You can use the Metanames in your queries to limit the search
to just the words contained in that META name of your document.
For example,
you might have a META-tagged field called subjects in your documents.
This would let you search your documents for the word "foo",
but only return documents where "foo" is within the subjects META tag.

Document Properties are somewhat related:
Properties allow the content of a META tag in a source document
to be stored within the index,
and that text to be returned along with search results.

To continue with our sample Swish-e.conf file,
add the following lines:

# Define META tags
MetaNames meta1 meta2 meta3

Reindex to include the changes:

swish-e -c swish-e.conf

Now search, but this time limit your search to META tag meta1:

swish-e -w 'meta1=metatest1'

Again, please see SWISH-RUN and SWISH-CONFIG
for complete documentation of the various indexing and searching options.

Spidering and Searching with a Web form.

This example demonstrates how to spider a web site
and set up the included CGI script to provide a web-based search page.
This example uses Perl programs that are included in the Swish-e distribution:
spider.pl will be used for reading files from the web server;
swish.cgi will provide the web search form and display results.

Also note that Swish-e knows to find spider.pl
at /usr/local/lib/swish-e/spider.pl.
The script installation directory (called libexecdir)
is set at configure time.
You can see your setting by running swish-e -h:

Swish-e does not include a web server.
So, you must use your locally installed web server.
Apache is highly recommended, of course.

Locate your web server's CGI directory.
This may be a cgi-bin directory in your home directory
or a central cgi-bin directory set up by the web server administrator.
Once this is located,
copy the swish.cgi script into the cgi-bin directory.

Where CGI scripts can be located
depends completely on the web server that is being used
and how it has been configured.
See your web server's documentation or your site's administrator
for additional information.

This example will use a site cgi-bin directory,
located at /usr/lib/cgi-bin.
Copy the swish.cgi script into the cgi-bin directory.
Again, we will need the location of the libexecdir directory:

If your operating system supports symbolic links
and your web server allows programs to be symbolic links,
then you may wish to create a link to the swish.cgi program, instead.

/usr/lib/cgi-bin# ln -s /usr/local/lib/swish-e/swish.cgi

We need to tell the swish.cgi script where to look
for the index created in the previous step.
It's also recommended to enter the path to the swish-e binary.
Otherwise, the swish.cgi script will look for the binary in the PATH,
and that may change when running under the CGI environment.

Notice that the CGI script returns the HTTP header (Content-Type)
and the body of the web page,
just like a well behaved CGI scrip should do.

Now, test using the web server
(this step depends on the location of your cgi-bin directory).
This example uses the "GET" command that is part of the LWP Perl library,
but any web browser can run this test.

The script reports errors to stderr,
so consult the web server's error log if problems occur.
The message "Service currently unavailable",
reported by running swish.cgi,
typically indicates a configuration error;
the exact problem will be listed in the web server's error log.

Detailed instructions on using the swish.cgi script
and debugging tips can be found by running:

$ perldoc swish.cgi

while in the cgi-bin directory where swish.cgi was copied.

The spider program spider.pl also has a large number
of configuration options.

Documentation is also available
in the directory $prefix/share/doc/swish-e or at
http://swish-e.org.

Note: Also check out the search.cgi script,
found at the same location as the swish.cgi script.
This is more of a skeleton script,
for those that want to create a custom search script.

Now you are ready to search.

Indexing Other Types of Documents - Filtering

Swish-e can only index HTML, XML, and text documents.
In order to index other documents,
such as PDF or MS Word documents,
you must use a utility to convert or "filter" those documents.

How documents are filtered with Swish-e has changed over time.
This has resulting in a bit of confusion.
It's also a somewhat complex process,
as different programs need to communicate with each other.

Filtering Overview

There are two ways to filter documents with Swish-e.
Both are described in the SWISH-CONFIG man page.
They use the FileFilter directive
and the SWISH::Filter Perl module.

The FileFilter directive is a general-purpose method of filtering.
It allows running of an external program for each document processed
(based on file extension),
and requires one or more external programs.
These programs open an input file, convert as needed,
and write their output to standard output.

Previous versions of Swish-e (before 2.4.0)
used a collection of filter programs
for converting files such as PDF or MS Word documents.
The external programs call other program to do the work of filtering
(e.g. pdftotext to extract the contents from PDF files).
Although these filter programs are still included
with the Swish-e distribution as examples,
it is recommended to use the SWISH::Filter method, instead.

One disadvantage of using FileFilter
is that the filter program is run once
for every document that needs to be filtered.
This can slow down the indexing process substantially.

The SWISH::Filter Perl module works very much like the old system
and uses the same helper programs.
Convieniently, however,
it provides a single interface for filtering all types of documents.
The primary advantage of SWISH::Filter
is that it is built into the program used for spidering web sites (spider.pl),
so all that's required is installing the filter programs that do the
actual work of filtering (e.g. catdoc, xpdf).
(The Windows binary includes some of the filter programs.)

But, Swish-e will not use SWISH::Filter by default
when using the file system method of indexing.
To use SWISH::Filter when indexing by file system method (-S fs),
you can use a FileFilter directive with the swish_filter.pl filter
(which is just a program that uses SWISH::Filter)
or use the -S prog method of indexing
and use the DirTree.pl program for fetching documents.

DirTree.pl is included with the Swish-e distribution
and is designed to work with SWISH::Filter.
Using DirTree.pl will likely be a faster way to index,
since the SWISH::Filter set of modules
does not need to be compiled for every document
that needs to be filtered.

See the contents of swish_filter.pl and DirTree.pl
for specifics on their use.

Filtering Examples

The FileFilter directive can be used in your config file
to convert documents, based on their extensions.
This is the old way of filtering,
but provides an easy way to add filters to Swish-e.

For example:

FileFilter .pdf pdftotext "'%p' -"
IndexContents TXT* .pdf

will cause all .pdf files to be filtered through the pdftotext program
(part of the Xpdf package)
and to parse the resulting output (from pdftotext)
with the text ("TXT") parser.

The other way to filter documents
is to use a -S prog prograam
and convert the documents before passing them onto Swish-e.

For example, spider.pl makes use of the SWISH::Filter" Perl module,
included with the Swish-e distribution.
SWISH::Filter is passed a document and the document's content type;
it looks for modules and utilities to convert the document
into one of the types that Swish-e can index.

For example, the Swish-e distribution includes a module called
SWISH::Filter::Pdf2HTML
that uses the pdftotext and pdfinfo utilities
provided by the Xpdf package.

This means that if you are using spider.pl
to spider your web site
and you wish to index PDF documents,
all that is needed is to install the Xpdf package
and Swish-e (with the help of spider.pl) will begin indexing your PDF files.

Ok, so what does all that mean?
For a very simple site,
you should be able to run this:

which is running the spider with default spider settings,
indexing the Web server on localhost,
and piping its output into Swish-e (using the default indexing settings).
Documents will be filtered automatically,
if you have the required helper applications installed.

Most people will not want to just use the default settings
(for one thing, the spider will take a while
because its default is to delay a few seconds between every request).
So, read the documentation for spider.pl,
to learn how to use a spider config file.
Also read SWISH-CONFIG
to learn about what configuration options can be used with Swish-e.

The SWISH::Filter documentation provides more details on filtering
and hints for debugging problems when filtering.