A query that generates the union of documents produced by its subqueries, and that scores each document with the maximum
score for that document as produced by any subquery, plus a tie breaking increment for any additional matching subqueries.

Package org.apache.lucene.search Description

Table Of Contents

Search Basics

Lucene offers a wide variety of Query implementations, most of which are in
this package, its subpackages (spans, payloads),
or the queries module. These implementations can be combined in a wide
variety of ways to provide complex querying capabilities along with information about where matches took place in the document
collection. The Query Classes section below highlights some of the more important Query classes. For details
on implementing your own Query class, see Custom Queries -- Expert Level below.

Once a Query has been created and submitted to the IndexSearcher, the scoring
process begins. After some infrastructure setup, control finally passes to the Weight
implementation and its Scorer instances. See the Algorithm
section for more notes on the process.

Query Classes

Of the various implementations of
Query, the
TermQuery
is the easiest to understand and the most often used in applications. A
TermQuery matches all the documents that contain the
specified
Term,
which is a word that occurs in a certain
Field.
Thus, a TermQuery identifies and scores all
Documents that have a
Field with the specified string in it.
Constructing a TermQuery
is as simple as:

TermQuery tq = new TermQuery(new Term("fieldName", "term"));

In this example, the Query identifies all
Documents that have the
Field named "fieldName"
containing the word "term".

SHOULD — Use this operator when a clause can occur in the result set, but is not required.
If a query is made up of all SHOULD clauses, then every document in the result
set matches at least one of these clauses.

MUST — Use this operator when a clause is required to occur in the result set. Every
document in the result set will match
all such clauses.

MUST NOT — Use this operator when a
clause must not occur in the result set. No
document in the result set will match
any such clauses.

Phrases

Another common search is to find documents containing certain phrases. This
is handled three different ways:

PhraseQuery
— Matches a sequence of
Terms.
PhraseQuery uses a slop factor to determine
how many positions may occur between any two terms in the phrase and still be considered a match.
The slop is 0 by default, meaning the phrase must match exactly.

MultiPhraseQuery
— A more general form of PhraseQuery that accepts multiple Terms
for a position in the phrase. For example, this can be used to perform phrase queries that also
incorporate synonyms.

While the
PrefixQuery
has a different implementation, it is essentially a special case of the
WildcardQuery.
The PrefixQuery allows an application
to identify all documents with terms that begin with a certain string. The
WildcardQuery generalizes this by allowing
for the use of * (matches 0 or more characters) and ? (matches exactly one character) wildcards.
Note that the WildcardQuery can be quite slow. Also
note that
WildcardQuery should
not start with * and ?, as these are extremely slow.
Some QueryParsers may not allow this by default, but provide a setAllowLeadingWildcard method
to remove that protection.
The RegexpQuery is even more general than WildcardQuery,
allowing an application to identify all documents with terms that match a regular expression pattern.

A
FuzzyQuery
matches documents that contain terms similar to the specified term. Similarity is
determined using
Levenshtein (edit) distance.
This type of query can be useful when accounting for spelling variations in the collection.

Scoring — Introduction

Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides
almost all of the complexity from the user. In a nutshell, it works. At least, that is,
until it doesn't work, or doesn't work as one would expect it to work. Then we are left
digging into Lucene internals or asking for help on
java-user@lucene.apache.org to figure out
why a document with five of our query terms scores lower than a different document with
only one of the query terms.

While this document won't answer your specific scoring issues, it will, hopefully, point you
to the places that can help you figure out the what and why of Lucene scoring.

These models can be plugged in via the Similarity API,
and offer extension hooks and parameters for tuning. In general, Lucene first finds the documents
that need to be scored based on boolean logic in the Query specification, and then ranks this subset of
matching documents via the retrieval model. For some valuable references on VSM and IR in general refer to
Lucene Wiki IR references.

The rest of this document will cover Scoring basics and explain how to
change your Similarity. Next, it will cover
ways you can customize the lucene internals in
Custom Queries -- Expert Level, which gives details on
implementing your own Query class and related functionality.
Finally, we will finish up with some reference material in the Appendix.

Scoring — Basics

Scoring is very much dependent on the way documents are indexed, so it is important to understand
indexing. (see Lucene overview
before continuing on with this section) Be sure to use the useful
IndexSearcher.explain(Query, doc)
to understand how the score for a certain matching document was
computed.

Generally, the Query determines which documents match (a binary
decision), while the Similarity determines how to assign scores to
the matching documents.

Fields and Documents

In Lucene, the objects we are scoring are Documents.
A Document is a collection of Fields. Each Field has
semantics about how it is created and stored
(tokenized,
stored, etc). It is important to note that
Lucene scoring works on Fields and then combines the results to return Documents. This is
important because two Documents with the exact same content, but one having the content in two
Fields and the other in one Field may return different scores for the same query due to length
normalization.

Score Boosting

Lucene allows influencing search results by "boosting" at different times:

Index-time boost by calling
Field.setBoost() before a document is
added to the index.

Query-time boost by setting a boost on a query clause, calling
Query.setBoost().

Indexing time boosts are pre-processed for storage efficiency and written to
storage for a field as follows:

All boosts of that field (i.e. all boosts under the same field name in that doc) are
multiplied.

The boost is then encoded into a normalization value by the Similarity
object at index-time: computeNorm().
The actual encoding depends upon the Similarity implementation, but note that most
use a lossy encoding (such as multiplying the boost with document length or similar, packed
into a single byte!).

Decoding of any index-time normalization values and integration into the document's score is also performed
at search time by the Similarity.

You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its
parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can
extend by plugging in a different component (e.g. term frequency normalizer).

Finally, you can extend the low level Similarity directly
to implement a new retrieval model, or to use external scoring factors particular to your application. For example,
a custom Similarity can access per-document values via FieldCache or
NumericDocValues and integrate them into the score.

Custom Queries — Expert Level

Custom queries are an expert level task, so tread carefully and be prepared to share your code if
you want help.

With the warning out of the way, it is possible to change a lot more than just the Similarity
when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by
three main classes:

Weight — The internal interface representation of
the user's Query, so that Query objects may be reused.
This is global (across all segments of the index) and
generally will require global statistics (such as docFreq
for a given term across all segments).

Scorer — An abstract class containing common
functionality for scoring. Provides both scoring and
explanation capabilities. This is created per-segment.

Details on each of these classes, and their children, can be found in the subsections below.

The Query Class

In some sense, the
Query
class is where it all begins. Without a Query, there would be
nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
is often responsible
for creating them or coordinating the functionality between them. The
Query class has several methods that are important for
derived classes:

The Weight Interface

The
Weight
interface provides an internal representation of the Query so that it can be reused. Any
IndexSearcher
dependent state should be stored in the Weight implementation,
not in the Query class. The interface defines five methods that must be implemented:

topLevelBoost: A query-boost factor from any wrapping queries that should be multiplied into every
document's score. For example, a TermQuery that is wrapped within a BooleanQuery with a boost of 5 would
receive this value at this time. This allows the TermQuery (the leaf node in this case) to compute this up-front
a single time (e.g. by multiplying into the IDF), rather than for every document.

norm: Passes in a a normalization factor which may
allow for comparing scores between queries.

The Scorer Class

The
Scorer
abstract class provides common scoring functionality for all Scorer implementations and
is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not
yet abstract, but will be in future versions and should be considered as such now) methods which
must be implemented (some of them inherited from DocIdSetIterator):

nextDoc() — Advances to the next
document that matches this Query, returning true if and only if there is another document that matches.

score() — Return the score of the
current document. This value can be determined in any appropriate way for an application. For instance, the
TermScorer simply defers to the configured Similarity:
SimScorer.score(int doc, float freq).

freq() — Returns the number of matches
for the current document. This value can be determined in any appropriate way for an application. For instance, the
TermScorer simply defers to the term frequency from the inverted index:
DocsEnum.freq().

advance() — Skip ahead in
the document matches to the document whose id is greater than
or equal to the passed in value. In many instances, advance can be
implemented more efficiently than simply looping through all the matching documents until
the target document is identified.

getChildren() — Returns any child subscorers
underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained
details on the scoring process.

Why would I want to add my own Query?

In a nutshell, you want to add your own custom Query implementation when you think that Lucene's
aren't appropriate for the
task that you want to do. You might be doing some cutting edge research or you need more information
back
out of Lucene (similar to Doug adding SpanQuery functionality).

Appendix: Search Algorithm

This section is mostly notes on stepping through the Scoring process and serves as
fertilizer for the earlier sections.

In the typical search application, a Query
is passed to the IndexSearcher,
beginning the scoring process.

Once inside the IndexSearcher, a Collector
is used for the scoring and sorting of the search results.
These important objects are involved in a search:

The Weight object of the Query. The
Weight object is an internal representation of the Query that allows the Query
to be reused by the IndexSearcher.

A Sort object for specifying how to sort
the results if the standard score-based sort method is not desired.

Assuming we are not sorting (since sorting doesn't affect the raw Lucene score),
we call one of the search methods of the IndexSearcher, passing in the
Weight object created by
IndexSearcher.createNormalizedWeight(Query),
Filter and the number of results we want.
This method returns a TopDocs object,
which is an internal collection of search results. The IndexSearcher creates
a TopScoreDocCollector and
passes it along with the Weight, Filter to another expert search method (for
more on the Collector mechanism,
see IndexSearcher). The TopScoreDocCollector
uses a PriorityQueue to collect the
top results for the search.

If a Filter is being used, some initial setup is done to determine which docs to include.
Otherwise, we ask the Weight for a Scorer for each
IndexReader segment and proceed by calling
Scorer.score().

At last, we are actually going to score some documents. The score method takes in the Collector
(most likely the TopScoreDocCollector or TopFieldCollector) and does its business.Of course, here
is where things get involved. The Scorer that is returned
by the Weight object depends on what type of Query was
submitted. In most real world applications with multiple query terms, the
Scorer is going to be a BooleanScorer2 created
from BooleanWeight (see the section on
custom queries for info on changing this).

Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the coord()
factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query.
Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the
Scorer.nextDoc() method. The nextDoc() method advances
to the next document matching the query. This is an abstract method in the Scorer class and is thus
overridden by all derived implementations. If you have a simple OR query your internal Scorer is most
likely a DisjunctionSumScorer, which essentially combines the scorers from the sub scorers of the OR'd terms.