From java-dev-return-47514-apmail-lucene-java-dev-archive=lucene.apache.org@lucene.apache.org Tue Mar 09 23:17:23 2010
Return-Path:
Delivered-To: apmail-lucene-java-dev-archive@www.apache.org
Received: (qmail 38961 invoked from network); 9 Mar 2010 23:17:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
by 140.211.11.9 with SMTP; 9 Mar 2010 23:17:23 -0000
Received: (qmail 91085 invoked by uid 500); 9 Mar 2010 23:16:54 -0000
Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org
Received: (qmail 91037 invoked by uid 500); 9 Mar 2010 23:16:54 -0000
Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: java-dev@lucene.apache.org
Delivered-To: mailing list java-dev@lucene.apache.org
Received: (qmail 91030 invoked by uid 99); 9 Mar 2010 23:16:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 23:16:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 23:16:51 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
by brutus.apache.org (Postfix) with ESMTP id A4E03234C4B5
for ; Tue, 9 Mar 2010 23:16:30 +0000 (UTC)
Message-ID: <1247671828.165181268176590674.JavaMail.jira@brutus.apache.org>
Date: Tue, 9 Mar 2010 23:16:30 +0000 (UTC)
From: "Michael McCandless (JIRA)"
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-2089) explore using automaton for
fuzzyquery
In-Reply-To: <1780563173.1258902039668.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org
[ https://issues.apache.org/jira/browse/LUCENE-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843354#action_12843354 ]
Michael McCandless commented on LUCENE-2089:
--------------------------------------------
YEAH!
> explore using automaton for fuzzyquery
> --------------------------------------
>
> Key: LUCENE-2089
> URL: https://issues.apache.org/jira/browse/LUCENE-2089
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: Flex Branch
> Reporter: Robert Muir
> Assignee: Robert Muir
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: ContrivedFuzzyBenchmark.java, createLevAutomata.py, gen.py, gen.py, gen.py, gen.py, gen.py, gen.py, Lev2ParametricDescription.java, Lev2ParametricDescription.java, Lev2ParametricDescription.java, Lev2ParametricDescription.java, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089.patch, LUCENE-2089_concat.patch, Moman-0.2.1.tar.gz, moman-57f5dc9dd0e7.diff, TestFuzzy.java
>
>
> we can optimize fuzzyquery by using AutomatonTermsEnum. The idea is to speed up the core FuzzyQuery in similar fashion to Wildcard and Regex speedups, maintaining all backwards compatibility.
> The advantages are:
> * we can seek to terms that are useful, instead of brute-forcing the entire terms dict
> * we can determine matches faster, as true/false from a DFA is array lookup, don't even need to run levenshtein.
> We build Levenshtein DFAs in linear time with respect to the length of the word: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.16.652
> To implement support for 'prefix' length, we simply concatenate two DFAs, which doesn't require us to do NFA->DFA conversion, as the prefix portion is a singleton. the concatenation is also constant time with respect to the size of the fuzzy DFA, it only need examine its start state.
> with this algorithm, parametric tables are precomputed so that DFAs can be constructed very quickly.
> if the required number of edits is too large (we don't have a table for it), we use "dumb mode" at first (no seeking, no DFA, just brute force like now).
> As the priority queue fills up during enumeration, the similarity score required to be a competitive term increases, so, the enum gets faster and faster as this happens. This is because terms in core FuzzyQuery are sorted by boost value, then by term (in lexicographic order).
> For a large term dictionary with a low minimal similarity, you will fill the pq very quickly since you will match many terms.
> This not only provides a mechanism to switch to more efficient DFAs (edit distance of 2 -> edit distance of 1 -> edit distance of 0) during enumeration, but also to switch from "dumb mode" to "smart mode".
> With this design, we can add more DFAs at any time by adding additional tables. The tradeoff is the tables get rather large, so for very high K, we would start to increase the size of Lucene's jar file. The idea is we don't have include large tables for very high K, by using the 'competitive boost' attribute of the priority queue.
> For more information, see http://en.wikipedia.org/wiki/Levenshtein_automaton
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org