Reranking
The reranker is a collection of algorithms that detect exact and near-exact token matches in the query, lifting them to the top of the result list. It complements the core pattern recognition which always runs first. It is on by default (Query.Ranking.Rerank), in either search mode. Before 6.0 it was called Coverage, and the API's names said so (EnableCoverageReport, CoverageReport, Field.CoverageWeight); they are EnableMatchReport, MatchReport and Field.RerankWeight now.
How It Works
The search first counts the documents that hold every query word, exactly or within one or two edits (the candidates, see Query.Candidates). The reranker then scores each of them 0–65535 by how much of the query the document covers, and blends that with the pattern recognition score (Query.Ranking.Share).
How the Reranker Ranks
The score is the share of the query text the document covers, so 65535 means all of it. Among documents that cover the query equally, three small deductions decide the order:
| Deduction | Amount | When |
|---|---|---|
Word out of order | 257 per word | An exactly matched word breaks the query's word order. Order is relative: the phrase in order anywhere in the text is in order |
Not at the start | 16 | The document does not begin with the query's first word |
Ends inside a word | 8 | The whole query was found as one string, but its last character is followed by more letters |
When the whole query appears in the document as one string, it sets a floor under the score, by where that string starts:
| Whole query found | Score at least |
|---|---|
At the start of the document | 65535 |
At the start of a word further in | 65519 |
Inside a word | (q − 1) / q × 65535, where q is the query length |
This is what makes search-as-you-type rank well. For knut h, the final h is too short to count as a word, but knut hamsun contains the whole query and scores 65527 (starts the document, ends inside a word), while knut faldbakken covers only knut and scores 43690.
Diacritic folding follows the TextNormalizer. In one-way mode (the default) a query is folded only against a document whose text folds, so blåbær fully matches blåbær but not blabar, while blabar fully matches both. In two-way mode both sides always fold, and the two spellings are the same query.
The settings
What the reranker reads is spread over the query's three parts (6.0; until then one CoverageSetup):
var query = new Query("knut hamsun", 20)
{
Match = { Tolerance = Tolerance.OneEdit, JoinedWords = true }, // what counts as a hit
Candidates = { Fallback = true }, // who comes in
Ranking = { Share = 0.2f, WholeQueryFloor = true }, // the order
};| Setting | Default | Description |
|---|---|---|
Match.Tolerance | TwoEdits | The most edits a word may differ by in this search ( Exact, OneEdit, TwoEdits, ThreeEdits). It caps each field's own Field.Tolerance (default OneEdit), never raises it |
Match.OneEditFrom, Match.TwoEditsFrom, Match.ThreeEditsFrom | 3, 6, 9 | The shortest word given one edit, two and three. 0 is never. Edits reach words of 64 characters at most; a longer word is met as typed only |
Match.JoinedWords | true | Words written together or apart count as the word ( blueberry for blue berry) |
Ranking.Rerank | true | Whether the reranker orders the candidates, in either search mode ( Query.Mode). false: the pattern score alone |
Ranking.Share | 0.2 | Share × reranker score + (1 − Share) × pattern score. 1 is the reranker alone |
Ranking.TypingShift | 0.5 | How far the share moves towards the reranker alone while the last word is being typed |
Ranking.WholeQueryFloor | true | The whole query found as one string sets a floor under the score (see above) |
Ranking.LongFieldFrom, Ranking.LongFieldWeightFloor | 200, 0 | From this length, in characters, a record's field counts less in the reranker: its weight is multiplied by this length over the field's own to the power 0.18 (735 characters count 0.79, 2 000 count 0.66), never below the floor. 0 is never |
Ranking.FieldWeights | empty | Per field, its weight in the reranker in this search in place of Field.RerankWeight; 0 leaves the field out of the reranker and the word lookup |
Candidates.Fallback | true | When no record holds every query word: the records holding the most of the words, then pattern matches ( Result.IsFallback). false gives an empty list |
Whole words, and a word as the start or end of a longer one, are always detected; the shortest
word is TokenizerSetup.MinimumWordSize. CoverWholeWords, CoverPrefixSuffix,
PrefixSuffixWhileTyping, MinWordSize and the per-field entries of CoverageSetup.Fields are
gone, as are the cut and its settings: the list is the counted set, cut only to the rows asked for.
Common Patterns
Reranking should be kept on (Ranking.Rerank, the default) for nearly all use cases, including search-as-you-type. It improves result quality across multiple searchable fields, at some cost per search: turn it off where speed matters more than the order inside the counted set.
Agent/tool usage (only records holding every word):
query.Candidates.Fallback = false;More candidates than the rows times four:
query.Candidates.MaxCandidates = 5000;Constructors
MatchSetup()CandidateSetup()RankingSetup()Properties
| Name | Type | Access | Description |
|---|---|---|---|
Tolerance | Tolerance | get/set | The most a word may differ by in this search:,or, the default,
which leaves each field its own (): a field's own is
capped by it, never raised. |
Rungs | Rungs | get/set | Which rungs are counted:, the default, the first that
gives hits;, every one up to the tolerance, the nearer
ranked first. |
OneEditFrom | int | get/set | The shortest query word given one edit, 3 by default; 0 is never. At three characters one
edit already reaches a large share of the vocabulary, below it nearly everything. |
TwoEditsFrom | int | get/set | The shortest query word given two edits, 6 by default; 0 is never. Two edits are a small share of a long word and a large one of a short
word (foris two in eleven). |
ThreeEditsFrom | int | get/set | The shortest query word given three edits, 9 by default; 0 is never. Only where the
search'sand the field's are. A word of 10 letters or more typically meets one
term within three edits; one of 6 or 7 letters meets dozens, and at worst thousands.
Edits of any number reach words of 64 characters at most; a longer word is met exactly
only (). |
CursorAt | int? | get/set | TextWhere the cursor is in, as a character index: the
word it is in or right after is the word being typed, met by its start as well as within
its edits; every other word is finished, met within its edits only. A cursor at 0 or right
after a delimiter is in no word. Null, the default: the last word is being typed when no
delimiter follows it. A search box that knows the cursor sends it, so that a user going
back to mend an earlier word ("kyll filet", the cursor after "kyll") gets that word's
starts. |
TypingPrefixFrom | int | get/set | The word being typed is also met as the start of a longer word from this many characters,
2 by default ("le" finds Leonardo); 0 is never. |
TypingPrefixOneEditFrom | int | get/set | That start within one edit from this many characters, 6 by default; 0 is never. |
JoinedWords | bool | get/set | Words written together or apart count as the word ("blueberry" for "blue berry",
"hotsauce" for "hot sauce"), on by default. |
CandidatesPerRow | int | get/set | The candidates a search takes per row asked for, 4 by default: the bound is the rows
times this, at most 5 000, never fewer than the rows. Below 1 is taken as 1. |
MaxCandidates | int? | get/set | CandidatesPerRowThe bound itself, when set: how many candidates the search takes, outright, in place of
the rows timesand past the floor of 100 and the roof of
5 000 the engine keeps that between. Null, the default. Set it whenis aboveand more of the counted should be ranked. |
MinCandidates | int | get/set | The fewest candidates a search takes when the rows timesare fewer, 100 by default: a search asking for few rows still has candidates enough to
rank. Taken between 1 and the roof of 5 000;, when set, sets
the bound outright past it. |
NoFallbackForNumbersFrom | int | get/set | No fallback for a number no record holds: a query that is one number of at least this
many digits (a final X allowed, for an ISBN-10) and that no record holds answers asfalse does, no records and a count of 0, rather than the records
sharing some of its digits. 0, the default, is off. Only the word lookup can say that no
record holds a word, so the field holding the numbers must be matched by its words
(); a number held only in a field matched by pattern is
never found by the lookup, and with this on its searches would answer empty. |
IndexTerms | IndexTerms | get/set | BothWhich of the index's terms BM25F scores with:, the
default, as the index holds them; the words alone; or the pieces alone. The index is not
rebuilt: a term knows whether it is a word. |
Fallback | bool | get/set | IsFallbackWhen no document holds every query word: the fallback, the documents holding the most of
the words, then BM25F's best (). On by default; off gives an
empty list, the counts still said. |
FillWithMostWords | bool | get/set | When too few documents hold every query word to fill the rows asked for:
below the counted documents and the next rungs' fill, the documents holding the most of
the words, those lacking one word first and, within that, those lacking the commonest,
shown and not counted (;, the rungs
and the facets stay the counted set's). Off by default: it gains most in a product
search with few hits, and is noise below the one record a title search wants
("matrix reloaded": many films hold one of the words). Apart from, so a dataset can keep the fallback for a query with a large typo
and leave short pages short. |
FillReach | int | get/set | How many rungs past the counted one the word lookup climbs to fill a page the counted
documents leave short (with), shown below them and not
counted (): 0, the default, no fill, the page as short as the
counted documents leave it; 1 the next rung only ("chili" counted, "chilli" below it;
"dark night" counted, The Dark Knight below it); 2 up to two edits. Below 0 is taken as
0, above 2 as 2. Off by default: on short titles a reach of 0 ranks as 1 does, and a
search with typo matching is much cheaper, since with the exact rung counted no edit is
searched for. A product search with few hits per query gains from 1. |
Rest | bool | get/set | In, BM25F's best not counted below the counted set,
up to the candidate bound: the rest, a fourth rung after two edits, in the last band,
counted () and in the facets, so count, facets and list
still agree. Off by default: the list is the counted set. On, it ranks as a relevance
search does. Not read in. |
Rerank | bool | get/set | Whether the candidates are reranked by Coverage (the field match level and the blend,), in either mode; on by default. Off:is BM25F alone, andorders inside a rung by BM25F's score. |
Share | float | get/set | RerankHow Coverage and BM25F make one score:, both
between 0 and 1; 0.2 by default. Coverage says how much of the query a document holds,
letter for letter; BM25F knows the field weights, how rare a word is and how long a field
is. Values outside 0..1 are taken as the nearest end. Not reranked
(off) there is no Coverage, and BM25F alone orders. |
TypingShift | float | get/set | How far the share moves towards Coverage alone while the last word is being typed:, 0.5 by default. 0 is no move, 1 is Coverage
alone. A word the index does not hold whole ("warr"), or, in a query of one word, any
word being typed ("war"). BM25F credits the letters typed wherever they stand, Coverage
where a word starts, which is what is being typed. |
LongFieldFrom | int | get/set | LongFieldWeightFloorThe text length, in characters, from which a field counts less in Coverage: where a
record's field is longer, its Coverage weight there is multiplied by this length over its
own to the power 0.18, never below; 735 characters
count 0.79, 2 000 count 0.66. 200 by default; 0 is never. |
LongFieldWeightFloor | float | get/set | LongFieldFromThe least a field's length may bring its Coverage weight down to
(), as a share of its own, 0 by default: the damped fall
seldom reaches a floor. |
WholeQueryFloor | bool | get/set | The whole query as one string in a field puts a floor under the document's Coverage
score, on by default. |
FieldWeights | Dictionary<string, float> | get/set | Per field, its weight in Coverage in this search, in place of its own
(): what a character confirmed there counts against the
heaviest field. 0 or more; a name that is not a searchable field, or a weight below 0,
makes the search return no records and say why in. Empty by
default. |
FieldBoosts | Dictionary<string, float> | get/set | Per field, a boost inside BM25F's saturation in this search (doubles
the title's influence). Typical range 0.5 to 3; above about 5 a field overwhelms the IDF.
Null or empty is no boosts. A field that is not searchable, or a boost below 0, is refused. |
Boosts | Boost[] | get/set | The boosts of this search: a document a boost's filter holds is scored higher by its
strength. Null or empty is none; there is no switch beside. |
Buries | Bury[] | get/set | The buries of this search, the boosts' counterpart: a document a bury's filter holds is
scored lower by its strength; nobody else moves. Null or empty is none. |
Toleranceenum
The rungs the word lookup meets a query word on, from the nearest. A document is on its worst word's rung.
| Name | Value |
|---|---|
Exact | 0 |
OneEdit | 1 |
TwoEdits | 2 |
ThreeEdits | 3 |
Rungsenum
RungsWhich of the word lookup's rungs a search counts (): the first that gives hits, or every one up to the tolerance.
| Name | Value |
|---|---|
Fewest | 0 |
All | 1 |