ConfigurationParameters
PatternMatchingEverything an engine needs to know about turning text into an index that is not a field's own: stop-term limit, the index and search text ceilings, case sensitivity, tokenizer and text-normalizer setup, filter cache size, word matching, embeddings and the scoring mode. How a field's text is indexed - its index sizes and start and stop characters - is set on the field ().
Constructors
ConfigurationParameters()The production configuration, as a fresh instance: every setting taken from its own type's. Equal toin every respect but identity, so preferunless you specifically want your own object.
ConfigurationParameters(IndexerSetup indexer)The indexer values of your choice, everything else as shipped.
IndexerSetupindexer— The engine's indexing values:, or that passed through.
ConfigurationParameters(IndexerSetup indexer, TokenizerSetup tokenizerSetup, TextNormalizer textNormalizer, CompoundSetup compoundSetup, int filterCacheSize = 200000)Builds a configuration from its parts. Every setup object must be stated - passfor the ones you do not care about, which is shorter and safer than copying their values out by hand.
IndexerSetupindexer— DefaultThe engine's indexing values;unless you know otherwise.TokenizerSetuptokenizerSetup— DefaultWord-token delimiters and sizes.is the 20-delimiter production set. Note thatis a single space delimiter, NOT the default.TextNormalizertextNormalizer— DefaultDiacritic folding and string replacement.folds the full table; one built with empty tables folds nothing, which is how normalization is turned off.CompoundSetupcompoundSetup— DefaultHow the index links its compounds to their parts.links them, parts of 3 letters, s or e between. It replaces 5.x's WordTrieSetup.intfilterCacheSize— DefaultFilterCacheSizeFilter cache entries. Omit for. Below the number of live filter keys the cache starts evicting and rebuilding them on next use.
ConfigurationParameters(IndexerSetup indexer, TokenizerSetup tokenizerSetup, TextNormalizer textNormalizer, CompoundSetup compoundSetup, EmbeddingSetup embeddingSetup, int filterCacheSize = 200000)DefaultAs the constructor above, with how vector fields are held stated too. The one above takes.
IndexerSetupindexer— As above.TokenizerSetuptokenizerSetup— As above.TextNormalizertextNormalizer— As above.CompoundSetupcompoundSetup— As above.EmbeddingSetupembeddingSetup— DefaultHow vector fields are held and searched.keeps the search graph's vectors as 16-bit integers.intfilterCacheSize— As above.
ConfigurationParameters(IndexerSetup indexer, TokenizerSetup tokenizerSetup, TextNormalizer textNormalizer, CompoundSetup compoundSetup, EmbeddingSetup embeddingSetup, ScoringSetup scoringSetup, int filterCacheSize = 200000)DefaultAs the constructor above, with how the searchable fields are scored stated too. The one above takes, BM25F: an overload, for the same reason as the embedding one.
IndexerSetupindexer— As above.TokenizerSetuptokenizerSetup— As above.TextNormalizertextNormalizer— As above.CompoundSetupcompoundSetup— As above.EmbeddingSetupembeddingSetup— As above.ScoringSetupscoringSetup— ScoringSetupBM25F or BM25PerField, and BM25F's k1. See.intfilterCacheSize— As above.
TokenizerSetup(Char delimiter = )Chardelimiter— Default:
TokenizerSetup(Char[] delimiters, int minimumWordSize = 2)Char[]delimitersintminimumWordSize— Default: 2
TextNormalizer(ValueTuple<string, string>[] stringReplacements, ValueTuple<Char, Char>[] charReplacements, bool oneWayMode, string[] keepOriginalOf = null)ValueTuple<string, string>[]stringReplacementsValueTuple<Char, Char>[]charReplacementsbooloneWayModestring[]keepOriginalOf— Default: null
TextNormalizer(StringReplacer stringReplacer, CharReplacer charReplacer, string[] keepOriginalOf = null)StringReplacerstringReplacerCharReplacercharReplacerstring[]keepOriginalOf— Default: null
CompoundSetup(bool Compounds = true, int MinimumPartLength = 3, string LinkingLetters = "se")How the index links its compounds to their parts, fixed with the index: every word of letters that is made of words of the index, each ofletters or more, one ofallowed between two ("fisk-e-boller", "dag-s-avis"). A word then meets the compounds it ends ("ost": "brunost"), a compound the records holding its parts ("hvitløksmør": "hvitløk smør"), and two words the compound they make.
boolCompounds— Link the compounds. Off, a word meets only itself and its variants within the edits.intMinimumPartLength— The shortest part, in letters. 3 makes "ost" a part of "brunost", and also "rom" of "syndrom".stringLinkingLetters— The letters that may stand between two parts.
CharReplacer(ValueTuple<Char, Char>[] replacements, bool oneWayMode)ValueTuple<Char, Char>[]replacementsbooloneWayMode
CharReplacer(Char[] toBeReplaced, Char[] replaceBy, bool oneWayMode)Char[]toBeReplacedChar[]replaceBybooloneWayMode
StringReplacer(string[] toReplace, string[] replaceBy)string[]toReplacestring[]replaceBy
StringReplacer(ValueTuple<string, string>[] replaceMents)ValueTuple<string, string>[]replaceMents
IndexerSetup(int stopTermLimit, bool doCaseSensitiveSearching, int maxIndexTextLength, int maxSearchTextLength)Builds the engine's indexing values. Most callers want, orthrough.
intstopTermLimit— How many documents a single term may hit before it is treated as a stop word and stops contributing to ranking. A term in nearly every document separates nothing.booldoCaseSensitiveSearching— When false — the usual choice — query text is lower-cased before matching.intmaxIndexTextLength— Ceiling on the text ONE DOCUMENT contributes to the index. Over it the text is cut andis raised; the stored record is untouched. A safety valve, not a tuning knob — the default sits far above any real document.intmaxSearchTextLength— Ceiling on. Over it the search is REFUSED, not truncated. A far smaller number than the index ceiling, and deliberately so: a document may legitimately be enormous, a query is typed by a person.
ScoringSetup(ScoringMode mode, float bm25fK1 = 1,2)ScoringModemode— ModeSee.floatbm25fK1— BM25FK1See.
EmbeddingSetup(VectorStorage graphVectors, DocumentVectorStorage documentVectors = Float32, bool automaticRebuild = true, double rebuildOwnerlessShare = 0,5, int rebuildOwnerlessMinimum = 1000)VectorStoragegraphVectors— GraphVectorsSee.DocumentVectorStoragedocumentVectors— DocumentVectorsSee.boolautomaticRebuild— AutomaticRebuildSee.doublerebuildOwnerlessShare— RebuildOwnerlessShareSee.intrebuildOwnerlessMinimum— RebuildOwnerlessMinimumSee.
Properties
| Name | Type | Access | Description |
|---|---|---|---|
Indexer | IndexerSetup | get | PatternMatchingThe stop-word threshold, case sensitivity, and the indexed-text and search-text
ceilings. How a field's text is indexed is the field's (). |
FilterCacheSize | int | get | Size of the filter cache, in entries. |
TextNormalizer | TextNormalizer | get | How text is normalized for the index and the query, for example 'ø' replaced by
'oe'; the stored documents are not changed. |
TokenizerSetup | TokenizerSetup | get | How text is cut into the words (tokens) that are indexed. |
CompoundSetup | CompoundSetup | get | MatchHow the index links its compounds to their parts. The word lookup's own
settings are per search (). |
EmbeddingSetup | EmbeddingSetup | get | How vector fields are held and searched. |
Scoring | ScoringSetup | get | How the searchable fields are scored:(the
default) or, and BM25F's k1. Fixed for the
engine's life, as everything here is. |
Default | ConfigurationParameters | get | PatternMatchingThe standard production configuration: full diacritic normalization, the production
delimiters, and the compounds linked to their parts. This is what an engine gets when no
configuration is supplied. How each field is indexed is the field's own
(). |
Delimiters | Char[] | get | The token delimiters. Returns a COPY: the array is handed to callers, and a get-only
property protects the reference, never the contents — without the copy,would retune every engine in the
process that took the defaults. |
MaxSizeOfWordToken | int | get/set | A word shorter than this is indexed as a word term, and is a word to Coverage; a longer
one is found through the field's pattern matching alone. It keeps strings that are no
words (base64, hashes, URLs without delimiters) out of the term dictionary. |
MinimumWordSize | int | get | The shortest word token, 2 by default: the one rule for how short a word may be, in the
index (every field's words, matched by pattern or not), in a query's words and in the
word lookup, which looks no shorter word up. A field'sconcern pattern matching and say nothing about
words. |
AllowedWordTerminators | Char[] | get/set | The delimiters after which a word ends whole. A delimiter
outside this set joins: "øl/vinglass" is beer glasses and wine glasses, so "øl" there
is met as the start of a longer word ("ølglass"), not as the word itself, in Coverage
and in the field match level. The end of the text always ends a word. Returns a copy,
as;unless set, and so
for a configuration file written before. |
DefaultWordTerminators | Char[] | get | DefaultThe delimiters ofafter which a word ends whole: white space,
punctuation and brackets. Left out, and so joining:. A hyphen
is a space before the tokenizer sees it (the default normalizer's string rules). |
CharReplacer | CharReplacer | get | return the CharReplacer class. |
StringReplacer | StringReplacer | get | return the StringReplacer. |
KeepOriginalOf | string[] | get | The string rules, by what they look for, whose original is kept: the index
holds the text both with every rule and without these, and the query is met in the form
without them. Case normalization is done on the original as on the ruled form: both are
lower-cased unless the search is case-sensitive. Empty when every rule replaces. |
Compounds | bool | get/set | Link the compounds. Off, a word meets only itself and its
variants within the edits. |
MinimumPartLength | int | get/set | A part of at least one letter: a part of none would make every word a compound. |
LinkingLetters | string | get/set | Letters only, none if empty. |
OneWayMode | bool | get/set | When i.e. 'ä' is replaced with 'a' the system will not be able to distinguish a search for
Q = "grä" and a search for Q ="gra". Setting OneWayMode = true will overcome this problem
on the cost of slightly higher resource usage. The overhead will depend on the fraction
of the documents which differs after replacement. |
ToBeReplaced | Char[] | get | StringReplacerThe characters replaced, and what each becomes — the same parallel-array shapeuses. Exposed so a normalizer can be written out and read
back: without it the replacement table was unreachable from the public surface, and a
configuration that round-tripped through JSON came back silently unable to fold
diacritics. It would still index and still search, just differently. |
ReplaceBy | string[] | get | The replacement targets. Returns a COPY — a get-only property protects the reference,
never the contents, and these arrays are reachable from a shared configuration. |
ToReplace | string[] | get | |
StopTermLimit | int | get | Largest number of documents a single term may hit before it is treated as a stop word. |
DoCaseSensitiveSearching | bool | get | Whether indexing and matching distinguish case. |
MaxIndexTextLength | int | get | Ceiling on the text a single document contributes to the index. A safety valve, not a
tuning knob: the default (100 000) sits far above any real document, so reaching it means
the input is wrong rather than merely long. |
MaxSearchTextLength | int | get | Ceiling on the length of. A query over it is REFUSED — an emptycarrying a, plus—
rather than truncated: a query that long cannot match more by being cut, and a silently
cut query answers a different question than the one asked. |
Mode | ScoringMode | get | : one index over all searchable fields, weights and
length normalisation per field inside one saturation.:
one BM25F index per field, each with that field's k1 and b, merged by. |
BM25FK1 | float | get | The k1 of aengine, for every field. Typical range
1.0 to 2.0; lower saturates repeated terms sooner. Not read in, where each field has its own. |
AutomaticRebuild | bool | get | Whether a vector field's search graph is rebuilt by itself when removed vectors have
gathered in it past the two thresholds below. True by default. |
RebuildOwnerlessShare | double | get | How many removed vectors a graph may hold for each vector it still has before a
rebuild is due. 0.5 by default: half as many removed as there are left, a third of
the graph. |
RebuildOwnerlessMinimum | int | get | The fewest removed vectors for which a rebuild is due, whatever their share. 1 000
by default, so that a small dataset is not rebuilt for a handful of deletes: there
the places cost next to nothing to pass over. |
DocumentVectors | DocumentVectorStorage | get | GraphVectorsHow a document's own copy of a vector is held in memory. Independent of: this is the copy a client is handed back in the
document's JSON, that one is the copy a search runs on. |
GraphVectors | VectorStorage | get | How the search graph of a vector field holds its vectors. It decides what a vector
search costs in memory and time, and nothing a client is handed: the graph's copy of
a vector is never returned, and a document's own JSON is not touched by this. |
Methods
With
ConfigurationParameters With(IndexerSetup indexer = null, TokenizerSetup tokenizerSetup = null, TextNormalizer textNormalizer = null, int? filterCacheSize = null, CompoundSetup compoundSetup = null, EmbeddingSetup embeddingSetup = null, ScoringSetup scoringSetup = null)IndexerSetupindexer— Default: nullTokenizerSetuptokenizerSetup— Default: nullTextNormalizertextNormalizer— Default: nullint?filterCacheSize— Default: nullCompoundSetupcompoundSetup— Default: nullEmbeddingSetupembeddingSetup— Default: nullScoringSetupscoringSetup— Default: null
Returns: ConfigurationParameters
Save
Writes this configuration as JSON. The counterpart to: what comes back is equal to what went in, property for property, including the diacritic table inside.
void Save(Stream stream)Streamstream
Load
Stream)Reads a configuration written by. The constructor validates it on the way in, so a file that has been hand-edited into nonsense throws here rather than producing an engine that behaves oddly later.
static ConfigurationParameters Load(Stream stream)Streamstream
Returns: ConfigurationParameters
CreateDefaultOneWayNormalizer
Builds the standard Indx one-way text normalizer (diacritic folding + delimiter string replacements). Kept because it is public API; the table itself moved to, which is now its only definition. Returns a fresh instance, as it always did —is the shared one.
static TextNormalizer CreateDefaultOneWayNormalizer()Returns: TextNormalizer
CreateDefaultOneWay
DefaultBuilds a fresh copy of the standard one-way normalizer.is one of these, cached; call this instead when you intend to own the instance.
static TextNormalizer CreateDefaultOneWay()Returns: TextNormalizer
ToString
string ToString()Returns: string
GetHashCode
int GetHashCode()Returns: int
Equals
bool Equals(object obj)objectobj
Returns: bool
<Clone>$
CompoundSetup <Clone>$()Returns: CompoundSetup
Deconstruct
void Deconstruct(Boolean& Compounds, Int32& MinimumPartLength, String& LinkingLetters)Boolean&CompoundsInt32&MinimumPartLengthString&LinkingLetters
Replace
The text with the rules applied in order, as a new string.
string Replace(string text)stringtext
Returns: string