Field
Fields are discovered automatically via Init() and then configured before Load(). The engine is schema-less — you never declare fields up front.
engine.Init(fstream);
var name = engine.GetField("name")!;
name.WordMatching = true; // matched by its whole words; counts
name.PatternMatching = PatternMatchingSetup.Default; // and inside its words
name.Weight = 2.0f;
engine.GetField("description")!.WordMatching = true;
engine.GetField("description")!.PatternMatching = PatternMatchingSetup.Default;
engine.GetField("category")!.Filterable = true;
engine.GetField("category")!.Facetable = true;
engine.GetField("price")!.Filterable = true;
engine.GetField("price")!.Sortable = true;A configuration from Init() searches nothing. Every field starts with neither WordMatching nor PatternMatching; give at least one field WordMatching, or no search narrows.
Configuration Properties
| Property | Type | Default | Description |
|---|---|---|---|
WordMatching | bool | false | Match the field by its whole words, exactly and within the edits its Tolerance allows. Its words go in the word lookup, so the field counts: a search narrows to the records holding every query word. Every word from TokenizerSetup.MinimumWordSize (2) is indexed |
Filterable | bool | false | Allow this field to be used in filters and boosts. Refused with an ArgumentException on a field with no type |
Facetable | bool | false | Return value/count pairs for this field with search results. On a number field it makes the field Filterable too (a range over the facet, a slider's, is a range filter); set both, and if Filterable is forgotten it is set. Not the other way |
Sortable | bool | false | Allow sorting by this field. Score is always the primary sort key. Refused with an ArgumentException on a field with no type |
PatternMatching | PatternMatchingSetup? | null | Match the field inside its words: a query finds it by any part of a word. The setup's IndexSizes (default [3], each 1 to 64), NumberOfStartCharacters (2) and NumberOfStopCharacters (0) say how; new PatternMatchingSetup([2, 3]) matches short values (tags, codes) by two characters, and a stop character anchors the end. Alone, without WordMatching, the field orders records but cannot narrow them. WordMatching with PatternMatchingSetup.Default is how every searchable field was matched in 5.0. Changing the setup of a searched field takes effect at the next Index() |
Tolerance | Tolerance | OneEdit | The most edits the field allows a query word: Exact, OneEdit, TwoEdits, ThreeEdits. Capped by the search's Match.Tolerance |
RerankWeight | float | 1.0f | The field's weight in the reranker. 0 leaves it out of the reranker and the word lookup; it still counts through BM25F |
TypingPrefixFrom | int? | null | From how many letters the word being typed matches the start of a longer word in this field. null takes the search's Match.TypingPrefixFrom |
CaseSensitive | bool | the engine's | Whether the field is matched with case. Defaults to the engine's IndexerSetup.DoCaseSensitiveSearching. Set at any time (or with FieldProxy.CaseSensitive); it takes effect at the next Load, and DocumentFields.RequiresReload says when a change waits for one. Saved with the field configuration |
TextNormalizer | TextNormalizer | the engine's | The string rules and character folding the field's text and the query against it go through: the engine's ConfigurationParameters.TextNormalizer unless the field is given its own (an ISBN or code field that deletes - and spaces, say). Set at any time; it takes effect at the next Load. Saved with the field configuration, as its differences from the shipped tables |
Embeddable | bool | false | Mark this field for embedding-based (vector) search. Embeddable fields are excluded from the text search index |
PreloadFilters | bool | false | Cache a filter for each distinct value during Load. Accessible via SearchEngine.PreLoadedFilters. Faster repeated filtering at the cost of memory |
Weight | float | 1.0f | Relative score weight for this field. Higher values make matches in this field rank higher |
BM25b | float | 0.75f | Length normalisation parameter. 0 = no normalisation, 1 = full normalisation. Applies in both BM25F and BM25PerField modes |
BM25k1 | float | 1.2f | Term-frequency saturation, read only when the engine scores with ScoringMode.BM25PerField. Typical range 1.0–2.0. In BM25F, the default, k1 is the engine's (ScoringSetup.BM25FK1): a value set here is ignored and Status.FieldK1Ignored says so, and TrySetFieldConfiguration refuses a changed one |
Read-Only Properties
| Property | Type | Description |
|---|---|---|
Name | string | Field name as it appears in the JSON data |
Type | JsonValueKind | Value kind: String, Number, True/False, Array, Object |
IsArray | bool | Whether this field holds an array of values |
Optional | bool | true if the field is absent or null in at least one document |
EmbeddingDimensions | int | Detected vector dimension. 0 until data is loaded with Embeddable = true |
Setting
PreloadFilters = trueautomatically setsFilterable = trueon the same field.
Fields With No Type
A field that was null or absent in every document Init() read has no type. Filterable and
Sortable are the only two roles that need one, to choose between a numeric and a categorical
index and to pick a comparator, so both are refused on such a field: TrySetFieldConfiguration returns
false with the reason, and the setters on Field throw ArgumentException.
Clearing either is always allowed.
WordMatching, PatternMatching and Facetable do not read the type and stay available. Setting
WordMatching on an empty field prepares it: a record inserted later that does carry a value is
indexed and found.
Changing Configuration After Load
A change needs one of three things, and you can ask which before applying it:
var proposed = new[] { new FieldProxy { FieldName = "category", Facetable = true } };
if (engine.DocumentFields.RequiresReload(proposed))
{
// Whether a field is searched at all (WordMatching or PatternMatching turned on or
// off), Embeddable and PreloadFilters. Only Load consumes these. Applied in place they return the engine to Created,
// so searches fail with a reason instead of answering as if the field were empty.
}
else if (engine.DocumentFields.RequiresReindex(proposed))
{
engine.TrySetFieldConfiguration(proposed, out string error);
engine.Index();
}
else
{
engine.TrySetFieldConfiguration(proposed, out string error); // takes effect at query time
}RequiresReindex covers which way a searched field is matched, a changed pattern setup, and a field getting its first role. A document keeps values only for the
fields in use when it was loaded, so such a field has none until Index() reads them back from the
stored JSON. A second role on a field that already had one takes effect at query time.
To apply a reload-only change without taking the engine out of service, build the new index beside
the live one with CreateInMemoryClone(monitor, fieldOverrides) and swap it in.
FieldProxy
FieldProxy is used by GetFieldConfiguration() (returns FieldProxy[]) and TrySetFieldConfiguration(FieldProxy[], out string error). All boolean and float properties are nullable — null on a set call means "leave unchanged"; every property is populated on a get call.
// Read the full configuration
FieldProxy[] config = engine.GetFieldConfiguration();
// Apply changes — only non-null values are written
engine.TrySetFieldConfiguration([
new FieldProxy { FieldName = "title", WordMatching = true, PatternMatching = true, Weight = 2.0f },
new FieldProxy { FieldName = "description", WordMatching = true, PatternMatching = true },
new FieldProxy { FieldName = "sku", WordMatching = true, Tolerance = Tolerance.Exact },
new FieldProxy { FieldName = "category", Filterable = true, Facetable = true },
new FieldProxy { FieldName = "price", Filterable = true, Sortable = true },
], out string error);| Property | Type | Description |
|---|---|---|
FieldName | string | Must match a field name discovered by Init() |
FieldType | string? | Read-only on GET ( "String", "Number", "Boolean"). Ignored on SET |
IsArray | bool? | Read-only on GET. Ignored on SET |
WordMatching | bool? | Matched by its whole words; counts |
Filterable | bool? | Allow use in filters and boosts |
Facetable | bool? | Return value/count pairs with results |
Sortable | bool? | Allow sorting by this field |
PatternMatching | bool? | On (the field's setup, or PatternMatchingSetup.Default) or off |
IndexSizes | int[]? | With pattern matching on, the setup's index sizes. Sent for a field that stays off, ignored with a warning in Status.ErrorMessage |
NumberOfStartCharacters | int? | With pattern matching on, the setup's start characters; as IndexSizes otherwise |
NumberOfStopCharacters | int? | With pattern matching on, the setup's stop characters; as IndexSizes otherwise |
Embeddable | bool? | Vector embedding field |
PreloadFilters | bool? | Cache a filter per distinct value during Load. Also forces Filterable = true |
Weight | float? | Score weight multiplier |
BM25b | float? | Length normalisation parameter |
BM25k1 | float? | TF saturation parameter (BM25PerField only) |
Tolerance | Tolerance? | The most edits the field allows |
RerankWeight | float? | Weight in the reranker; 0 leaves the field out of it |
TypingPrefixFrom | int? | When the word being typed matches by its beginning |
CaseSensitive | bool? | Match with case; in effect at the next Load |
TextRules | TextRule[]? | The field's own string rules before the engine's ( new TextRule("-", "")); [] the engine's again; in effect at the next Load |
A FieldProxy from a 5.0 client or file may carry Searchable: it is read as WordMatching alone, so such a field loses pattern matching. WordIndexing and HighResolution bind to nothing.
Scoring Modes
The scoring mode is part of the engine's configuration, chosen when the engine is built. It decides where k1 is set:
// BM25F (default): one index over all searched fields, one k1 for all of them
var bm25f = new SearchEngine(string.Empty, null,
ConfigurationParameters.Default.With(scoringSetup: new ScoringSetup(ScoringMode.BM25F, bm25fK1: 1.2f)));
// BM25PerField: one index per searched field, each with its own k1 and b, merged by Weight
var perField = new SearchEngine(string.Empty, null,
ConfigurationParameters.Default.With(scoringSetup: new ScoringSetup(ScoringMode.BM25PerField)));
perField.GetField("title")!.BM25k1 = 2.0f; // read in this mode onlyengine.Scoring (and engine.ScoringMode) read the choice back. Until 6.0 the mode was inferred from whether BM25k1 differed across the searchable fields; it no longer is.
Field Weights
Weight is a plain float. The practical range is 0.5–3.0. The ratio between fields matters — 2.0 and 1.0 produces the same relative effect as 4.0 and 2.0.
engine.GetField("title")!.Weight = 3.0f;
engine.GetField("heading")!.Weight = 2.0f;
engine.GetField("metadata")!.Weight = 1.5f;
engine.GetField("body")!.Weight = 1.0f;There is no hard maximum, but values above 5× the lowest weight in your set offer diminishing returns. If one field needs to dominate that strongly, it is usually better to make it the only searched field.
Constructors
Field()Parameterless constructor for deserialization.
Field(string name, JsonValueKind type, object value, bool isArray)stringnameJsonValueKindtypeobjectvalueboolisArray
DocumentFields()Default ctor.
FieldProxy()PatternMatchingSetup()DefaultThe setup as shipped:'s values.
PatternMatchingSetup(int[] indexSizes, int numberOfStartCharacters = 2, int numberOfStopCharacters = 0)int[]indexSizesintnumberOfStartCharacters— Default: 2intnumberOfStopCharacters— Default: 0
Properties
| Name | Type | Access | Description |
|---|---|---|---|
CaseSensitive | bool | get/set | Whether this field's text is searched with its case: the index keeps the case of the
field's text and the query is matched as typed. False as shipped, the engine's, given to every field at Init. Set at
any time; it takes effect at the next Load, which the engine reads the field with until
then (says when a change waits for one).
Saved with the field configuration; a configuration saved without it takes the engine's
default. |
TextNormalizer | TextNormalizer | get/set | The normalizer this field's text and the query against it go through (the string rules,
the character folding): the engine'sunless the field is given its own. Set at any time; it takes effect at the next Load, asdoes. Saved with the field configuration, as its exceptions and
additions over the shipped tables, the way a configuration file saves the engine's. |
BM25b | float | get/set | Per-field length-normalization parameter b for BM25 / BM25F scoring.
Range [0, 1]: 0 = no length normalization, 1 = full normalization.
Typical values: 0.3 for short fields (title), 0.75 for longer fields (body).
Defaults to 0.75. Applies in both(per-field BM25) and(multi-field BM25F). |
BM25k1 | float | get/set | BM25PerFieldThe field's term-frequency saturation k1, read only when the engine scores with. Typical range 1.0–2.0; lower values saturate
repeated terms sooner. Defaults to 1.2. Takes effect at the next. |
PatternMatching | PatternMatchingSetup | get/set | How the field is matched by pattern, inside its words (:
the index sizes and the start and stop characters), or null, the default: not by
pattern. One of the two ways a field is searched, with; a
field with either is searched, a field with neither is not. Takes effect at the next, or at the nextwhen it is what makes the field searched or
not (). |
Embeddable | bool | get/set | When true, this field holds a vector embedding — a fixed-length array of
float values produced by an embedding model. Fields must be set embeddable
after Init during the fieldconfiguration.
Embeddable fields are excluded from the standard text search index.
During Load their float[] values are collected into a per-field list that
is passed to the HNSW index for approximate nearest-neighbour search.
The expected dimensionality is fixed on the first document loaded and
validated on all subsequent ones.
Setting this to false on a field whose name matches the auto-detection
pattern overrides the automatic classification. |
EmbeddingDimensions | int | get/set | The expected number of dimensions for this embedding field.
Set automatically from the first document loaded during Load.
Zero means not yet determined. Ignored when Embeddable is false. |
Facetable | bool | get/set | Set to true if this field is going to be used as a facetable field. |
Filterable | bool | get/set | Set to true if this field is going to be used as a filterable field.
Refused on a field with no type (null in every analyzed document). |
WordMatching | bool | get/set | Whether the field is matched by its whole words: exactly, within the edits Coverage
allows, by prefix and affix, through the word lookup. One of the two ways a field is
searched, with; a field with either is searched, a field
with neither is not. With both, the field is matched inside its words and by them. Takes
effect at the next, or at the nextwhen it is what makes the
field searched or not (). |
Sortable | bool | get/set | Must be set in order to sort on this field.
This will be a secondary search key – score is always first.
Both this field and the sort order must be passed to Query. |
IsArray | bool | get/set | Is true for a possibly flattened array. |
Name | string | get/set | The name of the field. The name must be unique. |
Optional | bool | get/set | Optional field |
SampleValue | string | get/set | A short, display-ready sample of this field's content, captured from the first document
that carried a non-empty value for it. Surfaced asso a
caller configuring the field can see what it actually holds. |
PreloadFilters | bool | get/set | Setting this will make engine setup a valuefilter for each distinct value of a field.
The filters will be created and loaded during Load and may be accessed via
SearchEngine.GetPreloadedFilters. |
Type | JsonValueKind | get/set | Only primitive types and their arrays are accepted. Note that for fields containing
flattened arrays they will all have primitive types, and not JsonArray.
Use the IsArray property to check. |
Value | object | get/set | Actual value of the field. |
Weight | float | get/set | BM25PerFieldHow much this field contributes to the relevance score, relative to the other
searchable fields —in the BM25F formula, and the weight the per-field
scores are averaged by in.
Defaults to 1.0. Typical range: 1.0–3.0, where 1.0 is neutral and higher values
increase the field's contribution. Values below 1.0 de-emphasize the field.
Applies only to searchable fields; set it after Init() and before Index(). |
Tolerance | Tolerance | get/set | The most a query word may differ by in this field, in the word lookup and in Coverage:,(the default, as
Coverage has always worked),or(long words). A search'scaps it, never raises it. Body text is usually best at: a misspelling in a long description confirms the
wrong document. Read at every search; no Index() needed. Applies only to searchable
fields. Until 6.0 CoverageStrictness (Ldd0, Ldd1, Ldd2); its None is aof 0. |
TypingPrefixFrom | int? | get/set | From how many characters the word being typed meets the longer words it starts in this
field, in the word lookup: null (the default) as the search says
(), 0 never, a number never before that many
characters. It caps the search's value, never lowers it: the larger of the two counts,
and 0 in either is never. A field of codes (an ISBN, an article number) sets 0, found by
the whole code only, or a length such as 5, so that three typed digits do not count every
code they begin; withExact the field meets only the word as
written. The reranker still credits a word met as a start in the field. Read at every
search; no Index() needed. Applies only to fields matched by their words. |
AverageLength | float | get/set | LongFieldFromThe field's mean text length in characters over the documents holding it, worked out
by(0 before). Coverage reads it for: a field whose text is long on average
counts less there. Inserts and updates afterleave it as it was until the
next. Not saved with the field configuration: it is the data's, not a setting. |
RerankWeight | float | get/set | WeightThe field's weight in the reranker (Coverage; until 6.0 CoverageWeight), against the
heaviest searchable field in it: a query
word counts with its characters times this weight over the heaviest's, so a document
whose words are only in a field of half the heaviest's weight is at most half covered.
Defaults to 1; 0 leaves the field out of Coverage and the word lookup: it confirms no
word, and counts through BM25F only (until 6.0 CoverageStrictness.None). Not, which is BM25F's and acts inside its
saturation: Coverage asks whether a field names what was typed, and the fields that do
(title, author, ISBN) should weigh alike here whatever BM25F gives them; lower it for
body text. Read at every search; no Index() needed. |
NameOfDocumentKeyField | string | get/set | Name of the key field used to identify documents.
Required for operations such as document deletion. |
TextSettingsWaitForLoad | bool | get | Whether a field'sor, set since the last Load, waits for the next: the
engine searches with what that Load took until then. |
AnalyzedDocumentCount | int | get | Number of documents the analyze pass (Init) saw in the stream. A value of 1 on a
large file is the classic "export envelope" symptom: a root object wrapping the
real document array, which the parser reads as one giant document. Callers can
use this to warn before Load/Index is attempted. |
FieldName | string | get/set | Name of the field. Must match an existing field discovered by Analyze. |
FieldType | string | get/set | JSON value kind of the field (e.g. String, Number, True/False). Read-only on GET; ignored on SET. |
IndexSizes | int[] | get/set | With pattern matching on, the sizes the field's text is indexed in, e.g.(the default) or; see.
Not empty, every size 1 to(64). Null on
GET for a field not matched by pattern. On SET null leaves the value as it is, and
a value for a field that stays without pattern matching is ignored, which the engine says
inand its log. Takes effect at the next. |
NumberOfStartCharacters | int? | get/set | With pattern matching on, the leading characters marked as the start of the
field's text (default 2, 0 for none); see. Null and ignored as. |
NumberOfStopCharacters | int? | get/set | With pattern matching on, the trailing characters marked as the end of the
field's text (default 0); see.
Null and ignored as. |
TextRules | TextRule[] | get/set | TextNormalizerThe field's own string rules, in their order, each before the engine's of its kind
(): a rule of more than one character before the
engine's of more than one, a rule of one before the engine's of one, so a field's rule
wins for the same pattern and the rules keep the shape their fast tables need. An engine
rule of more than one character therefore runs before a field's of one ("\r\n" to a space,
then a field's space rule). The engine's character folding is kept. A rule
withadds a form rather than replacing one: the field is indexed both
with it and without it, and the query is met in the form without it, as every other
field meets it.makes a code field (an ISBN, a part number) found however it
is written: "978-0-306-40615-7" is indexed as "9780306406157" and as the words of
"978 0 306 40615 7", so a search with or without the hyphens, or for a part, finds it.
Case normalization is done on the original as on the ruled form: both are lower-cased
unless the field is case-sensitive. The rules run after it, so a rule is written in
lower case:meets "ISBN"; in a case-sensitive field a rule meets only the
case it is written in. Read on GET as the rules the field has before the
engine's, empty for none. On SET a list gives the field these rules, an empty one the
engine's normalizer again; taken at once and in effect at the next Load. Null leaves it
as it is. A rule needs a non-emptyand a(empty deletes). |
Default | PatternMatchingSetup | get | The setup as shipped: sizes, two start characters, no stop
characters - what every field had before pattern matching became a choice. |
Methods
Clone
Creates a deep copy of this field. The clone is detached: no NotifyChanged subscribers are carried over. Backing fields for WordMatching/Sortable/Weight/PreloadFilters are assigned directly so the clone's NotifyChanged event is silent during construction.
Field Clone()Returns: Field
ValidateFieldValue
Validates that a value is compatible with the field's type and array requirements.
static bool ValidateFieldValue(Field field, object value, String& error)Fieldfield— The field definition to validate againstobjectvalue— The value to validateString&error— Output error message if validation fails
Returns: True if value is valid for the field, false otherwise
Dispose
void Dispose()GetFacetableFieldList
Returns the list of fields marked as Facetable.
List<Field> GetFacetableFieldList()Returns: List<Field>
GetField
Returns the field with the given name, or null if not found.
Field GetField(string fieldName)stringfieldName
Returns: Field
RequiresReindex
Returns true if the proposed configuration needsbefore it takes effect. Two quite different reasons, and both answer here so a caller has one question to ask:Filterable, Facetable, Sortable, Weight and PreloadFilters on a field that already has a role take effect at query time, and those still answer false.
bool RequiresReindex(FieldProxy[] proposed)FieldProxy[]proposed
Returns: bool
RequiresReload
Returns true when the proposal changes something a newhas to consume, socannot make it take effect however often it is called. Three things: whether a field is searched at all (or, either way; which of the two, once searched, is Index()'s),, and. All three are read inside Load and nowhere else.
bool RequiresReload(FieldProxy[] proposed)FieldProxy[]proposed
Returns: bool
GetFieldList
Returns a list of all fields.
List<Field> GetFieldList()Returns: List<Field>
GetFilterableFieldList
Returns all fields marked as filterable.
List<Field> GetFilterableFieldList()Returns: List<Field>
GetSerialized
Returns the instance serialized as a JSON string.
string GetSerialized()Returns: string
Haskey
Returns true when this instance has a key field of the expected type.
bool Haskey()Returns: bool
SaveToFile
Serializes the field configuration to a file.
void SaveToFile(string fileName)stringfileName
Analyze
Analyzes a JSON string and extracts its field structure, repairing control characters and trailing commas on the way.
static DocumentFields Analyze(string jsonDocument, String& errorMessage)stringjsonDocumentString&errorMessage
Returns: DocumentFields
AnalyzeAsync
String@)Async variant of.
static Task<ValueTuple<DocumentFields, string>> AnalyzeAsync(Stream jsonStream)StreamjsonStream
Returns: Task<ValueTuple<DocumentFields, string>>
ReadFromFile
Deserializes an instance from a previously saved file.
static DocumentFields ReadFromFile(string fileName)stringfileName
Returns: DocumentFields
Equals
By value, the sizes element by element (a record compares an array by reference).
bool Equals(PatternMatchingSetup other)PatternMatchingSetupother
Returns: bool
GetHashCode
int GetHashCode()Returns: int
ToString
string ToString()Returns: string
<Clone>$
PatternMatchingSetup <Clone>$()Returns: PatternMatchingSetup
ScoringModeenum
ScoringHow an engine scores its searchable fields. Chosen in the configuration the engine is built with () and fixed for its life:With one searchable field the two are the same index and score alike.
| Name | Value |
|---|---|
BM25PerField | 0 |
BM25F | 1 |