Field

Fields are discovered automatically via Init() and then configured before Load(). The engine is schema-less — you never declare fields up front.

csharp
engine.Init(fstream);

var name = engine.GetField("name")!;
name.WordMatching = true;                              // matched by its whole words; counts
name.PatternMatching = PatternMatchingSetup.Default;   // and inside its words
name.Weight = 2.0f;
engine.GetField("description")!.WordMatching = true;
engine.GetField("description")!.PatternMatching = PatternMatchingSetup.Default;
engine.GetField("category")!.Filterable = true;
engine.GetField("category")!.Facetable = true;
engine.GetField("price")!.Filterable = true;
engine.GetField("price")!.Sortable = true;

A configuration from Init() searches nothing. Every field starts with neither WordMatching nor PatternMatching; give at least one field WordMatching, or no search narrows.

Configuration Properties

PropertyTypeDefaultDescription
WordMatching
bool
false
Match the field by its whole words, exactly and within the edits its Tolerance allows. Its words go in the word lookup, so the field counts: a search narrows to the records holding every query word. Every word from TokenizerSetup.MinimumWordSize (2) is indexed
Filterable
bool
false
Allow this field to be used in filters and boosts. Refused with an ArgumentException on a field with no type
Facetable
bool
false
Return value/count pairs for this field with search results. On a number field it makes the field Filterable too (a range over the facet, a slider's, is a range filter); set both, and if Filterable is forgotten it is set. Not the other way
Sortable
bool
false
Allow sorting by this field. Score is always the primary sort key. Refused with an ArgumentException on a field with no type
PatternMatching
PatternMatchingSetup?
null
Match the field inside its words: a query finds it by any part of a word. The setup's IndexSizes (default [3], each 1 to 64), NumberOfStartCharacters (2) and NumberOfStopCharacters (0) say how; new PatternMatchingSetup([2, 3]) matches short values (tags, codes) by two characters, and a stop character anchors the end. Alone, without WordMatching, the field orders records but cannot narrow them. WordMatching with PatternMatchingSetup.Default is how every searchable field was matched in 5.0. Changing the setup of a searched field takes effect at the next Index()
Tolerance
Tolerance
OneEdit
The most edits the field allows a query word: Exact, OneEdit, TwoEdits, ThreeEdits. Capped by the search's Match.Tolerance
RerankWeight
float
1.0f
The field's weight in the reranker. 0 leaves it out of the reranker and the word lookup; it still counts through BM25F
TypingPrefixFrom
int?
null
From how many letters the word being typed matches the start of a longer word in this field. null takes the search's Match.TypingPrefixFrom
CaseSensitive
bool
the engine's
Whether the field is matched with case. Defaults to the engine's IndexerSetup.DoCaseSensitiveSearching. Set at any time (or with FieldProxy.CaseSensitive); it takes effect at the next Load, and DocumentFields.RequiresReload says when a change waits for one. Saved with the field configuration
TextNormalizer
TextNormalizer
the engine's
The string rules and character folding the field's text and the query against it go through: the engine's ConfigurationParameters.TextNormalizer unless the field is given its own (an ISBN or code field that deletes - and spaces, say). Set at any time; it takes effect at the next Load. Saved with the field configuration, as its differences from the shipped tables
Embeddable
bool
false
Mark this field for embedding-based (vector) search. Embeddable fields are excluded from the text search index
PreloadFilters
bool
false
Cache a filter for each distinct value during Load. Accessible via SearchEngine.PreLoadedFilters. Faster repeated filtering at the cost of memory
Weight
float
1.0f
Relative score weight for this field. Higher values make matches in this field rank higher
BM25b
float
0.75f
Length normalisation parameter. 0 = no normalisation, 1 = full normalisation. Applies in both BM25F and BM25PerField modes
BM25k1
float
1.2f
Term-frequency saturation, read only when the engine scores with ScoringMode.BM25PerField. Typical range 1.0–2.0. In BM25F, the default, k1 is the engine's (ScoringSetup.BM25FK1): a value set here is ignored and Status.FieldK1Ignored says so, and TrySetFieldConfiguration refuses a changed one

Read-Only Properties

PropertyTypeDescription
Name
string
Field name as it appears in the JSON data
Type
JsonValueKind
Value kind: String, Number, True/False, Array, Object
IsArray
bool
Whether this field holds an array of values
Optional
bool
true if the field is absent or null in at least one document
EmbeddingDimensions
int
Detected vector dimension. 0 until data is loaded with Embeddable = true

Setting PreloadFilters = true automatically sets Filterable = true on the same field.

Fields With No Type

A field that was null or absent in every document Init() read has no type. Filterable and Sortable are the only two roles that need one, to choose between a numeric and a categorical index and to pick a comparator, so both are refused on such a field: TrySetFieldConfiguration returns false with the reason, and the setters on Field throw ArgumentException. Clearing either is always allowed.

WordMatching, PatternMatching and Facetable do not read the type and stay available. Setting WordMatching on an empty field prepares it: a record inserted later that does carry a value is indexed and found.

Changing Configuration After Load

A change needs one of three things, and you can ask which before applying it:

csharp
var proposed = new[] { new FieldProxy { FieldName = "category", Facetable = true } };

if (engine.DocumentFields.RequiresReload(proposed))
{
    // Whether a field is searched at all (WordMatching or PatternMatching turned on or
    // off), Embeddable and PreloadFilters. Only Load consumes these. Applied in place they return the engine to Created,
    // so searches fail with a reason instead of answering as if the field were empty.
}
else if (engine.DocumentFields.RequiresReindex(proposed))
{
    engine.TrySetFieldConfiguration(proposed, out string error);
    engine.Index();
}
else
{
    engine.TrySetFieldConfiguration(proposed, out string error);   // takes effect at query time
}

RequiresReindex covers which way a searched field is matched, a changed pattern setup, and a field getting its first role. A document keeps values only for the fields in use when it was loaded, so such a field has none until Index() reads them back from the stored JSON. A second role on a field that already had one takes effect at query time.

To apply a reload-only change without taking the engine out of service, build the new index beside the live one with CreateInMemoryClone(monitor, fieldOverrides) and swap it in.


FieldProxy

FieldProxy is used by GetFieldConfiguration() (returns FieldProxy[]) and TrySetFieldConfiguration(FieldProxy[], out string error). All boolean and float properties are nullable — null on a set call means "leave unchanged"; every property is populated on a get call.

csharp
// Read the full configuration
FieldProxy[] config = engine.GetFieldConfiguration();

// Apply changes — only non-null values are written
engine.TrySetFieldConfiguration([
    new FieldProxy { FieldName = "title",       WordMatching = true, PatternMatching = true, Weight = 2.0f },
    new FieldProxy { FieldName = "description", WordMatching = true, PatternMatching = true },
    new FieldProxy { FieldName = "sku",         WordMatching = true, Tolerance = Tolerance.Exact },
    new FieldProxy { FieldName = "category",    Filterable = true, Facetable = true },
    new FieldProxy { FieldName = "price",       Filterable = true, Sortable = true },
], out string error);
PropertyTypeDescription
FieldName
string
Must match a field name discovered by Init()
FieldType
string?
Read-only on GET ("String", "Number", "Boolean"). Ignored on SET
IsArray
bool?
Read-only on GET. Ignored on SET
WordMatching
bool?
Matched by its whole words; counts
Filterable
bool?
Allow use in filters and boosts
Facetable
bool?
Return value/count pairs with results
Sortable
bool?
Allow sorting by this field
PatternMatching
bool?
On (the field's setup, or PatternMatchingSetup.Default) or off
IndexSizes
int[]?
With pattern matching on, the setup's index sizes. Sent for a field that stays off, ignored with a warning in Status.ErrorMessage
NumberOfStartCharacters
int?
With pattern matching on, the setup's start characters; as IndexSizes otherwise
NumberOfStopCharacters
int?
With pattern matching on, the setup's stop characters; as IndexSizes otherwise
Embeddable
bool?
Vector embedding field
PreloadFilters
bool?
Cache a filter per distinct value during Load. Also forces Filterable = true
Weight
float?
Score weight multiplier
BM25b
float?
Length normalisation parameter
BM25k1
float?
TF saturation parameter (BM25PerField only)
Tolerance
Tolerance?
The most edits the field allows
RerankWeight
float?
Weight in the reranker; 0 leaves the field out of it
TypingPrefixFrom
int?
When the word being typed matches by its beginning
CaseSensitive
bool?
Match with case; in effect at the next Load
TextRules
TextRule[]?
The field's own string rules before the engine's (new TextRule("-", "")); [] the engine's again; in effect at the next Load

A FieldProxy from a 5.0 client or file may carry Searchable: it is read as WordMatching alone, so such a field loses pattern matching. WordIndexing and HighResolution bind to nothing.


Scoring Modes

The scoring mode is part of the engine's configuration, chosen when the engine is built. It decides where k1 is set:

csharp
// BM25F (default): one index over all searched fields, one k1 for all of them
var bm25f = new SearchEngine(string.Empty, null,
    ConfigurationParameters.Default.With(scoringSetup: new ScoringSetup(ScoringMode.BM25F, bm25fK1: 1.2f)));

// BM25PerField: one index per searched field, each with its own k1 and b, merged by Weight
var perField = new SearchEngine(string.Empty, null,
    ConfigurationParameters.Default.With(scoringSetup: new ScoringSetup(ScoringMode.BM25PerField)));
perField.GetField("title")!.BM25k1 = 2.0f;   // read in this mode only

engine.Scoring (and engine.ScoringMode) read the choice back. Until 6.0 the mode was inferred from whether BM25k1 differed across the searchable fields; it no longer is.

Field Weights

Weight is a plain float. The practical range is 0.5–3.0. The ratio between fields matters — 2.0 and 1.0 produces the same relative effect as 4.0 and 2.0.

csharp
engine.GetField("title")!.Weight    = 3.0f;
engine.GetField("heading")!.Weight  = 2.0f;
engine.GetField("metadata")!.Weight = 1.5f;
engine.GetField("body")!.Weight     = 1.0f;

There is no hard maximum, but values above 5× the lowest weight in your set offer diminishing returns. If one field needs to dominate that strongly, it is usually better to make it the only searched field.

Constructors

csharp
Field()

Parameterless constructor for deserialization.

csharp
Field(string name, JsonValueKind type, object value, bool isArray)
  • string name
  • JsonValueKind type
  • object value
  • bool isArray
csharp
DocumentFields()

Default ctor.

csharp
FieldProxy()
csharp
PatternMatchingSetup()

DefaultThe setup as shipped:'s values.

csharp
PatternMatchingSetup(int[] indexSizes, int numberOfStartCharacters = 2, int numberOfStopCharacters = 0)
  • int[] indexSizes
  • int numberOfStartCharacters — Default: 2
  • int numberOfStopCharacters — Default: 0

Properties

NameTypeAccessDescription
CaseSensitive
bool
get/set
Whether this field's text is searched with its case: the index keeps the case of the field's text and the query is matched as typed. False as shipped, the engine's, given to every field at Init. Set at any time; it takes effect at the next Load, which the engine reads the field with until then (says when a change waits for one). Saved with the field configuration; a configuration saved without it takes the engine's default.
TextNormalizer
TextNormalizer
get/set
The normalizer this field's text and the query against it go through (the string rules, the character folding): the engine'sunless the field is given its own. Set at any time; it takes effect at the next Load, asdoes. Saved with the field configuration, as its exceptions and additions over the shipped tables, the way a configuration file saves the engine's.
BM25b
float
get/set
Per-field length-normalization parameter b for BM25 / BM25F scoring. Range [0, 1]: 0 = no length normalization, 1 = full normalization. Typical values: 0.3 for short fields (title), 0.75 for longer fields (body). Defaults to 0.75. Applies in both(per-field BM25) and(multi-field BM25F).
BM25k1
float
get/set
BM25PerFieldThe field's term-frequency saturation k1, read only when the engine scores with. Typical range 1.0–2.0; lower values saturate repeated terms sooner. Defaults to 1.2. Takes effect at the next.
PatternMatching
PatternMatchingSetup
get/set
How the field is matched by pattern, inside its words (: the index sizes and the start and stop characters), or null, the default: not by pattern. One of the two ways a field is searched, with; a field with either is searched, a field with neither is not. Takes effect at the next, or at the nextwhen it is what makes the field searched or not ().
Embeddable
bool
get/set
When true, this field holds a vector embedding — a fixed-length array of float values produced by an embedding model. Fields must be set embeddable after Init during the fieldconfiguration. Embeddable fields are excluded from the standard text search index. During Load their float[] values are collected into a per-field list that is passed to the HNSW index for approximate nearest-neighbour search. The expected dimensionality is fixed on the first document loaded and validated on all subsequent ones. Setting this to false on a field whose name matches the auto-detection pattern overrides the automatic classification.
EmbeddingDimensions
int
get/set
The expected number of dimensions for this embedding field. Set automatically from the first document loaded during Load. Zero means not yet determined. Ignored when Embeddable is false.
Facetable
bool
get/set
Set to true if this field is going to be used as a facetable field.
Filterable
bool
get/set
Set to true if this field is going to be used as a filterable field. Refused on a field with no type (null in every analyzed document).
WordMatching
bool
get/set
Whether the field is matched by its whole words: exactly, within the edits Coverage allows, by prefix and affix, through the word lookup. One of the two ways a field is searched, with; a field with either is searched, a field with neither is not. With both, the field is matched inside its words and by them. Takes effect at the next, or at the nextwhen it is what makes the field searched or not ().
Sortable
bool
get/set
Must be set in order to sort on this field. This will be a secondary search key – score is always first. Both this field and the sort order must be passed to Query.
IsArray
bool
get/set
Is true for a possibly flattened array.
Name
string
get/set
The name of the field. The name must be unique.
Optional
bool
get/set
Optional field
SampleValue
string
get/set
A short, display-ready sample of this field's content, captured from the first document that carried a non-empty value for it. Surfaced asso a caller configuring the field can see what it actually holds.
PreloadFilters
bool
get/set
Setting this will make engine setup a valuefilter for each distinct value of a field. The filters will be created and loaded during Load and may be accessed via SearchEngine.GetPreloadedFilters.
Type
JsonValueKind
get/set
Only primitive types and their arrays are accepted. Note that for fields containing flattened arrays they will all have primitive types, and not JsonArray. Use the IsArray property to check.
Value
object
get/set
Actual value of the field.
Weight
float
get/set
BM25PerFieldHow much this field contributes to the relevance score, relative to the other searchable fields —in the BM25F formula, and the weight the per-field scores are averaged by in. Defaults to 1.0. Typical range: 1.0–3.0, where 1.0 is neutral and higher values increase the field's contribution. Values below 1.0 de-emphasize the field. Applies only to searchable fields; set it after Init() and before Index().
Tolerance
Tolerance
get/set
The most a query word may differ by in this field, in the word lookup and in Coverage:,(the default, as Coverage has always worked),or(long words). A search'scaps it, never raises it. Body text is usually best at: a misspelling in a long description confirms the wrong document. Read at every search; no Index() needed. Applies only to searchable fields. Until 6.0 CoverageStrictness (Ldd0, Ldd1, Ldd2); its None is aof 0.
TypingPrefixFrom
int?
get/set
From how many characters the word being typed meets the longer words it starts in this field, in the word lookup: null (the default) as the search says (), 0 never, a number never before that many characters. It caps the search's value, never lowers it: the larger of the two counts, and 0 in either is never. A field of codes (an ISBN, an article number) sets 0, found by the whole code only, or a length such as 5, so that three typed digits do not count every code they begin; withExact the field meets only the word as written. The reranker still credits a word met as a start in the field. Read at every search; no Index() needed. Applies only to fields matched by their words.
AverageLength
float
get/set
LongFieldFromThe field's mean text length in characters over the documents holding it, worked out by(0 before). Coverage reads it for: a field whose text is long on average counts less there. Inserts and updates afterleave it as it was until the next. Not saved with the field configuration: it is the data's, not a setting.
RerankWeight
float
get/set
WeightThe field's weight in the reranker (Coverage; until 6.0 CoverageWeight), against the heaviest searchable field in it: a query word counts with its characters times this weight over the heaviest's, so a document whose words are only in a field of half the heaviest's weight is at most half covered. Defaults to 1; 0 leaves the field out of Coverage and the word lookup: it confirms no word, and counts through BM25F only (until 6.0 CoverageStrictness.None). Not, which is BM25F's and acts inside its saturation: Coverage asks whether a field names what was typed, and the fields that do (title, author, ISBN) should weigh alike here whatever BM25F gives them; lower it for body text. Read at every search; no Index() needed.
NameOfDocumentKeyField
string
get/set
Name of the key field used to identify documents. Required for operations such as document deletion.
TextSettingsWaitForLoad
bool
get
Whether a field'sor, set since the last Load, waits for the next: the engine searches with what that Load took until then.
AnalyzedDocumentCount
int
get
Number of documents the analyze pass (Init) saw in the stream. A value of 1 on a large file is the classic "export envelope" symptom: a root object wrapping the real document array, which the parser reads as one giant document. Callers can use this to warn before Load/Index is attempted.
FieldName
string
get/set
Name of the field. Must match an existing field discovered by Analyze.
FieldType
string
get/set
JSON value kind of the field (e.g. String, Number, True/False). Read-only on GET; ignored on SET.
IndexSizes
int[]
get/set
With pattern matching on, the sizes the field's text is indexed in, e.g.(the default) or; see. Not empty, every size 1 to(64). Null on GET for a field not matched by pattern. On SET null leaves the value as it is, and a value for a field that stays without pattern matching is ignored, which the engine says inand its log. Takes effect at the next.
NumberOfStartCharacters
int?
get/set
With pattern matching on, the leading characters marked as the start of the field's text (default 2, 0 for none); see. Null and ignored as.
NumberOfStopCharacters
int?
get/set
With pattern matching on, the trailing characters marked as the end of the field's text (default 0); see. Null and ignored as.
TextRules
TextRule[]
get/set
TextNormalizerThe field's own string rules, in their order, each before the engine's of its kind (): a rule of more than one character before the engine's of more than one, a rule of one before the engine's of one, so a field's rule wins for the same pattern and the rules keep the shape their fast tables need. An engine rule of more than one character therefore runs before a field's of one ("\r\n" to a space, then a field's space rule). The engine's character folding is kept. A rule withadds a form rather than replacing one: the field is indexed both with it and without it, and the query is met in the form without it, as every other field meets it.makes a code field (an ISBN, a part number) found however it is written: "978-0-306-40615-7" is indexed as "9780306406157" and as the words of "978 0 306 40615 7", so a search with or without the hyphens, or for a part, finds it. Case normalization is done on the original as on the ruled form: both are lower-cased unless the field is case-sensitive. The rules run after it, so a rule is written in lower case:meets "ISBN"; in a case-sensitive field a rule meets only the case it is written in. Read on GET as the rules the field has before the engine's, empty for none. On SET a list gives the field these rules, an empty one the engine's normalizer again; taken at once and in effect at the next Load. Null leaves it as it is. A rule needs a non-emptyand a(empty deletes).
Default
PatternMatchingSetup
get
The setup as shipped: sizes, two start characters, no stop characters - what every field had before pattern matching became a choice.

Methods

Clone

Creates a deep copy of this field. The clone is detached: no NotifyChanged subscribers are carried over. Backing fields for WordMatching/Sortable/Weight/PreloadFilters are assigned directly so the clone's NotifyChanged event is silent during construction.

csharp
Field Clone()

Returns: Field

ValidateFieldValue

Validates that a value is compatible with the field's type and array requirements.

csharp
static bool ValidateFieldValue(Field field, object value, String& error)
  • Field field — The field definition to validate against
  • object value — The value to validate
  • String& error — Output error message if validation fails

Returns: True if value is valid for the field, false otherwise

Dispose

csharp
void Dispose()

GetFacetableFieldList

Returns the list of fields marked as Facetable.

csharp
List<Field> GetFacetableFieldList()

Returns: List<Field>

GetField

Returns the field with the given name, or null if not found.

csharp
Field GetField(string fieldName)
  • string fieldName

Returns: Field

RequiresReindex

Returns true if the proposed configuration needsbefore it takes effect. Two quite different reasons, and both answer here so a caller has one question to ask:Filterable, Facetable, Sortable, Weight and PreloadFilters on a field that already has a role take effect at query time, and those still answer false.

csharp
bool RequiresReindex(FieldProxy[] proposed)
  • FieldProxy[] proposed

Returns: bool

RequiresReload

Returns true when the proposal changes something a newhas to consume, socannot make it take effect however often it is called. Three things: whether a field is searched at all (or, either way; which of the two, once searched, is Index()'s),, and. All three are read inside Load and nowhere else.

csharp
bool RequiresReload(FieldProxy[] proposed)
  • FieldProxy[] proposed

Returns: bool

GetFieldList

Returns a list of all fields.

csharp
List<Field> GetFieldList()

Returns: List<Field>

GetFilterableFieldList

Returns all fields marked as filterable.

csharp
List<Field> GetFilterableFieldList()

Returns: List<Field>

GetSerialized

Returns the instance serialized as a JSON string.

csharp
string GetSerialized()

Returns: string

Haskey

Returns true when this instance has a key field of the expected type.

csharp
bool Haskey()

Returns: bool

SaveToFile

Serializes the field configuration to a file.

csharp
void SaveToFile(string fileName)
  • string fileName

Analyze

Analyzes a JSON string and extracts its field structure, repairing control characters and trailing commas on the way.

csharp
static DocumentFields Analyze(string jsonDocument, String& errorMessage)
  • string jsonDocument
  • String& errorMessage

Returns: DocumentFields

AnalyzeAsync

String@)Async variant of.

csharp
static Task<ValueTuple<DocumentFields, string>> AnalyzeAsync(Stream jsonStream)
  • Stream jsonStream

Returns: Task<ValueTuple<DocumentFields, string>>

ReadFromFile

Deserializes an instance from a previously saved file.

csharp
static DocumentFields ReadFromFile(string fileName)
  • string fileName

Returns: DocumentFields

Equals

By value, the sizes element by element (a record compares an array by reference).

csharp
bool Equals(PatternMatchingSetup other)
  • PatternMatchingSetup other

Returns: bool

GetHashCode

csharp
int GetHashCode()

Returns: int

ToString

csharp
string ToString()

Returns: string

<Clone>$

csharp
PatternMatchingSetup <Clone>$()

Returns: PatternMatchingSetup

ScoringModeenum

ScoringHow an engine scores its searchable fields. Chosen in the configuration the engine is built with () and fixed for its life:With one searchable field the two are the same index and score alike.

NameValue
BM25PerField
0
BM25F
1