02 · REFERENCE

Config TOML

XERJ reads one TOML file. Every key has a production-ready default, so the smallest working config is an empty file. The table below is the full surface; the sections below it walk each group with a runnable example.

The config file is read from --config /path/to/xerj.toml on the command line, else from the path in XERJ_CONFIG. Nothing else is searched: with neither set, the built-in defaults are used even when ./xerj.toml or /etc/xerj/xerj.toml exists.

Full key table

KEY

TYPE

DEFAULT

DESCRIPTION

[server]

rest_port

u16

8080

Native REST API listener port. All native endpoints (/v1/*) live here.

grpc_port

u16

8081

gRPC API (xerj.v1.XerjSearch, the same engine calls as REST, same authorization). Cleartext h2c even when [tls] is on: off loopback with TLS enabled the node refuses to start unless tls.allow_insecure_grpc_h2c = true.

es_compat_port

u16

9200

Elasticsearch-compatible wire port. Point Kibana, Logstash, or the ES client here. 0 binds an OS-assigned port (logged at startup); it does not disable the listener.

bind_address

string

"127.0.0.1"

Interface to bind every listener to. Loopback by default, so an unconfigured node is unreachable from the network. Set "0.0.0.0" or a private address to expose it (also --bind / XERJ_BIND_ADDRESS). Must be an IPv4 or IPv6 literal — host names are not resolved, and a node given one refuses to start.

allow_insecure_network_bind

bool

false

Permit a non-loopback bind_address while tls.enabled = false. Startup refuses that combination otherwise: every listener would serve plain HTTP, so the API key in every Authorization header crosses the network in cleartext. Set true only when a proxy, sidecar, mesh or container boundary terminates TLS in front of the node. Env: XERJ_ALLOW_INSECURE_NETWORK_BIND.

data_dir

path

"./data"

Root for indices, WAL files, and segments. Needs fast I/O and enough free space.

[auth]

enabled

bool

true

Require an API key on every request. Set to false only on a trusted network.

admin_api_key

string

Static admin key. Left empty, a 256-bit key is generated on first run and written to <data_dir>/admin.key.

[tls]

enabled

bool

false

Terminate TLS at the server. Requires cert_path + key_path. If you terminate at a proxy, leave false.

cert_path

path

PEM X.509 certificate. Use a CA-signed cert in production.

key_path

path

PEM private key. Permissions should be 0600.

[storage]

wal_sync

enum

"batched"

"sync" · "batched" · "async". Durability vs throughput. Batched is the recommended default.

wal_batch_ms

u32

100

Fsync cadence when wal_sync="batched" (ms). Range 1–10000. Lower = smaller loss window.

wal_max_size_mb

u32

1024

WAL rollover threshold (MiB). Larger means fewer rollovers, longer crash recovery.

flush_size_mb

u32

512

Memtable buffer size that triggers a segment flush (MiB).

flush_interval_secs

u32

Maximum wall-clock interval between flushes even if the buffer is not full.

[merge]

strategy

enum

"size_tiered"

Only "size_tiered" is implemented; "log_structured" is refused at startup rather than silently running the other policy.

min_segments

u32

Accepted and validated (must be ≥ 2) but not wired: the merge trigger is per-tier and comes from min_merge_count. Setting it away from the default logs a warning at startup.

max_segment_mb

u32

8192

Upper bound on a mergeable segment (MiB). Segments at or above this size are never merged again.

io_rate_mb_per_sec

u32

100

Accepted but not wired: merge I/O is not throttled in this build — the rate limiter that honours it sits in an unused storage-crate merge executor. Setting it away from the default logs a warning at startup.

max_concurrent

Accepted but not wired: merge parallelism comes from the XERJ_MERGE_PARALLELISM environment variable, which also defaults to 1. Setting it away from the default logs a warning at startup.

[compression]

enabled

bool

true

Accepted but not wired: every durable artifact is a compressed envelope with no uncompressed write path, so false changes nothing on disk. Setting it away from the default logs a warning at startup.

level

enum

"balanced"

"fast" (Zstd L1), "balanced" (Zstd L3, default), "best" (Zstd L6, cold storage). Applied when segments are re-encoded at merge; flush always writes L3, because raising the flush level collapses sustained ingest. Force-merge to apply a change to data already on disk.

block_size_docs

u32

128

Accepted but not wired: the stored codec is columnar over the whole segment section, not blocked by document count. Range 16–4096 is enforced at startup, and a non-default value logs a warning.

[fts]

default_analyzer

enum

"standard"

"standard" · "whitespace" · "simple" · "english". Override per-field in the mapping. Not wired: nothing reads this key — a text field uses its own analyzer, else the index's analysis.analyzer.default, else standard. Setting it away from the default logs a warning at startup.

[vector]

default_metric

enum

"cosine"

"cosine" · "dot_product" · "euclidean". Cosine for text embeddings is the usual choice. Not wired: similarity comes from the field mapping, and a mapping without one gets cosine whatever this says. Setting it away from the default logs a warning at startup.

hnsw_m

u32

Accepted and validated; the built-in HNSW graph currently builds with fixed M=16 — this key is not yet wired to the build.

hnsw_ef_construction

u32

200

Accepted; must be ≥ hnsw_m. The graph currently builds with fixed ef_construction=200 — this key is not yet wired to the build.

hnsw_ef_search

u32

100

Accepted but not read by the serving path — the ANN beam width comes from the request's num_candidates (floored at 800).

default_quantization

enum

"none"

"none" (the default) or "scalar8" = kNN scored from 1-byte-per-dimension codes, 1–2% recall loss. scalar8 buys precision, not RAM: the scan reads the full-precision vector from _source and quantizes it per query, so resident memory is unchanged and the field also gives up HNSW-served kNN (#392). "binary" has no quantizer behind it and is refused at startup rather than silently stored at full precision; "scalar4" is not a value this key accepts at all (the 4-bit quantizer in the vector crate is unreachable from config or mapping). Not wired as a node-wide default: only the "binary" refusal reads it; quantization comes from each field's index_options in the mapping. Setting it away from the default logs a warning at startup.

max_dimensions

u32

16384

Upper bound on vector dimensionality. 4× the Elasticsearch limit of 4096. Not wired: validated (≥ 1), but no mapping or document is checked against it. Setting it away from the default logs a warning at startup.

[logs]

retention_days

u32

Accepted but not wired: no retention job runs and nothing deletes log data in this build — delete old indices or documents yourself. Setting it away from the default logs a warning at startup.

time_partition

enum

"1h"

"1m" · "5m" · "15m" · "1h" · "6h" · "1d". Time-slice granularity for retention pruning. Not wired: log indices are not partitioned by time. Setting it away from the default logs a warning at startup.

[embedding]

default_endpoint

string

OpenAI-compatible embeddings URL. Empty disables auto-embedding on ingest. Example: "https://api.openai.com/v1/embeddings".

default_model

string

Model name passed to the endpoint. Example: "text-embedding-3-small" or "nomic-embed-text".

batch_size

u32

Docs per embedding API call. Range 1–2048. Bigger batches amortise round-trip cost.

timeout_ms

u32

5000

HTTP timeout for each embedding request attempt (ms, at least 1). A timed-out attempt is retried up to 3 times with backoff before ingest fails with the error.

[limits]

max_query_memory_mb

u32

512

Per-query memory cap (MiB). Queries that exceed it are cancelled.

max_concurrent_searches

u32

Global in-flight search ceiling. Extras are queued.

max_fields_per_index

u32

500

Field-explosion protection. ES default is 1000; 500 is intentionally stricter.

max_result_window

u64

10000

Largest from + size a search may ask for, for any index whose settings carry no index.max_result_window (that setting wins when present, above or below this). 1–2147483647. Page past it with search_after or a point-in-time cursor.

[indexing]

turbo_batch_size

u32

1000

Docs per batch in turbo mode. Range 500–5000. Larger = higher throughput, slightly higher latency. Not wired: a turbo request is indexed as one batch. Setting it away from the default logs a warning at startup.

turbo_parallel

bool

true

Parallel tokenisation on Rayon threads. Disable only for debugging. Not wired: turbo ingest no longer pre-tokenises, so there is nothing to parallelise. Setting it away from the default logs a warning at startup.

turbo_fast_analyzer

bool

false

Skip stemming and stop-word removal in turbo mode. Trades recall for speed. Not wired: the turbo path ignores it; fields keep their analyzers. Setting it away from the default logs a warning at startup.

[cluster]

enabled

bool

false

Enable multi-node cluster mode. When true, the Raft state machine and cluster transport are started on port.

port

u16

9300

TCP port for intra-cluster Raft and search messages. Must be reachable from every peer.

peers

array

[]

Peer list, one "host:port=host:port" entry per node — the label must equal the address, or the node refuses to start (#1171). The local node identifies itself from the entry matching bind_address and cluster.port. Example: ["10.0.0.1:9300=10.0.0.1:9300","10.0.0.2:9300=10.0.0.2:9300","10.0.0.3:9300=10.0.0.3:9300"].

tick_ms

u64

Raft tick interval (ms). Lower = faster leader election at the cost of CPU.

auth_secret

string

Cluster-wide shared secret authenticating every control frame on port (HMAC-SHA256 over a per-connection challenge). Mandatory when enabled = true — with cluster mode on and no secret here or in XERJ_CLUSTER_AUTH_SECRET the node refuses to start. Same value on every node; minimum 16 chars; generate with `openssl rand -hex 32`. Authenticates, does not encrypt.

[wal_tap]

enabled

bool

false

Push a filtered subset of indices to an external ES-compatible target (Elasticsearch, OpenSearch, or another XERJ node) in near-real-time. One-directional and single-node — this is not cross-cluster replication. Off by default.

target_url

string

Base URL of the target cluster; the tap POSTs to {target_url}/_bulk. Example: "https://central:9200". Empty disables the tap even with enabled = true. Must not carry credentials in the URL (user:pass@host is refused at startup): target_url is echoed by GET /_xerj/wal_tap, by _stats and in the log. Put the credential in target_auth.

target_auth

string

Verbatim Authorization header for the target, e.g. "ApiKey abc123". Write-only: GET /_xerj/wal_tap reports whether one is set, never its value.

indices

array

[]

Index allowlist, glob patterns. Empty ships nothing; ["edge-*","metrics"] ships those. A wildcard never expands to a hidden index, and XERJ's own .xerj* system indices are never shipped whatever this says.

poll_interval_ms

u64

500

How often each allowlisted index's WAL is polled (ms). Range 50–60000. This is the floor on end-to-end latency; the tap adds nothing to the write path.

max_batch_docs

usize

1000

Maximum WAL entries read and shipped per poll, per index. Must be at least 1.

max_batch_bytes

usize

5242880

Maximum _bulk body size (bytes). Bounds one HTTP request against the target's http.max_content_length — a larger batch is split across requests, never truncated.

request_timeout_secs

u64

Per-request timeout against the target. A timed-out batch is retried; the cursor does not advance, so nothing is dropped.

max_retry_backoff_secs

u64

Ceiling on the exponential, jittered retry backoff when the target is down. Range 1–86400 (one day), enforced at startup as well as by PUT /_xerj/wal_tap.

min_retained_generations

u64

Rotated WAL generations kept per shard after every entry in them is durable in a segment, so a tap whose target is briefly unreachable still finds them. At the default 0, WAL retention never waits for the target: an outage longer than storage.flush_interval_secs loses entries and reports them as gaps in GET /_xerj/wal_tap/_stats. This is a bounded floor, not a retention lease — the extra cost is at most n × storage.wal_max_size_mb per WAL shard per index, whether or not a tap is running, so a stalled target can never fill this node's disk. Capped at 64 and range-checked at startup, not only by PUT /_xerj/wal_tap — the config file must not be a way around the bound.

[rerank]

enabled

bool

true

Allow the rerank search stage. It is the only search-time feature that sends document text off this node, and it does nothing until a key is set. false refuses every rerank request with HTTP 403, whatever the environment holds, and nothing is sent.

api_key

string

Rerank provider key. Empty falls back to the TYPESAFE_API_KEY environment variable; a value here wins over the environment. Empty in both places means reranking is not configured and a rerank request gets HTTP 503. Write-only: GET /_xerj/rerank reports whether a key is set and where it came from, never its value. There is no per-request key.

endpoint

string

Where judged document text is POSTed. Empty falls back to TYPESAFE_ENDPOINT, then to https://api.typesafe.ai/v1/systemone. Must be an absolute http:// or https:// URL with no credentials in it (user:pass@host is refused at startup, whether it came from this file or from TYPESAFE_ENDPOINT): the endpoint is echoed by GET /_xerj/rerank.

[server]

Network listeners and the data directory. Most deployments only touch data_dir and the bind address.

[server]
rest_port      = 8080            # native /v1/* API
es_compat_port = 9200            # ES wire-compatible API
grpc_port      = 8081            # gRPC (xerj.v1.XerjSearch), plaintext h2c
bind_address   = "0.0.0.0"       # not a default — the default is "127.0.0.1" (loopback only)
allow_insecure_network_bind = true   # not a default (false) — required for a non-loopback bind while TLS is off
data_dir       = "/var/lib/xerj" # not a default ("./data") — an absolute path is strongly recommended

The default bind is loopback, so an unconfigured node is not reachable from the network. Exposing it while tls.enabled = false refuses to start unless allow_insecure_network_bind says the cleartext exposure is intended — enable TLS and neither line is needed.

[auth]

Static API-key authentication. The first-run admin key is written to <data_dir>/admin.key; subsequent starts reuse it. Clients pass Authorization: ApiKey <key> on every request.

[auth]
enabled       = true
admin_api_key = ""               # blank → auto-generated on first run

# Or provide your own:
# admin_api_key = "ak_live_c8f9a4…"

[tls]

TLS termination at the server. In Kubernetes or behind a load balancer, leave this off and terminate at the proxy instead — one place to rotate certs, one place to log handshakes.

[tls]
enabled   = true                         # not a default (false) — this block switches TLS on
cert_path = "/etc/xerj/certs/server.crt" # not a default ("") — your certificate
key_path  = "/etc/xerj/certs/server.key" # not a default ("") — your private key

[storage]

The WAL and flush tuning. wal_sync is the durability knob everyone looks for — pick "sync" for financial/compliance workloads, "batched" for everything else, and "async" only in benchmarks.

[storage]
wal_sync            = "batched"
wal_batch_ms        = 100        # fsync every 100 ms
wal_max_size_mb     = 1024       # roll WAL every 1 GiB
flush_size_mb       = 512        # flush memtable at 512 MiB
flush_interval_secs = 30

[merge]

Segment compaction. size_tiered is the only implemented policy: it merges same-size segments, which is cheap and write-optimal. log_structured is a name in the enum with no merge policy behind it, so the server refuses to start on it rather than silently running size-tiered merging for an operator who picked a levelled policy for its read amplification. If full-range scans are what hurts, that is a real gap in this build — setting the key will not change it.

[merge]
strategy           = "size_tiered" # the only one implemented — log_structured is refused at startup
min_segments       = 10          # accepted, not wired
max_segment_mb     = 8192        # 8 GiB cap
io_rate_mb_per_sec = 100         # accepted, not wired — merges are not throttled
max_concurrent     = 1           # accepted, not wired (see XERJ_MERGE_PARALLELISM)
tier_floor_mb      = 4
min_merge_count    = 4           # the real per-tier merge trigger
max_merge_count    = 16          # caps peak merge RAM

[compression]

See Compression for the encoding catalog. This section picks the outer block codec effort only — the inner per-column encodings are chosen automatically at write time.

[compression]
enabled         = true           # accepted, not wired — no uncompressed write path exists
level           = "balanced"     # Zstd L1 / L3 / L6, applied at MERGE (flush is always L3)
block_size_docs = 128            # accepted, not wired — 16-4096, range-checked at startup

[fts]

This key is accepted but not wired: a text field without an analyzer of its own uses the index's analysis.analyzer.default when the index declares one, and standard otherwise. Set the analyzer per field or per index in the mapping when creating an index. See Analyzers for the built-ins.

[fts]
default_analyzer = "standard"    # accepted, not wired — see the table

[vector]

Vector search settings. Unfiltered kNN is served by a persisted HNSW graph with exact rescoring (measured recall@10 1.00 on the official bench query); filtered kNN and SQ8 fields run the exact scan. The hnsw_* keys are accepted and validated but not yet wired: the graph builds with fixed M=16 / ef_construction=200, and the query-time beam width comes from the request's num_candidates (floored at 800). Quantization trades recall for scoring precision, not for memory — see default_quantization below and #392.

[vector]
default_metric         = "cosine"    # accepted, not wired — similarity comes from the mapping
hnsw_m                 = 16          # accepted, not wired — the graph builds with M=16
hnsw_ef_construction   = 200         # accepted for compat; no effect on results
hnsw_ef_search         = 100         # accepted for compat; no effect on results
default_quantization   = "none"      # "scalar8" = int8 precision, 1–2% recall loss.
                                     # NOT a RAM saving today — see issue #392
max_dimensions         = 16384       # accepted, not wired — nothing checks dims against it

[logs]

Time-series retention is not implemented in this build: both keys are accepted and nothing reads them, no retention job runs, and log data is kept until you delete it. Setting either away from its default logs a warning at startup.

[logs]
retention_days = 90              # accepted, not wired — nothing deletes log data
time_partition = "1h"            # accepted, not wired

[embedding]

Delegates vector generation to an OpenAI-compatible endpoint. Leave default_endpoint empty if clients provide vectors themselves. Token limits are model-specific; the chunker in the ai crate splits long documents to fit the model's window.

[embedding]
# OpenAI:
default_endpoint = "https://api.openai.com/v1/embeddings"  # not a default ("" disables auto-embedding)
default_model    = "text-embedding-3-small"                # not a default ("")
batch_size       = 64
timeout_ms       = 5000

# Or a local Ollama:
# default_endpoint = "http://localhost:11434/v1/embeddings"
# default_model    = "nomic-embed-text"

[limits]

Hard caps to protect the server from runaway queries and mapping explosions. Lower these on shared nodes, raise max_query_memory_mb for aggregation-heavy workloads. Leave max_process_memory_mb out to auto-size the process-wide cap from the machine (8 / 16 / 32 GiB tiers); set an explicit MiB ceiling only to raise headroom on a large host — for example if a big xerj autoindex trips 429 circuit_breaking_exception.

[limits]
max_query_memory_mb     = 512
max_concurrent_searches = 64
max_fields_per_index    = 500
# max_process_memory_mb = 0   # omit → AUTO (8/16/32 GiB by machine); 0 = whole machine

[indexing]

Turbo mode knobs. Turbo is opt-in per-request via POST /v1/indices/:name/turbo-ingest or the X-Turbo: true header on _bulk; none of the three settings below changes anything in this build (each logs a warning at startup when set away from its default).

[indexing]
turbo_batch_size    = 1000       # accepted, not wired
turbo_parallel      = true       # accepted, not wired
turbo_fast_analyzer = false      # accepted, not wired

[cluster]

Multi-node mode. Default is off — single-node doesn't need a consensus layer. When enabled, the embedded Raft implementation replicates metadata only (schemas, shard assignments, node roster). See Clustering for the full story.

[cluster]
enabled = true                   # not a default (false) — this block switches cluster mode on
port    = 9300                   # intra-cluster gRPC + Raft
peers   = [                      # not a default ([]) — your node roster; label = address (#1171)
  "10.0.0.11:9300=10.0.0.11:9300",
  "10.0.0.12:9300=10.0.0.12:9300",
  "10.0.0.13:9300=10.0.0.13:9300",
]
tick_ms = 50
auth_secret = "<secret>"         # not a default ("") — required when enabled, 16+ chars, same on every node

[rerank]

This feature sends your data off the machine. A search that carries a rerank block POSTs the question and the text of up to window hits (default 30, maximum 300) to a third-party relevance judge, and reorders the hits by the probability that comes back. It is the only search-time feature that sends document text off the node. Two other features send text off the node, both operator configuration and off by default: [embedding] default_endpoint (proxy embeddings) and the WAL tap. The neural model download and cluster Raft traffic carry no document text. Nothing is sent until an operator sets a key, a request only triggers it by asking for it, and enabled = false forbids it outright. Settings are resolved once at startup; a value in this file wins over the environment. Ranking quality with the provider's model has not been verified by this project. Full reference: docs/RERANK.md.

[rerank]
enabled  = true
api_key  = "ts-..."                                  # not a default ("") — or set TYPESAFE_API_KEY
endpoint = "https://api.typesafe.ai/v1/systemone"    # not a default ("") — this is where "" resolves to

Check it without printing the key: GET /_xerj/rerank (superuser only) returns configured, whether a key is set and whether it came from config or env, the endpoint with any query string removed, and the server-side limits.

Source · engine/xerj.default.toml · engine/crates/xerj-common/src/config.rs

◀ PREVCLI

NEXT ▶Environment