Does XERJ support S3 storage, alerting or custom plugins yet?

TL;DR — The scoreboard moved twice since this page was written: an index can live in an S3-compatible bucket since v1.0.0-rc.77, record corpus packs shipped in rc.78, and alerting became PARTIAL in rc.80 (_watcher evaluates exactly one watch shape and refuses every other loudly). User-supplied ingest plugins remain built-in-only. This page says exactly what exists, with the file that proves it. The authoritative list is ROADMAP.md.

Why this page exists

A search engine that speaks a familiar wire protocol invites assumptions. An agent that sees a _watcher route may plan around full Elasticsearch watcher semantics — cron schedules, transforms, webhooks — and it should not: since v1.0.0-rc.80 XERJ evaluates exactly one watch shape and refuses the rest. An operator who sees backend = "s3" in a config schema may expect a bucket to work — since v1.0.0-rc.77 it does, and this page is the one place that used to say otherwise.

XERJ's rule is that an input is either honoured or refused loudly. _watcher was the one surface that broke the rule (accepted a watch, never evaluated it); v1.0.0-rc.80 closed that — #1082, closing #1062. All statuses on this page were checked against main by reading the named file: 2026-09-18 originally, S3 re-verified 2026-09-26 against v1.0.0-rc.77, alerting and packs re-verified 2026-09-30 against v1.0.0-rc.80.

S3 and object storage: both directions work

This section said the opposite until 2026-09-26 — it described S3Backend as a local-directory simulation with no network client, and the startup refusal as permanent. That was true when it was checked (2026-09-18, before rc.77) and stopped being true in v1.0.0-rc.77. The correction is stated in the open rather than quietly rewritten.

Reading documents out of a bucket works. xerj autoindex s3://bucket/prefix lists the prefix, streams in each object whose ETag or size changed, and indexes it with the same extractors it uses for a folder. r2:// and any S3-compatible store behind --endpoint-url work the same way. The S3 indexing page has the commands and the request arithmetic.

Storing the index in a bucket works, since v1.0.0-rc.77 (#1008, closing #965). Set storage.backend = "s3" with an existing storage.s3_bucket, and:

• the segment path packs each segment family into one immutable ZBM1 bundle object — not one PUT per file (the layout it replaced would have issued ~104 PUTs per segment);
• a per-index snapshot.json catalogue is the publication point: merges publish their outputs before retiring their inputs, and a fresh node adopts the bucket by reading it;
• the client is a real S3-compatible client over aws-sdk-s3 — Cloudflare R2, MinIO and AWS S3 — in engine/crates/xerj-storage/src/s3.rs. The local-directory simulation survives only as a test double;
• the read-through segment cache, per-request cost accounting by billing class, and the budget that stops rather than warns landed in v1.0.0-rc.75;
• the WAL stays local. docs/OBJECT_STORAGE.md is the full design page.

One refusal remains, and it is the loud kind: with storage.backend = "s3" and no storage.s3_bucket, the server does not start. It prints that storage.backend = "s3" requires storage.s3_bucket to name an existing bucket — XERJ never creates one. That check is in engine/crates/xerj-common/src/config.rs.

Alerting: one real watch shape since v1.0.0-rc.80

This section said "there is none" until 2026-09-30 — and that was true when it was checked (2026-09-18): PUT /_watcher/watch/{id} accepted every body, replied that the condition was met, and no code ever evaluated anything. v1.0.0-rc.80 (#1082) replaced the accepted-and-ignored surface with evaluate-or-refuse.

What evaluates. Exactly one watch shape: trigger.schedule.interval (>= 1s), input.search over exactly one index, condition.xerj_decide (a question, a positive label, p_min, k, an optional decide index) and an index_alert action. A per-watch background task prefilters candidates with the watch's own query (ascending _seq_no keyset paging, 100 per page), runs each rendered question through the real /_decide, and writes a fire record into .xerj_alert_fires when the positive label wins non-abstaining at or above p_min. Fire records carry the RAW decide confidence, plus p_cal with calibrated: true when the node has a fitted calibration — the threshold stays on the raw value.

What refuses. Everything else — cron schedules, transforms, other action types, multi-index input — is a 501 that names what is not evaluated. Malformed supported shapes are 400.

Honest limits. A restart does not resume watches (evaluators are spawned by put_watch only); the per-pass decide budget is 1000, with the cursor held below a failure so the backlog is retried, never dropped; the console's .xerj_alert_rules index still has no writer — watches are authored through the API, not the UI.

The percolate query underneath is unchanged and real: you store queries as documents, and a percolate search returns the stored queries that match a document you supply. The watcher's prefilter is that idea, productised for one condition type.

Custom ingest plugins: built-in only

The ingest pipeline does run transforms on _bulk. They are built-in native Rust code: rename, drop, add, JSON parse, timestamp parse, PII redaction, grok and route.

The crate that holds them is named xerj-wasm, which suggests more than is there. Its Cargo.toml has no wasmtime dependency and no wasm feature. You cannot load your own module.

The refusal rule holds here. A pipeline that names a processor this build does not implement is stored as unrunnable, and every ingest through it is refused. It is never run as a shorter pipeline in silence.

Two more that are planned, not shipped

A log-specific index mode. The xerj-logs crate is in the workspace and is compiled in as a dependency, but it is not wired: it has zero call sites outside its own tests. Logs you index today go through the general segment format and the ordinary aggregations. The low-volume log search page describes what does work.

Downloadable corpus packs — the RECORD half shipped in v1.0.0-rc.78 (#1046, #1048; this section said "no code exists" before, which was true then). xerj corpus build turns a declarative recipe into a checksummed portable pack of records — a format_version readers refuse when unknown, licence and provenance on every record — corpus add --from <pack> verifies the per-file SHA-256 and --verify-sig a detached ed25519 signature, and the rust-vulns pack is rebuilt, signed and published daily as a dated GitHub Release. Still unbuilt: the PRE-INDEXED half (an already-indexed bundle you mount instead of indexing records yourself) and a hub directory beyond that one pack.

What to do instead today

What this page does not claim

It does not claim more alerting than one watch shape: no cron, no transforms, no notification delivery, no watch authoring in the console, and watches do not survive a restart. Those remain planned. It does not claim pre-indexed packs or a pack hub: only record packs exist. S3 as an index home shipped in v1.0.0-rc.77; before that release the same setting refused to start, and this page said so.

It does not claim the search side is limited in the same way. Full-text search, the hybrid query and vector search are shipping, and the default embedder is lexical feature hashing, not a neural model. The hybrid retrieval page covers that.

XERJ is single-node. Everything on this page describes one process on one host.

FAQ

Can XERJ store its index in S3, send alerts, or run my own ingest plugin?

S3 fully, since v1.0.0-rc.77 (storage.backend = "s3"); alerting PARTIALLY, since v1.0.0-rc.80 (one watch shape evaluates, everything else 501s); record corpus packs since rc.78. User-supplied ingest plugins are the one still not implemented, and the xerj-wasm crate name makes it look closer to done than it is. This page says exactly what exists.

Does XERJ support S3 or object storage?

Yes, in both directions. xerj autoindex s3://bucket/prefix reads documents OUT of a bucket as a source, and since v1.0.0-rc.77 storage.backend = "s3" stores the index itself in the bucket: one immutable ZBM1 bundle object per segment family plus a snapshot.json catalogue, written by a real S3-compatible client (Cloudflare R2, MinIO, AWS S3). Before rc.77 the index side refused to start on purpose — an earlier version of this page said it always would.

Does XERJ have alerting or a working watcher?

Partially, since v1.0.0-rc.80. One watch shape is real: PUT /_watcher/watch/{id} with an interval trigger, a one-index search input, a condition.xerj_decide question and an index_alert action runs a background evaluator that writes fires into .xerj_alert_fires when the positive label wins non-abstaining at or above p_min. Everything else — cron, transforms, other actions, multi-index input — is refused with a 501 that names what is not evaluated. Restarts do not resume watches.

Can I write my own ingest plugin for XERJ?

Not yet. The ingest pipeline runs built-in native transforms only. The crate is named xerj-wasm, but the wasmtime backend is not in the tree.

Why does the server refuse to start when I set the storage backend to s3?

Only one case refuses now, and it is deliberate: storage.s3_bucket is empty or does not name an existing bucket. XERJ never creates a bucket, so the config check fails loudly instead of writing to local disk in silence. With a named, existing bucket the server starts and the index lives in it (since v1.0.0-rc.77).

Is there anything real to build a detection on today?

Yes, one piece. The percolate query is a dispatched query type and matches stored queries against a document you supply. The judging and alerting stages that would sit after it do not exist.

Can I download a pre-indexed corpus for XERJ?

Record packs, yes, since v1.0.0-rc.78: xerj corpus build turns a recipe into a checksummed, signable pack of records and corpus add --from <pack> verifies it — the rust-vulns pack is published daily. PRE-INDEXED packs (an indexed bundle you mount) are still unbuilt, and there is no hub beyond that one pack.

Where is the authoritative status?

ROADMAP.md in the repository. If this page and the roadmap disagree, the roadmap wins, and the disagreement is a bug worth an issue.

Evidence

• storage.backend = "s3" stores the index in the bucket: one immutable ZBM1 bundle object per segment family plus a per-index snapshot.json catalogue; merges publish before retiring inputs; a fresh node adopts the bucket. — docs/OBJECT_STORAGE.md
• The one remaining startup refusal is the loud kind: storage.backend = "s3" requires storage.s3_bucket to name an existing bucket — XERJ never creates one. — engine/crates/xerj-common/src/config.rs
• The source side is implemented: xerj autoindex s3://bucket/prefix lists a prefix and streams each changed object into a local mirror — the mirrored object bytes land on local disk under the state directory. — docs/OBJECT_STORAGE.md
• Since v1.0.0-rc.80, PUT /_watcher/watch/{id} either accepts a watch it will evaluate (interval trigger, one-index input.search, condition.xerj_decide, index_alert action — a background task prefilters with the watch's query, runs /_decide per rendered question, writes .xerj_alert_fires on a non-abstaining win at >= p_min) or refuses the body with a 501 naming what is not evaluated. An earlier version of this claim said no code evaluates a stored watch — true before rc.80, false since. — engine/crates/xerj-api/src/es_compat.rs
• .xerj_alert_fires is written by the _watcher evaluator since rc.80 (fire records carry the RAW decide confidence plus p_cal when a calibration is fitted); .xerj_alert_rules still has no writer — the console does not author watches. — engine/crates/xerj-console-api/src/indices.rs
• Ingest transforms are built-in native Rust plugins; xerj-wasm has no wasmtime dependency and no wasm feature. — engine/crates/xerj-wasm/Cargo.toml
• The xerj-logs crate is compiled in as a dependency and is not wired: it has zero call sites in non-test code. — ROADMAP.md
• The roadmap section that carries these statuses, and the design page behind it. — docs/ZERO_TOKEN_DIRECTION.md

Related

• How do I search files that live in an S3 bucket?
• What is XERJ?
• How does XERJ combine BM25 and kNN?
• Why did my filtered vector search get slower when I added a filter?
• What can I use for low-volume log search?
• I want local search that works offline, no cloud embeddings. What are my options?
• Is there an Elasticsearch-compatible engine that isn't Elasticsearch? · same intent: search engine alternatives
• Is there a single-binary local search engine I can just run? · same intent: search engine alternatives
• The best lightweight Elasticsearch-compatible search servers in 2026 · same intent: search engine alternatives