THE SEARCH · DATA ·
MEMORY LAYER
FOR AI AGENTS.

An agent needs to turn raw files into something it can query, find things, find them by meaning, remember what it learns, and recall it later. XERJ is one Elasticsearch-compatible binary that does all five — zero-config folder indexing (xerj autoindex) plus keyword, semantic, vector, and hybrid search plus a per-agent memory store — over plain HTTP on :9200. No SDK, no signup, no external embedding key. If your agent can make an HTTP request, it can use XERJ.

Handed a pile of files nobody documented? xerj autoindex content-sniffs every file (extensions are never trusted) across 13 format families — JSONL, JSON, dialect-sniffed CSV, logs, SQL dumps, SQLite, PDF, DOCX, HTML, XML, YAML, plain text, gzip — infers types and date encodings, writes explicit mappings, and streams everything in with idempotent IDs. The agent flow is three steps:

# 1. index — one command, no per-corpus flags; junk is recorded, never fatal
$ xerj autoindex ./folder

# 2. orient — the engine describes the folder back to the agent
$ xerj autoindex map                       # or: POST /autoindex-catalog/_search
#    → datasets, counts, per-field types with real examples, time ranges,
#      cross-dataset key correlations, junk report, verified engine gotchas

# 3. query — plain ES queries over the new ax-* indices
$ curl -sXPOST localhost:9200/ax-logs/_search -H 'content-type: application/json' \
    -d '{"size":0,"query":{"term":{"level":"ERROR"}},
         "aggs":{"by_service":{"terms":{"field":"service"}}}}'

In the recorded evaluation, the XERJ-backed agent reached for terms, cardinality, sum / max, date_histogram, _count, and full-text queries — and all 10 of its runs consulted the catalog first. Verified: 80/81 ground-truth checks on a 1,995-file / 518 MB secret-manifest corpus (the one miss: a Shift-JIS file indexed as mojibake); 31 datasets / 2,018,398 records live in 38.1 s; kill -9 resume converges to identical counts. Honest result from the controlled agent exam: a XERJ-backed agent tied a fair grep/python baseline on accuracy (9 + 1 partial of 10 vs 10/10) on that local corpus — XERJ's wins are structural: orientation in 4 API calls, sub-second aggregations over millions of rows, uniform access to SQLite / DOCX / gzip / decimal-comma CSV, and remote/API-only access. Full walkthrough: the zero-config indexing recipe.

Store what an agent learned, then recall it with a question that shares no keywords — no vectors supplied, no index to create first. This is a real run against an empty XERJ.

$ curl -sXPOST localhost:9200/_memory/agent-demo \
    -H 'content-type: application/json' \
    -d '{"text":"The user prefers metric units and a dark UI theme.","metadata":{"kind":"preference"}}'
{"created":true,"id":"77eff57b-e432-431c-8b49-8a16b33ab551","namespace":"agent-demo"}

$ curl -sXPOST localhost:9200/_memory/agent-demo/_recall \
    -H 'content-type: application/json' \
    -d '{"query":"what display settings does the user like?","semantic":true,"k":1}'
{"hits":[{"id":"77eff57b-e432-431c-8b49-8a16b33ab551",
          "metadata":{"kind":"preference"},
          "score":0.655571460723877,
          "text":"The user prefers metric units and a dark UI theme."}],
 "namespace":"agent-demo"}

"semantic":true tells XERJ to embed the question server-side and match by similarity — the recall text never contains the words "display" or "settings".

Exact request and response shapes for every HTTP operation are on the endpoint contract; xerj_autoindex is a CLI subcommand of the xerj binary — its full contract is in the recipe and /llms-full.txt.

Point the agent at a messy folder of real documents — PDF, DOCX, HTML, Markdown, plain text. One indexing pass walks the tree, extracts the text from every format, chunks it, and bulk-indexes into XERJ with an auto-embedded semantic_text body. After that the agent just asks questions over :9200 and gets back ranked, cited passages — no whole-file re-reads, no format blind spots.

# one-time: walk ./corpus, extract every format, chunk, bulk-index into `docfolder`
$ node xerj-index.mjs ./corpus          # 29 files → 420 chunks · 1339 ms

# the agent asks a question — hybrid = BM25 on body_text, fused (rrf)
# with semantic recall on the auto-embedded body
$ curl -sXPOST localhost:9200/docfolder/_search \
    -H 'content-type: application/json' \
    -d '{"size":5,"query":{"hybrid":{"queries":[
          {"query":{"match":{"body_text":"weeks of paid parental leave?"}}},
          {"query":{"semantic":{"field":"body","query":"weeks of paid parental leave?","k":5}}}],
        "fusion":"rrf"}}}'

On a 22-question set over that folder, this beat a fair shell-only agent that greps the raw files (searching the question's own terms, not a rigged keyword list): 21/22 (95.5%) answered versus 14/22 (63.6%). The decisive, capability-based gap is on answers that live only inside a PDF or DOCX — grep sees compressed bytes and matches nothing: 6/7 vs 0/7. On the plaintext buckets a fair grep keeps up: differently-phrased answers tie 5/5 vs 5/5, and the literal control ties 6/6 vs 6/6 — so XERJ is not winning the cases grep can actually read. What it adds there is single-query, ranked retrieval and, on large documents, far less context: over the large-document set it returns 17.24× fewer bytes than the whole answer files a fair grep must open — where the baseline is charged only the single answer-containing file it must open to read the line, nothing else. Query latency was p50 61.23 ms (max 116.09 ms).

Honest scope, the same standard as everywhere on this page: the built-in semantic_text embedder is a lexical feature-hashing model (384-dim cosine), not neural, so its matching is word/sub-word overlap, not deep understanding — which is exactly why the differently-phrased set is a tie, not a XERJ win; knn is exact brute-force at this corpus size. And the context win is scale-dependent: on the large-document set it is 17.24×, but on tiny plaintext files it INVERTS to 0.23× — returning a few ranked passages can cost more than reading the small file whole — so we don't claim it there. The full corpus, query set, scorecard, and run steps are in the document-folder-index recipe.

XERJ describes only what actually runs. Two things worth knowing:

• The built-in embedder is deterministic and local — zero-config and no external key, which is why every example here reproduces offline. For production-grade semantic quality, point a semantic_text field at your own OpenAI-compatible embeddings endpoint, or pass your own vectors. The request shapes do not change.
• knn is HNSW-served (approximate) when unfiltered, exact brute-force otherwise — unfiltered kNN on a full-precision cosine field (≥1,024 docs) runs a persisted-HNSW beam search with exact rescoring (measured recall@10 1.00 on the official bench query, 100-probe mean 0.976); num_candidates sets the beam width. Filtered kNN, non-cosine metrics, SQ8 fields, and small indexes are scored exhaustively (recall 1.00).
• Unimplemented paths fail loudly — e.g. hybrid fusion "learned" returns a clear error instead of silently substituting another strategy.

Point a crawler or an agent's tool-discovery step at these. Both are served from the site root and describe the same operations documented above.

• /llms.txt — a plain-text manifest of the base URL, the seven agent operations (autoindex included), and the honesty caveats, written for an LLM to read directly.
• /openapi.json — an OpenAPI description of the endpoints, for tool/codegen pipelines.