How do I rerank search results with a relevance judge?

TL;DR — Add "rerank": {} to a XERJ _search body. The top 30 hits go to an external judge, which returns a 0 to 1 relevance probability per document. That probability replaces _score. Order by it, never threshold it with min_score — the measured probabilities are not calibrated (ECE 0.10–0.31). It sends document text off the machine. Ranking quality is measured: PR #975 scored median nDCG@10 of 0.7410 SciFact, 0.3312 NFCorpus and 0.3638 FiQA over 1,271 queries.

What the rerank stage does

The engine retrieves and ranks as usual. The stage then sends the question and the top window hits to a relevance judge. The judge answers one yes-or-no question per document and returns a probability that the document is relevant.

XERJ reorders the hits by that probability. The probability replaces _score, and hits.max_score becomes the top hit's probability.

{
  "query": { "match": { "body": "vitamin d bone density" } },
  "size": 5,
  "rerank": { "window": 50 }
}

The provider is TypeSafe AI's Jev model, reached through its System One API. XERJ asks one question per document rather than one multiple-choice question across all of them, because a real corpus has more than one relevant document. That request shape is the approach used by the hev/jev-rerank project, which is Apache-2.0.

The probability orders; it does not threshold

A BM25 score orders results but has no absolute meaning. A score of 7.2 on one query is not comparable to 7.2 on another query, so you cannot use it as a threshold.

The judge's probability turns out to need the same caution: PR #975 (merged 2026-09-20) measured it against graded human-relevance labels on 1,271 judged queries. Expected calibration error is 0.10–0.31, worst on FiQA: documents the judge rated around 0.93 confidence were relevant 34% of the time. The curve still rises to the right — the model discriminates, which is what the ranking gains ride on — but a 0.9 is not a 90% chance of relevance. Order by the scores; never threshold the raw probability with min_score, which cuts documents the judgments call relevant. If you need an absolute cut-off, make sure the threshold holds against your own data first. The top-level min_score stays the engine's threshold on engine scores, and the two compose.

This sends your data off the machine

Reranking is the only search-time feature that sends document text to a third party. Two other features send text off the node, and both are off by default. Proxy embeddings ([embedding] default_endpoint) send document text at indexing time and query text at search time to an external embeddings API. The WAL tap replays every write on tapped indices to an external _bulk endpoint. An operator has to turn either one on.

The node has two more outbound connections, and neither carries document or query text. Neural mode downloads its model from the HuggingFace Hub once. Cluster mode, which a default single-node deployment never starts, sends Raft messages, such as index names and mappings, to your own peers. The reference, docs/RERANK.md, lists every outbound connection a node can open.

A rerank request POSTs the question and the text of up to window hits to the provider.

Three controls exist. Reranking does nothing until an operator sets a provider key. A search only triggers it by carrying a rerank block. An operator can forbid it with enabled = false under [rerank], which refuses every rerank request with HTTP 403.

The judge sees exactly what the response returns, and nothing else. Source filtering is therefore also the control on what leaves the machine: "_source": ["title"] sends titles only. rerank.fields is exhaustive, so ["body"] sends the body and not even the title. Document ids, index names, numbers and vectors are never sent.

Set the provider key

Put the key in the config file or in the environment. A value in the config file wins over the environment. A stray variable therefore cannot silently change where document text is sent.

[rerank]
enabled  = true
api_key  = "ts-..."
endpoint = "https://api.typesafe.ai/v1/systemone"

The environment fallbacks are TYPESAFE_API_KEY and TYPESAFE_ENDPOINT. There is no per-request key, because a key in a search body would land in logs. GET /_xerj/rerank reports whether a key is set and where it came from. It never returns the key.

Read the _rerank block

Every reranked response carries a _rerank block ahead of hits. Read applied before you trust the order.

hits.total and aggregations stay the engine's. They describe the full match set, not the window. When four documents match and two are judged irrelevant, total is 4 and pruned_below_min_score is 2.

What happens when the provider fails

The policy is split: degrade on deadline, surface on contract.

A slow third party is not a reason to deny you results you already have. A wrong key does not fix itself, and hiding it would mislead you about which ranking you hold. This split follows the approach Meilisearch uses in its personalization module (MIT). The approach and the retry back-off constants were adapted from it, and the code cites the lines.

Cost facts

Every document in the window is a paid judgement. The window defaults to 30 and the server caps it at 300. A provider call carries at most 30 documents, so window: 35 is two calls. Scores from different calls stay comparable, because each one is an absolute probability rather than a rank inside its batch.

XERJ sends 8 calls at once by default and 16 at most. The usage field is the provider's own token count for the search. XERJ does not price it.

Every request is judged from scratch. There is no verdict cache, so three page requests over one 30-document window are three provider calls and 90 paid judgements. To keep the cost down, fetch the window once and cut the pages out of it in your own code. Page two continues page one only if the provider returns the same probabilities on a repeat call. The provider is not deterministic: PR #975 ran every query three times and measured a mean per-document probability drift of 0.0077–0.0096 (max 0.08). Any comparison tighter than about ±0.01 nDCG@10 is noise.

The strings have ceilings too. instructions is limited to 2,000 characters, because the provider's wire format repeats it once per judged document. The question is limited to 4,000 characters and model to 128. A longer value is an HTTP 400 that names the field and the limit.

What is measured, and what is not

When this page was updated on 2026-09-20, ranking quality with the real Jev model was genuinely unverified. XERJ had no provider key, and every test of the stage ran against a test double that scores by a table the test supplies.

PR #975, merged later that day, changed that: the real Jev reranker judged 1,271 BEIR queries ranking off a XERJ node. The run found and fixed an integration defect of XERJ's own — the SciFact pilot scored 0.3822 before the fix and 0.8299 after it. The full runs scored median nDCG@10 of 0.7410 on SciFact (300 queries), 0.3312 on NFCorpus (323) and 0.3638 on FiQA (648), against BM25 first stages of 0.6572, 0.3016 and 0.2382. The numbers and the reliability curves are published in benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot, -rerank-full and -rerank-full-fiqa, with the write-up in landing/blog/does-xerj-beat-jev.html. The stage's own HTTP tests still use a test double; the benchmark runs carry the quality claim.

The hev/jev-rerank README publishes nDCG@10 of 0.768 on SciFact and 0.358 on NFCorpus for Jev. XERJ did not run those systems, and they rerank that project's own first-stage shortlist, not XERJ's. XERJ's own hybrid search scored 0.6993 and 0.3448 on the same datasets. Those two figures were measured with --embed-mode neural and the all-MiniLM-L6-v2 model. The default embedder is lexical feature hashing, which has no model in it. This is not a controlled comparison.

When reranking is not worth it

Do not rerank an exact identifier lookup, a filter, or an aggregation-only request. The first stage already answers those, and the stage adds a paid network call per search. Rerank when the right document is probably in the top 30 but not at the top. If you also need an absolute relevance cut-off, make sure the threshold holds against your own data first — the judge's probabilities order well but are not calibrated.

FAQ

How do I rerank search results in XERJ?

Add a rerank object to the body of POST /{index}/_search. An empty object uses every default: the top 30 hits are judged and reordered by relevance probability.

Does reranking send my documents to a third party?

Yes. It is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when an operator configures them, and all three are off by default. The question and the text of up to window hits go to the provider. Only fields the response returns are sent.

What happens when the rerank provider is slow?

The search still answers with HTTP 200. You get the engine's order and scores, and the _rerank block says applied: false with the reason. A slow provider degrades; it does not fail the search.

What happens when the provider key is wrong or missing?

The search fails loudly. A missing key is HTTP 503, a provider 401 is HTTP 502, both with type rerank_exception and no hits. XERJ does not hand back lexical order as if it had been reranked.

Is the ranking quality of the rerank model verified?

Yes, since PR #975 (merged 2026-09-20). The real Jev reranker judged 1,271 BEIR queries ranking off a XERJ node: median nDCG@10 of 0.7410 on SciFact, 0.3312 on NFCorpus and 0.3638 on FiQA. The stage's own HTTP tests still use a test double; the benchmark runs carry the quality claim.

Can I combine rerank with sort or aggregations?

Aggregations work and still describe the full match set. sort, search_after, collapse, scroll and size: 0 are refused with an HTTP 400 that names the conflict.

Can an AI agent ask for reranking through MCP?

Yes. The xerj_search and xerj_hybrid_search tools take an optional rerank argument. It only works when the operator has set a provider key on the node.

Evidence

• Reranking is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when configured; the neural model download and cluster Raft messages carry no document or query text. A test checks that list against every outbound network client in the engine source. — engine/crates/xerj-rerank/tests/egress_inventory.rs
• The rerank stage defaults to a window of 30 hits, allows at most 300, sends at most 30 documents per provider call, runs 8 calls in flight by default and 16 at most, and defaults to a 10000 ms stage budget capped at 60000 ms. — engine/crates/xerj-rerank/src/lib.rs
• The failure policy, every refused combination, the 400, 403, 502 and 503 statuses, paging inside the window, and the rule that only returned text is sent are each pinned by an HTTP test against an in-process stub of the provider wire format. — engine/crates/xerj-api/tests/rerank_stage_http.rs
• The nDCG@10 figures for Jev (0.768 SciFact, 0.358 NFCorpus), Voyage rerank-3 (0.755, 0.357) and Cohere rerank-v3.5 (0.745, 0.340) are published in the hev/jev-rerank README. XERJ did not run them, and they rerank that project's own first-stage shortlist. — https://github.com/hev/jev-rerank
• XERJ hybrid RRF scored nDCG@10 0.6993 on SciFact and 0.3448 on NFCorpus with --embed-mode neural and all-MiniLM-L6-v2, not with the default lexical embedder. — benchmarks/beir-hybrid/README.md
• Ranking quality with the real Jev model is measured (PR #975, merged 2026-09-20): the reranker judged 1,271 BEIR queries ranking off a XERJ node — median nDCG@10 0.7410 SciFact, 0.3312 NFCorpus, 0.3638 FiQA, against BM25 first stages of 0.6572, 0.3016 and 0.2382 — after an integration defect of XERJ's own was found and fixed (SciFact pilot 0.3822 to 0.8299). The same runs measured calibration (ECE 0.10-0.31, worst on FiQA: documents rated ~0.93 confidence were relevant 34% of the time) and provider non-determinism (mean per-document drift 0.0077-0.0096, max 0.08; comparisons tighter than ~±0.01 nDCG@10 are noise). — benchmarks/beir-hybrid/results

Related

• What is Reciprocal Rank Fusion?
• How much better is hybrid search than BM25? Measured nDCG numbers
• How does XERJ combine BM25 and kNN?
• Config TOML
• Can I route tickets or filter spam from labelled history, without an LLM call? · same intent: retrieval research and eval
• When should I turn on neural embeddings, and when is the lexical default enough? · same intent: retrieval research and eval
• How do I improve a basic FAISS RAG pipeline? · same intent: retrieval research and eval