--- title: "How do I rerank search results with a relevance judge?" canonical: "https://xerj.org/answers/rerank-search-results-calibrated-judge" updated: "2026-10-19" source: "content/answers/rerank-search-results-calibrated-judge.md" --- # How do I rerank search results with a relevance judge? **Short answer** (updated 2026-10-19): How do I rerank search results with a relevance judge? Add a rerank block to a XERJ _search body. The top 30 hits go to an external judge, which returns a 0 to 1 relevance probability that replaces _score. Order by it, but never threshold it with min_score: measured calibration error was 0.10 to 0.31. **Key number:** The rerank stage defaults to a 30-hit window and allows at most 300. (as of 2026-10-19; source: [engine/crates/xerj-rerank/src/lib.rs](https://github.com/xerj-org/xerj/blob/main/engine/crates/xerj-rerank/src/lib.rs)) **TL;DR** — Add `"rerank": {}` to a XERJ `_search` body. The top 30 hits go to an external judge, which returns a 0 to 1 relevance probability per document. That probability replaces `_score`. Order by it, never threshold it with `min_score` — the measured probabilities are not calibrated (ECE 0.10–0.31). It sends document text off the machine. Ranking quality is measured: [PR #975](https://github.com/xerj-org/xerj/pull/975) scored median nDCG@10 of 0.7410 SciFact, 0.3312 NFCorpus and 0.3638 FiQA over 1,271 queries. ## Agent prompt ```text Act as a coding agent. Read https://xerj.org/llms.txt, ask the operator whether a rerank provider key is set on the XERJ node and whether document text may leave the machine, then run one search without the rerank block and one with it, and report the _rerank block of the second response, including whether applied is true. ``` ## Commands ### Command 1 Note: Ask the node whether a rerank provider is configured. The answer names where the key came from and never prints the key. A node started with --insecure needs no credentials. On any other node the route is superuser-only, so send the admin key from the admin.key file in the data directory. ```sh curl -s 'http://127.0.0.1:9200/_xerj/rerank' ``` ### Command 2 Note: Rerank the top 50 hits; they come back ordered by the judge's probability. Order by the scores, never threshold the raw probability with min_score. This request sends the text of those hits to the provider. ```sh curl -s -XPOST 'http://127.0.0.1:9200/kb/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"vitamin d bone density"}},"size":5,"rerank":{"window":50}}' ``` ### Command 3 Note: Send titles only. The judge sees exactly what the response returns, so source filtering is also the control on what leaves the machine. ```sh curl -s -XPOST 'http://127.0.0.1:9200/kb/_search' -H 'content-type: application/json' -d '{"query":{"match":{"body":"vitamin d bone density"}},"_source":["title"],"rerank":{}}' ``` ## What the rerank stage does The engine retrieves and ranks as usual. The stage then sends the question and the top `window` hits to a relevance judge. The judge answers one yes-or-no question per document and returns a probability that the document is relevant. XERJ reorders the hits by that probability. The probability replaces `_score`, and `hits.max_score` becomes the top hit's probability. ```json { "query": { "match": { "body": "vitamin d bone density" } }, "size": 5, "rerank": { "window": 50 } } ``` The provider is TypeSafe AI's Jev model, reached through its System One API. XERJ asks one question per document rather than one multiple-choice question across all of them, because a real corpus has more than one relevant document. That request shape is the approach used by the `hev/jev-rerank` project, which is Apache-2.0. ## The probability orders; it does not threshold A BM25 score orders results but has no absolute meaning. A score of 7.2 on one query is not comparable to 7.2 on another query, so you cannot use it as a threshold. The judge's probability turns out to need the same caution: [PR #975](https://github.com/xerj-org/xerj/pull/975) (merged 2026-09-20) measured it against graded human-relevance labels on 1,271 judged queries. Expected calibration error is 0.10–0.31, worst on FiQA: documents the judge rated around 0.93 confidence were relevant 34% of the time. The curve still rises to the right — the model discriminates, which is what the ranking gains ride on — but a 0.9 is not a 90% chance of relevance. Order by the scores; never threshold the raw probability with `min_score`, which cuts documents the judgments call relevant. If you need an absolute cut-off, make sure the threshold holds against your own data first. The top-level `min_score` stays the engine's threshold on engine scores, and the two compose. ## This sends your data off the machine Reranking is the only search-time feature that sends document text to a third party. Two other features send text off the node, and both are off by default. Proxy embeddings (`[embedding] default_endpoint`) send document text at indexing time and query text at search time to an external embeddings API. The WAL tap replays every write on tapped indices to an external `_bulk` endpoint. An operator has to turn either one on. The node has two more outbound connections, and neither carries document or query text. Neural mode downloads its model from the HuggingFace Hub once. Cluster mode, which a default single-node deployment never starts, sends Raft messages, such as index names and mappings, to your own peers. The reference, `docs/RERANK.md`, lists every outbound connection a node can open. A rerank request POSTs the question and the text of up to `window` hits to the provider. Three controls exist. Reranking does nothing until an operator sets a provider key. A search only triggers it by carrying a `rerank` block. An operator can forbid it with `enabled = false` under `[rerank]`, which refuses every rerank request with HTTP 403. The judge sees exactly what the response returns, and nothing else. Source filtering is therefore also the control on what leaves the machine: `"_source": ["title"]` sends titles only. `rerank.fields` is exhaustive, so `["body"]` sends the body and not even the title. Document ids, index names, numbers and vectors are never sent. ## Set the provider key Put the key in the config file or in the environment. A value in the config file wins over the environment. A stray variable therefore cannot silently change where document text is sent. ```toml [rerank] enabled = true api_key = "ts-..." endpoint = "https://api.typesafe.ai/v1/systemone" ``` The environment fallbacks are `TYPESAFE_API_KEY` and `TYPESAFE_ENDPOINT`. There is no per-request key, because a key in a search body would land in logs. `GET /_xerj/rerank` reports whether a key is set and where it came from. It never returns the key. ## Read the _rerank block Every reranked response carries a `_rerank` block ahead of `hits`. Read `applied` before you trust the order. | Field | Meaning | | --- | --- | | `applied` | `true` means the order is the judge's. `false` means the engine's order stands, and `reason` says why. | | `score_kind` | `probability` or `engine`. | | `judged` | Documents the provider returned a verdict for. | | `pruned_below_min_score` | Judged hits removed by `rerank.min_score` — a coarse filter, not a calibrated cut-off (the probabilities are not calibrated). | | `fields_without_text` | Named fields that no hit in the window returned text for. | | `usage` | Tokens the provider reported for this search. | `hits.total` and aggregations stay the engine's. They describe the full match set, not the window. When four documents match and two are judged irrelevant, `total` is 4 and `pruned_below_min_score` is 2. ## What happens when the provider fails The policy is split: degrade on deadline, surface on contract. | What happened | Result | | --- | --- | | The provider missed `timeout_ms` | HTTP 200, engine order, `applied: false`. | | The `rerank` block is malformed or the combination is refused | HTTP 400, nothing sent. | | The operator disabled reranking | HTTP 403, nothing sent. | | No provider key on the node | HTTP 503, nothing sent. | | The provider answered 401 or a malformed body | HTTP 502, no hits. | A slow third party is not a reason to deny you results you already have. A wrong key does not fix itself, and hiding it would mislead you about which ranking you hold. This split follows the approach Meilisearch uses in its personalization module (MIT). The approach and the retry back-off constants were adapted from it, and the code cites the lines. ## Cost facts Every document in the window is a paid judgement. The window defaults to 30 and the server caps it at 300. A provider call carries at most 30 documents, so `window: 35` is two calls. Scores from different calls stay comparable, because each one is an absolute probability rather than a rank inside its batch. XERJ sends 8 calls at once by default and 16 at most. The `usage` field is the provider's own token count for the search. XERJ does not price it. Every request is judged from scratch. There is no verdict cache, so three page requests over one 30-document window are three provider calls and 90 paid judgements. To keep the cost down, fetch the window once and cut the pages out of it in your own code. Page two continues page one only if the provider returns the same probabilities on a repeat call. The provider is not deterministic: [PR #975](https://github.com/xerj-org/xerj/pull/975) ran every query three times and measured a mean per-document probability drift of 0.0077–0.0096 (max 0.08). Any comparison tighter than about ±0.01 nDCG@10 is noise. The strings have ceilings too. `instructions` is limited to 2,000 characters, because the provider's wire format repeats it once per judged document. The question is limited to 4,000 characters and `model` to 128. A longer value is an HTTP 400 that names the field and the limit. ## What is measured, and what is not When this page was updated on 2026-09-20, ranking quality with the real Jev model was genuinely unverified. XERJ had no provider key, and every test of the stage ran against a test double that scores by a table the test supplies. [PR #975](https://github.com/xerj-org/xerj/pull/975), merged later that day, changed that: the real Jev reranker judged 1,271 BEIR queries ranking off a XERJ node. The run found and fixed an integration defect of XERJ's own — the SciFact pilot scored 0.3822 before the fix and 0.8299 after it. The full runs scored median nDCG@10 of 0.7410 on SciFact (300 queries), 0.3312 on NFCorpus (323) and 0.3638 on FiQA (648), against BM25 first stages of 0.6572, 0.3016 and 0.2382. The numbers and the reliability curves are published in `benchmarks/beir-hybrid/results/2026-09-20-rerank-pilot`, `-rerank-full` and `-rerank-full-fiqa`, with the write-up in `landing/blog/does-xerj-beat-jev.html`. The stage's own HTTP tests still use a test double; the benchmark runs carry the quality claim. The `hev/jev-rerank` README publishes nDCG@10 of 0.768 on SciFact and 0.358 on NFCorpus for Jev. XERJ did not run those systems, and they rerank that project's own first-stage shortlist, not XERJ's. XERJ's own hybrid search scored 0.6993 and 0.3448 on the same datasets. Those two figures were measured with `--embed-mode neural` and the all-MiniLM-L6-v2 model. The default embedder is lexical feature hashing, which has no model in it. This is not a controlled comparison. ## When reranking is not worth it Do not rerank an exact identifier lookup, a filter, or an aggregation-only request. The first stage already answers those, and the stage adds a paid network call per search. Rerank when the right document is probably in the top 30 but not at the top. If you also need an absolute relevance cut-off, make sure the threshold holds against your own data first — the judge's probabilities order well but are not calibrated. ## FAQ ### How do I rerank search results in XERJ? Add a `rerank` object to the body of `POST /{index}/_search`. An empty object uses every default: the top 30 hits are judged and reordered by relevance probability. ### Does reranking send my documents to a third party? Yes. It is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when an operator configures them, and all three are off by default. The question and the text of up to `window` hits go to the provider. Only fields the response returns are sent. ### What happens when the rerank provider is slow? The search still answers with HTTP 200. You get the engine's order and scores, and the `_rerank` block says `applied: false` with the reason. A slow provider degrades; it does not fail the search. ### What happens when the provider key is wrong or missing? The search fails loudly. A missing key is HTTP 503, a provider 401 is HTTP 502, both with type `rerank_exception` and no hits. XERJ does not hand back lexical order as if it had been reranked. ### Is the ranking quality of the rerank model verified? Yes, since PR #975 (merged 2026-09-20). The real Jev reranker judged 1,271 BEIR queries ranking off a XERJ node: median nDCG@10 of 0.7410 on SciFact, 0.3312 on NFCorpus and 0.3638 on FiQA. The stage's own HTTP tests still use a test double; the benchmark runs carry the quality claim. ### Can I combine rerank with sort or aggregations? Aggregations work and still describe the full match set. `sort`, `search_after`, `collapse`, `scroll` and `size: 0` are refused with an HTTP 400 that names the conflict. ### Can an AI agent ask for reranking through MCP? Yes. The `xerj_search` and `xerj_hybrid_search` tools take an optional `rerank` argument. It only works when the operator has set a provider key on the node. ## Evidence - Reranking is the only search-time feature that sends document text off the node. Proxy embeddings and the WAL tap also send text off the node when configured; the neural model download and cluster Raft messages carry no document or query text. A test checks that list against every outbound network client in the engine source. — `engine/crates/xerj-rerank/tests/egress_inventory.rs` - The rerank stage defaults to a window of 30 hits, allows at most 300, sends at most 30 documents per provider call, runs 8 calls in flight by default and 16 at most, and defaults to a 10000 ms stage budget capped at 60000 ms. — `engine/crates/xerj-rerank/src/lib.rs` - The failure policy, every refused combination, the 400, 403, 502 and 503 statuses, paging inside the window, and the rule that only returned text is sent are each pinned by an HTTP test against an in-process stub of the provider wire format. — `engine/crates/xerj-api/tests/rerank_stage_http.rs` - The nDCG@10 figures for Jev (0.768 SciFact, 0.358 NFCorpus), Voyage rerank-3 (0.755, 0.357) and Cohere rerank-v3.5 (0.745, 0.340) are published in the hev/jev-rerank README. XERJ did not run them, and they rerank that project's own first-stage shortlist. — [https://github.com/hev/jev-rerank](https://github.com/hev/jev-rerank) - XERJ hybrid RRF scored nDCG@10 0.6993 on SciFact and 0.3448 on NFCorpus with --embed-mode neural and all-MiniLM-L6-v2, not with the default lexical embedder. — `benchmarks/beir-hybrid/README.md` - Ranking quality with the real Jev model is measured (PR #975, merged 2026-09-20): the reranker judged 1,271 BEIR queries ranking off a XERJ node — median nDCG@10 0.7410 SciFact, 0.3312 NFCorpus, 0.3638 FiQA, against BM25 first stages of 0.6572, 0.3016 and 0.2382 — after an integration defect of XERJ's own was found and fixed (SciFact pilot 0.3822 to 0.8299). The same runs measured calibration (ECE 0.10-0.31, worst on FiQA: documents rated ~0.93 confidence were relevant 34% of the time) and provider non-determinism (mean per-document drift 0.0077-0.0096, max 0.08; comparisons tighter than ~±0.01 nDCG@10 are noise). — `benchmarks/beir-hybrid/results` ## Related - [What is Reciprocal Rank Fusion?](/answers/reciprocal-rank-fusion-when-to-use) - [How much better is hybrid search than BM25? Measured nDCG numbers](/answers/hybrid-search-quality-measured-ndcg) - [How does XERJ combine BM25 and kNN?](/answers/how-xerj-combines-search) - [Config TOML](/docs/config) - [Can I route tickets or filter spam from labelled history, without an LLM call?](/answers/classify-text-from-labelled-history-without-llm) (same intent: retrieval research and eval) - [When should I turn on neural embeddings, and when is the lexical default enough?](/answers/do-search-embeddings-help) (same intent: retrieval research and eval) - [How do I improve a basic FAISS RAG pipeline?](/answers/improve-basic-faiss-rag-pipeline) (same intent: retrieval research and eval)