Vector search · kNN

Goal: you have a set of items — products, articles, images, support tickets — and each one is represented by an embedding: a fixed-length list of floats. You want the classic operation behind recommendations, "more like this", dedup, and semantic retrieval: given one vector, find the N most similar items.

XERJ answers this natively. It stores a dense_vector field, maintains a persisted HNSW graph over it, and serves the Elasticsearch-8 top-level knn query — the same request bodies (and ES client libraries) work unchanged. No sidecar vector database, no separate service.

This recipe uses tiny, hand-made 4-dimensional vectors so every result is reproducible and you can see why each neighbor came back. In production your vectors come from a real embedding model (768, 1024, 1536 dims…) — the API is identical.

The toy feature space

Real embeddings are opaque. To make kNN legible, we hand-build vectors in a 4-axis "feature space":

[ citrus , berry , engine , wheels ]

Fruits load on the fruit axes and are ~0 on the vehicle axes; vehicles do the reverse. So a query sitting in the citrus corner should return oranges and lemons first and never a truck. That's our recall check.

1. Map a dense_vector field

curl -X PUT http://localhost:9200/catalog -H 'Content-Type: application/json' -d '{
  "mappings": {
    "properties": {
      "name":      { "type": "text" },
      "category":  { "type": "keyword" },
      "in_stock":  { "type": "boolean" },
      "embedding": { "type": "dense_vector", "dims": 4, "similarity": "cosine" }
    }
  }
}'

Two things matter:

• dims must equal the length of every vector you index. Indexing a wrong-length vector is silently skipped from the kNN scan.
• similarity defaults to cosine (magnitude-invariant — the right default for most normalized embeddings). Other options: l2, dot_product. Scores come back as ES-normalized cosine in [0, 1]: identical direction ≈ 1.0, orthogonal ≈ 0.5.

Load the rows with the Bulk API ({"index":{...}} + source line pairs), adding ?refresh=true so they're searchable immediately.

2. Plain kNN — "find items most like a citrus fruit"

Put the query vector right in the citrus corner and ask for the 3 nearest:

curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
  "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 }
}'

Real response (trimmed):

{
  "hits": { "hits": [
    { "_id": "1", "_score": 0.9998569, "_source": { "name": "Navel Orange", "category": "fruit" } },
    { "_id": "2", "_score": 0.9997083, "_source": { "name": "Meyer Lemon",  "category": "fruit" } },
    { "_id": "3", "_score": 0.9948,    "_source": { "name": "Blood Orange", "category": "fruit" } }
  ] }
}

The three nearest are exactly the three citrus fruits, in order of closeness, and no vehicle leaked in. k is how many neighbors you want back; num_candidates is the ANN beam width on the HNSW-served (unfiltered, ≥1,024-doc) path; a small corpus like this one runs the exact brute-force scan, which returns the true top-k regardless of this value.

Query the other corner and the space cleanly separates:

curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
  "knn": { "field": "embedding", "query_vector": [0.0, 0.0, 0.9, 0.85], "k": 2, "num_candidates": 10 }
}'
# → Pickup Truck (1.0), Sports Car (0.9992) — only vehicles

3. kNN + a filter — "…but only what's in stock"

This is the request people actually get wrong. To constrain a kNN search (price range, in-stock, tenant, language…), wrap the knn in a bool and put the constraint in a sibling filter clause. bool.filter runs as a pre-filter: XERJ narrows the candidate set first, then ranks the survivors by similarity.

curl -X POST http://localhost:9200/catalog/_search -H 'Content-Type: application/json' -d '{
  "size": 3,
  "query": {
    "bool": {
      "must":   [ { "knn": { "field": "embedding", "query_vector": [0.92, 0.08, 0.0, 0.0], "k": 3, "num_candidates": 10 } } ],
      "filter": [ { "term": { "in_stock": true } } ]
    }
  }
}'

Real response (trimmed):

1  0.9999  Navel Orange   in_stock=True
2  0.9997  Meyer Lemon    in_stock=True
4  0.5969  Strawberry     in_stock=True

Blood Orange (id 3) is the 3rd-closest citrus but is out of stock, so it drops out and a farther in-stock item (a berry) takes the slot. The filter genuinely restricted the pool — it didn't just re-score.

Heads-up on the ES-8 shorthand. Elasticsearch also lets you inline the filter inside the knn clause: "knn": { …, "filter": {…} }. XERJ parses that form but currently ignores the inlined filter on the top-level-knn path (it returns unfiltered neighbors). Until that's wired up, use the bool + filter wrapper shown above — it's the reliable way to get pre-filtered kNN today, and it's also how XERJ's own agent-memory recall builds filtered vector queries internally.

Reproduce it yourself

Boot XERJ on its default port and run the self-checking script — stdlib only, no pip, no network calls:

# 1. start a throwaway node (default ES-compat port 9200)
./engine/target/release/xerj --insecure --data-dir ./data

# 2. in another shell, from the repo root:
python3 docs/examples/vector-search-knn/knn_demo.py

The script reads its server URL from XERJ_URL (default http://localhost:9200); the older BASE variable still works as an alias. Point it at any node with XERJ_URL=http://host:port python3 ….

It creates the index, bulk-loads the 8-row catalog, runs all three searches, and — beyond the pass/fail assertions — computes recall@k against a brute-force exact-cosine ground truth built in-process, so the "the result is exact" claim below is measured, not asserted. Exact tail from a real run:

=== kNN k=3, query ~citrus ===
   ('1', 0.9999, 'Navel Orange', 'fruit', True)
   ('2', 0.9997, 'Meyer Lemon', 'fruit', True)
   ('3', 0.9948, 'Blood Orange', 'fruit', False)
  exact top-3 (brute-force cosine): ['1', '2', '3']
  recall@3 = 1.000 (3/3 of the true nearest returned)
  OK: top-3 are exactly the citrus fruits, no vehicles leaked in

=== kNN + bool.filter in_stock:true ===
   ('1', 0.9999, 'Navel Orange', 'fruit', True)
   ('2', 0.9997, 'Meyer Lemon', 'fruit', True)
   ('4', 0.5969, 'Strawberry', 'fruit', True)
  OK: out-of-stock Blood Orange excluded; only in-stock neighbors returned

=== kNN k=2, query ~vehicle ===
   ('6', 1.0, 'Pickup Truck', 'vehicle', True)
   ('7', 0.9992, 'Sports Car', 'vehicle', False)
  exact top-2 (brute-force cosine): ['6', '7']
  recall@2 = 1.000
  OK: vehicle query returns only vehicles

MEASURED recall vs brute-force exact across queries: 1.000 (exact)
ALL ASSERTIONS PASSED

The scores and recall are deterministic for this 8-row corpus — every run prints the same numbers (recall@3 = recall@2 = 1.000). The cosine scores above are the ES-normalized values (1 + cos)/2; unrounded they are 0.99985695, 0.99970835, 0.99479961 for the citrus top-3.

How the search works: HNSW-served kNN with an exact fallback

On a full-precision cosine field with ≥1,024 docs, an unfiltered knn is served by a beam search over a persisted HNSW graph, and the candidates are exact-rescored so returned scores match the exact path bit-for-bit — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937). num_candidates sets the beam width (floored at 800 to match ES's per-segment candidate semantics). Everything else — a bool.filter pre-filter, non-cosine similarity, SQ8 quantization, or a corpus under 1,024 docs like this recipe's — runs the exact brute-force scan: every surviving candidate is compared and the true nearest neighbours returned (recall 1.00; latency scales with vectors scanned).

This recipe's 8-row corpus is far below the 1,024-doc ANN threshold, so its queries run on the exact path — the script measures recall@k = 1.000 against an independent brute-force cosine ground truth on every query, which is exactly what the exact path must produce. The hnsw_m / hnsw_ef_construction config keys are accepted but not yet wired — the graph currently builds with fixed M=16 / ef_construction=200.

Raw kNN vs. semantic_text: which do I use?

Both end up doing vector search. The difference is who makes the vectors.

• Raw dense_vector + knn (this recipe) — you supply the embeddings, on both the ingest side and the query side. Use it when you already run an embedding model (OpenAI, a local sentence-transformer, a CLIP image encoder, your own recommender) and just need fast storage and nearest-neighbor retrieval. Maximum control; works for any modality, not just text.
• semantic_text field + semantic query — XERJ embeds text for you at ingest time and at query time, so you send plain strings and never touch a vector. Great when your data is text and you want semantic search with zero embedding plumbing. Honest caveat: the built-in embedder is lexical, not neural — excellent for demos and keyword-semantic overlap, but for production-grade semantic quality point XERJ at an external OpenAI-compatible /v1/embeddings endpoint (embedding.default_endpoint), after which both ingest and query use that higher-quality model.

Rule of thumb: already have embeddings → raw knn. Have text and want it embedded for you → semantic_text. Need to blend keyword relevance (BM25) with vector similarity in one request? Reach for the hybrid query.

Gotchas

• dims must match your vectors exactly — a length mismatch means the doc is skipped from the scan, not an error.
• Filter a kNN with bool + filter, not the inlined knn.filter (currently ignored on the top-level path — see §3).
• k vs num_candidates: k = results returned; num_candidates = the ANN beam width on the HNSW-served (unfiltered, ≥1,024-doc) path, floored at 800 — raise it to trade latency for recall. On the exact scan (filtered kNN, or small corpora like this recipe's) it has no effect.
• Scores are ES-normalized cosine in [0,1] (≈1.0 identical, ≈0.5 orthogonal), not raw cosine in [-1,1].