Agent memory

Use case: your agent learns things during a session — user preferences, project facts, decisions — and needs to remember them on a later turn to answer well. The usual answer is to bolt a separate vector database next to your search stack. XERJ folds that job into the engine you already run: a namespaced, offline agent-memory API (/_memory/{ns}) that stores text + an embedding + metadata and recalls the most relevant memories by vector similarity or plain text, with metadata filtering, forgetting, and hard per-agent isolation.

This recipe builds a tiny simulated coding-assistant agent (no LLM required) that:

1. stores what it learns as memories (POST /_memory/{ns}),
2. recalls the right memory to answer a new question — semantically (kNN) and lexically (BM25),
3. does a metadata-filtered recall ("only surface team preferences"),
4. forgets a memory that is no longer true, and
5. proves multi-namespace isolation: a second agent can never see the first agent's memories.

Everything below was run end-to-end against a live XERJ. No pip installs — the example is Python 3 stdlib only.

Why this replaces a bolt-on vector DB

A memory namespace is just a reserved XERJ index (.xerj-memory-{ns}) under the hood, so recall reuses the same exact dense_vector kNN, BM25, and metadata-filter paths that serve the rest of the engine. That means:

• One system to run and back up. No second datastore, no sync job keeping a vector index consistent with your source of truth.
• kNN and BM25 in the same store. Recall by embedding similarity, by keyword, or filter both by metadata — no glue code.
• Namespaces are physical isolation. Each agent (or tenant, or user) gets its own backing index; a recall in namespace A literally cannot read namespace B.
• Offline and zero-config. You supply the vectors (from your own model); XERJ never phones out to embed.

The memory API, in four calls

_recall picks the path from the body: a vector runs kNN; otherwise query runs BM25 over the text; an empty body returns recent memories (match_all). filter is a normal ES query clause applied as a bool filter, so it narrows without affecting the score. Recall of an unknown namespace returns {"hits": []} — that is how isolation stays clean.

Storing a memory (raw wire)

curl -XPOST localhost:9200/_memory/agent-ada -H 'content-type: application/json' -d '{
  "text": "The team prefers Rust over Go for new backend services.",
  "vector": [0.12, 0.0, 0.34],
  "metadata": {"kind": "preference", "topic": "language"},
  "id": "demo-1"
}'

{"created":true,"id":"demo-1","namespace":"agent-ada"}

Recalling by vector, filtered to preferences (raw wire)

curl -XPOST localhost:9200/_memory/agent-ada/_recall -H 'content-type: application/json' -d '{
  "vector": [0.12, 0.0, 0.34],
  "k": 2,
  "filter": {"term": {"metadata.kind": "preference"}}
}'

{"hits":[{"id":"demo-1",
          "metadata":{"kind":"preference","topic":"language"},
          "score":1.0,
          "text":"The team prefers Rust over Go for new backend services."}],
 "namespace":"agent-ada"}

Note metadata.kind filters straight out of the box — nested metadata subfields are queryable with a normal term clause, no explicit mapping needed.

The full agent

The script below is the whole example. The one thing to understand about it: it carries a tiny self-contained "embedder" (deterministic hashed character trigrams) so the demo needs no model download and cosine similarity is meaningful.

Honesty check. That toy embedder is lexical, not neural: it rewards shared character n-grams (so service/services, prefer/prefers land near each other), but it does not know that "car" ≈ "automobile". For real semantic recall you pass vectors from your own model (OpenAI/Cohere/a local model) — the store and recall calls are byte-for-byte the same. XERJ's own built-in semantic_text embedder is likewise lexical unless you configure an external /v1/embeddings endpoint; the memory API deliberately takes your vectors so quality is your choice.

The full example script is on GitHub — docs/examples/agentic-memory/agent_memory.py (Python 3 stdlib only, no dependencies).

Reproduce it yourself

Start XERJ (single node, no TLS/auth) on its default port and run the script. Nothing to install — the client is Python 3 stdlib only.

# 1. boot XERJ on the default port 9200
./engine/target/release/xerj --insecure --data-dir /tmp/xerj-mem &

# 2. run the agent (defaults to http://localhost:9200)
python3 docs/examples/agentic-memory/agent_memory.py

# Point it at a different node/port with XERJ_URL if you like:
#   XERJ_URL=http://localhost:9481 python3 docs/examples/agentic-memory/agent_memory.py

What a customer should see. The client exits 0 and prints All assertions passed.. The recall-quality line is the headline number, and it is deterministic — the toy embedder and BM25 are fixed functions of this corpus, so the values below reproduce exactly, run to run:

• Semantic (kNN): recall@1 = 83.3% (5/6), recall@3 = 100% (6/6)
• Lexical (BM25): recall@1 = 100% (6/6), recall@3 = 100% (6/6)

The single semantic recall@1 miss is honest and instructive: on "Which region is production deployed in?" the toy trigram embedder scores "Deploys are gated on CI…" (0.790) a hair above the correct "Production runs on AWS…" (0.780) — shared deploy/Deploys character trigrams. The right memory is still in the top 3, and BM25 recalls it first (6/6), which is exactly why the recipe offers both recall paths.

Real output

== Storing what the agent learned this session ==
  stored ada-1    created=True
  stored ada-2    created=True
  stored ada-3    created=True
  stored ada-4    created=True
  stored ada-5    created=True
  stored ada-6    created=True

== Semantic recall for: "What programming language should I use for a new backend service?"
  [0.779] (preference) The team prefers Rust over Go for new backend services.
  [0.741] (fact      ) The payments service owns the Postgres 'ledger' database.
  [0.731] (preference) The team prefers tabs over spaces and a 100-column line limit.

== Lexical (BM25) recall for: "Which region is production deployed in?"
  [3.746] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [1.010] (fact      ) Deploys are gated on the full integration suite passing in CI.

== Preference-only recall for: "How should I set up the new service?"
  [0.765] (preference) The team prefers Rust over Go for new backend services.
  [0.692] (preference) The team prefers tabs over spaces and a 100-column line limit.
  [0.666] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.

== Recall quality over 6 labeled probe questions ==
  semantic (kNN):  recall@1 =  83.3%  (5/6)   recall@3 = 100.0%  (6/6)
  lexical  (BM25): recall@1 = 100.0%  (6/6)   recall@3 = 100.0%  (6/6)

== Namespace isolation ==
  agent-billing recall for a coding question -> ['Invoices over $10k require CFO approval.']
  OK: agent-billing only ever sees its own namespace.

== Forgetting a memory that is no longer true ==

  before forget:
  [0.755] (preference) The team prefers OpenTelemetry for tracing, not vendor SDKs.
  [0.692] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [0.683] (preference) The team prefers Rust over Go for new backend services.
  forget -> {'forgotten': True, 'id': 'ada-6', 'namespace': 'agent-ada'}

  after forget:
  [0.692] (fact      ) Production runs on AWS in us-east-1; staging is us-west-2.
  [0.683] (preference) The team prefers Rust over Go for new backend services.
  [0.661] (preference) The team prefers tabs over spaces and a 100-column line limit.
  OK: the retracted memory is gone.

All assertions passed.

Read the run top to bottom:

• Semantic recall put the Rust preference on top for a question that never says "Rust" or "prefer" — trigram overlap on backend / service / new pulled it up over unrelated facts.
• BM25 recall answered the region question exactly, no vectors involved.
• Preference-only recall returned three memories, all kind:"preference" — the metadata.kind filter did its job.
• Recall quality over the 6 labeled probes is measured, not asserted by hand: BM25 recall@1 = 6/6, semantic recall@1 = 5/6, both recall@3 = 6/6 — a real number this run computed and printed.
• Isolation held: agent-billing, asked a coding question, only saw its own single finance memory.
• Forgetting removed ada-6 (the tracing preference); the follow-up recall no longer surfaces it.

Notes, limits, and going to production

• Bring your own embeddings. The memory API stores whatever vector you pass and kNN-searches it with cosine similarity. Swap the toy embed() for a call to your model's embeddings endpoint and everything else is unchanged. Keep the vector dimension consistent within a namespace — the backing index's dense_vector field is sized from the first stored vector.
• Vector or text. A memory needs at least text or a vector. Text is always BM25-indexed; a vector additionally enables kNN recall. Recall with vector uses kNN; without it, query uses BM25.
• Recency (opt-in). Recall is pure relevance by default. Pass recency_weight in [0, 1] on _recall to blend recency: the engine over-fetches candidates and re-ranks by (1 - w) * norm_relevance + w * norm_recency (both min-max normalized across the candidate set), so w = 1 surfaces the newest memory and w = 0 is exactly the default relevance order. Verified: two same-vector memories tie on relevance; with recency_weight: 1.0 the newer one is returned first. Useful when an agent's latest observation should win ties.
• Isolation is structural, not a filter. Namespaces are separate physical indices, so there is no query you can write in one namespace that reads another. Use one namespace per agent/tenant/user.
• GET /_memory/{ns} lists up to 100 most-recent entries for debugging/audit; it is bounded and recent-first, not a full export.
• Wire-compatible. Because a namespace is a real index, you can inspect or operate on it with the standard ES-compatible surface if you ever need to — the memory API is a thin, honest adapter over the same machinery.

Before you trust a recall number

The recall@1 and recall@3 above come from six hand-labelled probes. That is a smoke test. Before you use a recall number to choose an embedder, a chunking scheme or a recall path for your own agent, apply these field lessons. They were contributed by Chunxiao Wang (Assay / nautilus-compass) in #1138, from a production agent-memory system, and are written up with their measurements in Six eval scars from a production agent-memory system.

• Probe with situations, not questions. An agent recalls from memory using its current state ("deploy failed again with 401"), not an information need ("how do 401s work?"). This recipe's probes are questions. Add probes taken from your own agent's transcripts.
• Score on your own memories. In their runs, slicing memories by topic instead of by token window moved recall@1 by double digits, a bigger effect than moving from dense-only to hybrid retrieval. Their own data also ranked two embedding models in the opposite order to their MTEB scores. Freeze a sample of your own memories and probes, and compare candidates on that sample.
• If every probe scores zero, check the harness first. A masking bug that drops the gold item from the candidate pool turns an eval into a constant zero, or a constant pass. In this recipe, the equivalent check is that each probe's expected id was stored in the namespace you recall from.
• Small n needs a baseline and an interval. With six probes, 5/6 against 6/6 is a single probe. Report counts next to percentages. Compute what a trivial strategy scores, for example recency_weight: 1.0 (always the newest memory). Below about 100 probes, give a confidence interval.
• Compare recall paths on the same probes. kNN against BM25 on one probe set is a paired comparison. Count the probes where the two disagree (McNemar) instead of comparing two percentages.
• Freeze the criteria before the run. Write down the metric, the threshold and the artifact that will prove each claim before you run anything. Criteria may tighten mid-run but must not loosen.

The same team donated a corpus of real multi-session agent trajectories, with the failure turns kept, to the corpus hub as agent-session-trajectories. It is a starting point for testing recall on agent history instead of reference docs.