HOME·ANSWERS·XERJ CORPUS INDEX --FRESH FAILED. HOW DO I REBUILD A CORPUS INDEX?
ANSWERS · CODING AGENTS: REFERENCE RETRIEVAL

xerj corpus index --fresh failed. How do I rebuild a corpus index?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-10-19
SHORT ANSWER · UPDATED 2026-10-19

How do I rebuild a reference-coding corpus index from scratch? Run xerj corpus index with --fresh. It builds a replacement beside the existing index, checks that it holds records, switches the state file, and only then deletes the old indices by exact name. A failed build costs you nothing; the old index keeps serving.

Key number: On v1.0.0-rc.74, xerj autoindex --fresh over a committed generation aborted with exit 1; --fresh now swaps instead of discarding. (as of 2026-10-19 · source: benchmarks/autoindex-resilience/fresh-before-rc74.committed-generation.txt)

TL;DR — Run xerj corpus index <corpus> --fresh. It builds a replacement beside the existing index, checks that it holds records, switches the state file, and only then deletes the old indices by exact name. A build that fails costs you nothing. Older releases' wrapper scripts failed on every corpus that had been indexed before.

PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent maintaining a reference-coding corpus. Read https://xerj.org/llms.txt. When xerj code reports an index older than 30 days, or a plain `xerj corpus index` run says the state directory cannot become generation authority, run `xerj corpus index <corpus> --fresh`. Do not delete the old indices yourself and do not pass --fresh to xerj autoindex directly. Read the command's last lines: it states the record count of the verified replacement, or that the existing index was NOT touched.
RUN THIS XERJ COMMAND xerj corpus index xerj-storage Index a corpus, or update in place the build an earlier run recorded.
RUN THIS XERJ COMMAND xerj corpus index xerj-storage --fresh Rebuild from scratch: build beside the old index, verify, switch, then retire.
RUN THIS XERJ COMMAND bash -c 'cat ~/.xerj-code/state/xerj-storage.json' Read the ledger. prefix is the whole namespace; index_prefix is the one verified build.
RUN THIS XERJ COMMAND curl -s 'http://127.0.0.1:9200/xc-xerj-storage*/_count' Count the records yourself. The command already did, and refused to switch on zero.
RUN THIS XERJ COMMAND env XERJ_URL=http://127.0.0.1:9200 xerj code xerj-storage 'merge policy segment selection' Query the rebuilt corpus. xerj code addresses the verified build, not the whole namespace.

Why the old --fresh failed

The wrapper scripts used to forward --fresh to xerj autoindex. That flag means something narrower than its name suggests. It discards the resume journal. It never removes records from the node. And it is refused once a durable corpus generation exists.

So the rebuild failed in both states a previously indexed corpus can be in.

A state directory written before the generation-v1 format cannot be adopted:

error: this state directory contains a legacy nonempty plan that cannot become generation authority

A state directory written since refuses the flag outright:

error: this attempt made no remote mutations. `--fresh` cannot discard committed corpus generation 1 under the same destination

Both messages give the same advice: build into a new --state-dir and a new --prefix, validate, then switch readers. A command that owns the whole xc-<corpus> namespace can do exactly that. Since rc.76 it is built in: xerj corpus index.

What --fresh does now

StepWhat happensWhat can go wrong
Buildxerj autoindex runs with prefix xc-<corpus>-b<stamp> and a state directory of its own. --fresh is not forwarded.Nothing an earlier run left behind can be adopted, or can refuse.
Verifyautoindex exited 0 or 3, and _count for the new prefix is above zero.A zero count fails the build, even on exit 0.
Switchstate/<corpus>.json is rewritten by atomic rename.A reader sees the old build or the new one, never half.
RetireThe old indices are deleted by exact name, then their catalog documents and state directory.A crash between switch and retire leaves a duplicate, never a gap.

The old indices are listed before the build starts. The retire step can therefore never include what the run itself created.

A failed build costs nothing

If autoindex aborts, or exits 0 with zero records, the command removes only what that build created. The old indices, the old state file and the old state directory stay exactly as they were, and the command says so:

xerj corpus index: the existing index was NOT touched and is still what `xerj code` serves.

There is one case where a failed build is kept: it wrote records, and there is no working index to fall back to. Throwing it away would leave no corpus at all. autoindex can abort in finalisation after every document was written, and it can also stop part-way through a large corpus, for example when the node keeps rejecting writes. The command cannot tell the two apart, so it treats both the same way. It records the build, stores the real exit code in autoindex_exit with salvaged: true, and prints a warning that coverage is not guaranteed. It never swaps a working index out for a build that failed.

A kept build can be partial, so the reader is told. While the state file says salvaged, or records an exit code other than 0 or 3, xerj code prints this on stderr with every query, and xerj corpus list marks the corpus INCOMPLETE:

WARNING: the index for 'xerj-storage' was NOT verified complete (autoindex exit 1, kept unverified). A miss here is not evidence that the code is absent. Re-run `xerj corpus index xerj-storage` to resume or confirm it.

A count the node does not answer is not zero

Every decision above rests on a record count, and a busy node does not always answer one. A timeout, a 5xx or a refused connection is "did not say". It is not zero. The command asks up to 6 times, 5 seconds apart (XC_COUNT_TRIES, XC_COUNT_PAUSE). Only a number, or a 404 because no index matches, is believed.

When the node never says how many records the existing index holds, the command presumes it is a working index. A failed build is then not kept over it.

When the node never says how many records the new build holds, nothing is deleted and nothing is switched. The build's indices and its state directory are kept. If there is a working index it keeps serving, and the next --fresh that verifies retires the leftover. If there is none, the build is recorded as unverified, xerj code warns as above, and a plain re-run confirms it once the node answers.

An earlier draft of this command read "did not say" as zero. One 503 on one request then retired a working 500-record index in favour of a failed build, and another deleted a finished build together with its resume state. Both are reproduced in the retired script's test file and fail against that draft.

A plain xerj corpus index <corpus> resumes a kept build. The kept build has its own prefix and state directory, so the re-run passes both back to autoindex, which continues from its journal. When that run exits 0 or 3 the salvaged mark is cleared and the warning stops.

Two prefixes in the state file

{"corpus":"xerj-storage","prefix":"xc-xerj-storage","index_prefix":"xc-xerj-storage-b20260918101500", ...}

prefix is always xc-<corpus>. It matches every build's indices, so anything that globs xc-<corpus>* keeps working across a rebuild.

index_prefix is the one build that was verified. xerj code queries it. During a rebuild the namespace holds two builds, and a query against the whole namespace would return passages twice, or from a half-built index.

A state file written before builds existed has no index_prefix. xerj code falls back to prefix for it.

Siblings are safe

xc-battle-* also matches the indices of a corpus named battle-terse. The command knows which corpus names extend this one, from corpora/ and from state/, and leaves their indices out of every list. Deletes are by exact index name. There is no wildcard delete anywhere in the command.

A plain re-run updates in place

xerj corpus index <corpus> without --fresh re-runs autoindex against the recorded build's prefix and state directory. Additions, edits, deletions and renames reconcile incrementally, and the index prefix does not change.

If that run fails with the generation authority message, the corpus predates the generation format. Run --fresh once.

What to plan for

A rebuild holds the corpus on the node twice until the old build is retired. Plan disk for double. This was not measured.

The old state directory of a corpus indexed before builds existed lives under ~/.xerj/autoindex/, in a folder named by a hash. The command cannot compute that name, so it leaves the folder behind. It is harmless, and you can delete it by hand.

How this page was checked

The two error messages are captures from v1.0.0-rc.74, committed under benchmarks/autoindex-resilience/. Local paths in them were shortened; nothing else was changed.

The state-ledger contracts the warning section rests on — the salvaged warning on every query, exit 3 not reported as incomplete, index_prefix preferred over the bare namespace glob, the staleness refusal — are pinned by the tests in engine/crates/xerj-common/src/xccode/state.rs.

The full --fresh swap contract was pinned by 81 offline checks in the retired script suite (tools/xerj-code/tests/test_xc_index_fresh.py, deleted with the scripts): nothing deleted before the replacement's count was read, deletes by exact name, a failed or empty build leaving the old index untouched, sibling corpora never touched, an interrupted first build kept and resumable, unknown counts never read as zero. Those contracts are re-pinned as ten Rust tests in engine/crates/xerj-autoindex/src/xc.rs (issue #1004): the build/verify/swap flow runs unchanged against a fake node and a fake autoindex runner behind two injected seams, so every delete and every state-file switch is asserted on an audit trail rather than on stdout prose. What the old suite shelled out to a PATH-shimmed fake binary, the tests now inject; what it checked over HTTP against a fake node, they check against the same four operations the real node client implements.

Writing the old suite found five defects in drafts of the new command. An index listing that never reached its filter, a record count that read as zero when the node put a space after the colon, two rebuilds inside one second retiring the build they had just verified, and the two unanswered-count defects described above. All five were fixed in the port; the unanswered count was exercised against a fake node only, with a 503. It was not provoked on a real node.

FAQ

How do I rebuild a reference-coding corpus index from scratch?

Run xerj corpus index <corpus> --fresh. It builds a replacement beside the existing index, verifies that it holds records, switches the state file, and only then deletes the old indices by exact name.

xerj corpus index --fresh failed. How do I rebuild a corpus index?

Update the binary. Older releases shipped wrapper scripts that forwarded --fresh to xerj autoindex, which refuses it once a corpus generation has committed. The built-in command never forwards it and rebuilds under a new prefix instead.

Does --fresh delete my working index first?

No. The old index stays live until the replacement exited 0 or 3 and its record count is above zero. A build that fails or comes back empty is removed, and the old index and state file are left exactly as they were.

Why does the state file have both prefix and index_prefix?

prefix is always xc-<corpus>, the whole namespace, so anything that globs on it keeps working. index_prefix is the one verified build, and it is what xerj code queries.

Is xerj corpus index --fresh the same as xerj autoindex --fresh?

No. xerj autoindex --fresh only discards the resume journal, never removes records, and is refused once a durable generation exists. The corpus command's --fresh is a full rebuild under a new prefix and state directory.

What happens if the first build of a corpus is interrupted?

With no working index to fall back to, the command keeps the partial build, records salvaged: true and the real exit code in the state file, and warns. xerj code then warns on every query that coverage is incomplete. A plain xerj corpus index <corpus> resumes the run under the same prefix and state directory.

How much disk does a rebuild need?

Room for the corpus twice. Both builds exist on the node from the start of the rebuild until the old one is retired. This was not measured; plan for double.

Evidence

Related