Troubleshooting

The short list of things that go wrong in production, what they look like, and what to do about them. Every symptom links to the metric, the log line, and the config key you turn.

Symptom: ingest throughput drops after a few hours

Likely cause: merge pressure. Segments are piling up faster than the merger can consolidate them, and the memtable is flushing small segments.

Check:

$ curl -s http://127.0.0.1:8080/v1/metrics | grep -E 'segment_count|merge_duration'
xerj_segment_count{index="logs"} 2847
xerj_merge_duration_seconds_count{index="logs"} 183
xerj_merge_duration_seconds_sum{index="logs"} 1847.2

Fix: raise [storage] flush_size_mb so flushes produce bigger starter segments, and raise [merge] max_merge_count so each pass consumes more of them (watch RSS — peak merge memory scales with it). Set XERJ_MERGE_PARALLELISM above 1 for more concurrent merges on fast NVMe. [merge] max_concurrent and io_rate_mb_per_sec are accepted but not wired in this build, so neither will change anything here.

Symptom: queries time out under load

Likely cause: too many concurrent queries or a single query that grew too large.

Check: look for "query cancelled: max_query_memory_mb exceeded" in the logs or active_searches pegged at max_concurrent_searches.

$ journalctl -u xerj --since "10 min ago" | grep -E 'cancel|timeout|rejected'

Fix: if it's memory — raise [limits] max_query_memory_mb (512 → 1024 or 2048 for aggregation-heavy workloads). If it's concurrency — raise max_concurrent_searches, but check that the host actually has headroom first.

Symptom: node RAM climbs forever

Likely cause: large vector fields, or too many indices with large flush_size_mb.

Check:

$ curl -s http://127.0.0.1:8080/v1/metrics | grep memory_usage
xerj_memory_usage_bytes 14200000000  # 14 GB and climbing

Fix: lower flush_size_mb so memtables don't grow unbounded, and set an explicit [limits] max_process_memory_mb if the automatic cap does not suit the host. Quantization is not a memory fix today: scalar8 (opted into per field with index_options) shrinks the kNN scoring working set, not resident memory (#392), and the node-wide [vector] default_quantization key is not wired.

Symptom: WAL warnings on restart after a crash

Likely cause: the server was killed mid-write (power loss, OOM kill, a full disk), leaving a torn or corrupt WAL file.

Check: the first boot log. XERJ repairs the WAL on its own and never refuses to start over it: a torn tail is truncated (WAL open: healed torn tail (crash/ENOSPC leftover) by truncation), a file with mid-file corruption is frozen and appends move to a fresh generation (WAL open: mid-file corruption detected), and unreadable frames are skipped during replay (skipping corrupt WAL entry during replay).

Fix: nothing to run — there is no manual WAL repair command. Writes in the torn or corrupt region are lost; everything before it is replayed. If these warnings appear without a crash or power loss, check the disk and file an issue.

Symptom: "disk full" during a merge

Likely cause: no reservation for merge scratch space. A merge of two N-sized segments needs 2N free until the merge completes.

Fix: lower [merge] max_segment_mb so individual merges stay smaller, or free disk. Once the merge retries and succeeds, old segments are unlinked.

Symptom: cluster flapping — leader changes every few seconds

Likely cause: network latency between peers is high enough that heartbeats miss. Raft responds by calling a new election.

Check:

$ journalctl -u xerj --since "5 min ago" | grep xerj_cluster
INFO xerj_cluster::raft: Starting election node=10.0.0.12:9300 term=17
INFO xerj_cluster::raft: Higher term observed — stepping down to follower

Fix: raise [cluster] tick_ms from 50 to 150 or 250 — gives heartbeats more room on a slow network. Never drop the tick interval below the RTT between your worst pair of nodes. Sustained churn that survives a tick raise is a defect, not a tuning problem — capture RUST_LOG=xerj_cluster=debug and file it (the leader-death churn class was #1168).

Symptom: a WAL-tap target shows stale data

Likely cause: the tap polls on [wal_tap] poll_interval_ms (default 500) and ships at most max_batch_docs per poll — end-to-end lag is at least one poll interval, more if the target is rate-limiting or was briefly down (the tap retries with backoff and never advances past undelivered entries).

Fix: check GET /_xerj/wal_tap/_stats for gaps and retry backoff, then lower poll_interval_ms or raise max_batch_docs. There are no replicas to be stale on a single node — search reads the active memtable plus segments, so local results are always current.

Getting help

Collect diagnostics before filing a bug. There is no single bundle command; these five read what a report needs into one directory you can attach to the issue. Ports and paths are the defaults from the production guide — adjust them if your config differs:

$ mkdir xerj-report && cd xerj-report
$ export XERJ_API_KEY=$(sudo cat /var/lib/xerj/admin.key)
$ xerj --version > version.txt
$ curl -s http://127.0.0.1:8080/v1/health > health.json
$ curl -s -H "Authorization: ApiKey $XERJ_API_KEY" http://127.0.0.1:8080/v1/metrics > metrics.txt
$ curl -s -H "Authorization: ApiKey $XERJ_API_KEY" http://127.0.0.1:9200/_cluster/health > cluster-health.json
$ journalctl -u xerj --since "1 hour ago" --no-pager > xerj.log

Leave the config file and admin.key out of the report — the key is the node's admin credential. Paste the relevant [section] of the config by hand instead.

Source · engine/crates/xerj-server/src/main.rs · engine/crates/xerj-common/src/metrics.rs