03 · OPERATIONS

Upgrades

XERJ follows SemVer. Patch releases (0.1.x) are drop-in replacements. Minor releases (0.x.0) can introduce backward-compatible schema fields. Major releases (x.0.0) may change the on-disk format — when they do, the binary ships a one-shot migration tool.

Single-node upgrade

# 1. Back up (see Backup & restore)
$ curl -sXPOST -H "Authorization: ApiKey $XERJ_API_KEY" \
    http://127.0.0.1:8080/v1/admin/backup \
    -d '{"destination":"s3://my-backups/xerj/pre-upgrade"}'

# 2. Fetch the new release
$ curl -sLO https://xerj.dev/releases/xerj-0.2.0-linux-x86_64.tar.gz
$ tar xzf xerj-0.2.0-linux-x86_64.tar.gz

# 3. Replace the binary in place
$ sudo systemctl stop xerj
$ sudo install -m 0755 xerj /usr/local/bin/xerj
$ sudo systemctl start xerj

# 4. Confirm
$ curl -s http://127.0.0.1:8080/v1/health | jq .version
"0.2.0"

Downtime is the time it takes the WAL to replay — usually a few seconds for workloads in the low millions of live docs.

Rolling cluster upgrade

Upgrade one node at a time. The cluster tolerates floor((N-1)/2) unavailable nodes, so a 3-node cluster can lose one, a 5-node cluster can lose two.

# For each node, in order:
$ curl -sXPOST .../v1/cluster/nodes/<id>/drain
# Wait for "shards_remaining": 0 (30s-5min depending on data size)
$ ssh node-<id> sudo systemctl stop xerj
$ ssh node-<id> sudo install -m 0755 xerj-new /usr/local/bin/xerj
$ ssh node-<id> sudo systemctl start xerj
# Wait for the node to return to "active" in cluster health
$ curl -sf .../v1/cluster/health | jq '.nodes[] | select(.id=="<id>")'
$ curl -sXPOST .../v1/cluster/nodes/<id>/activate

Mixed-version clusters are supported within one minor — you can run 0.1.4 and 0.1.5 side by side during a rolling upgrade. Across minors, upgrade every node within the same maintenance window.

On-disk format migrations

A major release that changes the segment format ships a migrate subcommand. Run it after stopping the old binary and before starting the new one:

$ sudo systemctl stop xerj
$ xerj migrate --from 1.x --to 2.x --data-dir /var/lib/xerj
Scanning segments... 842 found
Rewriting segment 1/842: logs/seg-000123 ... ok
...
Migration complete. Verify with: xerj verify --data-dir /var/lib/xerj
$ sudo systemctl start xerj

The migrator is idempotent — if it crashes halfway, re-run it. It never touches source segments until the replacement is fully written and fsynced.

Rolling back

Patch releases roll back by reinstalling the previous binary. Across a format migration, rolling back requires the pre-migration backup — there is no inverse migrator.

Source · engine/crates/server/src/main.rs