Clustering

XERJ ships an embedded Raft implementation — no etcd, no ZooKeeper, no Consul. One binary is still one binary; cluster mode is a config switch, not a separate process. Where the ring stands today, plainly: leader election, heartbeats, and the authenticated inter-node transport are real and tested; nothing in the production write path rides on the log yet. Indices, mappings, and documents are node-local — a document indexed on one node is not visible on another until the replication work lands (public tracking: xerj-org/xerj#1170). If you need a second live copy of your data today, use the WAL tap or per-node backups.

What cluster mode does not give you yet

Treat a clustered node's data durability exactly like a single node's.

What you get

• Leader election — exactly one leader per term, randomized 150–300 ms election timeouts, automatic failover. Measured on a 3-node loopback ring (tick 50 ms): a new leader within ~250 ms of killing the leader, then election-stable for minutes. Leader death used to cause perpetual election churn — fixed by #1168.
• Authenticated transport — every inter-node connection is HMAC-SHA256 over a per-connection challenge using the cluster secret. There is no unauthenticated mode; the node refuses to start without a secret.
• Dead-peer isolation — a hung or dead peer costs one WARN on the transition and a retry ladder backing off 100 ms → 6.4 s; it never delays messages to live peers.
• Durable local Raft log — each node persists its term, votes, and committed entries, and replays them on restart.

Bootstrapping a 3-node cluster

Start three nodes, each with its own data dir, and point them at each other. On first startup they hold a Raft election and one becomes leader:

# /etc/xerj/node-a.toml  (run on 10.0.0.11)
[server]
data_dir     = "/var/lib/xerj"
bind_address = "10.0.0.11"

[tls]                                     # required off loopback (or terminate TLS in front
enabled   = true                          # and set server.allow_insecure_network_bind = true)
cert_path = "/etc/xerj/certs/node-a.crt"
key_path  = "/etc/xerj/certs/node-a.key"
allow_insecure_grpc_h2c = true            # gRPC is cleartext h2c; required off loopback with TLS on

[cluster]
enabled     = true
port        = 9300                        # intra-cluster Raft + search port
auth_secret = "<the same secret, 16+ chars, on every node>"   # or XERJ_CLUSTER_AUTH_SECRET
peers       = [
  "10.0.0.11:9300=10.0.0.11:9300",       # this node
  "10.0.0.12:9300=10.0.0.12:9300",
  "10.0.0.13:9300=10.0.0.13:9300",
]
tick_ms = 50

# /etc/xerj/node-b.toml  (run on 10.0.0.12)
[server]
data_dir     = "/var/lib/xerj"
bind_address = "10.0.0.12"

[tls]                                     # required off loopback (or terminate TLS in front
enabled   = true                          # and set server.allow_insecure_network_bind = true)
cert_path = "/etc/xerj/certs/node-b.crt"
key_path  = "/etc/xerj/certs/node-b.key"
allow_insecure_grpc_h2c = true            # gRPC is cleartext h2c; required off loopback with TLS on

[cluster]
enabled     = true
port        = 9300                        # intra-cluster Raft + search port
auth_secret = "<the same secret, 16+ chars, on every node>"   # or XERJ_CLUSTER_AUTH_SECRET
peers       = [
  "10.0.0.11:9300=10.0.0.11:9300",
  "10.0.0.12:9300=10.0.0.12:9300",       # this node
  "10.0.0.13:9300=10.0.0.13:9300",
]
tick_ms = 50

Each peers entry is "<host>:<port>=<host>:<port>": node ids are socket addresses, and the node refuses to start on an entry whose label differs from its address (#1171 — a friendly label made that node's vote replies undeliverable). The current node is the entry whose address matches its own bind_address and cluster.port. Three more startup checks apply to this example and each refuses to boot when unmet: cluster.auth_secret (or XERJ_CLUSTER_AUTH_SECRET) must be set; a non-loopback bind_address needs [tls] or server.allow_insecure_network_bind = true; and with TLS on off loopback, the cleartext gRPC listener needs tls.allow_insecure_grpc_h2c = true. Start all three boxes:

$ xerj --config /etc/xerj/node-a.toml     # on 10.0.0.11
$ xerj --config /etc/xerj/node-b.toml     # on 10.0.0.12
$ xerj --config /etc/xerj/node-c.toml     # on 10.0.0.13

Within a second the ring elects a leader. The honest place to watch it today is the log — the health endpoints report the node answering the request, not the ring's membership (that plumbing is part of the wiring work):

$ journalctl -u xerj -f | grep xerj_cluster
INFO xerj_cluster::transport: TCP transport listening (authenticated) node=10.0.0.11:9300
INFO xerj_cluster::raft: Starting election node=10.0.0.12:9300 term=1
INFO xerj_cluster::raft: Became leader node=10.0.0.12:9300 term=1

$ curl -sH "Authorization: ApiKey $XERJ_API_KEY" \
    http://10.0.0.11:9200/_cluster/health | jq '.status, .number_of_nodes'
"green"
1

Indexes and data are node-local today

Index creation, mappings, and documents do not cross the wire — every node's API serves that node's storage only. There is no shard assignment to other members, no replica set, and no drain/region API; the sections that used to document them were removed because the binary has never shipped them.

For a second live copy of your data, point the WAL tap at another XERJ or any ES-compatible target: it ships indexed writes near-real-time from the WAL, one-directional, without touching the write path.

Failure modes

When not to cluster

Today, almost always. Until the replication wiring lands, cluster mode gives you a leader-election ring — not copies of your data. One node avoids network round-trips, needs no quorum, and is trivial to back up; add a WAL tap to a second node if you want a live spare. The default config ships with [cluster] enabled = false.

Source · engine/crates/xerj-cluster/src/raft.rs · engine/crates/xerj-cluster/src/transport.rs · engine/crates/xerj-cluster/src/runner.rs