Clarity implements an active-passive HA cluster for on-premise deployments. Two nodes share a Virtual IP (VIP); the Leader owns the VIP and handles all writes. The Secondary continuously replicates data and can be promoted if the Leader fails.
When HA_ENABLED is not set, a single node boots directly as Leader and all HA gates are pass-through — single-node behaviour is unchanged.
Source module: clarity:backend/src-tauri/src/ha/
clarity:backend/src-tauri/src/ha/mod.rs:1-17
| Variable | Default | Description |
|---|---|---|
HA_ENABLED |
(unset / disabled) | Set to true to enable HA mode |
HA_NODE_ID |
hostname | Unique identifier for this node |
HA_LISTEN_ADDR |
0.0.0.0:3031 |
Inbound peer TCP bind address |
HA_PEER_ADDR |
(empty) | Static address of peer; if empty, peer is discovered via mDNS |
HA_PEER_NODE_ID |
(empty) | Logical peer identifier for log messages |
Six roles defined in clarity:backend/src-tauri/src/ha/role.rs:
| Role | u8 | Writable | Singletons | Promotable |
|---|---|---|---|---|
Leader |
0 | Yes | Yes | — |
SecondaryHealthy |
1 | No | No | Yes |
SecondaryStale |
2 | No | No | No |
Recovering |
3 | No | No | No |
Promoting |
4 | No | No | — |
Fenced |
5 | No | No | No |
Startup safety rule: When HA is enabled, a node always boots as Recovering. It must complete the full resync path before it can serve as Secondary or be promoted.
Single-node mode: HA_ENABLED not set → node boots directly as Leader.
Legal transitions (from ha-approach.txt §3):
Recovering → SecondaryHealthy (resync complete)
Recovering → SecondaryStale (resync partial)
Recovering → Fenced (stale term / admin action)
SecondaryHealthy → Promoting (peer presumed dead)
SecondaryHealthy → SecondaryStale (lag exceeds threshold)
Promoting → Leader (promotion committed)
Promoting → SecondaryHealthy (promotion lost race)
Leader → SecondaryHealthy (demoted by higher term)
Leader → Fenced (admin action)
clarity:backend/src-tauri/src/ha/role.rs:54-end
HaStateclarity:backend/src-tauri/src/ha/state.rs
Thread-safe via atomics. Holds:
role: AtomicU8 — current roleterm: AtomicU64 — monotonically increasing leadership termha_enabled: bool — whether HA is activenode_id: String — this node's unique IDleader_node_id: RwLock<Option<String>> — current leader's IDfenced: AtomicBool — fencing flaglast_tsdb_sync_seq: AtomicU64 — TSDB replication sequence counterlast_sqlite_snapshot_id: AtomicU64 — SQLite snapshot IDevents: RwLock<VecDeque<HaEvent>> — bounded ring of recent HA eventslast_promotion_at / last_demotion_at: AtomicU64 — epoch-ms timestampscontrol_store: ControlStore — durable ha_state.json persistenceclarity:backend/src-tauri/src/ha/state.rs:23-51
clarity:backend/src-tauri/src/ha/peer_connection.rs
Dual TCP transport between two nodes:
Peer discovery: if HA_PEER_ADDR is set, used directly. Otherwise, mDNS discovery (clarity:backend/src-tauri/src/ha/mdns_discovery.rs) finds the peer by its node ID on the LAN.
As of c41a03b, inbound frames are broadcast as Arc<InboundFrame> (subscribers clone a pointer, not the payload bytes; consumer on_frame signatures take &InboundFrame), and the connection tracks a conn_generation: AtomicU64 incremented on every control-plane (re)establishment — the invalidation key for the digest-skip caches in both replicators. clarity:backend/src-tauri/src/ha/peer_connection.rs:76-79,262,286,465-467,713
Firewall helper (clarity:backend/src-tauri/src/ha/firewall.rs) opens inbound rules for HA control port and data port (control + 1) on Windows Defender. No-op on macOS/Linux. As of a18d35c it invokes netsh.exe by absolute path (C:\Windows\System32\netsh.exe) and returns an error if the binary is missing, instead of relying on PATH. In c41a03b, ensure_inbound_allowed gained a protocol parameter (threaded into netsh) enabling a UDP/5353 rule for mDNS discovery, and spawns hide their console windows. clarity:backend/src-tauri/src/ha/firewall.rs:41,83,123
clarity:backend/src-tauri/src/ha/tsdb_replicator.rs
Every successful Storage::flush_sparse_writes on the Leader emits a TsdbDelta to all registered DeltaListener sinks. The TsdbDeltaPublisher forwards the delta to the Secondary over the peer data plane.
Wire format (ha-approach.txt §7.2):
offset size field
0 8 seq u64
8 8 day_midnight_sec u64 (file stem)
16 2 path_len u16
18 path_len collection_path ("org/site/unit/grid")
.. 2 interval_ms hint u16
.. 4 pair_count u32
.. pair_count × { u16 tag_index, u32 grid_index, i32 scaled_value }
Bandwidth estimate: ~6 bytes per (tag_index, scaled_i32) pair + header → ~6 KB/s at 1000 tags × 1 Hz.
V2 storage-format awareness (new in c41a03b). Delta values carry the disk encoding, so the applier writes bytes verbatim; the follower resolves the collection's codec from metadata (disk_sentinel() / disk_xor()) when it must interpret values. Fill-on-grow is now conditional: V2 collections (zero disk-sentinel) skip the sentinel fill entirely — set_len's zeroed bytes already mean "missing" — while V1 still writes the sentinel pattern. Followers also keep the lastlist cache warm by XOR-decoding delta values back to memory encoding (update_last_values_replicated). A late delta targeting an already-sealed day now unseals the replica day first (restoring the hot .bin) before applying, so cold-tier writes don't leave a stale sealed file shadowing fresh data. clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:687-745 — see Storage Engine § On-disk format V2.
In-memory ring of recent deltas. Allows a reconnecting Secondary to replay a short gap without a full file copy.
DEFAULT_RING_CAPACITY defined in tsdb_replicator.rs.
A day rolls into the cold tier when it is sealed (numeric {day}.bin → {day}.bin.czst; blob heaps {day}.blob/.bref → .blob.zst/.bref.czst). On a timer the Secondary reconciles whole day units against the Leader to catch historic backfill writes and corruption.
Reconciliation works on a unit model — ReplicaDayKind { Numeric, BlobPair } (clarity:backend/src-tauri/src/api/storage/mod.rs:171-260). A Numeric unit is the {day}.bin/.bin.czst pair; a BlobPair is the {day}.bref(+.czst) ref index together with the {day}.blob(+.zst) heap — valid only as a set from the same snapshot, because refs are byte offsets into the heap.
Digests are computed over decompressed logical bytes, not the on-disk file, so a hot day and its sealed .czst/.zst counterpart hash identically — seal-state skew between the two nodes no longer produces a false mismatch (compute_unit_digest, clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:853-876; rationale at :1183-1190). When a unit diverges, the Leader ships logical bytes via read_replica_day (hot-preferred, else unseals the cold file — clarity:backend/src-tauri/src/api/storage/ha.rs:422-494) and the Secondary installs them as hot files with install_replicated_day, deleting any stale sealed counterpart so it can't resurrect (clarity:backend/src-tauri/src/api/storage/ha.rs:495-593). The candidate set the reconciler enumerates includes .bin.czst, .bref.czst, and .blob.zst (clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:1525-1536).
Scan modes (clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:1380-1507):
DayListRequest/DayListResponse wire messages), unions with local days, and reconciles the entire archive — so a fresh or long-dead Secondary backfills complete history.clarity.ha.reconcile_days window (default 14 days, walking back from today).Divergence detection is memoized: an unchanged unit's digest is cached by its (mtime, size) fingerprint and recomputed only when the file changes or the connection generation bumps (unit_digest_cached, clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:1509-1579).
NOTE: This supersedes the earlier "reconciliation is inert once a day seals" limitation flagged at
c41a03b. As of45a686e(commit6061ec1, "seal/blob-aware replication") sealed cold-tier days are decompressed, checksummed, and repaired between HA peers; the two tiers can no longer diverge undetected via this path.clarity.storage.seal.hot_daysstill defaults to0(every historic day sealed), but those sealed days are now covered by the deep/rolling reconcile above.
Components:
TsdbDeltaPublisher — producer on LeaderTsdbDeltaApplier — consumer on SecondarySealedReconciler — scan/reconcile driver on Secondary; both sides handle framesRecoveryCoordinator — Leader answers recovery requests with ring replay or resync planRecoveryInitiator — Secondary sends recovery request on first peer connectCollectionCreateReplicator ships create_collection events and tag-list extensions from Leader to Secondary. Also bootstraps all existing collections on peer link-up.
Changes in c41a03b: CollectionCreate frames carry a trailing [u32 storage_format] field (decoded as optional, defaulting to 0 = legacy V1, so pre-V2 peers interoperate) — clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:335-366,437-441. Metadata reconcile now digest-skips: a per-path (blake3, conn_generation) cache avoids re-shipping unchanged CollectionCreate frames to the same peer session; a reconnect forces a full pass. clarity:backend/src-tauri/src/ha/tsdb_replicator.rs:554-600
clarity:backend/src-tauri/src/ha/shadow_cache.rs
During a failover the VIP is briefly unowned, so a write issued in that gap would be lost. The shadow cache closes that gap: clients (and optionally the Leader itself) mirror every write to the Secondary's static IP, where it is buffered in memory; on promotion the new Leader drains the buffer and backfills any timestamps it is missing. Disabled by default.
Endpoint — POST /exactapi/shadow_write (clarity:backend/src-tauri/src/main.rs:4064-4077, handler clarity:backend/src-tauri/src/main.rs:1344-1354). It requires a bearer token but is not behind the leader-write gate — it is mounted in read_routes precisely so a Secondary can accept it. On the Leader (or when the feature is disabled) the handler is a no-op; on a Secondary it appends the raw request body verbatim to an in-memory Mutex<VecDeque<Entry>> (clarity:backend/src-tauri/src/ha/shadow_cache.rs:36-58,111-127), bounded by a time window + entry cap + byte cap, evicting the oldest entry on each push (clarity:backend/src-tauri/src/ha/shadow_cache.rs:160-183).
Two independent feeders (both dual-write to the peer's static IP; neither blocks the primary write):
/api/shadow_write, but the server serves only /exactapi/shadow_write; this path mismatch is tracked on the JS SDK page.clarity.ha.shadow_targets is set, store_datapoints_write_fast spawns a best-effort, 1 s-timeout mirror of each batch to POST {target}/exactapi/shadow_write (clarity:backend/src-tauri/src/processing_api/ingest.rs:32-43,1124-1148).Drain on promotion / clear on demotion. ShadowDrainSubsystem is a gated subsystem (clarity:backend/src-tauri/src/main.rs:480-564, registered at :3588-3592). On promotion → Leader it drains every buffered body, parses each as a normal write, and applies it via Storage::flush_sparse_writes_fill_missing — which fills only currently-missing timestamps and never overwrites a value the dead Leader already persisted (idempotent gap-fill). On demotion it clears the cache.
Configuration (clarity:backend/src-tauri/src/config.rs:340-349,484-488,758-769; properties block clarity:backend/src-tauri/clarity.properties:515-547):
| Key | Default | Effect |
|---|---|---|
clarity.ha.shadow_cache.enabled |
false |
Master switch for the Secondary buffer |
clarity.ha.shadow_cache.window_seconds |
60 |
Max age of buffered entries |
clarity.ha.shadow_cache.max_entries |
2000000 |
Entry-count cap |
clarity.ha.shadow_cache.max_mb |
512 |
Byte-size cap |
clarity.ha.shadow_targets |
`` (empty) | Peer static IP(s) the server-side forwarder mirrors to (SDK clients configure their own targets and ignore this key) |
clarity:backend/src-tauri/src/ha/sqlite_replicator.rs
Periodically sends snapshots of pulse-db.sqlite and process_registry.db from Leader to Secondary. Configured via SqliteReplicatorConfig:
pulse_db_filename: "pulse-db.sqlite"interval_secs: DEFAULT_SNAPSHOT_INTERVAL_SECSSqliteSnapshotProducer (Leader) → SqliteSnapshotConsumer (Secondary).
Digest-skip (new in c41a03b): the producer keeps last_shipped: HashMap<DbSlot, ([u8;32], u64)> (blake3 + connection generation); ship_file returns Ok(false) and skips the transfer when identical bytes were already shipped this peer session — snapshot id/log advance only on a real ship. clarity:backend/src-tauri/src/ha/sqlite_replicator.rs:246,355-362,372-390,441-445
When a new snapshot is applied, any existing HA process instance row may be wiped. The main thread log consumer detects this via instance_exists() and re-registers before writing the next log entry.
clarity:backend/src-tauri/src/main.rs:2379-2413
clarity:backend/src-tauri/src/ha/vip_manager.rs
Manages a floating VIP (Virtual IP) so clarity.local always resolves to the current Leader. On promotion, the Leader acquires the VIP. On demotion or shutdown, the VIP is released.
When HA_ENABLED=true and HA_VIP_ADDR is configured, mDNS advertises the VIP address instead of the node's physical IP.
Configuration via VipConfig::from_config() reads clarity.properties.
On Windows, the VIP manager invokes all system binaries by absolute path (net.exe for the elevation check, netsh.exe for address add/remove, powershell.exe for the ARP-clear step, ping.exe -S <vip> for the gratuitous-ARP announce) and fails/skips with a logged error if any is missing — a18d35c hardening consistent with the rest of the codebase. clarity:backend/src-tauri/src/ha/vip_manager.rs:358-545
As of 45a686e, the gratuitous-ARP announce sweeps the whole subnet: build_announce_ping pings up to MAX_SWEEP_HOSTS addresses (subnet_hosts / sweep_hosts) unioned with any configured announce_targets, so stale ARP entries still pointing clients at the dead Leader are flushed faster (tightening the window the shadow cache exists to cover). On Windows these pings use absolute ping.exe with a hidden console. clarity:backend/src-tauri/src/ha/vip_manager.rs:510-519
verify_owned)Two additions harden the in-process VIP owner against ghost/stripped addresses:
store=active (non-persistent). The VIP is added with store=active so it is not written to the Windows registry — a node that died unclean while Leader does not come back up owning a ghost VIP. clarity:backend/src-tauri/src/ha/vip_manager.rs:1136-1150verify_owned() self-heal. A new trait method (default = acquire()) with a Windows override that bypasses the cached fast-path, probes the OS directly, and re-acquires + re-announces if an external event (adapter reset, netsh int ip reset, driver reload) stripped a still-running Leader's VIP. The Leader tick calls it every ~30 ticks (~3 s), acquire() otherwise. clarity:backend/src-tauri/src/ha/vip_manager.rs:90-100, 1241-1256, clarity:backend/src-tauri/src/ha/agent.rs:269-284windows_vip_is_present / windows_vip_delete reuse the plumbing without side effects (also used by the guard below). clarity:backend/src-tauri/src/ha/vip_manager.rs:965-996vip_guard.rs)clarity:backend/src-tauri/src/ha/vip_guard.rs
The in-process VIP release (Ctrl+C, Tauri ExitRequested, Drop) does not run on a hard kill — taskkill /f, End Task, power loss, watchdog termination — leaving a dead node's NIC still answering ARP for the VIP and producing a split-brain / duplicate floating IP once the Secondary is promoted. clarity:backend/src-tauri/src/ha/vip_guard.rs:1-23
The guard is a Windows out-of-process SYSTEM sidecar that enforces one rule: if the Clarity app is not alive on this machine, this machine must not hold the VIP.
spawn_heartbeat_writer) that writes unix-millis + pid every 1 s to %ProgramData%\Clarity\ha_vip_guard.heartbeat (a machine-wide path). clarity:backend/src-tauri/src/ha/vip_guard.rs:63-106run_guard) polls every 1 s; if the heartbeat is older than STALE_AFTER = 3s (three missed beats) and the VIP is present, it strips the VIP via windows_vip_delete. Safe because a VIP is only ever legitimately held by a live Leader. clarity:backend/src-tauri/src/ha/vip_guard.rs:51, 144-188ClarityHaVipGuard (/sc onstart /ru SYSTEM /rl HIGHEST) via the CLI subcommand ha-vip-guard, and also started immediately. Windows-only; non-Windows are stubs. clarity:backend/src-tauri/src/ha/vip_guard.rs:204-257Distinct from vip_manager (the in-process, role-aware owner): the guard is a minimal, role-agnostic liveness enforcer running as a separate process, reusing only the thin netsh helpers.
clarity:backend/src-tauri/src/ha/singleton_gate.rs
Ensures Leader-only subsystems (MQTT client, monitoring agent, Python services) start only on the Leader and shut down on demotion. Components register themselves via gate.register().
MqttSubsystem is registered at startup. Additional subsystems (PI driver, monitoring) are added at their respective call sites.
clarity:backend/src-tauri/src/main.rs:2679
clarity:backend/src-tauri/src/ha/agent.rs
Owns the role state machine and drives promotion/demotion. Starts with HaAgent::start(). As of c41a03b the tick loop is supervised: a panic no longer permanently disables promotion — the loop re-enters after a capped backoff (2 × min(n,5) seconds, tracking consecutive_panics). clarity:backend/src-tauri/src/ha/agent.rs:145-171
Responsibilities:
clarity:backend/src-tauri/src/main.rs:2679-2686
clarity:backend/src-tauri/src/main.rs:2500-2516
When HA is enabled, a WAL file (write_buffer.wal) is created at data_path. The WriteBuffer writes entries to the WAL before placing them in the in-memory channel. This ensures crash durability: a recovering node can replay the WAL on restart.
In single-node mode, wal_path = None and the WAL is skipped.
clarity:backend/src-tauri/src/ha/api.rs
All 13 endpoints are mounted at /exactapi/ha/* and remain reachable regardless of role — operators need visibility even on non-leader nodes. Read-only endpoints (status, events, health, peer, lag, ops/recovery_status) are open; the mutating endpoints (promote, demote, fence, unfence, ops/switchover, ops/force_promote, ops/force_fence) are now bearer-token gated via require_auth().
clarity:backend/src-tauri/src/ha/api.rs:5-9,18-28,53-72
| Method | Path | Description |
|---|---|---|
GET |
/exactapi/ha/status |
Current role, term, node ID, and full HaState snapshot |
POST |
/exactapi/ha/promote |
Manual promotion: SecondaryHealthy → Promoting → Leader; 409 if wrong role |
POST |
/exactapi/ha/demote |
Manual demotion: Leader → Recovering; 409 if invalid transition |
POST |
/exactapi/ha/fence |
Set fenced flag; node stops accepting writes regardless of role |
POST |
/exactapi/ha/unfence |
Clear fenced flag |
GET |
/exactapi/ha/events |
Last 100 HA events (role transitions, promotions, fences) |
GET |
/exactapi/ha/health |
Load-balancer probe: 200 if writable (Leader), 503 otherwise |
GET |
/exactapi/ha/peer |
Peer connection stats: connected, missed_heartbeats, last_seen_ms_ago, peer_term, peer_role, frames_in/out |
GET |
/exactapi/ha/lag |
Replication lag: tsdb_sync_seq, sqlite_snapshot_id, peer_term, peer_connected |
POST |
/exactapi/ha/ops/switchover |
Coordinated graceful switchover (requires active peer connection) |
POST |
/exactapi/ha/ops/force_promote |
Force-promote without peer coordination |
POST |
/exactapi/ha/ops/force_fence |
Force-fence without peer coordination |
GET |
/exactapi/ha/ops/recovery_status |
Recovery progress (requires active peer connection) |
Wired in clarity:backend/src-tauri/src/main.rs:
1. init_config() — load clarity.properties
2. HaState::new() — start as Recovering
3. Storage::new() — binary TSDB storage
4. WriteBuffer::new_with_wal() — with WAL at write_buffer.wal
5. PeerConnection::start() — dual TCP, mDNS or static peer addr
6. TsdbDeltaPublisher.register() — Leader emits deltas to peer
7. TsdbDeltaApplier::start() — Secondary applies incoming deltas
8. SealedReconciler::start() — sealed file integrity check/copy
9. RecoveryCoordinator::start() — Leader handles recovery requests
10. RecoveryInitiator::start() — Secondary sends recovery request
11. CollectionCreateReplicator::start() — metadata replication
12. SqliteSnapshotProducer::start() — SQLite Leader → Secondary
13. SqliteSnapshotConsumer::start() — SQLite apply on Secondary
14. TagScopeWarmer::start() — keeps TAG_SCOPE_MAP warm
15. VipManager::make_vip_manager() — VIP control
16. SingletonGate::new() — Leader-only service gating
17. ShadowCache::from_config() — Secondary write buffer for zero-loss failover
18. ShadowDrainSubsystem.register() — drains/gap-fills the buffer on promotion
19. HaAgent::start() — role state machine + heartbeat
Shadow-cache init at clarity:backend/src-tauri/src/main.rs:3373-3377; drain-subsystem registration at clarity:backend/src-tauri/src/main.rs:3588-3592. (Other steps are wired across main.rs boot; individual line anchors shifted with the 45a686e boot reordering, which also moved build_tag_map off the critical path — each warp::serve now awaits wait_tag_map_ready() first.)
Last updated: 2026-07-17 from commit 6800acc