Type: how-to · Last reviewed: 2026-07-18
Applies to: Pulse Historian · edge/on-prem
Goal: run two Pulse Historian nodes as an active-passive pair so the plant keeps recording — and dashboards and alarms stay live — if one node fails.
Prerequisites:
clarity.properties / environment and the ability to restart both.HA and backups solve different problems — run both. HA is off by default; a node with
HA_ENABLEDunset simply boots as a single Leader with all HA gates pass-through.
Set these environment variables (or the matching properties) on each node, then restart it:
| Variable | Value | Notes |
|---|---|---|
HA_ENABLED |
true |
Enables HA mode |
HA_LISTEN_ADDR |
0.0.0.0:3031 |
Inbound peer TCP bind (default) |
HA_PEER_ADDR |
the other node's host:3031 |
Leave empty to discover the peer via mDNS on the LAN |
HA_PEER_NODE_ID |
a label for the peer | For log readability |
On node A set HA_PEER_ADDR to node B, and vice-versa. Full table → HA § Configuration.
Start both nodes. One wins the election and becomes Leader (owns the VIP, serves writes); the other becomes Secondary (replicates, services gated off). Confirm roles:
curl -k https://nodeA:3030/exactapi/ha/status -H "Authorization: Bearer $TOKEN"
{ "role": "Leader", "term": 3, "writable": true }
curl -k https://nodeB:3030/exactapi/ha/status -H "Authorization: Bearer $TOKEN"
{ "role": "SecondaryHealthy", "term": 3, "writable": false }
Reconfigure clients/SDKs to use the VIP, not either node's real IP. On failover the VIP moves to the new Leader, so clients need no reconfiguration. Writes sent to a Secondary are rejected with 503 (leader-write gate) — always target the VIP.
Plain replication can lose the handful of writes in flight the instant the Leader dies. To close that gap, enable the shadow cache and tell each node to mirror writes to its peer:
# clarity.properties (both nodes)
clarity.ha.shadow_cache.enabled=true
clarity.ha.shadow_targets=<peer-static-ip>:3030
On promotion the new Leader drains the buffer and gap-fills the lost writes. Defaults and caps (window_seconds, max_entries, max_mb) → HA § Secondary Shadow Cache. SDK clients configure their own shadow targets and ignore shadow_targets.
GET /exactapi/ha/status shows one Leader and one SecondaryHealthy on the same term.GET /exactapi/ha/peer shows the peer connected; GET /exactapi/ha/lag shows replication lag near zero.GET /exactapi/ha/status on the Secondary flips to Leader and the VIP moves to it; clients on the VIP reconnect and resume. Bring the old node back — it rejoins as Secondary and back-fills via reconciliation.POST /exactapi/ha/ops/switchover instead of killing the Leader.| Symptom | Cause | Fix |
|---|---|---|
| Both nodes think they're Leader (split-brain) | Peer link down (3031 blocked), or both static-addressed wrong | Open TCP 3031; verify HA_PEER_ADDR points each node at the other |
Secondary stuck SecondaryStale |
Large backlog being reconciled | Wait; watch GET /exactapi/ha/lag. Deep catch-up backfills full history on first connect |
Writes to a node return 503 |
That node is the Secondary | Send writes to the VIP, not a node IP |
| Peer never discovered | mDNS blocked on the subnet | Set HA_PEER_ADDR explicitly instead of relying on discovery |
| Gap in data across a failover | Shadow cache not enabled | Enable clarity.ha.shadow_cache.enabled + shadow_targets on both nodes |
status, promote, switchover, fence, …) → HA / Backup / Platform API § High Availability