📘 New here? Start with the plain-language explainer How monitoring & alarms work — this page is the reference it links down to.
Real-time rule evaluation against live time-series data with alarm event lifecycle management. Added February 2026.
Source directory: clarity:backend/src-tauri/src/monitor/
| File | Purpose |
|---|---|
types.rs |
Core type definitions: RuleDefinition, AlarmSnapshot, AlarmEvent, ComparisonOp, MissingBehavior, WindowData, etc. |
config.rs |
ConfigManager — rule and collection CRUD in SQLite; init_schema bootstraps three monitor tables |
rules.rs |
Four RuleEvaluator implementations: AllOfNRule, KOfNRule, PercentageRule, GOATRule; RuleDefinition::to_evaluator() dispatch |
reader.rs |
DataReader — reads time-series windows from binary .bin files via Storage's bounded LRU mmap cache |
evaluator.rs |
Evaluator::evaluate_collection — Rayon parallel evaluation across tags within a collection |
agent.rs |
MonitoringAgent — async background loop; per-collection tick scheduling, shutdown signaling, snapshot merge |
snapshot.rs |
SnapshotStore — in-memory alarm state backed by parking_lot::RwLock |
events.rs |
AlarmEventTracker — OPEN/CLOSED alarm lifecycle; persists to alarm_events and mirrors to deviations tables |
api.rs |
Warp route definitions for all 20 endpoints under /exactapi/monitor/ |
auto_rules.rs |
Automatic AllOfN rule creation triggered by tagmeta POST hooks (limLo/limHi limits) |
tests.rs |
Test module — currently empty (0 tests) |
NOTE:
tests.rsexists atclarity:backend/src-tauri/src/monitor/tests.rsbut contains 0 bytes. The claim of "57 unit tests" inARCHITECTURE.md(section 7) is incorrect — file is present but empty.
NOTE:
auto_rules.rsis a module not listed in the oldARCHITECTURE.mdsection 7's "10 submodules" count. The actual submodule count is 11.
All types defined in clarity:backend/src-tauri/src/monitor/types.rs.
ComparisonOpclarity:backend/src-tauri/src/monitor/types.rs:14-21
GreaterThan (default)
GreaterThanOrEqual
LessThan
LessThanOrEqual
Equal
NotEqual
check(value: f32, threshold: f32) -> bool performs the comparison. Equal/NotEqual use f32::EPSILON for float comparison.
MissingBehaviorclarity:backend/src-tauri/src/monitor/types.rs:44-55
| Variant | Behaviour when a data point is None |
|---|---|
FailOpen |
Counts the point as present but failing (triggers alarm on missing data) |
Skip (default) |
Ignores missing points entirely |
RequireMinPresent |
Like Skip but gates on missing_threshold minimum present points |
RuleDefinitionclarity:backend/src-tauri/src/monitor/types.rs:59-77
| Field | Type | Default | Notes |
|---|---|---|---|
id |
String | auto UUID | Assigned by ConfigManager::create_rule |
collection_id |
String | — | org/site/unit/grid format |
tag_index |
usize | — | Index into collection's tag array |
tag_name |
String | — | Denormalized for display |
name |
String | — | Human-readable rule name |
description |
String | "" |
— |
enabled |
bool | true |
Disabled rules are skipped by evaluator |
rule_type |
String | — | AllOfN / KOfN / Percentage / GOAT |
rule_config |
JSON | — | Type-specific config object (see schema below) |
window_duration_ms |
u64 | 10000 |
Evaluation window in milliseconds |
missing_behavior |
MissingBehavior | Skip |
How to treat absent data points |
missing_threshold |
Option<usize> | None |
Min points required (KOfN/Percentage) |
unitsId |
Option<i64> | None |
Owning unit id; persisted to the monitor_rules.unitsId column (added by migration). clarity:backend/src-tauri/src/monitor/types.rs:78 |
created_at |
i64 | auto | Unix ms |
updated_at |
i64 | auto | Unix ms |
AlarmSnapshotclarity:backend/src-tauri/src/monitor/types.rs:147-166
In-memory state written after every agent tick. Fields: timestamp_ms, collections: Vec<CollectionAlarmState>, tick_number, evaluation_duration_ms, total_triggered_tags.
AlarmEventclarity:backend/src-tauri/src/monitor/types.rs:216-257
Persisted record for each alarm transition. Key fields:
| Field | Type | Notes |
|---|---|---|
event_id |
String | UUID v4 |
status |
AlarmEventStatus |
OPEN or CLOSED |
opened_at_ms |
u64 | Unix ms when alarm opened |
closed_at_ms |
Option<u64> | Set on close |
duration_ms |
Option<u64> | closed_at_ms - opened_at_ms |
trigger_value |
f32 | Value that triggered the alarm |
clear_value |
Option<f32> | Last value before close |
peak_value |
f32 | Peak value during open period (updated on each tick) |
threshold |
f32 | Threshold from rule config |
comparison |
String | Serialised ComparisonOp |
metadata |
AlarmEventMetadata |
window_size, pass/present counts, rule_name, reason |
clarity:backend/src-tauri/src/monitor/snapshot.rs
pub struct SnapshotStore {
current_snapshot: Arc<RwLock<Arc<AlarmSnapshot>>>, // parking_lot::RwLock; Arc'd snapshot as of c41a03b
last_tick_number: Arc<AtomicU64>,
last_update_ms: Arc<AtomicU64>,
}
parking_lot::RwLock — confirmed at snapshot.rs:1 (use parking_lot::RwLock)c41a03b, readers share the snapshot instead of deep-cloning it: read() returns Arc<AlarmSnapshot> via Arc::clone and write() takes an Arc<AlarmSnapshot> (snapshot.rs:8,16,24-31); the agent builds Arc::new(AlarmSnapshot{…}) per tick (agent.rs:191-199) and the one affected API call site derefs (api.rs:155). Snapshot fields/semantics unchanged.AtomicU64 fields allow status() (tick + last_update) without acquiring the RwLockclarity:backend/src-tauri/src/monitor/events.rs
Manages OPEN/CLOSED alarm lifecycle. Two internal stores:
| Store | Type | Purpose |
|---|---|---|
active_alarms |
parking_lot::RwLock<HashMap<(collection_id, tag_index, rule_id), ActiveAlarm>> |
In-memory active alarms; used for O(1) transition detection |
db |
parking_lot::Mutex<Connection> |
SQLite persistence |
Event lifecycle:
| Transition | Action |
|---|---|
(false → true) |
Creates AlarmEvent with status OPEN; inserts row into alarm_events; inserts mirror row into deviations table; back-fills the new deviation_id + incident_id onto the alarm_events row |
(true → true) |
Updates peak_value in DB and deviations if current value is larger |
(true → false) |
Sets status CLOSED; sets closed_at_ms, duration_ms, clear_value; updates deviations mirror |
(false → false) |
No-op |
SQLite tables used:
alarm_events — primary event store (see schema in ConfigManager section)deviations — mirror table for cross-system queries (same SQLite DB as main app data)Linked record IDs (deviation_id / incident_id):
On open, after inserting the deviations (and incident) rows, the tracker issues UPDATE alarm_events SET deviation_id = ?, incident_id = ? WHERE event_id = ? so the linked record IDs are persisted on the alarm row. This means they survive a restart and can be used to close the linked records later. clarity:backend/src-tauri/src/monitor/events.rs:462-475
Alarm restore on restart:
load_active_from_db() queries alarm_events WHERE status = 'OPEN', now also selecting deviation_id and incident_id, and repopulates the active_alarms map with those IDs (previously always restored as None). clarity:backend/src-tauri/src/monitor/events.rs:45-85, clarity:backend/src-tauri/src/monitor/agent.rs:49-58
Force-close on threshold change:
close_open_alarms_for_rule(rule_id, now_ms) -> Result<u32, String> (added a18d35c) removes every active alarm whose key's rule_id matches, persists a CLOSED AlarmEvent for each (computing duration_ms, preserving peak_value, clear_value = None), and returns the count closed. Called by auto_rules when an auto-rule's threshold changes via a tagmeta PUT (see Threshold-change alarm close). clarity:backend/src-tauri/src/monitor/events.rs:863-925
Orphan sweep (
notification-enginebranch). After force-closing the per-event matches,close_open_alarms_for_rulenow always runs a blanketUPDATE alarm_events SET status='CLOSED' … WHERE rule_id=? AND status='OPEN'to catch DB rows left OPEN with no in-memory entry (the open-path race where the in-memory insert precedes the DB insert). The earlyif matching.is_empty() { return }guard was removed so the sweep always runs; the method returnsclosed + swept.clarity:backend/src-tauri/src/monitor/events.rs:908-925
Equipment / system enrichment on open (new in 67ac68c):
When opening an alarm, the tracker now looks up the triggering tag's tagmeta row (SELECT equipmentId, meta_data FROM tagmeta WHERE dataTagId = ?1 LIMIT 1) and uses it to populate the deviation/incident JSON fields that were previously always empty arrays: systems, systemName, equipments, equipmentName, and equipmentIds.
meta_data blob is read tolerantly (BLOB first, falling back to UTF-8 TEXT, mirroring read_meta_data_bytes), then parsed for equipmentName / systemName / equipmentId.equipmentId falls back to the blob value (parsed as an i64, accepting a string like "21") when the real equipmentId INTEGER column is NULL — which happens when the tag was loaded via meta_upload (which stores the id only as a string inside the blob). The id is emitted as a string in equipmentIds.[] (unchanged) rather than a [""] entry; a missing tagmeta row leaves all five fields empty.clarity:backend/src-tauri/src/monitor/events.rs:388-481
clarity:backend/src-tauri/src/monitor/config.rs
Wraps a std::sync::Mutex<Connection> (not parking_lot). Provides CRUD for three tables:
monitor_system_config — key-value store for global settings (clarity:backend/src-tauri/src/monitor/config.rs:13-17)
CREATE TABLE monitor_system_config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now') * 1000)
);
Default rows: enabled = 'true', max_evaluation_time_ms = '5000'
monitor_collections — per-collection monitoring config (clarity:backend/src-tauri/src/monitor/config.rs:19-25)
CREATE TABLE monitor_collections (
id TEXT PRIMARY KEY,
enabled INTEGER NOT NULL DEFAULT 1,
tick_rate_ms INTEGER NOT NULL DEFAULT 1000,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now') * 1000),
description TEXT DEFAULT ''
);
monitor_rules — persisted rule definitions (clarity:backend/src-tauri/src/monitor/config.rs:27-44)
CREATE TABLE monitor_rules (
id TEXT PRIMARY KEY,
collection_id TEXT NOT NULL,
tag_index INTEGER NOT NULL,
tag_name TEXT NOT NULL,
name TEXT NOT NULL,
description TEXT DEFAULT '',
enabled INTEGER NOT NULL DEFAULT 1,
rule_type TEXT NOT NULL,
rule_config TEXT NOT NULL,
window_duration_ms INTEGER NOT NULL DEFAULT 10000,
missing_behavior TEXT NOT NULL DEFAULT 'Skip',
missing_threshold INTEGER,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now') * 1000),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now') * 1000),
FOREIGN KEY(collection_id) REFERENCES monitor_collections(id),
UNIQUE(collection_id, tag_index, name)
);
Plus indexes on alarm_events: opened_at_ms, status, (collection_id, tag_index), rule_id.
Migrations (idempotent ALTER TABLE … ADD COLUMN, ignored if already present): monitor_rules.unitsId INTEGER, and on alarm_events: deviation_id INTEGER, incident_id INTEGER. clarity:backend/src-tauri/src/monitor/config.rs:73-76
delete_collection_cascade(collection_id)New in 67ac68c. Deletes all monitor data tied to a collection_id in one transaction (BEGIN IMMEDIATE … COMMIT, rollback on any error), in FK order: alarm_events → monitor_rules → monitor_collections. Returns (alarm_events_deleted, rules_deleted, collections_deleted). clarity:backend/src-tauri/src/monitor/config.rs:413-451
NOTE: The live
DELETE /exactapi/collections/:idflow does not call this method — it inlines the same three deletes into its own larger transaction (so the monitor cleanup shares one commit with theconnections/tagmetacleanup).ConfigManageropens a separate connection with its ownBEGIN/COMMIT, which would defeat that atomicity. See SQLite API § Collections route surface.
clarity:backend/src-tauri/src/monitor/reader.rs
Reads time-series windows from the binary storage layer.
NOTE: Rewritten — the old hand-rolled per-day mmap iteration (XOR codec,
SCALE_FACTOR, its own rayon fan-out) was deleted.read_windownow delegates toStorage::read_data_ultra_fast— the canonical merged read path (day.bin+ staged chunk log + flushed ring epochs) — so the monitor sees exactly the same data as every other reader, including un-materialized staged data.clarity:backend/src-tauri/src/monitor/reader.rs:10-16, 83-91
org/site/unit/grid — split into 4 path components (clarity:backend/src-tauri/src/monitor/reader.rs:24)Vec<Option<f64>> (was f32), matching the engine-wide f64 value path. None = missing.aligned_end_ms = current_time_ms - (current_time_ms % interval_ms) and back = (aligned_end_ms - ts) / interval_ms — so missing days leave None gaps instead of shifting older samples forward. clarity:backend/src-tauri/src/monitor/reader.rs:97-112clarity:backend/src-tauri/src/monitor/rules.rs
Four concrete rule evaluators, dispatched by RuleDefinition::to_evaluator():
rule_type |
Triggers when | Extra config fields |
|---|---|---|
AllOfN |
ALL present values pass the comparison | — |
KOfN |
At least k values pass |
k: usize |
Percentage |
At least percentage% of values pass |
percentage: f64 |
GOAT |
Value approaches or exceeds a known record | mode (Value|RateOfChange), known_record: f64, approach_margin_pct?: f64 |
All rule numeric fields (
threshold,percentage,known_record) and evaluator inputs (&[Option<f64>]) widenedf32→f64inc9fceb2— see the f64 value path.clarity:backend/src-tauri/src/monitor/rules.rs:9-14
Hysteresis (all rule types except GOAT): Once triggered (was_triggered = true), the alarm stays open unless zero present values pass the comparison. This prevents rapid open/close cycling on threshold boundaries.
NOTE: The old
ARCHITECTURE.mdsection 7 and the gap analysis list rule types asthreshold,range,rate_of_change— these names do not exist in the source code. The actual types areAllOfN,KOfN,Percentage,GOAT.
clarity:backend/src-tauri/src/monitor/evaluator.rs
Evaluator::evaluate_collection is called per collection per tick:
prev_triggered map from the previous snapshot (avoids per-rule linear scan)rule_config JSON once per rule (amortises JSON cost across tags)tag_indextag_entries.par_iter() — all tags in the collection are evaluated concurrently (clarity:backend/src-tauri/src/monitor/evaluator.rs:1,56-75)Returns a CollectionAlarmState with triggered_count and a TagAlarmState per tag.
clarity:backend/src-tauri/src/monitor/agent.rs
BASE_POLL_MS = 100 — the agent wakes every 100ms to check which collections are due for evaluation. Collections are only evaluated when now - last_eval_time >= tick_rate_ms. (clarity:backend/src-tauri/src/monitor/agent.rs:62-63)
The shipped
clarity.propertiesnow setsclarity.monitor.agent.poll_interval_ms=1000(was 100) — the file value overrides the code default, so deployed installs poll every 1 s.
Per tick (run_tick):
SnapshotStoredue_collections.par_iter() calls evaluate_collection per collection (clarity:backend/src-tauri/src/monitor/agent.rs:153-160)SnapshotStoreevent_tracker.process_tick() to detect transitions and persist alarm changesmonitor_collections.tick_rate_ms)ConfigManager::enable_collection and set_collection_tick_rate)PUT /collections/:id/tick-rate or the tick_rate_ms field in PUT /collections/:id/enableclarity:backend/src-tauri/src/monitor/config.rs:167-169Arc<AtomicBool> — the agent checks shutdown.load(Ordering::Relaxed) at the top of each loop iteration. (clarity:backend/src-tauri/src/monitor/agent.rs:22,65-68)
clarity:backend/src-tauri/src/monitor/auto_rules.rs
When a tagmeta record is created or updated (via tagmeta_hooks — see SQLite API § interceptors), this module automatically creates up to two AllOfN rules for the tag:
limLo (LessThan comparison)limHi (GreaterThan comparison)Each rule uses window_duration_ms = 60_000 (raised from 1000 in a18d35c). clarity:backend/src-tauri/src/monitor/auto_rules.rs:355,420
Two process-wide OnceLock handles are set once by main.rs after monitor init: init_global_config (the ConfigManager) and, new in a18d35c, init_global_event_tracker (the AlarmEventTracker, wired at main.rs:3991). Rule creation is fire-and-forget via tokio::spawn to avoid blocking the HTTP response. clarity:backend/src-tauri/src/monitor/auto_rules.rs:30-65
On a PUT that changes an existing Low/High limit, create_monitor_rules_for_tag fetches the existing rules for the tag (empty on POST), and when the stored threshold differs from the new value it calls AlarmEventTracker::close_open_alarms_for_rule(old_rule_id, now_ms) via the global tracker — so alarms opened under the old threshold are cleanly closed and re-evaluated under the new one on the next tick. clarity:backend/src-tauri/src/monitor/auto_rules.rs:308-340, 382-417
Logs route to data/logs/auto_rules.log via the target-based dispatcher in main.rs.
On each tick's alarm transitions, the agent fans out to the Notifications Engine (non-blocking, so it never touches the SQLCipher write lock):
NotificationService::try_send_event(NotificationEvent::MonitorAlarmOpened { rule_id, rule_name, tag, value, threshold, priority, orgs_id, sites_id, units_id }). priority is "danger" when the rule comparison is GreaterThan, else "warning". clarity:backend/src-tauri/src/monitor/agent.rs:213-229MonitorAlarmClosed { rule_id, rule_name, tag, duration_ms, orgs_id, sites_id, units_id }. clarity:backend/src-tauri/src/monitor/agent.rs:238-249The dispatcher maps these to Severity (Warn/Error for opened by priority, Info for closed), links them to /alarms/{id}, and delivers via OS toast + in-app banner + SSE (and optionally email).
Known gap: monitor alarm notifications set only
units_id;orgs_id/sites_idare alwaysNone(agent.rs:225-226, 245-246), so their RBAC visibility relies on unit scope or admin role. See Notifications § Monitor feed.
Mounted at clarity:backend/src-tauri/src/main.rs:3255-3262:
warp::path("exactapi").and(warp::path("monitor")).and(routes(...))
Authentication: None — no JWT filter is applied to monitor routes in main.rs.
All paths below are relative to /exactapi/monitor/.
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/alarm/snapshot |
None | Full current AlarmSnapshot — in-memory read, <1ms |
GET |
/alarm/summary |
None | Compact summary; ?collection_id= filters to one collection |
GET |
/alarm/triggered-only |
None | Only triggered collections/tags; ?collection_id= filter |
| Method | Path | Auth | Query params / Body | Description |
|---|---|---|---|---|
GET |
/alarm/events |
None | from_ms, to_ms, collection_id, tag_index, rule_id, status, limit (default 100, max 1000), offset (default 0) |
Paginated event query; returns {events, total_count, has_more} |
GET |
/alarm/events/active |
None | — | All currently OPEN alarm events |
GET |
/alarm/events/stats |
None | from_ms (required), to_ms (required) |
Aggregate stats: total, open/closed counts, avg/max duration, breakdown by collection and rule |
GET |
/alarm/events/:event_id |
None | — | Single event by UUID; 404 if not found |
DELETE |
/alarm/events/cleanup |
None | body: {"older_than_ms": <u64>} |
Deletes CLOSED events older than timestamp; returns {deleted_count} |
NOTE: An acknowledge endpoint
PUT /alarm/events/{id}/acknowledgeis listed inARCHITECTURE.mdsection 7 API references and gap analysis — not found inapi.rs. Do not document until confirmed.
NOTE: An encryption endpoint
GET /encryptionis mentioned in gap analysis — not found inapi.rs. Do not document until confirmed.
| Method | Path | Auth | Description |
|---|---|---|---|
POST |
/rule |
None | Create rule; returns {"id": "<uuid>"} with HTTP 201 |
GET |
/rule |
None | List rules; filter by ?collection_id=, ?tag_index=, ?enabled= |
GET |
/rule/:id |
None | Single rule by ID; 404 if not found |
PUT |
/rule/:id |
None | Update rule; 400 if not found |
DELETE |
/rule/:id |
None | Delete rule; 404 if not found |
| Method | Path | Auth | Body | Description |
|---|---|---|---|---|
GET |
/config |
None | — | Returns {"enabled": bool} from monitor_system_config |
PUT |
/config |
None | {"enabled": bool} |
Enable or disable the entire monitoring system |
| Method | Path | Auth | Body | Description |
|---|---|---|---|---|
GET |
/collections |
None | — | List all collections with id, enabled, description, tick_rate_ms |
PUT |
/collections/:id/enable |
None | {"tick_rate_ms"?: u64} (optional) |
Enable collection; sets tick rate (default 1000ms if omitted) |
PUT |
/collections/:id/disable |
None | — | Disable collection monitoring |
PUT |
/collections/:id/tick-rate |
None | {"tick_rate_ms": u64} |
Update evaluation interval; min 100ms |
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/status |
None | Returns {enabled, collection_count, rule_count, last_snapshot_ms, tick_number} |
NOTE: Old docs and
ARCHITECTURE.mdclaim 19 endpoints. Actual count fromapi.rsis 20 — thePUT /collections/:id/tick-rateendpoint was missing from the old count.
rule_config is an opaque JSON object stored in monitor_rules.rule_config. Schema varies by rule_type:
AllOfN — all values in window must pass:
{
"threshold": 100.0,
"comparison": "GreaterThan"
}
KOfN — at least k values must pass:
{
"threshold": 100.0,
"comparison": "LessThan",
"k": 3
}
Percentage — percentage of values must pass:
{
"threshold": 50.0,
"comparison": "GreaterThanOrEqual",
"percentage": 80.0
}
GOAT — triggers when metric approaches/exceeds known record:
{
"mode": "Value",
"known_record": 500.0,
"approach_margin_pct": 5.0
}
mode is "Value" (use last raw value) or "RateOfChange" (use abs delta between last two values). approach_margin_pct defaults to 0.0 if omitted.
comparison accepts the serialised enum names: "GreaterThan", "GreaterThanOrEqual", "LessThan", "LessThanOrEqual", "Equal", "NotEqual". Default is "GreaterThan".
window_duration_ms is a top-level RuleDefinition field, not inside rule_config. The evaluator uses the max window_duration_ms across all rules for a collection in one read call.
Source: clarity:backend/src-tauri/src/monitor/rules.rs:31-36,86-89,133-139,185-190
clarity:backend/src-tauri/src/monitor/tests.rs — file exists, 0 tests.
NOTE:
ARCHITECTURE.mdsection 7 claims "57 unit tests" — incorrect as of 2026-05-22. The file is empty.
End-to-end loop from agent wake-up through alarm event persistence.
Sources: clarity:backend/src-tauri/src/monitor/agent.rs, clarity:backend/src-tauri/src/monitor/reader.rs, clarity:backend/src-tauri/src/monitor/evaluator.rs, clarity:backend/src-tauri/src/monitor/events.rs
Diagram generated from source. If implementation has changed, update the Mermaid source in this file directly.
Last updated: 2026-07-17 from commit 6800acc
Range9847dba..c41a03bnote:config.rs,types.rs,rules.rs, andevaluator.rsgained#[cfg(test)]modules only — no behavior change;events.rs,auto_rules.rs, and the API surface are untouched. Behavioral deltas below are limited tosnapshot.rs,reader.rs, andagent.rs.