Terms as used in the Clarity + Pulse codebase.
collection
A named group of time-series tags sharing a common scope (organization/site/unit/grid) and sampling interval. Stored in data/{org}/{site}/{unit}/{grid}/ with a metadata.json file.
clarity:backend/src-tauri/src/api/storage/meta.rs
scope
The 4-tuple (organization, site, unit, grid) that addresses a collection. The Scope struct is used throughout clarity for routing writes and reads.
clarity:backend/src-tauri/src/api/tag_resolver.rs
tag
A named signal within a collection (e.g., a sensor measurement). Tags are stored as columns in the collection's mmap files. In PI terminology, equivalent to a PI Point.
TAG_SCOPE_MAP
A lazy-static in-memory map (DashMap<String, Arc<Scope>>) built by scanning the data/ directory at startup. Maps each known tag name to its containing scope, enabling tagname-only writes/queries without requiring callers to supply the full 4-tuple.
clarity:backend/src-tauri/src/api/tag_resolver.rs
mmap pool
Two bounded LRU pools of memory-mapped file handles managed by Storage — a read-mmap cache (default 512 entries) and a writer-mmap cache (default 1024). Allows concurrent readers/writers without re-opening files on each access.
clarity:backend/src-tauri/src/api/storage/mmap.rs
push-down op
An aggregation computed inside the storage engine rather than returned as raw points to the caller. Available ops: Mean, Sum, Min, Max, Count, First, Last.
clarity:backend/src-tauri/src/api/storage/read.rs:966
aggregation pipeline
A per-tag sequence of AggregationStep objects (op + params) applied after query retrieval. Ops include: Mean, Sum, Min, Max, Count, First, Last, Percentile, Histogram, LeastSquares, Rate, Scale, Score, filter/threshold.
clarity:backend/src-tauri/src/api/aggregator.rs
QueryEngine
Thread-safe wrapper around Storage that holds a DashMap of open mmap column handles for fast repeated reads.
clarity:backend/src-tauri/src/api/query.rs
PI tag / PI attribute / PI point / PI element
OSIsoft PI System terminology used in the WebPI connector. A PI Point is a named time-series value. A PI Attribute belongs to a PI Element (equipment model). Web IDs are opaque string identifiers used by the WebPI REST API.
clarity:backend/src-tauri/src/connectors/webpi/
collection (in PI context)
Not to be confused with Clarity's own collection concept. In WebPI connector code, "collections" refers to PI AF element collections (hierarchical equipment). In Clarity, a "collection" is always the storage scope 4-tuple container.
monitor rule
An expression evaluated periodically against tag values to detect alarm conditions. Rules are parsed and evaluated by the rules.rs engine and executed by the monitor agent loop.
clarity:backend/src-tauri/src/monitor/
alarm event
A record written to SQLite when a monitor rule condition is met. Tracked with timestamps and event metadata.
clarity:backend/src-tauri/src/monitor/agent.rs
ADK (Agent Development Kit)
Google's framework for building multi-agent AI systems. Pulse uses google-adk to define agents, tools, and session management.
pulse_multi_agents/index.py
agent
A Google ADK Agent or LlmAgent instance. The root agent in Pulse is pulse_manager. Sub-agents are specialized workers registered as AgentTool on the root.
pulse_multi_agents/pulse_manager/agent.py
sub-agent
One of six specialized agents delegated to by pulse_manager: meta_data_agent, data_analysis_agent, incident_agent, system_health_agent, dashboard_creation_agent, question_agent.
tool
A callable registered on an ADK agent. In Pulse, sub-agents are wrapped as AgentTool, and utility functions are wrapped as FunctionTool.
pulse_multi_agents/pulse_manager/agent.py
session
An ADK session tracking the conversation state between a user and the agent. Persisted in SQLite or PostgreSQL. State dict carries auth_token, base_url, and agent working memory.
pulse_multi_agents/index.py:47-69
RestrictedPython sandbox
The execution environment used by data_analysis_agent to run dynamically generated Python analysis code with restricted builtins to prevent unsafe operations.
pulse_multi_agents/pulse_manager/sub_agents/data_analysis_agent/agent.py
deployment topology (Historian-only / AI bundled / AI separate)
The retired [mode: historian | standalone | cloud-copilot] tags map to three technical topologies: Historian-only (clarity binary, no AI service), AI bundled (edge) (clarity + pulse_multi_agents in one on-prem install), and AI separate (cloud) (pulse_multi_agents as a managed-cloud service against an on-prem historian). Customer-facing pages instead use the product names Pulse Historian / Pulse Copilot with the deployment axis edge/on-prem vs managed cloud. See Overview § Technical topologies.
tag_resolver (clarity vs pulse)
Two distinct implementations with different purposes:
clarity:backend/src-tauri/src/api/tag_resolver.rs: Filesystem scope mapper — scans data/ directory to build TAG_SCOPE_MAPpulse_multi_agents/pulse_manager/sub_agents/meta_data_agent/tag_resolver.py: Semantic NLP resolver — uses sentence-transformers with 24h TTL cache to match natural language tag descriptionsKairos
A time-series database format. Clarity supports Kairos-compatible output via kairos=True flag in the Python SDK and equivalent in JS SDK. Timestamps are converted to milliseconds.
generatedDataTagId
A UUID-like identifier assigned to a tag spec in the tag_mappings SQLite table. Allows callers to reference tags by semantic spec (metricName + custom fields) instead of raw tag name strings. Used by /exactapi/tag_mappings and the mapped_tags SDK feature.
clarity:backend/src-tauri/src/main.rs:1040-1104
ProcessManager
The Rust subsystem that spawns, monitors, health-checks, and auto-restarts Python child processes (connectors, services) within the Tauri app.
clarity:backend/src-tauri/src/process_manager/supervisor.rs
write buffer
A lock-free queue in front of the storage engine for the buffered write path. Configuration: 100k capacity, 20ms flush interval, 10k batch size.
clarity:backend/src-tauri/src/main.rs:2288-2293
HA (High Availability)
Optional active-passive cluster for Clarity. Two nodes share a Virtual IP; only the Leader accepts writes and runs singleton services. Activated via HA_ENABLED=true. See HA Architecture.
clarity:backend/src-tauri/src/ha/
Leader
HA role: the authoritative node. Owns the VIP, accepts reads and writes, runs singleton services (MQTT, monitor, PI driver, Python services). Only Leader nodes run flush_sparse_writes and emit TSDB deltas to the Secondary.
clarity:backend/src-tauri/src/ha/role.rs
Secondary
HA role: standby node. SecondaryHealthy is promotable; SecondaryStale has fallen behind replication lag threshold. Applies incoming delta frames byte-for-byte.
clarity:backend/src-tauri/src/ha/role.rs
Recovering
HA startup role. A node always starts as Recovering when HA is enabled. Must complete a full resync before transitioning to SecondaryHealthy.
clarity:backend/src-tauri/src/ha/role.rs
TsdbDelta
A flush-aligned delta frame emitted by Storage::flush_sparse_writes on the Leader. Contains changed (tag_index, scaled_i32) pairs for one collection day file. Forwarded to the Secondary via the peer data plane.
clarity:backend/src-tauri/src/ha/tsdb_replicator.rs
DeltaRingBuffer
In-memory ring of recent TsdbDelta frames on the Leader. Allows a reconnecting Secondary to replay short gaps without a full file copy.
clarity:backend/src-tauri/src/ha/tsdb_replicator.rs
VIP (Virtual IP)
A floating IP address managed by VipManager. The current Leader acquires the VIP on promotion and releases it on demotion or shutdown. mDNS advertises the VIP so clarity.local resolves to the current Leader.
clarity:backend/src-tauri/src/ha/vip_manager.rs
SingletonGate
HA subsystem that starts leader-only services (MQTT, monitor, PI driver) on promotion and stops them on demotion. Components register with gate.register().
clarity:backend/src-tauri/src/ha/singleton_gate.rs
HaAgent
The HA role state machine driver. Monitors peer heartbeats, triggers promotion when peer is presumed dead, demotes on higher-term peer messages. Coordinates VipManager and SingletonGate.
clarity:backend/src-tauri/src/ha/agent.rs
SealedReconciler
Background task (Secondary side) that reconciles whole day units against the Leader by comparing blake3 digests; a mismatch triggers a chunked copy. As of 45a686e it is seal/blob-aware — digests are computed over decompressed logical bytes and cover both numeric (.bin/.bin.czst) and blob (.bref/.blob) units, so hot-vs-sealed skew no longer yields false mismatches and cold-tier days are actually repaired (deep scan on reconnect covers the whole archive; a rolling scan covers clarity.ha.reconcile_days, default 14). Note: "sealed" here originally meant immutable after UTC-day rollover — related to but not identical with the cold-tier seal below, which the reconciler now also handles. See HA § Sealed File Reconciliation.
clarity:backend/src-tauri/src/ha/tsdb_replicator.rs
seal / cold tier / .bin.czst
The cold-tier mechanism: an hourly task zstd-compresses day-files older than clarity.storage.seal.hot_days (default 0 — everything before today) into a sealed .bin.czst and deletes the original .bin. New seals use a V2 delta+varint codec (CLRTY_V2); older CLRTY_V1 (plain zstd) files stay readable. Reads transparently fall back to the cold file; writes unseal → write → reseal. POST /exactapi/seal_now forces a manual reclaim. Distinct from the HA "sealed file" concept (immutability), despite the shared word.
clarity:backend/src-tauri/src/api/storage/seal.rs:139-395
ColdDayReader
Reader over a .bin.czst cold file: mmaps the compressed file and decompresses one tag column at a time on demand (decompress_column(tag_idx)). Shared &self across rayon threads; the read path models each day as DayStorage::Hot(mmap) or DayStorage::Cold(ColdDayReader).
clarity:backend/src-tauri/src/api/storage/seal.rs:273-333
WAL (Write-Ahead Log)
The WriteBuffer journals entries to per-shard write_buffer_{n}.wal files (compact binary codec; legacy JSON still replays). As of c41a03b the WAL is a per-cycle journal written by the flusher thread (the enqueue path no longer touches it) and fsync is opt-in — clarity.write_buffer.wal_fsync defaults to false, so entries reach the OS page cache only. Enabled by default (clarity.write_buffer.wal_enabled).
clarity:backend/src-tauri/src/main.rs:2520-2536
secret pack
A single AES-256-GCM + HMAC-SHA256 encrypted file (~/.clarity/secrets/secrets-pack.bin, a JSON object) holding all application secrets (admin/opc passwords, MQTT broker creds). Keys are derived from the hardware fingerprint + build-baked JWT_SECRET, so the file decrypts only on the same machine with the same binary — same threat model as the license keystore. Read/written in one file access; provisioned from env vars on every startup. Replaces plaintext passwords in the seed data. (Before 1027dce this was an OS-keychain entry via the keyring crate; the keychain was removed.)
clarity:backend/src-tauri/src/secure_store.rs
obfuscation (Python services)
Build-time transform (build.rs) that encrypts each Python service .py to a .dat (zlib → XOR(key) → base64) and replaces it with a # obf-rs loader stub. The 32-byte key (.obf_key) is baked into the binary and injected as the _SK env var at service spawn. Originals are kept in .python_sources/ (never shipped).
clarity:backend/src-tauri/build.rs
integrity manifest / PYTHON_HASHES
A build-time static array of (path, sha256) for every Python service .py/.dat, generated into python_manifest.rs. The integrity module verifies each service directory against it before spawn; a mismatch rejects the service.
clarity:backend/src-tauri/src/integrity/mod.rs
SQLCipher v4 HMAC
The encrypted SQLite database is opened in SQLCipher v4 mode with per-page HMAC-SHA-256 (cipher_use_hmac=ON); a startup PRAGMA integrity_check fails-fast on any tampered/corrupt page.
clarity:backend/src-tauri/src/sqlite_api/db/mod.rs
account lockout
After the login rate limit (30 req/60 s per email) is exceeded, the email is locked for auth_lockout_seconds (default 900 s); further login attempts return HTTP 429 ACCOUNT_LOCKED until expiry. Tracked in the ACCOUNT_LOCKOUTS map.
clarity:backend/src-tauri/src/main.rs
shadow (last-timestamp query)
POST /sensordata/shadow returns the last data timestamp per unit. Each "<unitId>-shadow" key is resolved to its scope and answered by Storage::get_unit_last_timestamp, which scans the newest day file backwards for a non-sentinel value; units with no data are omitted.
clarity:backend/src-tauri/src/api/storage/read.rs:510
shadow cache / shadow_write (HA)
Not to be confused with the last-timestamp shadow query above. The HA secondary shadow cache (45a686e, disabled by default) is an in-memory buffer on the Secondary fed by clients/servers that mirror each write to POST /exactapi/shadow_write; on promotion the new Leader drains it and gap-fills any writes lost during the failover window (idempotent — never overwrites values the dead Leader already persisted). Gated by clarity.ha.shadow_cache.* / clarity.ha.shadow_targets. See HA § Secondary Shadow Cache.
clarity:backend/src-tauri/src/ha/shadow_cache.rs
storage format V2 / disk XOR codec
Per-collection on-disk value encoding (storage_format in metadata.json; new collections default to V2 as of c41a03b). V2 stores each scaled i32 XORed with i32::MIN, making the missing-sentinel zero on disk — so freshly allocated day-files need no sentinel-fill pass and can be born sparse. Legacy V1 (raw values, i32::MIN sentinel) remains readable forever; the format is fixed at collection creation.
clarity:backend/src-tauri/src/api/storage/mod.rs:56-78,437-449
blob store (.bref / .blob)
Sidecar files that hold string and array tag values (c41a03b): {day}.blob is an append-only value heap (crc-checked records; UTF-8 or JSON payloads) and {day}.bref is a sparse ref grid in the day-file column layout whose slots point into the heap. Served by the /exactapi/blob_* endpoints; sealed as whole-heap zstd (.blob.zst).
clarity:backend/src-tauri/src/api/blob_store.rs
hot ring buffer
Per-collection row-major RAM buffer (c41a03b) for collections sampling at ≤1 s: one contiguous row write per timestamp instead of N column-file page faults, drained to day files by a background compactor. Capped by clarity.hot_ring.max_bytes (128 MB default).
clarity:backend/src-tauri/src/api/hot_ring_buffer.rs
disk guard
Background thread (c41a03b) sampling free space every 10 s; blocks all ingest endpoints with HTTP 507 when free space drops below clarity.storage.min_free_disk_mb (default 2048 MB), unblocking at 1.25× (hysteresis). Reads, sealing, and metadata are unaffected.
clarity:backend/src-tauri/src/disk_guard.rs
MQTT fan-out
The decoupled, lossy, freshest-wins publisher (c41a03b) between ingest and the broker: batches are try_send-enqueued (queue capacity 8, whole-batch drop on overflow), coalesced per tag, and published at QoS 0. The historian write happens first and is the lossless source of truth.
clarity:backend/src-tauri/src/processing_api/mqtt_fanout.rs
staging tier / chunk log (staging.clog)
A per-collection sequential chunk log between the hot ring buffer and the column-major day file (497003f). Ring drains append to staging.clog (CRC'd, optionally fsync'd) instead of scattering one 4 KiB page per tag; a background materializer folds the log into {day}.bin once ~1024 slots/tag accumulate, so each day-file page is written once. Reads merge the day file with pending chunks. Gated by clarity.storage.staging.enabled (shipped true).
clarity:backend/src-tauri/src/api/storage/staging.rs:1-38
wire format version (binary query)
The value width in the binary query responses widened from f32 to f64 (c9fceb2), bumping each serializer's format_version: the basic format (/exactapi/fast_query_binary) went v1→v2, the optimized format (/exactapi/fast_query_optimised) went v2→v3. On-disk storage is unchanged (scaled i32); only the decoded value type widened. Clients branch on the per-payload format_version.
clarity:backend/src-tauri/src/api/binary_format.rs:43-226
ingest v2
A fast in-process ingest path (POST /ingest/v2/{client_id}/{config_id}) that streams a borrowed-key parse (no serde Value tree) against a per-(client_id, config_id) cached IngestSession mapping each key to a (scope, ring column) binding, revalidated per request via one atomic generation load. Wire format is identical to v1; exotic/unresolved payloads fall back to v1.
clarity:backend/src-tauri/src/processing_api/ingest_v2.rs:1-29
notification
A persistent, RBAC-scoped message (severity info|warn|error|critical; source monitor|process_manager|licensing|backup|system|admin|custom) delivered to the desktop app (OS toast + in-app banner) and to browser sessions over an SSE stream. Monitor alarm open/close is the only wired producer today. See Notifications Engine.
clarity:backend/src-tauri/src/notifications/model.rs:118-159
mail queue
The SQLite mail_queue store-and-forward table behind the outbound-email service. Producers enqueue; a background sender sweeps pending rows, sends via a lettre SMTP transport, and reschedules failures with exponential backoff + jitter until max_retries. See Mail / SMTP.
clarity:backend/src-tauri/src/mail/queue.rs
CloudSync
A background task that pulls recent time-series from the local instance and pushes it to a remote Clarity instance (local→remote data sync / migration), reconciling collections by scope. Cadence = lookback_minutes; stateless (no on-disk cursor); controlled via /exactapi/cloudsync/*. Not HA-leader-gated. See CloudSync.
clarity:backend/src-tauri/src/cloudsync/mod.rs:29-63
VIP guard
A Windows out-of-process SYSTEM sidecar (ClarityHaVipGuard scheduled task) that strips the HA virtual IP if the main app's heartbeat file goes stale (>3 s), covering hard-kill/power-loss cases where in-process VIP release never runs — preventing a dead node from keeping the floating IP after failover. Distinct from the in-process vip_manager.
clarity:backend/src-tauri/src/ha/vip_guard.rs:1-23
Last updated: 2026-07-17 from commit 6800acc