Append-only. Never edited manually.
Format:[YYYY-MM-DD] <repo> <commit-sha> — <summary>
[2026-05-21] INIT — initial wiki generated from clarity@4a3163c + pulse_multi_agents@e278054
[2026-05-22] gap-fill — backup-system.md created from code verification against BACKUP_SYSTEM.md; interval_minutes default discrepancy noted (runtime=180, stale test=10)
[2026-05-22] gap-fill — processing-api.md: added elog.rs section (5 endpoints, Excel generation, data flow) + complete EMS endpoint table + ingest endpoints; noted ems_routes() is unregistered dead code
[2026-05-22] gap-fill — monitor.md rewritten with full submodule breakdown and endpoint table (20 endpoints verified); conflicts with old docs noted: rule types (AllOfN/KOfN/Percentage/GOAT not threshold/range/rate_of_change), tests.rs empty (not 57 tests), auto_rules.rs is new 11th submodule
[2026-05-22] gap-fill — api-server.md: full endpoint index + 3-role auth confirmed + JWT_SECRET no-default clarified (old docs wrong) + token expiry 24h confirmed (SECURITY.md wrong at 1h) + auth/refresh endpoint does not exist + elog/ingest/monitor/backup/EMS groups added
[2026-05-22] gap-fill — license-portal.md rewritten with Rust licensing module + fingerprinting + JWT schema; two conflicts with LICENSING_LLD.md flagged: nvram_counter.bin is 68 bytes (AES-GCM+HMAC, hardware-bound) not 40 bytes (BLAKE3 keyed-hash, order-only); keystore test-mode uses encrypted token.bin not plain token.txt
[2026-05-22] gap-fill — pi-connector.md rewritten with full endpoint table (10 endpoints) + onboarding workflow + historic backfill params + live driver adaptive batching; 3 conflicts with old docs flagged: list_asset_servers response shape (PiItemSummary not {WebId,Name,Path}), onboard_unit is streaming NDJSON not non-blocking (old doc stale), verify_ssl defaults to true not false
[2026-05-22] gap-fill — python-sdk.md: constructor parameter table added (7 params verified), pipeline op reference table added (17 ops verified against aggregator.rs; sampler=placeholder, save_as=unimplemented noted), error hierarchy added, flush interval conflict documented (SDK docstring 50ms vs Rust backend 20ms); js-sdk.md: _contextFields behavior (undefined vs null), pipeline op reference table, error handling section, performance characteristics section added
[2026-05-22] post gap-fill cleanup — 3 TODO-VERIFYs resolved using cross-session context: backup auth (admin_middleware confirmed via api-server.md), elog auth (no JWT confirmed via api-server.md), EMS dead code (ems_routes() confirmed dead via processing-api.md); index.md and dev/index.md processing-api description updated to include elog; overview.md updated with backup system + elog subsystem; CLAUDE.md TEMPORARY rule 9 removed (hard rule 7 restored)
[2026-05-22] TODO-VERIFY pass — resolved all remaining source-resolvable TODO-VERIFYs: opcuser seeding confirmed via users.json + sqlite_api/db/mod.rs:607-627; entity list verified from schema.rs (28 entities); attachment endpoints confirmed (/exactapi/attachments/* with 9 routes, default containers: tasks/incidents/uploads/mail/pulselogo); process HTTP API corrected (only /process-logs + /process-logs/ws, no start/stop/status); port-kill implementation confirmed (lsof on Unix, netstat+taskkill on Windows, Python-only guard); sqlite-api entity table added with all relationships; python-services health_check_interval=30s/restart_policy="always" confirmed; run_python_binary Tauri command confirmed; MQTT broker Windows-service-only, QoS=AtMostOnce, topics clarity/test/; WebSocket /mqtt is raw TCP proxy not MQTT-over-WS; aggregator op serialization strings confirmed (leastsquares/trim/saveas/score + undocumented avg/dev/gaps/diff); tag_resolver model=all-mpnet-base-v2 threshold=0.0 fallback=[]; setup_persistence corrected to Windows autostart only; DashMap key/eviction confirmed; WriteBuffer params confirmed (20ms flush/10k batch/100k queue); 2 TODO-VERIFYs kept with annotation (license-portal auth + ML port 8200 require deployment config)
[2026-05-23] gap-fill — added 5 Mermaid LLD flowcharts (ingest, query, monitor, backup, agents) to processing-api.md, storage-engine.md, monitor.md, backup-system.md, agents.md; diagrams generated directly from source files
[2026-05-23] generated openapi.yaml from source (main.rs + handler files). swagger-ui.html added. Sources: clarity:backend/src-tauri/src/main.rs, auth.rs, storage.rs, query.rs, aggregator.rs, monitor/api.rs, backup/state.rs, backup/api.rs, processing_api/elog.rs, processing_api/ingest.rs, processing_api/ems.rs, connectors/webpi/webpi_meta_connector.rs, sqlite_api/schema.rs, api/google_adk.rs, order_portal/app/main.py + route files.
[2026-05-23] restructure — SDK docs moved to wiki/dev/integrations/; integrations index created
[2026-05-23] created wiki/hld/ — 5 views: system-context.md (5-layer stack diagram with verified sources/connections), component-architecture.md (internal component diagram + PI onboarding sequence), deployment-views.md (historian/standalone/cloud-chat process topology), security-model.md (trust boundary, role table, data at rest, hardening roadmap), network-ports.md (all 5 ports verified with exact source citations)
[2026-06-02] REINIT — wiki-init rerun from clarity@498c020 + pulse_multi_agents@e278054. New: created wiki/dev/architecture/ha.md documenting the entire HA subsystem (active-passive cluster, 6-role state machine, TsdbDelta replication wire format, SealedReconciler, DeltaRingBuffer, CollectionCreateReplicator, SqliteSnapshotProducer/Consumer, VipManager, SingletonGate, HaAgent, WAL for WriteBuffer). Updated overview.md, glossary.md (14 new HA terms), dev/index.md, index.md to include HA links. Clarity SHA advanced from 4a3163c → 498c020; pulse_multi_agents unchanged at e278054.
[2026-06-02] TODO-VERIFY resolved — ha.md Admin API section: verified all 13 /exactapi/ha/ endpoints from ha/api.rs (status, promote, demote, fence, unfence, events, health, peer, lag, ops/switchover, ops/force_promote, ops/force_fence, ops/recovery_status); noted endpoints are role-unrestricted per file-level comment.
[2026-06-02] gap-fill from rest-api.html cross-reference — storage-engine.md: AggregationOp table expanded from 14 to 22 entries; added Avg/Dev/Gaps/Diff/Div/Sampler/Trim/SaveAs with parameters and placeholder notes; verified against aggregator.rs:8-33. sqlite-api.md: added LoopBack filter syntax section, standard CRUD route table (14 methods), and full nested routes table (36 paths verified from warp_routes.rs:744-879); corrected route base to /exactapi/{entity} (no /v2 prefix). api-server.md: corrected SQLite CRUD route prefix error (/exactapi/v2 → /exactapi) with source citation to main.rs:3749.
[2026-06-02] gap-analysis fixes (A1,A2,B3,C6,C7,D9,D10) — CLAUDE.md: deployment label + filename reconciled cloud-copilot → cloud-chat (now matches wiki + .llmwiki/config.yml); module map rebuilt comprehensively (~40 rows across both repos); repo description marks OPC-UA as planned/not-implemented. agents.md: added meta_data_agent tool table (fetch_data / df_operations_tool / getTagMeta / getIncidents), documented per-session CSV isolation (inject_session_id → {session_id}_datafetch.csv) + CLARITY_APP data-source switch (Clarity query API vs KairosDB); corrected inject_auth_credentials to match live code (session-state branch commented out, HEADERS_STATUS guard); flagged that the semantic tag_resolver import is commented out (agent.py:1170) so getTagMeta resolves via LoopBack like queries. Created wiki/dev/adr/ with index + ADR-0001..0005 (HA 6-role FSM; JWT build/env secret; custom mmap binary TSDB; RestrictedPython sandbox; two-layer tag resolution), all source-cited. Renamed 'OPC UA.md' → opc-ua.md (kebab-case), rewrote as 'not implemented' after verification (no opcua module/crate; only GET /opc-network ID stub at main.rs:3581-3592 + opcuser.json seed); linked from index.md + dev/index.md.
CORRECTION: security-model.md JWT_SECRET claim fixed — auth.rs:28-41 reads the runtime env var with a compile-time env! fallback (no runtime startup panic), not the previously-documented .expect() panic.
NEW FINDING (security gap): backup admin routes /api/admin/backup/* are UNAUTHENTICATED — get_backup_routes wired at main.rs:3606 with no admin_middleware; backup/api.rs carries no inline auth. The earlier [2026-05-22] log claim that backup auth was "confirmed via admin_middleware" was incorrect. Logged in security-model.md hardening table; the stale TODO-VERIFY is now resolved.
[2026-06-11] new page — created wiki/dev/internals/capacity-planning.md (clarity@d7fa675) covering 4 facets for the time-series store: (1) on-disk sizing — dense i32 grid, 4 B/sample, implicit timestamps, full-day pre-allocation; formula4 × tags × (86400/interval_s) × days; worked example 8000 tags @ 1s × 3yr ≈ 3.03 TB (verified vs storage.rs:1103-1209 + test :2737-2738); (2) service overheads — negligible hierarchy/metadata/WAL vs material multipliers: timeseries backups never pruned (unbounded), HA replica ×2, alarm-events growth; (3) RAM/CPU — ~3 GB page cache for hot day, bounded heap caches (config.rs:254-259), query working-set OOM risk (raw all-tag/day ≈ 5.5 GB, TODO-VERIFY); (4) cold-storage/compression — confirmed NONE: no at-rest compression of .bin (only gzip in backup/), no tiering, no TS retention/TTL, no rollup (saveas/sampler unimplemented); only flate2 use is backups; FS-level zstd/lz4 flagged as the practical lever (ratio TODO-VERIFY). Linked from index.md, dev/index.md, and storage-engine.md See-also. Unstaged for human review.
CORRECTION (same day): capacity-planning.md cold-storage section refined re LZ4/zstd. Cargo.toml DOES declarelz4_flex = "0.11"(comment: "LZ4 compression for binary fast query endpoints") +zstd = "0.11"(added 2025-10-24, 84ab331). However both are DEAD DEPS — zero call sites in src (verified: grep lz4_flex|zstd|compress_prepend|decompress over backend/**/.rs = empty). Intent was wire-format (fast-query response) compression, NOT at-rest; even that was removed for performance (main.rs:3174,:3198 comments; binary_format.rs emits raw LE bytes). Net: no LZ4/zstd on disk or wire today; only live compression remains gzip in backup/. The earlier "zstd: None" wording was imprecise (crate present but unused) — now documented explicitly so a reader opening Cargo.toml isn't misled.
FOLLOW-UP (out of scope here): api-server.md may also describe the backup endpoints as admin-authed — needs a verify pass. And tag-resolver.md may overstate that meta_data_agent uses the semantic resolver.
[2026-06-02] A1 reversed per user decision — standardized on cloud-copilot (not cloud-chat) across the wiki + .llmwiki/config.yml + CLAUDE.md. Renamed wiki/dev/deployment/cloud-chat.md → cloud-copilot.md; flipped all[mode: cloud-chat]tags, "Cloud Chat"/"Cloud-Chat" display names, and file links in overview.md, index.md, dev/index.md, agents.md, monitor.md, adr/0004, and hld/{index,network-ports,system-context,deployment-views}.md. The canonical mode label is now[mode: cloud-copilot]everywhere. (wiki/.obsidian/workspace.json left untouched — editor state, auto-regenerated.)
[2026-06-05] clarity 498c020..f14ebda — SDK: added ClarityModbus class to python-sdk.md (Modbus TCP adapter built on ClarityClient; fetch_register_map, read_holding_registers, start_server, start_server_background; register encoding float32→2×big-endian 16-bit); added multi-language Modbus client reference (.NET/C# SDK/dotnet/, Java SDK/java/); added GAIL customer demo scripts section (SDK/demos/gail/ — Python/Java/C#/JS plot + ingest demos). python-services.md: added registered services table from python_services_config.json; documented new elog-group-data-report-api service (port 9003, /elog/group/data/* routes) added in this commit; noted reports-to-pdf submodule renamed to reports-to-pdf-python. TODO-VERIFY added: elog Python service vs Rust elog.rs route precedence.
[2026-06-05] DRAFT — created wiki/product/agentic-ai-website-copy-v1.md and -v2.md: consolidated Agentic AI marketing/website copy for the website Google Doc. Sourced from agents.md, overview.md, glossary.md, deployment/{cloud-copilot,standalone}.md, hld/system-context.md. v1 = cautious/accuracy-flagged; v2 = marketing-forward (multi-agent emphasis, unified Clarity+Pulse on-prem package framing per user direction to drive/test demand; some capabilities roadmap/in-development presented as available). Not code-cited wiki pages — marketing drafts for human review. wiki/product/index.md updated to list both. Not committed/staged.
[2026-06-05] DRAFT — created wiki/product/historian-website-copy.md: marketing/website copy for the Clarity Historian (features + performance), companion to the Agentic AI drafts. Mirrors AI v2 structure; positioning = modern PI-class historian + AI-ready data layer for Pulse. Bold claim posture per user direction — leads with README headline numbers (1M+ writes/sec, <100ms/100k tags, sub-ms buffered, 70-90% compression, OPC-UA listed) backed by an internal verify-before-publish note. Code-verified facts (storage.rs/write_buffer.rs/aggregator.rs/binary_format.rs/config.rs/Cargo.toml/main.rs) used for the spec table; flagged divergences from README in the internal note: OPC-UA not implemented (stub only), 22 ops/7 push-down (not "25+"), throughput/compression are targets/claims not repo benchmarks. Sources cross-checked against wiki overview/glossary/storage-engine/monitor/ha/backup/security-model + clarity SDK/README. wiki/product/index.md updated. Not committed/staged.
[2026-06-08] clarity f14ebda..d7fa675 (34 commits) — Timeseries scalability: storage-engine.md documents the cached tag-name→row-index Arc in metadata_cache (no per-call rebuild), get_metadata_and_index(), and the >30-day LRU-bypass via get_mmap_uncached(); WriteBuffer rewritten as 4 hash-sharded queues+flushers with per-shard WAL files. tag-resolver.md: TAG_SCOPE_MAP capacity 200k→2M (and type corrected DashMap→RwLock). api-server.md: Windows runtime hardening (SetErrorMode crash-dialog suppression, SetThreadExecutionState sleep prevention), mDNS dns-sd/mdns-sd dedup, WAL now always-on, Windows graceful shutdown handler (ctrl_shutdown/logoff/close → drain), watchdog start; Users alias + configurations TEXT-id notes. process-manager.md: 30-min auto-start watchdog, kill-instead-of-adopt leftover processes, idempotent restore, taskkill /F /T tree-kill, wmic→tasklist detection, 30s log-tailer retry, mdns_sd log suppression. python-services.md: per-request proxy timing logs + 502 on upstream fail; Windows python_runtime\python.exe preferred over uv trampoline shim + PYTHONPATH injection + pyvenv.cfg patch hardening + health_check_url validation; service ports re-assigned (data-calculation-b 443→17001, reports-to-pdf 5000→17000) and meta-ui-upload-service replaced by data-api (17002, /sensordata/spcplot). sqlite-api.md: String-Id CRUD model for configurations (StringIdCrudModel/impl_crud_string_id!/crud_routes_string_id!, TEXT PK + migration), User name→username rename + migration, /exactapi/Users alias, regex predicate tree (apply_filter_with_tree/RegexNode AND/OR/NOR) replacing flat deferred-regex list. python-sdk.md: added ClarityClient.query_dataframe(); REMOVED ClarityModbus, multi-language Modbus client, and GAIL demo sections (clarity_modbus.py, SDK/java, SDK/dotnet, SDK/demos/gail/* all deleted upstream). Skipped per config filter (outside backend/src-tauri/src + SDK): Cargo.toml (tauri-build 2.6.2), clarity.properties (TLS paths commented), seed_data/{configurations,users}.json, .gitmodules (data-api submodule), deleted tests/ + python_testing_scripts/.
[2026-06-12] clarity d7fa675..307efa7 (44 commits) — Major: SQLite CRUD migrated to a dynamic relation-discovery layer. sqlite-api.md: documented Phases 1–6 (introspect.rs PRAGMA FK discovery + RelationMap/OPT_OUT_TABLES; aliases.rs RELATION_NAME_OVERRIDES + SOFT_FK_EXCEPTIONS; api/dynamic.rs CRUD core + STRING_PK_TABLES/EXTRA_COLUMNS/pre_bind hooks, dormant Axum build_dynamic_router; api/paths.rs BFS nested-path builder; warp_dynamic_bridge.rs GET /exactapi/dyn/_schema; warp_dynamic_routes.rs full Warp surface incl. URL_ALIASES customers→orgs/Users→users + TABLE_URL_OVERRIDES faultTemplates/dashboard-plots/profiles-lookups; warp_routes DYNAMIC_TABLES gate via dyn_skip sentinel — all 28 paths now dynamic, typed layer is gated dead code). Also: discovery caches on Database; customerId dual-FK columns + backfill across 6 org-scoped tables (sync.rs writes both); users.username backfill; delete_nested_item (LoopBack F6, safety-belted) on both layers; WhereNode in-memory WHERE evaluator; heatrates relations removed from schema.rs. auth.rs: row-access policy extracted to check_row_access / check_row_access_with_db (FK traversal for dashboardplots) + AccessDenied enum — documented in api-server.md. license-portal.md: Windows fingerprinting now implemented via wmic (ProcessorId/baseboard SerialNumber/diskdrive SerialNumber + csproduct UUID fallback) — was placeholder; new lic_log.rs (clarity::licensing → licensing.log, never logs secrets); start_background_tasks(verified_order_id, on_brick) avoids get_order_id() key drift + Thread 2 sleeps 3630s; Check 2 iat grace last_seen+300<iat. api-server.md: single-instance LOCK now binds port server_port+10000 (atomic, debug+release) replacing connect-to-3030; logger init before license check + licensing.log routing; tray show/click gated on verify_license(). process-manager.md: Windows Job Object (JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) kills child processes on exit/crash/Task-Manager-kill; new windows-sys dep. python-services.md: proxy now plain HTTP on loopback (no TLS, removed danger_accept_invalid_certs) + pool_idle_timeout(60s); profiles-service submodule added (port-clarity branch) — flagged port 17002 COLLISION with data-api; corrected stale "data-api replaced meta-ui-upload-service" (both coexist; meta-ui-upload 5100/WAITRESS_THREADS=1, overview-kpi WAITRESS_THREADS=16); waitress dep + extra_modules clarity_sdk wheel installed into venv. python-sdk.md: query_binary now milliseconds + pipeline forwarded to binary endpoint (no JSON reroute); new ClarityDecoder.to_dataframe(); SDK packaged as wheel (pyproject.toml/build_wheel.sh); flagged query_dataframe seconds-vs-ms docstring inconsistency. Memory-watch check: Cargo.toml lz4_flex/zstd still have zero call sites (decompress-logic-change commit was Python-side clarity_sdk.py only) — deps remain dead. Skipped: configurations.json seed reshuffle (10k-line data churn, not documented); reqwest rustls-tls feature switch (internal build config, no consumer impact).
[2026-06-15] clarity 307efa7..e021b09 (7 commits) — Dynamic CRUD TEXT-PK + config/rename fixes. sqlite-api.md: dynamic Warp/:idGET/PUT/DELETE now takeStringparams (TEXT-PK tables likeconfigurationsreachable by id; previouslyi64silently failed) andcount/findOnereordered before/:id; PATCH upsert id-from-body accepts string|int andupsert_rowbranches on STRING_PK_TABLES; folder rename on orgs/sites/units name change (update_item_shim PUT + update_all_shim POST /update, which gained a Storage arg → renaming::handle_rename; find_actual_folder_fallback handles DB↔disk name drift, single-candidate only); INCLUDE now runs before FIELDS projection (parent-table resolved by segment parity) with relation-name preservation; apply_order_subquery emits unqualified ORDER BY for windowed include scope + parameterizedWHERE rn <= ?; EXTRA_COLUMNS gainedincidents.criticalTags(BLOB) via new read_extra_col_string (read, TEXT→BLOB lossy fallback) / extra_col_to_string (write, arrays/objects→JSON) helpers; utils link_resource_to_admin & sync_admin_access tolerate BLOB/TEXT meta_data; db/mod.rs DB init now CREATEs monitor_system_config/monitor_collections/monitor_rules/alarm_events (+4 indexes) — schema shared with monitor subsystem. api-server.md: get_user_hierarchy_access (now :401-505) tolerates BLOB/TEXT/Null userprofiles.meta_data and login_user fails open (issues JWT with empty units/sites/orgs on lookup error) — noted interaction with row-access deny for non-admins with empty units_id. python-services.md: data-api gained/sensordata/elog/datapoints/queryroute; profiles-service port 17002→17005 (resolves the previously-flagged collision — TODO-VERIFY removed). No new pages; no UNMAPPED files (python_services_config.json is under Assets/ not src/ but maps directly to python-services.md). Memory-watch: no lz4_flex/zstd changes this range — deps remain dead.
[2026-06-15] capacity-planning.md — added "Worked example — 15k / 25k / 50k tags @ 1 s, 5 years (with & without HA)" section (no new source read; scaled from the existing on-disk model at clarity@d7fa675). Per-tag-per-day 345,600 B; 1,826 days (5×365+1 leap). Raw .bin floor: 15k≈9.47 TB / 25k≈15.78 TB / 50k≈31.55 TB. Per-node provisioned (raw+~20% headroom, backups separate): 11.5/19/38 TB; RAM floor-comfortable 16-32 / 32-64 / 64-128 GB driven by hot-day page cache (tags×86400×4B = 5.18/8.64/17.28 GB); CPU 8 / 8-16 / 16 cores. HA doubles disk per active-passive full-replica model (cluster totals 23/38/76 TB), RAM/CPU per node unchanged; replication BW ~0.1-0.3 MB/s scaled from ha.md ~6 KB/s@1k tags. Re-flagged unbounded timeseries backups, FS zstd/lz4 lever (~2-4×, TODO-VERIFY), and query-OOM working set (tags×86400×8B/day). Header date 2026-06-11→2026-06-15, commit unchanged (d7fa675). Not committed/staged — for human review.
[2026-06-15] DRAFT — created wiki/product/roadmap/ (index + 2 scoping docs) from clarity@e021b09 + pulse_multi_agents@426c3fa. (1) local-llm-granite.md: PM scope for running the agent layer fully offline on a local model (e.g. IBM Granite). Key code-grounded findings: get_model() provider factory keyed on MODEL_NAME already supports GEMINI/OLLAMA/GROQ/OPEN_AI and has an existing GRANITE stub → LiteLlm(openai/granite @127.0.0.1:8080/v1) (utilis.py:485-490); litellm+google-adk already deps; tag_resolver already local (sentence-transformers/torch). Scope = finish/unify/harden/validate: duplicate get_model in dashboard agent has no GRANITE branch (silent cloud fallthrough); data_analysis_agent bypasses LiteLLM via genai.Client().generate_content + isinstance(str) guard → always cloud Gemini (agent.py:750-758,1689-1697); incident agent uses Gemini-native GenerateContentConfig; net-new = supervise+bundle inference server; gating risk = tool-calling quality. (2) plant-doc-rag.md: PM scope for RAG over SOPs/guides/datasheets/schematics. Foundation already bundled: offline embedder all-MiniLM-L6-v2-local (main.rs:1514), persisted qdrant_data (main.rs:1513,1835), /create_qdrant_collection route → agent binary (main.rs:3336-3342,418), PI connector auto-trigger (webpi_meta_connector.rs:1256), build packaging (build_and_deploy.sh:29-43). Gaps: agent-side consumer is a STUB (QdrantRequest defined index.py:43 but no route wired; no QdrantClient/upsert/search; no qdrant-client dep) + TODO-VERIFY whether shippedmainbinary differs; zero document ingestion (no loaders/OCR/chunking); MiniLM vs mpnet embedder inconsistency. Customization differentiator = unit-scoped Qdrant payload filtering (reuses existing resolve_unit orchestration) + hybrid+rerank + citation grounding. Tiered by doc type: T1 text-native (M), T2 datasheets/OCR (M-L), T3 schematics/P&ID = vision-model research (L, conflicts with offline goal). Both docs cross-link (RAG matters MORE under local inference). Indexes updated: product/index.md, master index.md. Not code-cited as dev pages but citations retained for grounding. Not committed/staged — for human review.
[2026-06-17] clarity e021b09..a18d35c (41 commits) — Security/secrets hardening + Rust-native Python obfuscation. Theme 1 — secrets out of binary/seed: newsecure_store.rssecret pack (singleclarity-secrets-packkeychain entry, 1 access;provision_packsyncs ADMIN/OPC/MQTT creds from env on every startup); plaintext passwords removed fromusers.json/configs.json,opcuser.jsondeleted; db/mod.rs user+config seeding reads from the pack; PI driver + ensure_admin_user read ADMIN_EMAIL/ADMIN_PASSWORD. Documented in api-server.md (new "Seed credentials & secret pack" + "Default Seeded Users" rewrite), sqlite-api.md (secret-pack seeding), mqtt.md (broker creds from pack), security-model.md, glossary. Theme 2 — Python obfuscation moved Python→Rust:obfuscate.pydeleted; build.rsrun_obfuscation(zlib→XOR→base64,# obf-rsstubs, .obf_key→_SK),restore_python_sources(CLARITY_RESTORE),generate_python_manifest(SHA-256→PYTHON_HASHES); newintegrity/mod.rsverify_python_file/verify_service_dir; service_manager verifies each service dir before spawn. New build deps sha2/flate2/rand/serde_json + wmi (windows). Documented in python-services.md (new "Build-time obfuscation & integrity"). Theme 3 — auth: JWT_SECRET runtime override only-if-non-empty (else compile-time env!); JWT_EXPIRY_MINUTES override; account lockout (auth_lockout_seconds default 900s, ACCOUNT_LOCKOUTS→429 ACCOUNT_LOCKED). api-server.md + security-model.md. Theme 4 — Windows hardening: all subprocess calls use absolute System32 paths w/ missing-binary guards (certutil/schtasks/reg/netsh/net/ping/sc/passwd/powershell/tasklist/netstat/taskkill); license fingerprint + python port-kill use three-tier wmic.exe→PowerShell→wmi-crate detection. Spread across api-server.md, license-portal.md, mqtt.md, ha.md (firewall+vip), storage-engine.md (persistence), python-services.md. Theme 5 — license keystore resilience: TOKEN_CACHE in-memory cache; use_file_keystore() (test/override/headless-Linux SecretService probe); KeystoreUnavailable (transient errors don't brick); CLIgenerate-challengesubcommand. license-portal.md + standalone.md. Theme 6 — SQLCipher v4 HMAC + startup PRAGMA integrity_check fail-fast. sqlite-api.md + security-model.md. Theme 7 — SQLite CRUD: new clients→ingestconfigs SOFT_FK + lookup_fk soft-FK fallback; tagmeta_hooks/tagmeta/update+/units/updateinterceptors (string-or-number tolerance, where-clause identity resolution); deferred-regex LIMIT/OFFSET correctness fix + stream fields projection + filter-body unwrap; rawlikeregex. sqlite-api.md. Theme 8 — monitor: init_global_event_tracker + threshold-change force-close (close_open_alarms_for_rule); auto-rule window 1000→60000ms. monitor.md + sqlite-api.md. Theme 9 — storage get_unit_last_timestamp powering POST /sensordata/shadow (new route, parallel per-unit). storage-engine.md + api-server.md. Theme 10 — branding: tauri.conf/dev productName "Clarity Engine"→EdgeLive, icon pulse.ico→tmx.ico (Thermax), bundle resources narrowed Assets//→python_services+scripts; startup diagnostics (panic hook + clarity_startup.log + WebView2/MSVC probes). standalone.md + api-server.md. SDK: numpy-based decoders (frombuffer/unpackbits, structured dtype), ClarityClient.login/query_binary_parallel/query_fast. python-sdk.md (TODO-VERIFY: numpy not declared in pyproject). Memory-watch: lz4_flex/zstd still zero call sites — deps remain dead. Skipped per filter: docs/developer/PYTHON_OBFUSCATION.md (docs/ excluded), reports-to-pdf-python submodule bump, binary .whl/.ico. No new pages; bookmark advanced e021b09→a18d35c. Unstaged for human review.
[2026-06-18] clarity a18d35c..dab6060 (11 commits) — License keystore: OS keychain removed (file-only); aggregator bucket alignment; nested SQL pushdown. license-portal.md: rewrote Token Storage — thekeyring/OS-keychain default +use_file_keystore()/SecretService-probe fallback +CLARITY_LICENSE_FILE_KEYSTORE+KeystoreUnavailablemapping are all GONE (keystore.rs); now two AES-256-GCM+HMAC encrypted files only on every platform:order_id.bin(key = hardware_fp ‖ JWT_SECRET) andtoken.bin(key = order_id ‖ hardware_fp ‖ JWT_SECRET) — JWT_SECRET now mixed into key material; 0o600 file perms; two-step unlock (fp→order_id→token); plaintext order_id.bin migration; store_order_id MUST precede store_token (activation.rs). Added: I/O-cost row (2 file reads, cached), brick_license() comment fix, runtime.rs hourly-check NotActivated streak tolerance (brick only on 3rd consecutive; KeystoreUnavailable arm now defensive/unreachable), threat-model row (copying .bin files fails on different hardware), CLI/end-to-end-flow/env-var (CLARITY_LICENSE_TEST_MODEno longer toggles keystore) fixes. keyring crate still in Cargo.toml but used only by secure_store.rs secret pack (unrelated). fingerprint.rs change (into_bytes→as_bytes().to_vec()) = no-op refactor, not documented. standalone.md: CLI subcommand note + header. storage-engine.md: new Aggregation Pipeline subsection — post-processingaggregate()(run_pipeline) now anchors buckets to querystart(was epoch-alignedts % bucket); pre-anchor points dropped to avoid u64 underflow; scoped NOTE that the push-down path in storage.rs is unchanged. sqlite-api.md: Phase 5b — new nested-alterPOST /{root}/:id/{rel}/alter(depth-1, merge_parent_constraint injects parent FK into where, dispatches to flat alter shim); 409 Conflict on unique-key violation (check_unique_constraintsDUPLICATE:prefix → conflict_error vs 500); leaf-table SQL pushdown for nested lists (LeafWhereValue enum + split_where_for_pushdown conservative eq/like-only, AND/OR rules, list_nested takes leaf_where_clauses, remaining filtered in memory) avoiding ~2850-row materialization; regex section — trailing-dot patterns (kwh.→kwh.*via has_trailing_dot/is_regex_like/trailing_dot_to_regex); WhereNode section — case-insensitive like/nlike via options flag (WhereOp::Like(String,bool)) + dotted field-path resolution (lookup_path, shared by where & regex trees). python-services.md: data-api gained/sensordata/elog/datapointsroute. env_vars.sh added doc comments for CLARITY_NO_OBFUSCATE/CLARITY_RESTORE — already in python-services Build-time toggles, no change. No new pages; no UNMAPPED files. Memory-watch: no Cargo.toml/lz4_flex/zstd changes in range — deps remain dead. Bookmark advanced a18d35c→dab6060. Unstaged for human review.
[2026-06-20] clarity dab6060..1027dce (12 commits) — secure_store.rs migrated OS keychain → encrypted file; SQLite nested-route any-depth POST + auto-collection; LIKE/NULL filter-semantics fixes; parallel sentinel fill. secure_store.rs: dropped thekeyringcrate (removed from Cargo.toml), now persists the secret pack as a single AES-256-GCM+HMAC-SHA256 file~/.clarity/secrets/secrets-pack.bin(0o600; %APPDATA%\clarity\secrets on Windows), keys = blake3::derive_key(hardware_fp ‖ JWT_SECRET) → same machine+binary binding as license keystore; atomic temp+rename write; legacy single-key API now writes per-key .bin files. db/mod.rs provision_pack ADMIN_PASSWORD pair gained build-timeoption_env!("ADMIN_PASSWORD")fallback (was ""); build.rs declares rerun-if-env-changed=ADMIN_PASSWORD. Documented in api-server.md ("Seed credentials & secret pack" rewrite + env-var table + seeded-users), glossary ("secret pack"), security-model.md (Data-at-Rest row, hardening table, seeded-user note — admin pw now: runtime env > on-disk > build-baked > empty), mqtt.md (broker creds), sqlite-api.md (secret-pack seeding), license-portal.md (keyring fully removed note). sqlite-api.md Phase 5b: nested POST now any-depth (build_nested_post_filter_any_depth via warp::path::tail(), even-count ≥2 tail; create_nested accepts odd-length ≥3 segments + storage arg, resolves FK from last 3 segments + derive_parent_fks); post_create_hook auto-creates default unit+collection forsitesand default collection forunits(lookup_name reads meta_data..name; ///default_grid; update_metadata_fields patches {organization,site,unit}; failures logged, don't fail insert); **dispatch_post_create_sync** centralises tags/tagmeta/equipment sync across flat/bulk/nested POST (nested previously had no sync). Leaf-table pushdown: list_nested arg now `&[LeafWhereClause]` boolean tree (Clause/Or/And via emit_leaf_where — preserves user `or` grouping that the old flat list silently AND-ed); LeafWhereValue Literal/JsonEq now hold serde_json::Value bound via bind_json_value (type-preserving — fixes TEXT-vs-INTEGER 0-row bug); real-column conditions now emit OR-of-both (real col OR json_extract) to match flat route. paths.rs: bfs_paths_with_max_depth root edge-detection fixed `belongs_to`→`has_many` (orgs etc. had empty belongs_to → misclassified as pure leaf → nested orgs/:id/sites|units POST/GET/DELETE never registered). utils/mod.rs eval_op/LIKE fixes: `neq` on NULL → false (SQL != excludes NULLs, was !is_null_value); `nlike` on NULL → false (regex+literal branches), regex-pattern branch defers to regex tree for non-null; `_` no longer suppresses LIKE auto-wrap — only `%` does, so `like:"VTP_G1"`→`%VTP_G1%` substring (like_to_regex + emit_like_inner). storage-engine.md: ensure_file_initialized now fills MISSING_SENTINEL via mmap + rayon par_chunks_mut(4) + flush_async (was single-threaded seek+write_all 1MB chunks) — perf-only, fill value/region unchanged. Skipped: Cargo.toml `Win32_Security` windows-sys feature (commit "new windows security crate added") — added but NO call sites in src yet (only an ACL mention in a secure_store comment); not documented per no-boilerplate rule (memory-watch lz4_flex/zstd unchanged this range — deps remain dead). No new pages; no UNMAPPED files. Bookmark advanced dab6060→1027dce. Unstaged for human review. [2026-06-21] DRAFT — created wiki/product/roadmap/ai-quality/index.md: a PLANNING GUIDE / thought-process capture (per user request) framing a proposed AI-quality roadmap cluster. Core insight documented: the three named topics (acceptable hallucination rate, retrieval quality threshold, inference cost vs feature richness) are coupled knobs sharing a hard prerequisite — a measurement substrate (offline eval harness + production telemetry) the system lacks today (it has enforcement: IncidentValidationRetryPlugin + the grounding instruction, but no measurement). Code-anchored to existing levers/gaps: search_tags similarity_threshold=0.0 default (no floor), top_k=10; getTagMeta uses LoopBack `like` (semantic resolver import commented out); data_analysis_agent direct genai.Client Gemini bypass (cost concentration); get_model() MODEL_NAME factory (cost lever); 15s heartbeat keepalive (latency evidence); CLARITY_APP tool exclusions (feature-richness dial per mode); RestrictedPython sandbox (action-safety). Proposes a 3-layer cluster under product/roadmap/ai-quality/: L0 foundations (evaluation-harness.md, observability.md) → L1 threshold/policy (hallucination-policy, retrieval-thresholds, inference-cost, latency-budget) → L2 governance (change-management ties to Granite roadmap; optional action-safety). Includes a trade-off-triangle framing (one per-turn budget per mode), recommended sequencing (harness first), three PM decisions (numeric-vs-qualitative hallucination target; wrong-tag = safety-class vs UX; cost-ceiling owner), and a self-contained glossary. Child docs NOT yet written (listed as proposed, no dead links). Linked from product/roadmap/index.md. Not code-cited as a dev page; citations retained for grounding. Not committed/staged — for human review. [2026-06-22] clarity 1027dce..f3f16e6 (6 commits) — **Response ID stringification + Windows DLL hardening + tokio stack bump + Cargo.lock untracked.** sqlite-api.md Phase 5b: new "ID fields stringified in responses" subsection — the dynamic `/exactapi` layer now emits **top-level `id`/`*Id` fields as JSON strings, not numbers** to satisfy the UI's `z.string().uuid()` schema validation (tables use INTEGER PK, no real UUIDs). `stringify_row_ids` (top-level only; i64/u64 always, whole floats `1.0`→`"1"`; skips `dataTagId` and `null`; recurses into arrays = include-relation child rows but not embedded sub-docs like meta_data/equipmentLoad/benchmark) + `stringify_response` (object or array) + `reply_value`/`reply_option_value` (stringify-applying replacements for reply_for/reply_option). Shims switched: list/create/bulk-create/update/upsert/findOne + nested-POST + odd-segment list_nested_shim return through reply_value; get_item_shim + even-segment list_nested_shim call stringify_row_ids directly. `warp_dynamic_routes.rs:1009-1100`. processing-api.md: new "Ingest-config ID parsing" subsection — `fetch_ingest_configs_from_api` now reads IDs via `parse_id` accepting both Number and String (the ingestconfigs response is now stringified), `clientsId` via parse_id with string `clientId` fallback (`ingest.rs:324-371`). api-server.md: Windows runtime hardening gained **DLL search-order hardening** — `SetDefaultDllDirectories(0x1000)` at top of `fn main()` drops CWD + legacy search paths from the DLL search order (CWE-427 / "vapt-dll fix"); also fixed stale SetErrorMode/SetThreadExecutionState citation (1654-1664 → 1885-1898). New "Tokio runtime — worker stack size" subsection: main() builds its own multi-thread runtime with `thread_stack_size(8 MB)` (up from 2 MB default), installed via `tauri::async_runtime::set` before `tauri::Builder` and leaked, to stop warp compression-middleware stack overflow when deflating the ~5-10 MB JS bundle (`main.rs:1853-1868`). Header dates → 2026-06-22 / f3f16e6 on all three pages. Cargo.lock deletion: build-hygiene only — the file is now in .gitignore (verified `git check-ignore`), no behavior change, not documented. Memory-watch: lz4_flex + zstd still in Cargo.toml with **zero call sites** in src (re-verified this pull) — deps remain dead. No new pages; no UNMAPPED files. Bookmark advanced 1027dce→f3f16e6. Unstaged for human review. [2026-06-24] clarity f3f16e6..57c553a (30 commits) — **TAG_SCOPE_MAP stale-map fix (warm upsert on tagmeta sync + write-path refresh/retry); push-down aggregation timezone fix; incidents.criticalTags JSON re-typing; alter-shim ID stringification; two new Python services.** tag-resolver.md: documented `upsert_tags`/`remove_tags` exports; new Lifecycle bullet — `sync_tagmeta_created`/`sync_tagmeta_bulk` now `upsert_tags` newly-`Added` tags into TAG_SCOPE_MAP immediately (so write_fast resolves them without waiting for next build_tag_map scan or restart); corrected `resolve_from_tags` return type (HashMap, not Vec) + `parse_write_request` ref (now main.rs:511); new write-path refresh-and-retry note. api-server.md: new "Scope resolution, refresh-and-retry, and skipped tags" subsection — `parse_write_request` now returns `(Vec, Vec)` (2nd = unresolved tags); `/write`+`/write_fast` clone body, rebuild TAG_SCOPE_MAP once + retry on any unresolved tag, fall back to partial parse; `/write_buffered` does same one-shot rebuild in single/bulk/batch branches (route now injects Storage); success appends "(skipped N unresolved tag(s): …)"; `execute_write` errors on zero writable batches (was vacuous ✅); `handle_write_mmap` now returns 400 on parse-fail / 500 on write-fail (was always 200 + ❌ body) so raise_for_status() callers see failures (main.rs:511,750-760,836-973,979-1256). storage-engine.md: amended the dab6060 NOTE — push-down path `storage.rs::read_data_aggregated` received the **same** start-anchored bucketing in 57c553a (PR #257 int-to-float): `bucket_start = start_ms`, `bucket_end = start_ms + ((end_ms-start_ms)/bucket_ms)*bucket_ms` (was epoch-aligned `start_ms % bucket_ms` → 05:30/06:30 IST); each of 5 per-op scan loops gained `ts_ms < bucket_start → continue` underflow guard (aligned_start_ms can precede start_ms) (storage.rs:2314,2351-2356,2463-2466). sqlite-api.md: new `parse_extra_col_value` bullet — read path re-parses extra-column strings starting with `[`/`{` back to JSON Value (fixes incidents.criticalTags returned as quoted `"[{…}]"`), plain strings (dataTagId/email) pass through; replaces `json!(s)` in list_rows/get_row/fetch_rows_for_include/list_nested + include branches (dynamic.rs:112-119). New alter-shim bullet — `dynamic_alter_shim` now stringifies IDs on both paths: `{result:items}` envelope via stringify_response (recursive into include arrays) + streaming json_insert path via stringify_row_ids after fields projection / on raw row (warp_dynamic_routes.rs:1192-1200,1326-1345); updated "applied uniformly across …" line to include /alter. New "Tagmeta sync warm-registers tags in TAG_SCOPE_MAP" subsection (sync.rs:584-597,1064-1078). python-services.md: added `equipment-status-v2` (port 17004, env Q_PORT=1883/BROKER_ADDRESS=127.0.0.1, health-check disabled, no routes) + `util-services` (port 17008, no env, no routes) to Registered Services table — both new submodules in 57c553a (.gitmodules, python_services_config.json now 213 lines, python_services.sh SUBMODULES/URLS + port-clarity checkout blocks); reports-to-pdf-python submodule commit bumped (no doc change). Skipped: SDK/python/clarity_sdk.py — a single per-tag `logger.debug` line commented out (no behavior change, commented-out code not documented). Memory-watch: no Cargo.toml / lz4_flex / zstd changes this range — deps remain dead. No new pages; no UNMAPPED files. Bookmark advanced f3f16e6→57c553a. Unstaged for human review. [2026-06-22] pulse_multi_agents 426c3fa..7cdfec7 (5 commits) — **matplotlib/WebP diagnostic plots + bulk tag-metadata prefetch for incident & system-health agents.** agents.md: new "Diagnostic plots — generate_base64_plot" section — shared plot helper now POSTs to `{public_datacenter_url}/sensordata/spcplot/matplotlib` (was `/sensordata/spcplot`) and expects/returns `image/webp` (was `image/png`); gained optional `tagmeta=`/`equipment=` passthrough so prefetched metadata skips per-plot lookups (utilis.py:884-958, 721-749). New "Bulk tag-metadata fetch — fetch_unit_tagmeta" subsection — now accepts a **list** of dataTagIds → `{"dataTagId":{"inq":[...]}}` returning a list (scalar still returns one dict); filter passed as request param not URL-interpolated (utilis.py:570-600). New "Model configuration — get_model" note — Gemini `gemini-3-flash-preview` gained `http_options=HttpOptions(timeout=600_000)` 600s timeout (utilis.py:444-470). incident_agent section: build_incident_context now issues ONE bulk fetch_unit_tagmeta over all deviating tags + one fetch_equipment, passed into generate_base64_plot, replacing per-tag fetch_tag_limits; get_deviating_tags gained `incident=` reuse param; long-term plot window **1yr→8 months** (525600→350400 min) + analyzer_agent prompt text "1 year"→"8 months"; artifacts now .webp (utils.py:367-525, 436, 511; analyzer_agent.py:104,112). system_health_agent section: same bulk-prefetch pattern (tagmeta_lookup + equipment_lookup → _bounded_plot_gen) replacing concurrent fetch_tag_limits gather; keeps 1yr long-term window; artifacts .webp (agent.py:109-253). NOT documented per no-boilerplate rule: the pervasive `[TIMING]` print instrumentation added across incident_agent/agent.py + utils.py (diagnostic logging only, no behavior change). No new pages; no UNMAPPED files. Header date → 2026-06-22 / 7cdfec7. Bookmark advanced 426c3fa→7cdfec7. Unstaged for human review. [2026-06-29] clarity 57c553a..67ac68c (9 commits) — **Collection-delete cascade + recursive parent delete; on-disk→DB hierarchy sync; any-depth nested alter; string-or-number ID tolerance in auth; alarm equipment/system enrichment; SDK query_fast raw-JSON return.** sqlite-api.md: new `## Recursive cascade delete — delete_item_shim` (gained Storage arg; `recursive_cascade_delete` walks relation graph + `conventional_child_fks` PRAGMA-scan for convention `{parent}Id`/`{parent}_id`/`{Parent}Id`/`{Parent}_Id` columns that `foreign_key_list` misses; cycle-guard HashSet; orgs/sites/units also cascade collection subtree; fixes pre-existing FK-constraint 500 deleting a parent with children) (warp_dynamic_routes.rs:1975-2003,2005-2048,2050-2177). New `## Collections route surface — /exactapi/collections` (create_collections_routes mounted LAST after dynamic_routes, not a SQLite table; `POST /collections` now idempotent — ensure_org/site/unit before metadata.json; `DELETE /collections/:id` 11-step cascade, steps 2-5+10 in one BEGIN IMMEDIATE txn with inlined monitor cascade [NOT ConfigManager, to share txn], 6-11 cache/FS after commit; reply reports per-step counts; `cascade_collections_for_parent` resolves parent path from DB names, list_collections_under_subtree walk, per-collection delete_collection_db_only + remove_tags, single subtree cache sweep, remove_dir_all; does NOT delete parent row) (collections.rs:79-170,467-505,595-733,804-819,908-1054; warp_routes.rs:1293,1487-1488). New `## Folder↔DB sync — sync.rs` (ensure_org/site/unit idempotent upserts linked to admin, site/unit also write customerId; `sync_folder_to_db` startup walk of base/////metadata.json → ensure_* per unique triple, recovers backup-restore hierarchy rows) (sync.rs:14-165,167-271). Nested-alter subsection rewritten depth-1→**any-depth** (`build_nested_alter_filter_any_depth` registered for depth()>=1 [was ==1, 405'd depth-2+]; warp::path::tail() up-front shape validation before body::json to avoid "request body already taken" across sibling alter filters; `dynamic_nested_alter_shim` now takes Vec<(parent_id,child_fk)> applying merge_parent_constraint per segment → AND-chain of FK constraints; GET + any-depth-POST nested filters now reject `alter`-suffixed tails) (warp_dynamic_routes.rs:743-810,855-872,909-940,1557-1612). ID-stringification: new `accessAllowed` special-case bullet — `normalize_access_allowed_units` stringifies numeric customerId/sitesId + every numeric `units[]` entry (legacy mixed int/string), write/response counterpart to auth parse_id (warp_dynamic_routes.rs:1174-1213). db/mod.rs: new "configurations seed file build-time override" — `CONFIGURATION_FILE` cfg-env selects Assets/seed_data/file at compile time (default configurations.json) (db/mod.rs:706-720; env_vars.sh:31-32). api-server.md: new "Startup folder→DB sync" subsection (main.rs:4121-4133); User-Hierarchy + Row-level-access gainedparse_idstring-or-number ID tolerance — accessAllowed sitesId/units[] and check_row_access id/unitsId accept "21"-style strings round-tripped from stringified responses into meta_data BLOBs (auth.rs:93-100,459-471,681-684). storage-engine.md: new "Cache & lock invalidation (collection-delete support)" — invalidate_metadata_cache / invalidate_metadata_cache_for_subtree / pub invalidate_mmap_cache / invalidate_day_locks_for / list_collections_under_subtree (storage.rs:1062-1190). monitor.md: RuleDefinition.unitsId field row (types.rs:78); ConfigManager.delete_collection_cascade (alarm_events→monitor_rules→monitor_collections single txn; NOTE live delete inlines instead, for txn-sharing) (config.rs:413-451); AlarmEventTracker open-path equipment/system enrichment — reads tagmeta equipmentId col + meta_data blob to fill systems/systemName/equipments/equipmentName/equipmentIds (was always []), equipmentId falls back to string blob value when INTEGER col NULL (meta_upload case), id emitted as string (events.rs:388-481). python-sdk.md: query_fast now returns raw response.json() verbatim (PR #260 "dumb bug fix") —(int,float)tuple coercion removed/commented; TODO-VERIFY stale return-type annotation + docstring still claim tuple shape (clarity_sdk.py:765,784-785,815-824). Skipped: connections.rs.boxed()(trivial); monitor/tests.rsunitsId: Nonefixture (test-only). Memory-watch: no Cargo.toml / lz4_flex / zstd changes this range — deps remain dead. No new pages; no UNMAPPED files. Bookmark advanced 57c553a→67ac68c. Unstaged for human review.
[2026-06-30] INGEST — benchmark_results.json (clarity_benchmark_version 3.0, TSBS-aligned, run 2026-06-30T06:39Z on Darwin arm64 / Apple Silicon, loopback, 500 tags @ 60s, year 2024, read_repeat=3). New pagedev/internals/benchmarks.mdholding measured write/read throughput, latency, wire-format efficiency, and value-integrity for the storage engine. Write:write_fast(store_data_mmap) = 1.86 M metrics/s single-stream (3 720 rows/s, p50 batch 206 ms, 0 errors, JSON payload 4.76× raw); concurrent scaling 1→4 writers = 1.76 M→3.33 M metrics/s (~1.9×, p99 tail 28→112 ms from day-lock/mmap contention). Reads across {1,5,10,50,100} tags × {1h,1d,1w,1mo,1y} for all three endpoints —fast_query(JSON, ~25 B/pt, 6.2× raw, peak scan 3.4 M pts/s, decode-bound on big responses),fast_query_binary(v1 per-tag (u64 ts, f32 val), 12 B/pt, 3.0×, peak 46.8 M pts/s),fast_query_optimised(v2 shared-timestamp + per-tag validity bitmap, →4.2 B/pt = 1.05× raw at 100 tags, peak 144 M pts/s, ~36× JSON). Integrity: max round-trip error 0.0005 = ½ the 0.001 i32×1000 step → empirically confirms the fixed-precision integer storage model. Aggregation flagged TODO-VERIFY — all 56 cross-checks FAILED (avg_points_returned = raw sample count, not bucket count; "count > max possible", "sum ≠ mean×count") → the benchmarked requests returned raw un-bucketed samples, not server-side aggregates; latencies recorded but NOT cited as aggregation perf. Caveats recorded: single run/3 samples-per-cell, page-cache-warm, no physical network,write_bufferednot tested,storagesection empty (no post-write disk size). Cross-updates: storage-engine.md (measured throughput in Write Methods; new "Binary response formats — binary_format.rs" subsection documenting v1/v2 + wire efficiency, citing binary_format.rs:32-90 + main.rs:3547/3582/3604; aggregation cross-check caveat banner; See-also + header → 2026-06-30); capacity-planning.md (measured ingest ceiling on the Ingest CPU row; header note that 0.0005 round-trip confirms the i32×1000 model; header → 2026-06-30); overview.md (Measured-performance note after read path); dev/index.md + index.md register the new page. Endpoints verified in clarity@67ac68c (main.rs:3483/3547/3582/3604 under /exactapi prefix main.rs:3968). Figures cite Docs/benchmark_results.json (a measured artifact, not source) + the relevant code paths. No code pull; bookmarks unchanged. Unstaged for human review.
[2026-06-30] INGEST — performance product docs from benchmark_results.json (v3.0). Newwiki/product/performance.md— benchmark-backed sales/proposal brief (hero stats, "why it's fast" prose, Clarity-internal Mermaid charts for ingest scaling / query latency / wire size / scan throughput, a cited industry-context table, "what it means for you", quotable snippets, and a methodology+caveats footer). Honesty constraints baked in: aggregation numbers EXCLUDED (v3.0 cross-checks invalid) and the "70–90% Zstd/LZ4 compression" claim EXCLUDED (not implemented — dead deps). Industry context uses cited public figures only (TDengine TSBS IoT 2.83M metrics/s on 32-vCPU Xeon; QuestDB 11.36M rows/s & TimescaleDB 1.24M rows/s on 32-vCPU EPYC — flagged as different hardware/datasets/units, not head-to-head; sources linked). Also produced a shareable visual one-pager Artifact (instrument-readout design: dark hero with animated signal-trace canvas, mono tabular numerals, amber=Clarity/best bar charts) at https://claude.ai/code/artifact/d09eee0e-b23d-4d49-8936-3b06770e91c2 — linked from performance.md + product/index.md. Cross-updates: product/index.md (new Performance section, header→2026-06-30); historian-website-copy.md Internal Note rewritten — claims #2 (throughput) & #5 (wire) now ✅ benchmarked, #3 aggregation-speed flagged invalid, #4 compression ❌ contradicted by code (remove before publish). Product docs cite the benchmark source file + dev/internals/benchmarks.md (no code citations per product-namespace rules). Not committed/staged — for human review.
[2026-06-30] clarity 67ac68c..1e53e3f (3 commits, PR #261 "compression-setup") — zstd cold-tier compression + SIMD NEON scan + two-phase write locking + WAL binary codec / 8 shards. storage-engine.md: new top-level "Cold-tier compression — sealed.bin.czstfiles" section — on-disk format[magic CLRTY_V1][n_cols u32][ppd u32][per-col (offset u64, comp_size u32)][zstd chunks], per-tag-column zstd level 3 (rayon-parallel);seal_day_file/unseal_day_file/seal_old_day_files(hot_days);ColdDayReader(mmap + lazy per-column decompress, &self thread-shared);DayStorage::Hot|Cold+ColData::Borrowed|Ownedso reads/aggregation handle both tiers (hot zero-copy, cold decompress-on-demand); writes unseal→write→reseal viaBulkWriteContext.to_reseal; nightly sealer task wired in main.rs (HOT_DAYS=7, first +5min then every 24h, spawn_blocking) (storage.rs:45-275,65-186,1697-1780,2480-3094; main.rs:3199-3221). New "Two-phase locking (store_data_mmap/store_data_fast)" subsection — phase 1 holds collection lock for metadata + acquires per-day locks in ascending sorted order (day_lock_arcs outer-scope outlives _day_guards), phase 2 does file I/O under per-day locks only so writers to other days proceed; deadlock-free (collection→days sorted order) (storage.rs:1963-2229). New "SIMD-accelerated scan (NEON)" subsection —scan_mmap_to_optionsuses aarch64 NEON 4-wide (scan_mmap_neon) with scalar fallback; perf-only, semantics unchanged (storage.rs:292-355). Write Buffer: NUM_SHARDS 4→8 (heading + flusher names {0..7}); new "Binary WAL codec + per-batch fsync" subsection —wal_encode/wal_decodebinary (~20-50 B/entry vs ~120 JSON; legacy JSON still replays via{detection), fsync moved per-append→per-batch viaflush_to_disk()(write_buffer.rs:18-127,216-241,270-285,610-618). Fixed the binary-format parenthetical (zstd no longer "reverted" — now on-disk cold-tier; lz4_flex still dead, wire still uncompressed). Header→1e53e3f. capacity-planning.md: rewrote "Cold storage & compression" — compression now EXISTS (the "engine does nothing for cold data" claim retracted for compression; tiering+on-disk compression yes, retention/rollup/value-recoding still no); resolved the lz4/zstd dead-deps note (zstd 3 call sites, lz4_flex still 0); 3-way hot/cold distinction with HA-sealed-vs-cold-tier-seal disambiguation; combined-sizing table now notes 3.03TB is all-hot worst case (cold compresses, ratio TODO-VERIFY); Queries/CPU row notes cold-read decompress (~0.1ms/day-file author-measured) + NEON; header→1e53e3f. api-server.md: cold-tier sealer added to startup-task list; WAL section notes 8 shards + binary codec + per-batch fsync; header→1e53e3f. glossary.md: new "seal / cold tier / .bin.czst" + "ColdDayReader" entries, disambiguated SealedReconciler ("sealed"=immutable UTC-rollover, not compression), fixed stale WAL entry (now always-on + binary codec, was "disabled in single-node"); header→1e53e3f. overview.md: write-path cold-tier note. TODO-VERIFY flagged: HA SealedReconciler checksums past-day.binfiles but the sealer turns those into.bin.czst(PR #261 doesn't touch ha/) — possible hot/cold divergence on an HA pair, needs confirmation. Memory-watch [[watch-lz4-zstd-wire-compression]] RESOLVED for zstd (on-disk cold-tier, not the watched wire compression); lz4_flex watch kept (lighter). Product: performance.md "does not compress at rest today" claim corrected to note the new cold tier (ratio still unmeasured). NOTE: the untracked draft website-copy files (historian-website-copy.md, agentic-ai-website-copy-v1/v2.md) are no longer on disk (never committed; HEAD's product/index.md references them — pre-existing dangling links, the user's commit-index/keep-drafts-local pattern) — so the planned historian-copy compression-claim correction is moot; left index.md/perf companion links untouched. No new pages; no UNMAPPED files. Bookmark advanced 67ac68c→1e53e3f. Unstaged for human review.
[2026-06-30] EDIT (follow-up, user-directed) — pruned dead links + simplified capacity-planning + specified benchmark hardware. (1) Pruned the dangling website-copy links the user deleted as stale: removed the "Marketing / website copy drafts" section (3 links: agentic-ai-website-copy-v1/v2, historian-website-copy) from product/index.md + the stale "no product documents ingested yet" line; removed the companion-copy line from performance.md. Verified no website-copy refs remain outside log.md (append-only history retains them). (2) Rewrote capacity-planning.md from scratch — shorter (259→~95 lines) and compression-aware. New structure: "The cost model" (dense i32 grid + cold-tier as the hot ceiling), "Disk sizing" (compact 3-row ceiling table + backups/HA multipliers), "RAM & CPU" (condensed, measured ingest ceiling, cold-read decompress, bulk-extraction OOM risk), "What the engine does NOT do" (retention/rollup/value-recoding/object-tiering still none; zstd live, lz4_flex dead), "Configuration knobs". Dropped the duplicate 8k-and-15k/25k/50k worked examples + the now-false "engine does no at-rest compression" caveat. Inbound anchor links to the old #cold-storage--compression repointed to #the-cost-model (storage-engine.md binary-format note, performance.md). (3) Specified benchmark hardware everywhere = MacBook Pro · Apple M5 (10-core CPU) · 16 GB RAM (per user; matches commit c6e4e55's "M5 mac, 16GB"): benchmarks.md (new Machine row + headline + caveats), storage-engine.md (cold-decompress note), capacity-planning.md (measured note + decompress note), performance.md (hero/measured-note/context-table/takeaway/methodology/snippets), and the HTML one-pager (hero, context row, methodology footer + cold-tier wording) — redeployed to same URL https://claude.ai/code/artifact/d09eee0e-b23d-4d49-8936-3b06770e91c2. (4) performance.md relook: compression claim corrected (cold tier exists, ratio unmeasured), added a "Long-term storage cost" benefit row, reviewer-checklist item updated. No code change — docs/product only; clarity bookmark stays 1e53e3f. Unstaged for human review.
[2026-07-01] clarity 1e53e3f..9847dba (13 commits, PR #265) — XSS input sanitization across the write surface; ingest scope-resolution & MQTT hardening; SQLite soft-FK completeness; MQTT broker auth; monitor init made non-fatal; attachment upload robustness. sqlite-api.md: new top-level "## Input sanitization (XSS)" section —common.rsport of the LoopBackmiddleware_xss(sanitize_stringCow fast-path + HTML-tag-gated 5-char entity encode, single-encode not double;sanitize_value/encode_html_valuerecursive;sanitized_json::<T>()→400-on-mismatch +sanitized_json_value()warp filters replacingwarp::body::json;parse_raw_querynow sanitizes keys+values) adopted across all crud_routes/nested_route macros, dynamic flat/nested-alter/nested-POST filters, connections/tag_mappings/collections + main.rs auth routes; documented the bulk-body shape change (Vec<Value>→Value+.as_array().unwrap_or_default()→ non-array body now silently empties instead of 400). db/mod.rs section: new "Soft-FK column backfill" (copiesingestconfigs.{clientsId,unitsId,siteId,orgsId}+tags.ingestconfigIdfrom meta_data JSON → real SQL cols, PRAGMA-guarded, non-fatal) + "Admin accessAllowed sync (rewritten)" (old correlatedUPDATE…GROUP BY o.idtruncated hierarchy → now compute full json_group_array once via query_row, json_insert into every admin profile). Phase 2 soft-FKs: added units/sites/orgs→ingestconfigs (unitsId/siteId/orgsId) + ingestconfigs→tags (ingestconfigId) — legacyPRAGMA foreign_key_list(ingestconfigs)can return empty. Phase 3:fk_columns_for_table/fk_references_for_tablenow merge SOFT_FK_EXCEPTIONS (so create_row stores value in the SQL col, not meta_data);list_nestedCTELEFT JOINfor soft-FK edges now(alias.fk=cur OR json_extract(alias.meta_data,'$.fk')=cur)(OR-of-both, matches flat route). Auto-collection creation: ID-based default namesunnamed_site_{id}/unnamed_unit_{id}(was fixedunnamed_site1/unnamed_unit1; insert temp name→UPDATE with row id) + new "Typed-layer org-create no longer duplicates defaults" (create_item_warporgs branch now only links-to-admin; post_create_hook owns default tree → fixes stale FS folders after rename). Folder-rename:rename_dirtarget-exists now removes stale source + returns Ok, plus cross-device old-path cleanup; newcleanup_stale_foldershelper documented as not-yet-wired (zero callers). Collections:Metadata.tags/descriptionsgained#[serde(default)](tolerant deserialization for ingest scope cache) + collection write routes use sanitizing filters. api-server.md: new "## Input sanitization (XSS)" cross-ref section (register/login/admin/user/delete/update_password/create_collection/update_collection/fast_query_tag_mapping/write_tag_mapping/create_qdrant_collection switched); JWT verification now stripsBearer/bearercase-insensitively (auth_middleware simplified to empty-check only; expired-token log warn→debug); "Non-fatal monitor init" note (DB-open/schema failure logs+returns None instead of killing the setup task → real routes+Leader-gated agent on success, else a 501monitoring unavailablefallback filter at /exactapi/monitor/); attachment upload boundary/filename resolution hardening (extract_boundary_from_content_type quoted/trailing-param aware, parse_multipart Err-on-missing-boundary, filename priority query→X-Filename→Content-Disposition[+RFC5987 filename]→autoupload_{ts}{ext}via guess_extension). mqtt.md: macOS/Linuxsetup_mosquitto_credentials(locate homebrew/distro conf dir,mosquitto_passwd -b -cpwfile, patchpassword_file+allow_anonymous false, systemctl/brew reload; env→secret-pack→defaults; non-fatal) + non-Windowsstart_mosquittonowis_external:true(broker = externally-managed system service, no spawn/kill); async clientinit_mqttnow authenticates viaload_mqtt_credentials(env→secret pack→admin/"")set_credentials. processing-api.md: ingest config now carries org/site/unit_id + reverse ORG/SITE/UNIT_NAME_BY_ID caches (fetch_orgs/fetch_sites);/collection→/collectionsendpoint fix (singular 404'd silently → empty TAG_SCOPE_MAP → every payload skipped_empty_unit); TAG_SCOPE_MAP refresh no longer overwrites a populated map with empty data; per-tag scope config-ID fallback (unresolved/empty-unit tag → scope from config org/site/unit_id via reverse caches, grid=default_grid); tag-prefix double-apply guard; secondary MQTT topic{clientId}/{configId}/{dataTagId}for ALL tags (primaryu/{unitUID}/{tag}/{r,sd,v/e}still needs UID); 30s timeout-based publish watchdog (was unconditional 5s warn); ingest response 204→200 OK (handle_ingest_v1 + handle_ingest_backfill); updated ingest mermaid diagram. Skipped:api/tag_resolver.rs(+18) — logging-only (build_tag_map collection/tag counts, no behavior change, no doc);docs/developer/openapi.yaml(+4423) +docs/developer/swagger-ui.html(+85) — under excludeddocs/(config.yml), and the wiki keeps its owndev/api/copies; main.rs route-composition.boxed()churn — compile-time/type boilerplate, no consumer impact. No new pages; no UNMAPPED files. Bookmark advanced 1e53e3f→9847dba. Unstaged for human review.
[2026-07-10] OVERHAUL (reader layer, augment-in-place) — added reading-guide.md (front door), concepts/{index,data-model}, guides/{index,pi-connector-setup}, CHANGELOG.md (Keep a Changelog); index.md wired to reader layer. CLAUDE.md: new "Reader layer" section — Diátaxis page-type rule, 3 page templates (how-to/concept/product), canonical running example (acme-power/plant-1/boiler-2), reader-layer citation rule (prose-first, link down to dev/), log-vs-CHANGELOG split, freshness.sh + citation-validator to-wire notes; directory diagram expanded to show reader vs reference layers. dev/ + hld/ pages untouched (citation engine intact). This is the PATTERN batch — for human approval before rolling the templates out to remaining connectors/concepts/product. No code pull; bookmarks unchanged. Unstaged for human review.
[2026-07-10] OVERHAUL cont. (taxonomy + naming + end-user orientation) — per user decisions after reviewing website mockup V10. CLAUDE.md: replaced "Deployment modes" with "Product, deployment & naming" (Pulse platform; Pulse Historian/Chronos; Pulse Copilot/Clarity AI; edge-onprem vs managed-cloud axes; dev/ topology mapping table; migration note retiring [mode:] tags), updated reader-layer conventions (end-user personas: API/terminal + UI/app; screenshot/graph/diagram placeholder convention 🖼️/📊/🎞️ + [VERIFY-UI]; dropped "tables over screenshots"), template Applies-to → product·deployment. reading-guide.md rewritten around products + reader types (adds Plant/app user and API/terminal user personas) + product×deployment table. New guides: query-data-terminal.md (end-user, fully code-grounded from rest-api.html + SDKs) and using-the-app.md (UI scaffold with screenshot placeholders, pending frontend access). Naming reconciled in concepts/data-model.md + guides/pi-connector-setup.md (Clarity→Pulse Historian). CHANGELOG updated. Flagged for rollout (not yet done): overview.md still says "Pulse is an APM platform on cloud" + uses "Clarity" as platform name; 86 [mode:] tags across dev/ pages; opc-ua.md reframe (native vs OPCConnect gateway). No code pull; bookmarks unchanged. Unstaged for human review.
[2026-07-11] OVERHAUL cont. (inherited-framing sweep — rollout item 1) — overview.md REWRITTEN to the Pulse taxonomy (products × deployment + technical-topology table mapping the retired modes; header re-stamped clarity@9847dba + pulse_multi_agents@7cdfec7; endpoint paths corrected /api/→/exactapi/ — verified main.rs:3993 wraps write/fast_query routes at :3491-3651; auth "all endpoints" claim softened with security-model link; OPC-UA connector claim corrected to gateway framing). ALL [mode:] tags migrated (0 remain outside log.md + intentional "(was …)" mappings): dev/ pages drop the tag or convert to one-line Deployment/Platform notes (api-server ×5, python-services ×5, processing-api ×3, monitor, pi-connector, ha, mqtt ×2, process-manager ×2, benchmarks, backup-system, capacity-planning, agents ×2, license-portal, ADR 0001/0002/0004); deployment/{historian,standalone,cloud-copilot}.md headers → topology lines (Historian-only / AI bundled (edge) / AI separate (cloud)); hld/deployment-views.md sections re-titled by topology, hld/system-context.md notes converted + stale "OPC-UA not implemented, only a seed file" fixed; reader/product →Applies to: <product> · <deployment>(pi-connector-setup guide, roadmap plant-doc-rag/local-llm-granite/ai-quality); glossary "historian mode" entry replaced by "deployment topology" mapping entry; index.md + dev/index.md labels updated. opc-ua.md REFRAMED "no native connector + external OPCConnect gateway": native absence re-verified at 9847dba (no module, no crate; /opc-network stub MOVED main.rs:3581-3592→4066-4077 — citation rot fixed); opcuser.json citation rot fixed (file deleted upstream a18d35c; opcuser seeded via users.json:13-17 + secret pack key opc-password, secure_store.rs:83, db/mod.rs:1025-1035); gateway documented from Docs/OPConnect.pdf (DEV-INT-01-OPC UA v1.0: Node.js node-opcua driver, southbound opc.tcp + user/pass or X.509, NodeId assembly ns/prefix/suffix, 60000ms default subscribe interval, northbound wss://…/mqtt + https://…/exactapi to Pulse cloud); endpoint-shape match with clarity documented (mqtt_ws_proxy.rs:6-7 reg. main.rs:4099; /exactapi main.rs:3993) with TODO-VERIFY on edge-install use + opcuser-as-gateway-credential. CLAUDE.md repo table + module map opcuser row updated; CHANGELOG Changed/Fixed entries added. No code pull; bookmarks unchanged (clarity 9847dba, pulse 7cdfec7). NOTE: .llmwiki/state/pending_update_clarity.md brief pending (9847dba..c41a03b, 85 commits) — needs a separate /wiki-update session; pulse brief is stale (bookmark already at its target 7cdfec7). Unstaged for human review.
[2026-07-12] clarity 9847dba..c41a03b (85 commits, PRs #283-#294) — MAJOR: blob storage (string/array values); on-disk V2 XOR codec; hot ring buffer; disk guard; sealing config (hot_days=0, hourly, delta+varint V2 seals, seal_now); ingest→MQTT decoupled (mqtt_fanout, lossy); write-buffer WAL redesign (fsync opt-in); monitor/pm/sqlite test-coverage wave; licensing hardening; Windows codesigning + perMachine installer. Analyzed via 7 parallel Explore agents + 1 follow-up (storage/sqlite/main/monitor/ingest-mqtt/licensing-pm-pysvc/ha + V2 addendum). storage-engine.md: NEW sections On-disk format V2 (STORAGE_FORMAT_V2, disk_xor=i32::MIN, zero sentinel → born-sparse files, no migration, storage.rs:26-48,650-672), Blob storage (.blob heap CLRTYBH1 + .bref ref grid, crash-ordering record-before-ref, BlobPredicate search, .blob.zst seal; blob_store.rs), Hot ring buffer (row-major RAM, ≤1s collections, 128MB cap), Read fan-out/rayon-8MB 50k fix (main.rs:2318); WriteBuffer rewritten (config 200k/50ms/500k, admit 10M backpressure, ahash shards, per-cycle flusher journal + wal_fsync=false, panic containment); sealing section revised (config keys, seal_now storage.rs:6100, SEAL_MAGIC_V2 :146, hourly main.rs:3575-3600); V1-only fill note; SealedReconciler TODO-VERIFY → RESOLVED-INERT. ha.md: Arc+conn_generation; digest-skip (tsdb :554-600, sqlite ship_file→Ok(bool) :372-390); CollectionCreate +u32 storage_format (:335-366,437-441); V2-aware apply + fill-skip (:733-772) + lastlist warm (:786-807); agent panic-supervised tick (:145-171); firewall protocol param UDP/5353; CONFIRMED LIMITATION: reconciler builds {day}.bin only (:1296-1310) → all sealed days silently no-op (both-None short-circuit :1108-1110) — cold-tier divergence undetected on HA pairs. monitor.md: snapshot Arc-shared (:8-31), reader dual-codec disk_xor decode (:55-56,180-183), poll 1000ms properties note, test-only wave note (config/types/rules/evaluator; events/auto_rules untouched). processing-api.md: ingest section rewritten (raw Bytes+413; bg config-cache loop CACHE_BOOTSTRAPPED; in-process refresh_tag_scope_map via handle_list_collections + REFRESH_IN_FLIGHT 200ms debounce; historian_write buffered|fast|http via init_ingest_backends main.rs:3290, write-before-MQTT, 503 backpressure; NEW § MQTT fan-out — mqtt_fanout.rs (cap 8 try_send, coalesce freshest-wins, QoS0 try_publish_nonblocking, topics u/{uid}/{tag}/{r,sd,v/e} + {clientId}/{configId}/{tag} — tag NAME not dataTagId, 10s summary, no watchdog); internal_http_client shared pool (elog×3, ems×6, webpi_meta×3); new mermaid). api-server.md: blob+seal_now route table (main.rs:3994-4208); disk-guard bullet (10s sampler, 507, 1.25× hysteresis, disk_guard.rs); sealer/hot-ring-compactor task bullets; scope-resolution section re-anchored (parse_write_request+find_binary_path REMOVED → resolve_write_scopes :177/build_scoped_writes :203/run_write :400); NEW § Config surface changes (keys, changed defaults 1024/3600 vs properties-pinned 256/120, properties renames incl. rate_limit/cors/auth.internal_token) + § Misc (start_adk.rs deleted dead code; win_console HideConsole; VBScript watchdog; asset_body zero-copy; perMachine + codesign; auth.rs OnceLock perf). sqlite-api.md: NEW § c41a03b (shallow_merge replaces json_patch everywhere + update_all single-txn; PATCH /:id; bind_json_value/value_to_opt_i64 coercions; WhereOp::All + contains/ncontains + in-alias; emit_neq/nin now OR-IS-NULL — LoopBack semantics, TODO-VERIFY in-memory parity; FkRef 3-pass + JSON_SOFT_FKS tagmeta INNER JOIN CAST json_extract + 3 functional indexes db/mod.rs:1319-1327; statuses.ingestconfigId col+backfill+dual-FK hook; error_to_reply 400s; get_item_shim include; alter-shim 16KB streaming + expand_critical_tags wiring; /_tables + /_schema; paths url_template configs/:id/configs shape change; invalidate_last_values on collection delete; REMOVED query_cache.rs + get_items_warp_cached + handle_rejection + HeatRate + sync_admin_access) + header test-only note. license-portal.md: fingerprint table updated (Linux /proc/cpuinfo fallback REMOVED, macOS hw.cpufamily→IOPlatformSerialNumber, machine-id board fallbacks) + anchor-quality block (is_placeholder/real_anchor_count/fp_matches :127-152, InsufficientEntropy activation.rs:72-79); nvram REWRITTEN → mirrored monotonic HWM (guard_dirs, read_hwm max, ROLLBACK_SLACK 6h); run_checks clock rules + build-manifest binding (pubkey_b3); HEADLESS_LINUX_LICENSING.md verified accurate (licensing_status Tauri command exists — doc's "no status" is CLI-only). process-manager.md: watchdog 30s/300s; c41a03b block (hide_console, sysinfo single-PID + exe-path match, Python-skip-on-restore, windowed restart counts, 256-line batched log tailer, logs_ui XSS). python-services.md: python_runner.rs section → REMOVED (32c6c65; papa_agent_app via ProcessManager main.rs:2005,5013-5018); work-dirs/data_dir/CLARITY_LOG_DIR/PDF_OUTPUT_DIR; sysinfo port-kill replaces three-tier; report-scheduler 18000 row; meta-upload +3 routes; Chrome-for-Testing + kaleido>=1.0.0. pi-connector.md: live connector logging-only note; meta connector data-dir override + shared client. mqtt.md: integrity-gated installer (verify_asset_file/dir refuses elevation) + ps1 ACL hardening; client log gating; try_publish_nonblocking + fanout xref. tag-resolver.md: FastMap retype; pub(crate) handle_list_collections; parse_write_request removal note. backup-system.md: stale interval test fixed (state.rs:242 now 180). SDKs: python-sdk.md (_ArrayBackedDict + _clarity_arrays DataFrame fast paths; shared requests.Session HTTPAdapter 16/32; query_optimised pipeline → /exactapi/fast_query_binary with shape-preserving reboxing :936-965; /api/→/exactapi/ in write table; all cites re-anchored) + js-sdk.md (both decoders rewritten to bulk typed-array views + scratch buffer + SWAR popcount; FastBinaryReader retained-for-workers note; TODO-VERIFY: JS client still targets /api/ but server serves only /exactapi/ (main.rs:4539, no alias) — Python migrated, JS not; cites re-anchored). capacity-planning.md: V2 born-sparse cost-model note; hot_days=0 steady state; blob sizing; disk-guard 2GB floor; caches update (dead query-cache knobs); new config-knob table. overview.md: sealer/cold-tier + buffered 50ms/200k + blob mention. glossary.md: +storage format V2, blob store, hot ring buffer, disk guard, MQTT fan-out; WAL entry updated (fsync opt-in). index.md + dev/index.md python_runner descriptions dropped. CLAUDE.md module map: +blob_store/hot_ring_buffer/disk_guard/fast_hash/mqtt_fanout/win_console rows; python_runner row DELETED. Cargo/profile: panic abort→unwind, release-max profile, +blake3/crc32fast/ahash/flume (noted in storage section). Memory-watch: lz4_flex still ZERO call sites (re-verified) — dead dep; zstd live (blob + cold tier). Skipped per filter: docs/developer/* + docs/user/* (excluded docs/, used as orientation only), order_portal/portal_ui.py (2 lines), test-only diffs, submodule bumps, seed/config data churn. No UNMAPPED files. Bookmark advanced 9847dba→c41a03b; both pending_update briefs deleted (pulse brief was stale — its range already processed). Unstaged for human review.
[2026-07-12] STRUCTURE — (1) index.md consolidation: deleted 5 thin sub-indexes (concepts/, guides/, hld/, dev/adr/, dev/integrations/ — 14-27 lines each), folding their content (incl. planned-page lists and the hld page table) into the master wiki/index.md + dev/index.md; kept 3 indexes: master, dev/ (30+ pages), product/ (required by /wiki-ingest). Inbound directory links fixed in reading-guide.md + master index. (2) Provenance-footer sweep: moved the> Last updated:stamp (+ attached> Rangeaudit notes) from the top to a---footer on 37 reference pages (dev/, hld/, glossary, overview's Sources line); tops now carry orientation only (Type/Applies-to/Deployment-Platform-Topology notes); CLAUDE.md "Page stamps" convention + quality rule updated (format unchanged —> Last updated:prefix stays greppable for the planned freshness tooling). (3) /wiki-update workflow hardened in CLAUDE.md: stale-brief check + brief deletion after bookmark advance; >20-commit fan-out guidance; reader-layer impact check (grep reader pages for links/claims about updated dev/ pages); CHANGELOG.md step added; hook backpressure snippet documented for the human to apply (Claude cannot write to source-repo hooks). (4) State-file cleanup: pulse_agents_repo_path duplicate deleted; CLAUDE.md + .llmwiki/config.yml now reference pulse_multi_agents_repo_path (matches hook). Unstaged for human review.
[2026-07-12] VISUAL — created wiki/hld/assets/pulse-stack.svg (hand-written SVG, ~150 lines): styled 5-layer stack rendering of the system-context architecture (OT sources → connectivity → data/Chronos [hot mmap + zstd cold tier + blob store + SQLite] → capabilities → application), amber ✦ marks agent-layer components with a topology legend; embedded in hld/system-context.md above the Mermaid (Mermaid remains the regenerable source of truth). Trial for the D2-vs-SVG showcase decision (rollout: product Introduction diagrams pending user verdict). Unstaged for human review.
[2026-07-12] clarity c41a03b..45a686e (35 commits) — HA secondary shadow cache + zero-lossPOST /exactapi/shadow_write(ha.md new §, api-server.md, processing-api.md, js-sdk.md shadow dual-write, glossary); cold-tier reconciliation now seal/blob-aware over logical bytes — earlier "reconciliation inert" limitation RESOLVED in code (ha.md, storage-engine.md, capacity-planning.md banners rewritten, glossary SealedReconciler); gaps-agg query-start anchoring fix (d79dca2) + serde lowercase-canonical trim/filter/order/boundary (storage-engine.md, both SDK op tables); tag_mappings routes mounted (were 405) + find-allGET ?filter, invalidate_schema_cache, CREATE TABLE, CLARITY_BU/default_bu, BelongsTo includes, update_all TEXT-PK fix (sqlite-api.md); Python SDK mapped_tags read→find-all (python-sdk.md); tag_resolver readiness gate off boot critical path (tag-resolver.md); Windows disk-guard strip_verbatim fix + requireAdministrator manifest + boot reordering (api-server.md); shipped clarity.properties ingest.write.method=fast, hot_ring drain 30s→60s; HA admin mutating endpoints now bearer-gated. binary_format.rs:32-90 cite corrected (v2 118-198); SDK per-method citation drift re-anchored (+6/+44 py, +39/+55 js). Footers → 45a686e. Unstaged for human review.
[2026-07-12] pulse_multi_agents 7cdfec7..88241b9 (2 commits) — system_health_agent UnboundLocalError fix (plot_tasks = []init in build_system_health_context); internal-only, no behavior change → no wiki page edit (agents.md unchanged).
[2026-07-12] READER (rollout item 2, partial) — created product/introduction.md (product front door: two-product framing Pulse Historian+Pulse Copilot / engines Pulse Chronos+Clarity AI; vocabulary table; Mermaid 5-layer stack; legacy-historian comparison table vs OSI PI/Honeywell/IP.21/Canary with honest-framing note + performance.md methodology link; migrate-vs-keep adoption fork; deployment matrix; reviewer notes on unreconciled footprint numbers + unwritten migration whitepaper) + product/copilot-use-cases.md (use cases by role mapped to shipping sub-agents per agents.md; 7-step→3-step shift; end-to-end worked example on canonical boiler-2/idfan_vibration; honest boundaries incl. anti-hallucination + CLARITY_APP reduced tool set; [VERIFY] on offline-AI specifics). Sourced from website/pulse-website-mockup_V10.html (V10) + agents.md + performance.md (product pages cite source docs, not code). Wired into product/index.md, master index.md, CHANGELOG (Added), reading-guide.md (Plant/app-user + Product/sales/RFP paths). All 18 relative links verified resolving. FLAGGED (not fixed — needs human decision): (a) wiki/overview.md was renamed to repo-root home.md (commit 7d082a9, Wiki.js landing convention) → ~6 inbound./overview.mdlinks now DEAD (reading-guide ×3, index.md, glossary, hld/, agents.md); (b) wiki/product/roadmap/ is gitignored so the item-2 roadmap Now/Next/Later reshape is not committable as-is. Unstaged for human review.
[2026-07-12] AUTOMATION (rollout item 3) — built.llmwiki/validate_citations.sh(hard gate: 904 citations checked across wiki/**/.md except log/CHANGELOG; asserts file exists + line/range/comma-list in-bounds; exits 1 on failure; skips docs/openapi.yaml) +.llmwiki/freshness.sh(advisory: reads footer clarity@/pulse@ sha, reports STALE pages whose cited source moved in sha..HEAD; exits 0) +.llmwiki/hooks/pre-commitsample (validator blocking + freshness advisory; install via symlink). Bash-3.2/BSD-grep portable. First run surfaced + FIXED 6 broken citations: mqtt.md ×2 (mqtt_process.rs refactor 472-602→185-317, 604-615→319-327), tag-resolver.md ×2 (heading shorthandclarity:tag_resolver.rs/pulse_multi_agents:tag_resolver.py→ plain filenames; full-path Source cites unchanged), local-llm-granite.md (.../incident_agentellipsis → full path), historian.md (clarity:docker-compose.ymldoesn't exist — reworded, dropped false cite + stale "0-byte placeholder" claim). CLAUDE.md Automation section updated (P0 marked BUILT + usage). Validator now green (0 failures). Unstaged for human review.
[2026-07-12] READER (rollout item 4, batch 1 — concepts explanation tier) — created concepts/historian.md (sample lifecycle connect→store hot/cold→serve→act; three write modes table; hot-vs-cold worked example; why-it's-fast; links down to storage-engine/api-server/capacity-planning/benchmarks), concepts/agent-layer.md (orchestrator+6 sub-agents diagram; L1/L2/L4 intent classification; anti-hallucination grounding rule; conceptual delegation example; edge-vs-cloud topology + CLARITY_APP reduced tools; links to agents.md + tag-resolver), concepts/high-availability.md (active-passive pair; hot delta + cold sealed-file reconciliation + SQLite snapshot replication; VIP failover; singleton gate; zero-loss shadow cache; failover worked example; links to ha.md + ADR 0001 + backup-system). All prose-first,> Type: explanation(+Applies toon agent-layer/HA), reader-layer citation rule (link down, no file:line). Wired: index.md concepts list (3 planned→real, time&timezones still planned), reading-guide reader paths (Operator→HA-explained, Backend-dev→historian, AI-dev→agent-layer), CHANGELOG. All 18 links verified resolving; validator green. Remaining item 4: concepts/time-timezones; guides (mqtt, opc-ua, enable-ha, backup-restore, monitor-rule, sdk-quickstart); start/ quick-starts. Unstaged for human review.
[2026-07-12] NAV FIX — repointed all 8 deadoverview.mdlinks → repo-roothome.md(the overview was renamed to home.md in 7d082a9 for the Wiki.js landing page, one level above wiki/, so links needed one extra../): reading-guide.md ×3 (../home.md), index.md, CHANGELOG.md, glossary.md (../home.md#technical-topologies), hld/deployment-views.md (../../home.md#…), dev/architecture/agents.md (../../../home.md#…). Anchor#technical-topologiesverified present in home.md. dev/hld footer stamps NOT bumped (internal link repair, not a source-content change). CLAUDE.md structure map annotated (overview.md is published as repo-root home.md — do not recreate wiki/overview.md). log.md audit entries left untouched (historical). NOTE: home.md sits outside wiki/ so the citation validator/freshness tooling (scoped to wiki/**) does not scan its handful ofclarity:citations — minor gap, left as-is. Unstaged for human review.
[2026-07-16] READER (user-review round 1, items 3+2) — API REFERENCE ADOPTED: created wiki/dev/api/ (index + 10 group pages: auth-users, collections-writes, queries, ingest, aggregation, entity-crud, special-resources, monitoring, connectors-pi, ha-backup-platform) by adopt-and-harden from clarity:docs/developer/07-rest-api.md + docs/developer/api/.md — internal cross-doc links repointed to wiki pages, provenance +Primary handlers:footers added, hot endpoints spot-verified vs warp routes (main.rs @bff451d: register/login 3799-3847, create_collection 3950, write_fast 4078, write_buffered 4103, shadow_write 4124, seal_now 4174, fast_query 4213-4292, /exactapi mount 4722). Fixed pre-existing DEAD links to nonexistent dev/api/openapi.yaml + swagger-ui.html in reading-guide.md ×3, index.md ×2, dev/index.md. CLAUDE.md "REST API documentation" section rewritten (wiki/dev/api/ canonical; docs/developer = upstream of record; Docs/rest-api.html retired). New API pages pass the citation validator. BENCHMARKS xlsx-only FULL REPLACE: benchmarks.md rewritten from Pulse_Historian_Benchmark_Report.xlsx (2 machines — Workstation PC Ryzen7-7800X3D/32GB/NVMe = edge, Cloud VM 4c8t/64GB = managed; T1 live 50k tags@1Hz 47.9k/46.6k pts/s 0-drop; T2 backfill 1.0/1.66M pts/s reduced-scale; T3 read grid; T4 concurrency 143/587 req/s 0-err); product/performance.md rebuilt customer brief on same report + Pulse naming (dropped TSBS/competitor/M5 one-pager artifact link); capacity-planning.md corroboration re-anchored off M5→xlsx (3 lines). Purged ALL M5/v3.0/1.86M/144M/wire-format-efficiency refs from storage-engine.md (5 spots, re-linked to new benchmark anchors), index.md, dev/index.md, product/index.md. All benchmark anchor links resolve. NOTE: validator currently RED on 51 PRE-EXISTING breakages (46× api/storage.rs → split to api/storage/ dir at repo HEAD bff451d, ahead of wiki bookmark 45a686e; + monitor/reader.rs:180-183 range drift; + 4 stale docs/developer/_LLD/_API.md cites) — a PENDING /wiki-update, NOT from this work. Unstaged for human review.
[2026-07-16] NAV — LANDING PAGE swap (user item 1, repo-root home.md filename mechanism): repo-root home.md is now THE READING GUIDE (was the overview). Moved reading-guide content into home.md with links rebased./x→./wiki/xand overview pointer→./wiki/overview.md(home.md sits above wiki/, uses standard relative resolution — the earlier overview→home.md rename had left home.md's own./dev/outbound links BROKEN; fixed by the rebase). Recreated wiki/overview.md from the home.md overview content (links./dev/correct from wiki/; also purged its stale M5 "1.86M/144M v3.0" perf line → xlsx framing). Deleted wiki/reading-guide.md. Repointed inbound: reading-guide refs→home.md (index.md, CHANGELOG.md, product/introduction.md ×2) ; overview refs→wiki/overview.md (index.md, glossary.md#technical-topologies, hld/deployment-views.md, concepts/agent-layer.md, dev/architecture/agents.md, introduction.md Sources prose). CLAUDE.md structure map + 3 workflow/prose refs updated (home.md=front door; overview back in wiki/). Validator: no new failures (overview.md clean; total 51→50 as the rewritten benchmarks.md dropped a storage.rs cite). ⚠ ASSUMPTION to verify on live Wiki.js: home.md's./wiki/...links assume standard/GitHub relative resolution (the repo's consistent convention) — eyeball the landing page renders its links before publish. Unstaged for human review.
[2026-07-17] READER (user-review round 1, items 4+5) — READABILITY: added 3 explanation-type concept pages that carry narrative + diagrams and link DOWN into the terse cited dev pages (Diátaxis — did NOT soften the reference pages): concepts/storage-engine.md (on-disk anatomy: dense-grid/i32×1000, .bin/.bin.czst/.blob/.bref files, 3 read encodings JSON/v1/v2, push-down agg — complements historian.md's journey), concepts/monitoring.md (rule model, 4 rule types AllOfN/KOfN/Percentage/GOAT, missing-behavior, OPEN→CLOSED lifecycle + deviations mirror, auto-rules from tagmeta limits, boiler-2 steam_temp>540 worked example), concepts/process-lifecycle.md (ProcessManager supervision, watchdog restart, leftover-PID + Job-Object orphan handling, per-service logs + /process-logs, pi-driver worked example). Added> 📘 New here?reader pointers atop the 3 dense dev pages (storage-engine.md, monitor.md, process-manager.md → their concept page). Wired into wiki/index.md concepts list, home.md reader paths (Backend-dev→storage-engine; Operator→process-lifecycle+monitoring), CHANGELOG. PI-CONNECTOR UI-FIRST: rewrote guides/pi-connector-setup.md from REST-call-driven to in-app flow (Connections→add PI; browse AF tree; onboard unit; map tags industry/equipment ≥0.5 verified; backfill range/resolution; verify via dashboard) with 🖼️ SCREENSHOT/📊 GRAPH placeholders + [VERIFY-UI] tags (frontend not visible); REST detail pushed to dev/api/connectors-pi.md. Also fixed 3 reader-layer stale openapi.yaml links (data-model.md, query-data-terminal.md ×2 → dev/api/index.md / aggregation.md / queries.md). All relative links resolve; validator unchanged at 50 pre-existing failures (concept/guide pages are reader-layer, 0 code cites). Unstaged for human review.
[2026-07-18] clarity 45a686e..6800acc (77 commits) — big update. NEW subsystems documented: dev/architecture/notifications.md (persistent RBAC notifications + SSE + LocalTransport toasts + mail bridge), mail.md (lettre SMTP + mail_queue store-and-forward + admin_smtp.html), cloudsync.md (local→remote sync, /exactapi/cloudsync/, not HA-gated), integrations/notification-sdks.md (py+ts clients). f32→f64 VALUE-PATH widening threaded through: storage-engine.md (wire basic v1→v2, optimised v2→v3; query f64), api/queries.md, python-sdk.md, js-sdk.md, monitor.md (rules/reader f64), processing-api.md (ingest_v2 + ems f64). storage.rs SPLIT into api/storage/ module — storage-engine.md rewritten + all 45 storage.rs cites repointed (mmap/meta/read/write/flush/ring_flush/seal/staging/blob/ha/last_values); added Staging (chunk-log) section. sqlite_api RESTRUCTURE — sqlite-api.md 59 broken cites repointed (warp_dynamic_routes.rs→dir mod/reply/scope/shims; utils/mod.rs→regexeng/inmem/sqlemit; db/mod.rs→db/migrations.rs; attachments→warp_attachments; rate_limit→rate_limiter) + LoopBack /collections CRUD section + ⚠ content-freshness NOTE (body still narrates removed typed-macro layer — TODO-VERIFY full rewrite). ha.md: vip_guard + non-persistent VIP + verify_owned. api-server.md: notifications/mail/cloudsync route+config sections, runtime Authenticode self-verify, auth log-redaction. monitor.md: notifications integration + orphan sweep + reader delegate. Mechanical repoints: glossary (7), adr-0003, capacity-planning, hld/system-context (storage.rs→storage/), license-portal+pi-connector (docs→docs/_archive), mqtt/opc-ua/api-server (db/mod→db/migrations, handlers→reply, warp_routes→warp_dynamic_routes/mod). glossary +8 terms (staging, wire v2/v3, ingest v2, notification, mail queue, CloudSync, VIP guard). index/dev-index/CHANGELOG/overview updated + footers bumped to 6800acc. Validator GREEN. Reader-layer: no f32/wire contradictions found. Open flags: sqlite-api.md body needs typed-layer content rewrite; JS-SDK batchQuery→queryBinary stub latent bug; login-limit code(10)/properties(30) mismatch; cloudsync not HA-gated. Unstaged for human review.
[2026-07-18] READER (post-6800acc pending-flags + reader-layer sweep) — cleared 2 actionable open flags from the 77-commit update + filled item-4 reader gaps. FLAG 1 (sqlite-api.md body): re-mapped the current sqlite_api module via a source Explore agent, then removed the stale typed-macro/DYNAMIC_TABLES per-path-gate narration (route macros crud_routes!/nested_route! + hooks.rs/service_auth.rs/db_helpers.rs confirmed GONE at 7948df9/1355333) and replaced the "Phase 6 gate" section with an accurate "Dispatch & precedence (no per-path gate)" + a new "The typed CrudModel layer today" subsection (trait + impl_crud!/invoke_schema! SURVIVE, still used by /incidents/raw, connections alter, tagmeta/units interceptors — NOT wholesale dead code); reframed String-Id table (route half removed, model half survives); dropped the TODO-VERIFY content-freshness note; footer→2026-07-18. FLAG 5 (dev/api re-adoption): created wiki/dev/api/notifications.md (send/list/mark-read/unread-count/SSE) adopt-and-hardened from docs/developer/NOTIFICATIONS_API.md against the mounted routes (aspirational config keys + list envelope flagged TODO-VERIFY); wired into api/index. Flags 2/3/4 (JS-SDK queryBinary stub, login-limit 10/30, CloudSync not HA-gated) are code bugs already documented in js-sdk/api-server/cloudsync — no new wiki action (sdk-quickstart reinforces the JS stub warning). READER-LAYER DRIFT: added concepts/notifications.md (how alerts reach a person — toast/bell/SSE/email + RBAC; boiler-2 example) + linked it from concepts/monitoring.md ("How you're notified" — closes the "where results go" gap). ITEM-4 GAP FILL: 5 guides (monitor-rule, backup-restore, mqtt-live-data, enable-ha, sdk-quickstart — each grounded in the cited dev pages + self-verifying examples), concepts/time-timezones.md (epoch-ms/grid/UTC day files/display-tz + NEW hand-authored assets/grid-slots.svg in the approved pulse-stack style), start/historian-quickstart.md + start/index.md (start/ tier now populated). Wired all into home.md reader paths + wiki/index.md + CHANGELOG. Diagram assessment: added one SVG where genuinely additive (grid/slot mapping, previously undiagrammed); left existing Mermaid (notifications flow, monitoring) as-is (adequate). Validator GREEN (1118 cites, 0 fail); all new-page relative links resolve (0 dead). Roadmap Now/Next/Later still BLOCKED (wiki/product/roadmap gitignored). Unstaged for human review.