Status: Accepted (documented from implementation)
Applies to: clarity binary deployments (all clarity modes)
Related: Storage Engine
Clarity is positioned as an industrial historian handling high-cardinality, high-rate tag data with sub-100ms write targets on commodity hardware, embedded inside a single binary.
Rather than embedding a general-purpose TSDB, clarity implements a custom binary, memory-mapped store:
Storage keeps bounded LRU pools of mmap handles (read cache 512, writer cache 1024) and persists per-collection metadata.json under data/{org}/{site}/{unit}/{grid}/ — clarity:backend/src-tauri/src/api/storage/mmap.rs, clarity:backend/src-tauri/src/api/storage/meta.rsstore_data (fsync per batch) and store_data_mmap (mmap flush, no fsync; <100ms for 100k+ tags) — clarity:backend/src-tauri/src/api/storage/write.rsQueryEngine with cached column handles and column masking — clarity:backend/src-tauri/src/api/query.rsPushDownOp: Mean/Sum/Min/Max/Count/First/Last); richer ops run as a post-processing pipeline — clarity:backend/src-tauri/src/api/storage/read.rs:966, clarity:backend/src-tauri/src/api/aggregator.rs:8-33f64 values as of c9fceb2 — clarity:backend/src-tauri/src/api/binary_format.rs:43-226WriteBuffer (200k capacity; 50ms flush; sharded) — clarity:backend/src-tauri/src/api/write_buffer.rsNOTE: The ~8,000-line
api/storage.rswas split into anapi/storage/submodule directory (497003f, 2026-07-13) — behavior-preserving; citations above point at the new files.
.bin files without the clarity decoders (the Python/JS SDKs ship decoders for the binary wire format)..bin files are not compressed or encrypted at rest (encryption is a roadmap item — see security-model); only backup archives are gzip-packed (.tar.gz).clarity:backend/src-tauri/src/api/storage/mod.rs (+ mmap.rs/meta.rs/read.rs/write.rs), query.rs, aggregator.rs:8-33, binary_format.rs:43-226, write_buffer.rsLast updated: 2026-07-17 from commit 6800acc