Sizing the clarity time-series store — disk, RAM/CPU, and overheads. Every number follows from the on-disk grid layout plus the zstd cold tier; clarity does not behave like a general-purpose TSDB.
Measured throughput/latency live in Benchmarks (a two-machine field run — an 8-core workstation PC and a 4-core cloud VM). This page is the sizing model; the benchmark corroborates it — a full 50,000-tag @ 1 Hz load sustained with zero dropped points on both machines, at 10–36% average CPU.
Sources: clarity:backend/src-tauri/src/api/storage/mod.rs (grid layout), clarity:backend/src-tauri/src/api/storage/seal.rs (cold tier), clarity:backend/src-tauri/src/config.rs (cache defaults), clarity:backend/src-tauri/src/monitor/ (steady load), backup/ + ha/ (multipliers). See also Storage Engine, Backup System, HA, ADR-0003.
clarity stores a dense, fixed-interval grid, not (timestamp, value) pairs. Three properties make cost deterministic:
i32, scaled ×1000 (fixed 0.001 resolution). The decoded value widened to f64 (c9fceb2), but the stored width is unchanged. clarity:backend/src-tauri/src/api/storage/mod.rs:156-168clarity:backend/src-tauri/src/api/storage/write.rsi32::MIN) — a day with one sample costs a full day. V2 collections (the default for anything created at ≥ c41a03b) use a zero missing-sentinel, so files are created with zero fill I/O and can be born sparse — physical usage grows with written extents, and the formula below becomes a logical ceiling rather than a guaranteed physical cost. See Storage Engine § On-disk format V2.So the hot on-disk size ceiling is a pure function of tags, rate, and span:
hot bytes ≈ 4 × tags × (86400 / interval_seconds) × days
Cold tier — now "everything before today" by default. The sealer runs hourly and compresses each day-file older than clarity.storage.seal.hot_days (default 0, was a hardcoded 7) into .bin.czst (new seals use a delta+varint V2 codec) and deletes the raw .bin; reads decompress on demand, writes unseal → write → reseal — see Storage Engine § Cold-tier compression. In steady state only today is uncompressed. The corpus-wide ratio is not yet measured — > TODO-VERIFY: capture the sealer's logged reduction before quoting a ratio.
Blob-typed tags (string/array values, new in c41a03b) cost extra: a second ref grid (.bref, same 4 B × slots layout) plus the append-only value heap (.blob, actual payload bytes; capped per-value by clarity.blob.max_bytes = 64 KB).
Disk-full guard: ingest is blocked with HTTP 507 when free space drops below clarity.storage.min_free_disk_mb (default 2048 MB; unblocks at 1.25× — hysteresis). Treat ~2 GB as a hard reserved floor when sizing volumes.
Per tag @ 1 s: 345,600 B/day (86,400 × 4) → ~126 MB/year. Scale by tags and years for the hot ceiling (1-minute data is 1/60th of these):
| Scenario | Hot ceiling (raw .bin) |
In steady state (default hot_days=0) |
|---|---|---|
| 8,000 tags · 1 s · 3 yr | ~3.03 TB | only today raw; rest zstd .bin.czst |
| 25,000 tags · 1 s · 5 yr | ~15.8 TB | ″ (ratio TODO-VERIFY) |
| 50,000 tags · 1 s · 5 yr | ~31.6 TB | ″ |
Two multipliers that change the total:
.tar.gz that is never pruned (cleanup_old_backups only touches SQLite archives). Put backups on a separate volume with an external retention policy. Backup SystemUpdated
45a686e: HA sealed-file reconciliation is now seal/blob-aware — it digests over decompressed logical bytes and reconciles whole day units (.bin.czst,.bref.czst,.blob.zst) regardless of seal state, so cold-tier divergence between HA peers is now detected and repaired (commit6061ec1). This closes the earlier gap where historic sealed days silently no-op'd. See HA § Sealed File Reconciliation. ("HA-sealed" = UTC-rollover immutability remains a different concept from the cold-tier zstd "seal".)
Negligible at TB scale: the directory tree, metadata.json (~100 B/tag), the bounded write-buffer WAL, app SQLite rows, and filesystem block rounding.
Driven by the live working set, not total history — cold history is paged in (and decompressed) only when queried.
CPU — practical 4–8 cores. Two parallelism layers: Tokio (HTTP/IO) and a Rayon pool (par_iter over tags for every heavy scan).
> TODO-VERIFY).RAM — dominated by OS page cache for the hot day:
hot page cache ≈ tags × (86400 / interval_s) × 4 B # 8,000 @ 1s ≈ 2.76 GB
This is reclaimable page cache, not heap; without it free, ingest/monitor fault against disk. Heap caches are small and bounded: mmap LRU 512; metadata cache (code defaults now 1024 entries / 3600 s, but the shipped clarity.properties still pins 256 / 120 s — the file wins); write buffer 200k capacity / 8 shards / 10 M-point admission cap; the hot ring buffer adds up to clarity.hot_ring.max_bytes = 128 MB for ≤1 s-interval collections. (The old SQLite query-result cache was removed in c41a03b; its config knobs are dead.) clarity:backend/src-tauri/src/config.rs
⚠️ The real risk is bulk extraction. A full-resolution, all-tag raw read transiently materializes ~tags × points × 8 B (Vec<Option<f32>> / Vec<(u64,f32)>) — ~5.5 GB for one day of 8,000 tags, tens of GB for a week: enough to OOM. Bound it with push-down aggregation, column masking, and capped time ranges. > TODO-VERIFY: the byte figures are Rust-layout estimates, not measured.
| Resource | 8,000 tags @ 1 Hz |
|---|---|
| CPU | 4–8 cores |
| RAM | ~3 GB hot page cache + ~0.5–1 GB heap + query headroom → floor 8 GB, comfortable 16 GB |
| Per HA node | sized identically (the replica does the same ingest/eval work) |
Rule of thumb at larger scale (RAM floor ≈ hot-day page cache + one query burst): ~16 GB @ 15k tags, ~32 GB @ 25k, ~64 GB @ 50k (all @ 1 s); 8–16 cores.
| Capability | Status |
|---|---|
| At-rest compression | Cold only — .bin.czst for every day before today (default hot_days=0); only the live day is uncompressed (and unencrypted). |
| Retention / TTL / auto-delete | None — sealing compresses, never deletes; old .czst is kept forever. |
| Rollup / downsampling | None — saveas / sampler ops are unimplemented placeholders; aggregation is query-time only. |
| Value re-encoding (Gorilla / delta-of-delta) | None — the cold tier uses generic zstd per column, not a time-series codec. |
| Object-storage / separate-volume tiering | None — hot .bin and cold .bin.czst live in the same data/{org}/{site}/{unit}/{grid}/ tree. |
zstd is live (cold tier, 3 call sites in storage.rs); lz4_flex remains a dead dependency — the reverted query-wire-compression experiment — so the binary query response is uncompressed. The only other compression is gzip in backup/.
clarity:backend/src-tauri/src/config.rs (defaults shown):
| Setting | Default (code) | Effect |
|---|---|---|
storage_mmap_cache_max_entries |
512 | Open day-file mmap LRU cap |
storage_metadata_cache_max_entries |
1024 (properties pins 256) | Metadata + tag-index cache cap |
storage_metadata_cache_ttl_seconds |
3600 (properties pins 120) | Metadata cache TTL |
clarity.storage.min_free_disk_mb |
2048 (0 disables) |
Disk-guard floor — ingest blocked (507) below it |
clarity.storage.seal.hot_days / .interval_secs / .grace_secs |
0 / 3600 / 3600 | Cold-tier sealing policy (now config-driven) |
clarity.hot_ring.max_bytes / .drain_interval_ms |
128 MB / 5000 (properties: 60000) | Hot ring buffer for ≤1 s collections |
clarity.write_buffer.capacity / .flush_interval_ms / .batch_size |
200000 / 50 / 500000 | Buffered-write path sizing |
clarity.write_buffer.max_pending_points |
10,000,000 | Admission cap (backpressure → 503) |
clarity.write_buffer.wal_enabled / .wal_fsync |
true / false | WAL journal; fsync off by default (page-cache durability only) |
clarity.blob.max_bytes / .max_array_elems / .fsync_on_write |
65536 / 4096 / false | Blob (string/array) value limits |
db_pragma_mmap_size_bytes |
1 GiB | SQLite mmap pragma |
Backup cadence/retention: Backup System. Monitor tick rate (1000 ms default, 100 ms min; properties sets the agent poll to 1000 ms): Monitor.
Last updated: 2026-07-12 from clarity@45a686e