π New here? Start with the plain-language explainer How Pulse Chronos stores data β this page is the byte-level reference it links down to.
The storage engine is a custom binary time-series store backed by memory-mapped files. It provides three write modes, a thread-safe query engine, a push-down aggregation pipeline, a blob store for string/array values, a two-tier staging path for high-rate ingest, and a zstd cold-tier that compresses every day-file before today (configurable).
NOTE: Refactor (
497003f, 2026-07-13) β the ~8,000-lineapi/storage.rswas split into anapi/storage/module (mod.rs,mmap.rs,meta.rs,read.rs,write.rs,flush.rs,ring_flush.rs,seal.rs,staging.rs,blob.rs,ha.rs,last_values.rs). The move was behavior-preserving β each submodule header records "no logic changes". TheStoragestruct,CollectionMetadata, constants and shared types stay inmod.rs; citations below point at the new files.
Sources:
clarity:backend/src-tauri/src/api/storage/mod.rs (Storage struct, metadata, constants)clarity:backend/src-tauri/src/api/storage/read.rs, write.rs, meta.rs, mmap.rsclarity:backend/src-tauri/src/api/query.rsclarity:backend/src-tauri/src/api/aggregator.rsclarity:backend/src-tauri/src/persistence.rsSee also: Capacity Planning β disk sizing formula, service overheads, RAM/CPU, and the cold-tier compression story Β· Benchmarks β measured ingest, backfill, read-latency and concurrency figures (two-machine field report).
c9fceb2)The decoded value path was widened from f32 to f64 end-to-end ("Widen value path f32βf64: exact 0.001 readback at all magnitudes"). On-disk storage is unchanged β each value is still a scaled i32 (value Γ 1000, missing = i32::MIN); only the decode type changed.
Every decode site must now use scaled as f64 / 1000.0 (divide, never multiply by 0.001). Rationale, documented in the code: an i32 is always exactly representable in f64, so the decode keeps full 0.001 resolution at any magnitude β decoding through f32 silently quantized large values (at magnitude ~350,000 the f32 grid is 0.03125, which destroyed totaliser deltas). Dividing by an exactly-representable 1000.0 also yields clean JSON digits (0.009, not 0.009000000000000001). clarity:backend/src-tauri/src/api/storage/mod.rs:156-168
This is why query.rs, aggregator.rs, the monitor reader, the SDK decoders and the binary wire formats all changed in the same release β see the wire-format version bumps below.
persistence.rsSource: clarity:backend/src-tauri/src/persistence.rs
setup_persistence(app_handle) is Windows-only and handles OS-level autostart registration, not data directory creation.
clarity:backend/src-tauri/src/persistence.rs
What it actually does:
HKCU\Software\Microsoft\Windows\CurrentVersion\Run for user-login autostartClarityEngineWatchdog that checks every 5 minutes and restarts the app if no instance is runningAs of a18d35c, both steps invoke their system binary by absolute path (C:\Windows\System32\reg.exe, β¦\schtasks.exe) and skip with a logged error if the binary is missing, rather than relying on PATH. clarity:backend/src-tauri/src/persistence.rs:14-67
On non-Windows, setup_persistence is a no-op. Data directory initialization is handled separately by the storage engine at startup (not by this function).
storage/mod.rs, mmap.rs, meta.rsSource: clarity:backend/src-tauri/src/api/storage/mod.rs
Storage maintains two bounded LRU pools of memory-mapped file handles so files are not reopened per operation:
clarity.storage.mmap_cache_max_entries (default 512). clarity:backend/src-tauri/src/api/storage/mmap.rs:290-333, default clarity:backend/src-tauri/src/config.rs:536storage_writer_mmap_cache_max_entries (default 1024). clarity:backend/src-tauri/src/api/storage/mmap.rs:127-137, default clarity:backend/src-tauri/src/config.rs:548The mmap_cache / day_locks / metadata_cache fields live on the Storage struct: clarity:backend/src-tauri/src/api/storage/mod.rs:510-594.
Read paths compute bypass_cache = day_range.len() > 30. When a query spans more than 30 day-files, each day's mmap is opened with get_mmap_uncached() (a direct MmapOptions::map that never touches the LRU cache) instead of get_or_create_mmap(). This prevents a single multi-month historic query from evicting the current-day mmaps that live ingest depends on.
clarity:backend/src-tauri/src/api/storage/read.rs:788-795, clarity:backend/src-tauri/src/api/storage/mmap.rs:290-333
Each collection stores a metadata.json file at data/{org}/{site}/{unit}/{grid}/metadata.json containing:
tags: list of tag namesdescriptions: list of tag descriptions (same length as tags)id: collection UUIDinterval_ms: sampling interval in millisecondsAs of a64043f, metadata.json also persists the org/site/unit/grid scope names and per-tag descriptions, and update_collection is append-only (descriptions are preserved on update); collections that were created without descriptions are no longer invisible to GET /collection. clarity:backend/src-tauri/src/api/storage/meta.rs:323-632
The TTL-bounded metadata_cache stores, alongside each CollectionMetadata, a pre-built Arc<FastMap<String, usize>> mapping every tag name to its row index. The cache entry is a 3-tuple (Arc<CollectionMetadata>, Arc<FastMap<String, usize>>, Instant).
clarity:backend/src-tauri/src/api/storage/mod.rs:522-525
build_tag_index(tags) is the single canonical place the index is constructed. clarity:backend/src-tauri/src/api/storage/meta.rs:264get_metadata_and_index(org, site, unit, grid) returns both the metadata and the index in one cache lookup; get_metadata() is a thin wrapper that discards the index. clarity:backend/src-tauri/src/api/storage/meta.rs:268-320store_data, store_data_fast, store_data_mmap, sparse flush) and read path (get_latest_data, read_data_ultra_fast, aggregated read) takes the index from the cache instead of rebuilding a HashMap per call. When new tags are auto-added, the index is rebuilt once and re-inserted into the cache. clarity:backend/src-tauri/src/api/storage/write.rs:319-430NOTE: Before this change,
prepare_bulk_writeand each read built the tag index from scratch on every call β an O(tags)HashMapallocation per write/read. The index is now built at most once per cache-fill.
| Method | Durability | Target throughput |
|---|---|---|
store_data |
fsync after each batch | General-purpose |
store_data_mmap |
mmap flush, no fsync | <100ms for 100k+ tags |
Write path: scope resolution (org/site/unit/grid) β locate or create mmap files β write timestamped values per tag. Entry points: store_data clarity:backend/src-tauri/src/api/storage/write.rs:290, store_data_fast :431, store_data_mmap :527.
Measured (field benchmark): the
store_data_mmappath behindPOST /exactapi/write_fastsustained a full 50,000 tags @ 1 Hz with 0 dropped points on both an 8-core workstation and a 4-core cloud VM, and reached 1.0β1.66 M points/s on bulk backfill. See Benchmarks.
Missing values are encoded as a sentinel (MISSING_SENTINEL = i32::MIN) to preserve alignment in the shared-timestamp binary format. clarity:backend/src-tauri/src/api/storage/mod.rs:52
Writes are refused with HTTP 507 while the disk guard has blocked ingest (free space below
clarity.storage.min_free_disk_mb, default 2048 MB).
c41a03b)storage_format is a per-collection metadata.json field (#[serde(default)] β absent = 0 = legacy V1): STORAGE_FORMAT_V1 = 1, STORAGE_FORMAT_V2 = 2, STORAGE_FORMAT_LATEST = V2. New collections are created V2; existing collections keep their format forever ("Fixed at collection creation β existing day files are unreadable under any other format"). clarity:backend/src-tauri/src/api/storage/mod.rs:74-78
The transform is a self-inverse XOR at the file boundary (CollectionMetadata::disk_xor() = i32::MIN for V2 else 0; disk_sentinel() = MISSING_SENTINEL ^ disk_xor()):
| on-disk value | on-disk "missing" | |
|---|---|---|
| V1 | raw scaled i32 | i32::MIN (0x8000_0000) |
| V2 | scaled ^ i32::MIN (sign bit flipped) |
0x0000_0000 |
Why: V1 requires every new day-file to be pre-filled with the sentinel so unwritten slots don't read as 0.0 β "at high tag counts this fill is a massive write burst at every day roll (337 KB/tag at 1 Hz)". With V2's zero sentinel, freshly-allocated bytes (set_len / sparse extents) already mean missing, so day files are created with zero fill I/O and can be born sparse; a real scaled 0 survives as 0x8000_0000. clarity:backend/src-tauri/src/api/storage/mod.rs:56-69, 437-449
All write paths XOR at the file boundary (store_data/store_data_fast/store_data_mmap in write.rs; ring drain in ring_flush.rs; sparse flush in flush.rs); readers pass disk_sentinel()/disk_xor() into scan_mmap_scalar/scan_mmap_neon (clarity:backend/src-tauri/src/api/storage/read.rs:74-131, backing all query/aggregation reads) and the monitor reader now delegates to the same read path. HA deltas carry the disk encoding verbatim. There is no v1βv2 migration β old collections read correctly via disk_xor = 0.
store_data_mmap / store_data_fast)As of 9416f9c, the two mmap write paths no longer hold the collection write lock for the duration of file I/O. write_phase1 splits the work into two phases to let concurrent writers targeting different days of the same collection proceed in parallel:
metadata.json (and replicate the tag list to HA listeners), compute the set of unique days the batch touches, then acquire each day's per-day write lock (get_day_lock) in ascending sorted order. The collection lock is released at the end of phase 1.prepare_bulk_write β write_to_mmaps_inline β flush_async. The collection lock is free, so a writer to other days does not block.Deadlock-free because every caller takes the collection lock first, then day locks in the same sorted order. The day_lock_arcs: Vec<Arc<RwLock<()>>> is declared in the outer scope so it outlives _day_guards (which borrows from it). clarity:backend/src-tauri/src/api/storage/write.rs:319-641
ensure_file_initialized(file, total_file_size) fills any new or newly-extended region of a .bin day-file with the disk sentinel so freshly added tags read as missing rather than 0.0, using rayon par_chunks_mut, then flush_async(). clarity:backend/src-tauri/src/api/storage/mmap.rs:417-478
V2 collections skip the fill entirely β their disk sentinel is
0, andset_len-grown regions are already zero (see On-disk format V2). The fill only runs for legacy V1 collections.
| Method | Purpose |
|---|---|
create_collection_with_descriptions(org, site, unit, grid, interval_ms, tags, descriptions) |
Create new collection, write metadata.json |
collection_exists(org, site, unit, grid) |
Check if collection directory exists |
update_metadata_fields(org, site, unit, grid, updates) |
Merge JSON updates into metadata.json |
get_collection_path(org, site, unit, grid) |
Return filesystem path for a collection |
cache_stats() |
Returns (mmap_count, meta_count, lock_count) for telemetry |
cleanup_stale_locks() |
Remove expired day_lock entries |
clarity:backend/src-tauri/src/api/storage/meta.rs:9-163, 486
Five Storage methods let the SQLite-API collection-delete flow tear down all in-memory state for a collection β or for a whole org/site/unit subtree β before the on-disk directory is removed. They back the DELETE /exactapi/collections/:id cascade and the parent-level cascade in recursive_cascade_delete β see SQLite API Β§ Collections route surface.
| Method | Effect |
|---|---|
invalidate_metadata_cache(metadata_path) |
Remove one metadata_cache entry by exact path |
invalidate_metadata_cache_for_subtree(parent_path) -> usize |
Remove every metadata_cache entry whose path starts with parent_path; returns the count evicted |
invalidate_mmap_cache(collection_path) |
Drop all mmap-read-cache entries under the collection dir |
invalidate_day_locks_for(collection_path) -> usize |
retain-drop every day_locks key prefixed by the collection path; returns the count evicted |
list_collections_under_subtree(parent_path) -> Vec<(String, Vec<String>, PathBuf)> |
Walk a parent subtree and return (collection uuid, tags, collection dir) for every metadata.json found |
clarity:backend/src-tauri/src/api/storage/meta.rs:633-760, clarity:backend/src-tauri/src/api/storage/mmap.rs:336
PushDownOp enum: Mean, Sum, Min, Max, Count, First, Last. Applied inside the storage read path before returning data, reducing data transfer for summary queries.
clarity:backend/src-tauri/src/api/storage/read.rs:966, 1308
query.rsSource: clarity:backend/src-tauri/src/api/query.rs
QueryEngine wraps Storage and adds a DashMap-backed cache of open column handles for fast repeated reads across concurrent requests.
pub struct QueryEngine { /* DashMap<column_key, mmap_handle> + storage */ }
impl QueryEngine {
pub fn new(storage: Storage) -> Self { ... }
// Returns HashMap<String, Vec<(u64, f64)>> β tag β [(timestamp, value)]
}
Thread-safe via Arc<QueryEngine> shared across warp route handlers.
Column masking: the query engine supports returning only a subset of columns (tags) from a collection, avoiding reads of unneeded data.
Return type: HashMap<String, Vec<(u64, f64)>> β maps tag name to a vector of (unix_timestamp, float64 value) tuples (widened from f32; see the f64 widening). clarity:backend/src-tauri/src/api/query.rs:14, 29-38
Storage uses two caches:
mmap_cache: Arc<RwLock<LinkedHashMap<PathBuf, Arc<Mmap>>>> β LRU-bounded cache for read mmaps, keyed by file path. clarity:backend/src-tauri/src/api/storage/mod.rs:516day_locks: Arc<DashMap<String, Arc<RwLock<()>>>> β per-day write locks, key format: "{org}/{site}/{unit}/{grid}/{day_midnight_timestamp}". clarity:backend/src-tauri/src/api/storage/mod.rs:520Eviction policy for day_locks: entries are dropped when their Arc strong count drops to 1 (no thread currently holds the lock). Called periodically to prevent unbounded growth.
Storage::get_unit_last_timestamp(org, site, unit) -> Option<u64> (exposed through QueryEngine::get_unit_last_timestamp) finds the most recent data timestamp for a unit across all its grids:
base_path/<org>/<site>/<unit>/<grid>/, read each grid's metadata.json for interval_ms and tag count, and pick the newest <day>.bin file across all grids.MISSING_SENTINEL, as day*1000 + slot*interval_ms.day*1000) as a best-effort; None if the unit has no data files.This backs POST /sensordata/shadow (see API Server), which fans out one spawn_blocking call per unit. clarity:backend/src-tauri/src/api/storage/read.rs:510, clarity:backend/src-tauri/src/api/query.rs:123
binary_format.rsSource: clarity:backend/src-tauri/src/api/binary_format.rs
The query result (HashMap<String, Vec<Option<f64>>> over a shared timestamp axis) can be returned in three wire encodings. All binary formats are little-endian and uncompressed on the wire β wire compression was tried via lz4_flex and reverted, and lz4_flex remains a dead dependency. (zstd is not dead: it compresses day-files at rest in the cold tier β see Cold-tier compression.)
Wire-format version bump (
c9fceb2) β values are now 8-bytef64. Each of the two binary serializers bumped itsformat_versionby one when values widened fromf32tof64. There is no single v1/v2/v3 sequence: the basic format went v1 β v2, the optimized format went v2 β v3. Clients must branch on the per-payloadformat_versionfield (the Python and JS SDK decoders both accept old and new).
| Endpoint | Format | Layout | Bytes/point |
|---|---|---|---|
GET\|POST /exactapi/fast_query |
JSON | {tag: [[ts, value], β¦]} |
~25 |
GET\|POST /exactapi/fast_query_binary |
Basic binary, v2 (was v1) | 12-byte header, then per tag: name + per-point (u64 ts, f64 value) |
16 |
GET\|POST /exactapi/fast_query_optimised |
Optimized binary, v3 (was v2) | Shared timestamp array (stored once) + per-tag f64 values + 1-bit-per-point validity bitmap |
floor β 8 + β per point at high tag counts |
serialize_basic_from_shared, writes format_version = 2; the 12-byte header's reserved field doubles as flags with bit 0 = contains_gaps (NaN markers where the gaps aggregator emitted empty buckets). clarity:backend/src-tauri/src/api/binary_format.rs:43-118serialize_optimized, writes format_version = 3; shared u64[] timestamps + per-tag valid_count + bitmap + f64[] non-null values. Its advantage only materializes for multi-tag queries (at 1 tag it is marginally larger than basic). clarity:backend/src-tauri/src/api/binary_format.rs:146-226Both serializers build their per-tag chunks with rayon::par_iter. Read-latency figures across tag-counts and windows: Benchmarks Β§ Read performance grid.
aggregator.rsSource: clarity:backend/src-tauri/src/api/aggregator.rs
run_pipeline(tag_data, pipeline, start, end) applies a sequence of AggregationStep objects to each tag's time-series after retrieval from storage. The whole pipeline is f64 as of c9fceb2 (run_pipeline, aggregate, compute_rate, compute_diff, compute_least_squares, gaps β no f32 remains). clarity:backend/src-tauri/src/api/aggregator.rs:97-345
β οΈ No validated aggregation benchmark exists yet. The pipeline below is documented from source; server-side aggregation throughput was not covered by the field benchmark. See Benchmarks Β§ Caveats.
clarity:backend/src-tauri/src/api/aggregator.rs:8-34
Enum uses #[serde(tag = "op", rename_all = "lowercase")] β all op names serialize to lowercase. The parameter sub-enums (TrimType, FilterOp, Order, Boundary) are likewise #[serde(rename_all = "lowercase")], so their canonical wire values are lowercase (the former UPPERCASE spellings are retained as aliases). clarity:backend/src-tauri/src/api/aggregator.rs:36-87
| Op (serialized) | Parameters | Description |
|---|---|---|
mean |
β | Arithmetic mean over bucket |
avg |
β | Alias for mean |
sum |
β | Sum over bucket |
min |
β | Minimum over bucket |
max |
β | Maximum over bucket |
count |
β | Count of non-null values |
first |
β | First sample in bucket |
last |
β | Last sample in bucket |
dev |
β | Standard deviation |
gaps |
bucket (ms), start (ms), end (ms) |
Fill time gaps with null on the query-start-anchored bucket grid; all three params required |
histogram |
percentile: f64 |
Frequency bucket distribution |
percentile |
percentile: f64 |
Nth percentile (e.g., 50.0 = median) |
leastsquares |
β | Linear regression trend line (also accepts the alias least_squares) |
diff |
β | Delta (Ξ) between consecutive samples |
div |
divisor: f64 |
Divide all values by a constant |
rate |
unit: String |
Rate of change per unit time |
scale |
factor: f64 |
Multiply all values by a constant |
trim |
trim: "first"\|"last"\|"both" |
Remove first/last/both boundary samples (UPPERCASE spellings also accepted) |
saveas |
metric_name, tags, ttl, add_saved_from |
Placeholder β unimplemented. Logs a warning and returns input unchanged. |
sampler |
unit: String |
Placeholder β unimplemented. Returns input unchanged. |
filter |
filter_op: FilterOp, threshold: f64 |
Drop samples outside threshold |
score |
order: Order, thresholds: Vec<Threshold> |
Normalized score aggregation |
{
"pipeline": {
"tag_name": [
{ "op": "mean", "params": { "window": 3600 } },
{ "op": "scale", "params": { "factor": 1.0 } }
]
}
}
Each step takes the output of the previous step as input. The pipeline is specified per-tag in the query request.
start anchored)run_pipeline takes the query window start. The bucketing helper aggregate() anchors every bucket to that start:
anchor = start.unwrap_or(data[0].0) // query start, or first sample if no start
bucket_for(ts) = anchor + ((ts - anchor) / bucket_size) * bucket_size
Two correctness guards accompany this: pre-anchor points (ts < anchor, which the storage layer sometimes returns) are dropped before bucketing to avoid u64 underflow; if filtering empties the series, aggregate() returns an empty result. clarity:backend/src-tauri/src/api/aggregator.rs:190-244
NOTE: This is the post-processing path. The push-down aggregation path inside
read.rs::read_data_aggregatedis a separate implementation that received the same anchoring fix β it builds its output bucket grid anchored to the querystart_ms, with an underflow guard per per-op scan loop.clarity:backend/src-tauri/src/api/storage/read.rs:956-1439
The gaps() post-processing op walks its grid from start (the same rule as aggregate()), so gap-filled null points coincide exactly with the upstream aggregate grid. clarity:backend/src-tauri/src/api/aggregator.rs:331-345
A lock-free, sharded queue in front of the storage engine for the /exactapi/write_buffered endpoint. Sizing is config-driven:
clarity.write_buffer.capacity β default 200,000 writesclarity.write_buffer.flush_interval_ms β default 50 msclarity.write_buffer.batch_size β default 500,000admit(points) enforces clarity.write_buffer.max_pending_points (default 10,000,000) β over-limit enqueues are rejected (surfaces as ingest 503). clarity:backend/src-tauri/src/api/write_buffer.rs:365-478enqueue_single(...) and enqueue_batch(BufferedWriteRequest)flush_guarded and respawn on panic, so a shard is never permanently wedged. clarity:backend/src-tauri/src/api/write_buffer.rs:252-286, 483-497The write buffer is split into NUM_SHARDS = 8 independent shards, each with its own bounded channel and background flusher thread. A collection is assigned to a shard by hashing its (organization, site, unit, grid) tuple with a process-static ahash::RandomState, modulo NUM_SHARDS, so one slow collection cannot block flushes for collections on other shards. clarity:backend/src-tauri/src/api/write_buffer.rs:12-18
buffer_size / NUM_SHARDS; per-shard batch size = batch_size / NUM_SHARDS. clarity:backend/src-tauri/src/api/write_buffer.rs:313-336total_queued, total_flushed, queue_full_errors, flush_errors (via GET /exactapi/write_buffer_stats).Each shard gets its own WAL file write_buffer_{shard_idx}.wal. The enqueue path no longer touches the WAL; the flusher thread journals each drain cycle via append_cycle before writing to storage. Durability is opt-in: clarity.write_buffer.wal_fsync defaults to false β appends reach the OS page cache only, so an OS crash (not just a process crash) can lose the last cycle unless fsync is enabled. Gate: clarity.write_buffer.wal_enabled (default true). clarity:backend/src-tauri/src/api/write_buffer.rs:72-128, 191
staging.rsNew module (staging.rs). A per-collection sequential chunk log that sits between the hot ring buffer and the column-major day file, added to break the write-amplification wall on second-level ingest.
Problem: a ring drain into a column-major day file dirties one 4 KiB page per tag per drain (tags are a full day-row apart), so the physical write rate is tags Γ 4 KiB / flush period β ~17k random-page IOPS at 1M tags on a 60 s drain, regardless of how few bytes actually changed. clarity:backend/src-tauri/src/api/storage/staging.rs:1-16
Fix: ring drains append to a per-collection staging.clog (sequential, CRC'd, optionally fsync'd) instead of scattering pages; a background materializer folds the log into {day}.bin only once ~1024 slots per tag have accumulated β so each 4 KiB day-file page is written once.
ring (RAM, β€ drain interval)
ββ drain ββ staging.clog (sequential append, CRC'd, fsync'd)
ββ materializer ββ {day}.bin (once β₯ ~1024 slots/tag)
Reads stay fresh: the query paths merge the day file with the pending chunks (staging_overlay / staging_latest) and with the ring buffer itself (ring_overlay / ring_latest); reads never trigger a drain. Record encode/parse clarity:backend/src-tauri/src/api/storage/staging.rs:183-333, append :514-631, read overlays :632-809, materializer :810-1011, scheduler/boot :1012-1157.
Config (clarity.storage.staging.*): enabled (code default false, shipped clarity.properties sets true), fsync (true), materialize_slots (1024), materialize_max_age_secs (1800), max_pending_bytes (1 GiB), tick_secs (30). clarity:backend/src-tauri/src/config.rs:279-298, 582-587, clarity:backend/src-tauri/clarity.properties:390-395
.bin.czst filesDay-files older than a hot window are compressed into a sealed .bin.czst and the original .bin is removed. Reads transparently fall back to the cold file; writes transparently unseal-then-reseal. Source: clarity:backend/src-tauri/src/api/storage/seal.rs.
Policy: the hot window is config-driven β clarity.storage.seal.hot_days default 0 ("seal everything before today"), seal_old_day_files takes (hot_days, grace_secs) (clarity:backend/src-tauri/src/api/storage/seal.rs:344) and a new seal_now(scope, include_today) supports forced/manual reclaim (clarity:backend/src-tauri/src/api/storage/seal.rs:366) behind POST /exactapi/seal_now (clarity:backend/src-tauri/src/main.rs:4503). New seals use a V2 codec β SEAL_MAGIC_V2 = "CLRTY_V2", delta+varint columns β while V1 (CLRTY_V1, plain zstd) files stay readable; SealCodec is selected from the file magic. clarity:backend/src-tauri/src/api/storage/seal.rs:24-126
Terminology β two unrelated meanings of "seal". This cold-tier "seal" = zstd-compress a
.binto.bin.czst. The HA "sealed file" = a day that has rolled over UTC midnight and is treated as immutable for checksum reconciliation. Different operations that share a word.
.bin.czst)czst_path_for(bin) appends .czst to the .bin name. The layout is a fixed 16-byte header + per-column offset table + one zstd chunk per tag column:
[magic: "CLRTY_V1"|"CLRTY_V2" (8B)] [n_cols: u32] [points_per_day: u32]
[ n_cols Γ (data_offset: u64, comp_size: u32) ]
[ chunk for col 0 ] [ chunk for col 1 ] β¦
Per-column framing makes column-masked reads cheap: a query touching one tag decompresses only that tag's chunk. clarity:backend/src-tauri/src/api/storage/seal.rs:8-126
| Function | Effect |
|---|---|
seal_day_file(bin, ppd) |
Compress .bin β .bin.czst atomically (write .tmp β rename), log the size reduction, then delete the original .bin. seal.rs:139 |
unseal_day_file(czst, bin) |
Decompress .bin.czst β .bin (write .tmp β rename). Keeps the .czst; caller reseals after writing. seal.rs:254 |
seal_old_day_files(hot_days, grace_secs) -> usize |
Walk org/site/unit/grid, seal every .bin whose day is older than hot_days. Skips already-sealed and still-hot files; holds the collection write lock per file; invalidates the mmap cache. seal.rs:344 |
clarity:backend/src-tauri/src/api/storage/seal.rs:139-395
main() spawns a background task that runs seal_old_day_files(hot_days, grace_secs) β first run 5 minutes after startup, then every storage_sealer_interval_secs.max(60) (hourly by default). With the default hot_days = 0, only today's day-file stays hot; every prior day is compressed. See API Server Β§ Background tasks.
Read paths model each day as a DayStorage:
DayStorage::Hot(Arc<Mmap>) β the live .bin, scanned zero-copy.DayStorage::Cold(Arc<ColdDayReader>) β a .bin.czst; ColdDayReader mmaps the compressed file and decompresses one tag column at a time on demand (decompress_column(tag_idx)), returning a fresh Vec per call so it is shared &self across rayon threads.When opening a day, the read path tries the hot .bin first and falls back to ColdDayReader::open. The aggregation path wraps each day-column in a ColData β Borrowed { ptr, len } (zero-copy into the hot mmap) or Owned(Vec<i32>) (decompressed cold column) β so the per-bucket loop is identical for both tiers. clarity:backend/src-tauri/src/api/storage/read.rs:8-31, clarity:backend/src-tauri/src/api/storage/seal.rs:273-333
When a write targets a day that is currently cold (hot .bin absent but .bin.czst present), the write path unseal_day_files it first, records it in the bulk-write context, performs the write, then reseals after the mmaps are flushed and dropped. This applies in store_data, store_data_fast, store_data_mmap, and the write-buffer flush path. clarity:backend/src-tauri/src/api/storage/write.rs, clarity:backend/src-tauri/src/api/storage/seal.rs:254
HA cold-tier reconciliation is seal/blob-aware.
storage/ha.rsexposes seal/blob-aware primitives βread_replica_day/install_replicated_daydigest, ship, and install whole day units as decompressed logical bytes regardless of seal state,list_replica_daysenumerates the archive, and the shadow-cache drain performs idempotent gap-fill. Cold-tier content is checksummed and repaired between peers.clarity:backend/src-tauri/src/api/storage/ha.rs:61-98, 422-593. Details in HA Β§ Sealed File Reconciliation.
The hot-path mmapβOption<f64> fill is vectorized: on aarch64 (Apple Silicon, ARM64 servers) it uses a NEON 128-bit path (scan_mmap_neon, 4 lanes/iteration), with a scalar tail and a scalar fallback (scan_mmap_scalar) on other architectures. Both take the collection's disk_sentinel/disk_xor and the f64 divisor, so the same paths decode both V1 and V2 files. Output is identical to the scalar loop. clarity:backend/src-tauri/src/api/storage/read.rs:74-131
NOTE: Performance-only β semantics (sentinel β
None, elseSome(raw as f64 / 1000.0)) are unchanged.
.bref + .blob sidecars)A blob-typed tag stores a reference in its day-file slot instead of a scaled numeric; the actual value lives in an append-only sidecar heap. The wire codec lives in blob_store.rs; the storage integration moved to storage/blob.rs. Two sidecar files per day per grid dir, beside {day}.bin:
{day}.blob β append-only value heap: 16-byte header (magic "CLRTYBH1" | version u16=1 | flags u16 | reserved u32), then 8-byte-aligned records len u32 | crc32 u32 | type u8 | payload | zero-pad. type 1 = UTF-8 string, type 2 = JSON (arrays/objects). clarity:backend/src-tauri/src/api/blob_store.rs:1-42, 140-181{day}.bref β sparse ref columns in the same column-major layout/codec as .bin; a slot holds record_offset / 8 (always β₯ MIN_VALID_REF = 2). clarity:backend/src-tauri/src/api/blob_store.rs:34-37Write path: Storage::store_blob_writes validates against clarity.blob.max_array_elems (default 4096), appends records (optional fsync per clarity.blob.fsync_on_write, default false), then writes refs under the per-day write lock. Crash-ordering contract: the record is durable before its ref is written; crc+length bounds turn a torn record into "missing", not a panic. clarity:backend/src-tauri/src/api/storage/blob.rs:61-252
Read path: storage-level readers read_blob_data, get_latest_blob_data, search_blob_data (BlobPredicate::{Eq,Contains,Regex}), and blob_state_durations surface through QueryEngine::fast_blob_read / fast_blob_read_latest / fast_blob_search / blob_state_durations. clarity:backend/src-tauri/src/api/storage/blob.rs:405-833; clarity:backend/src-tauri/src/api/query.rs:129-211
Endpoints (all JWT-authenticated): POST /exactapi/blob_write, GET|POST /exactapi/blob_query, GET|POST /exactapi/blob_lastlist, POST /exactapi/blob_search, POST /exactapi/blob_state_durations β see API Server.
Sealed form: {day}.blob.zst = whole-heap zstd ("CLRTYBZ1" | raw_len u64 | zstd(...)) so every ref stays valid; .bref seals via the same column codec as .bin. clarity:backend/src-tauri/src/api/blob_store.rs:255-329
Hot short-string keyed maps use FastMap/FastSet (std collections over ahash::RandomState, per-process seed β clarity:backend/src-tauri/src/fast_hash.rs:1-15).
hot_ring_buffer.rsA per-collection row-major RAM buffer for high-rate collections (interval_ms β€ 1000): "N tags at one timestamp" becomes one contiguous row write instead of N scattered page faults in the column-major day file. clarity:backend/src-tauri/src/api/hot_ring_buffer.rs:1-14
clarity.hot_ring.max_bytes. Values scaled Γ1000 to i32; the same divide-by-1000.0 f64 decode contract as the rest of the engine. clarity:backend/src-tauri/src/api/hot_ring_buffer.rs:23-28, 135-144peek_rows β persist elsewhere β confirm_drain, tracked by a DrainTicket (records slot + write-generation per epoch, so a write landing between peek and confirm keeps the epoch dirty). clarity:backend/src-tauri/src/api/hot_ring_buffer.rs:65-73, 486-553resolve_columns + write_indexed take pre-resolved column indices for /ingest/v2 sessions (see Processing API Β§ ingest v2). clarity:backend/src-tauri/src/api/hot_ring_buffer.rs:200-263read_epochs / read_latest let queries merge unflushed ring data. clarity:backend/src-tauri/src/api/hot_ring_buffer.rs:554-615flush_all_ring_buffers every clarity.hot_ring.drain_interval_ms (shipped properties 60000); reads force-flush dirty rows first; shutdown and seal_now call flush_all_ring_buffers_final. clarity:backend/src-tauri/src/api/storage/ring_flush.rs:174-386Query and aggregation flow through QueryEngine. The critical branching point is whether the request pipeline contains push-down eligible ops.
The read paths merge three tiers per day β the hot
.binmmap (or a coldColdDayReaderover.bin.czst), the pendingstaging.clogchunks, and the in-RAM ring epochs. Cold days decompress per-column on demand.
Diagram generated from source. If implementation has changed, update the Mermaid source in this file directly.
Last updated: 2026-07-17 from commit 6800acc