CloudSync is an in-app background task that pulls recent time-series data from the local Clarity instance's read API and pushes it to a remote Clarity instance's write API — i.e. local→remote sync/migration of time-series data plus collection metadata. It also reconciles collections (org/site/unit/grid scopes, tags, descriptions, interval) between the two instances so the remote has a landing place for incoming data (clarity:backend/src-tauri/src/cloudsync/mod.rs:1-5).
Deployment note: CloudSync runs on any install where
clarity.cloudsync.local_to_host_cloudsyncis set — it is independent of product/deployment topology. It is not gated by the HA singleton gate (see Known limitations below).
NOTE: There is no persistent on-disk cursor. State is only the in-memory
CloudSyncStatussnapshot; continuous mode is stateless and does not resume across a restart (clarity:backend/src-tauri/src/cloudsync/task.rs:124-157).
Defined in clarity:backend/src-tauri/src/cloudsync/mod.rs. The process name registered with the process manager is PROCESS_NAME = "clarity-cloudsync" (clarity:backend/src-tauri/src/cloudsync/mod.rs:23).
| Item | Location | Role |
|---|---|---|
CloudSyncConfig |
mod.rs:29-63 |
Mirrors the clarity.cloudsync.* keys |
CloudSyncStatus |
mod.rs:125-143 |
Last-cycle snapshot: running flag, cycle timestamps, last_error, tags_synced, points_written, failed_batches |
CloudSyncRuntime |
mod.rs:149-152 |
Holds RwLock<config> + Mutex<status> |
CloudSyncConfig::from_clarity_config() (clarity:backend/src-tauri/src/cloudsync/mod.rs:72-94); redacted() blanks passwords for API responses (clarity:backend/src-tauri/src/cloudsync/mod.rs:99-108).OnceLock<Arc<CloudSyncRuntime>> with init_runtime() (clarity:backend/src-tauri/src/cloudsync/mod.rs:166-187) and runtime() (clarity:backend/src-tauri/src/cloudsync/mod.rs:192-197).register_internal_task() (clarity:backend/src-tauri/src/cloudsync/mod.rs:205-212). build_process_config() sets auto_start: false and RestartPolicy::OnFailure (clarity:backend/src-tauri/src/cloudsync/mod.rs:227-239); register_only() (clarity:backend/src-tauri/src/cloudsync/mod.rs:247-249) and start_now() (clarity:backend/src-tauri/src/cloudsync/mod.rs:257-261) register and launch respectively.The client in clarity:backend/src-tauri/src/cloudsync/client.rs talks to a remote — or the local source — Clarity over /exactapi/*.
| Operation | Endpoint | Location | Notes |
|---|---|---|---|
login() |
POST {base}/exactapi/login |
client.rs:50-72 |
Sends {"email","password"}, returns a Bearer token |
fetch_collections |
GET /exactapi/collection |
client.rs:107-123 |
Lists collections on an instance |
create_collection |
POST /exactapi/create_collection |
client.rs:146-184 |
NOT idempotent — regenerates a UUID on each call |
update_collection |
POST /exactapi/update_collection |
client.rs:208-236 |
Shallow merge; preserves existing id |
fetch_window |
POST /exactapi/fast_query |
client.rs:256-289 |
Reads [start,end) by tag name |
write_window |
POST /exactapi/write_fast |
client.rs:344-380 |
Pushes aligned bulk data |
All reads and writes are by tag name only — no scope is sent on the wire; the target instance resolves scope via its own tag→scope map (see Tag Resolver). Raw (timestamp_ms, value) series are reshaped by align_to_shared_timestamps into shared-timestamp bulk batches before writing (clarity:backend/src-tauri/src/cloudsync/client.rs:318-339).
The loop entry point is run_sync_loop() at clarity:backend/src-tauri/src/cloudsync/task.rs:77-190.
PUT /config change applies without a restart (clarity:backend/src-tauri/src/cloudsync/task.rs:96).lookback_minutes — the task sleeps lookback_minutes * 60s between cycles (default 30 min). There is no separate poll-interval knob, so trailing windows never gap (clarity:backend/src-tauri/src/cloudsync/task.rs:175-182). Cancellation is cooperative via tokio::select! on a shutdown receiver.start_time_ms and end_time_ms are set, the task performs a one-shot backfill over the explicit window and then exits cleanly (clarity:backend/src-tauri/src/cloudsync/task.rs:101-110,159-165). Otherwise it runs continuous trailing-lookback mode (now - lookback → now).clarity:backend/src-tauri/src/cloudsync/task.rs:256-283):
collection_ids override → sync_collections_by_id() — one-directional: the remote conforms to the source, the source is never modified, and collections are matched by scope rather than id (clarity:backend/src-tauri/src/cloudsync/task.rs:389-524).tag_list override → skips discovery and syncs the listed tags.reconcile_collections() performs bidirectional create-missing-on-both-sides (clarity:backend/src-tauri/src/cloudsync/task.rs:536-630), then syncs every source collection.tag_batch_size (clarity:backend/src-tauri/src/cloudsync/task.rs:313-315); per-batch retry is MAX_BATCH_ATTEMPTS = 3 at 10 s (clarity:backend/src-tauri/src/cloudsync/task.rs:14-20); per-cycle login retry is MAX_LOGIN_ATTEMPTS = 3 at 30 s (clarity:backend/src-tauri/src/cloudsync/task.rs:28-33). Failed batches are counted and skipped, not treated as fatal (clarity:backend/src-tauri/src/cloudsync/task.rs:331-364).CloudSyncStatus snapshot updated each cycle (clarity:backend/src-tauri/src/cloudsync/task.rs:124-157). Continuous mode is stateless — each window is computed from now and lookback_minutes, relying on cadence == lookback to avoid gaps rather than a stored high-water mark. There is no resume-across-restart cursor.All routes are mounted under /exactapi/cloudsync/* and all require bearer-token auth (crate::auth::auth_middleware). Routes are assembled at clarity:backend/src-tauri/src/cloudsync/http_api.rs:24-55.
| Method | Path | Auth | Purpose |
|---|---|---|---|
| POST | /exactapi/cloudsync/start |
Bearer | Register + start (handle_start, http_api.rs:66) |
| POST | /exactapi/cloudsync/stop |
Bearer | Find task by PROCESS_NAME and stop (handle_stop, http_api.rs:90) |
| GET | /exactapi/cloudsync/status |
Bearer | Redacted config + cycle snapshot + process state (handle_status, http_api.rs:127) |
| GET | /exactapi/cloudsync/config |
Bearer | Redacted config (handle_get_config, http_api.rs:148) |
| PUT | /exactapi/cloudsync/config |
Bearer | Full replace, in-memory only (not persisted to disk) (handle_put_config, http_api.rs:163) |
See the API server page for how these routes compose into the warp router and the shared auth middleware.
register_internal_task + init_runtime at clarity:backend/src-tauri/src/main.rs:3212-3213.register_only at clarity:backend/src-tauri/src/main.rs:3225-3228.clarity:backend/src-tauri/src/main.rs:5074.config().cloudsync_local_to_host_cloudsync, inside the WARP_READY block at clarity:backend/src-tauri/src/main.rs:5703-5707, so it starts only after Warp is listening.cloudsync.log is set up at clarity:backend/src-tauri/src/main.rs:3367-3415.Config keys are prefixed clarity.cloudsync.* and loaded in clarity:backend/src-tauri/src/config.rs:397-448; the shipped example values live in clarity:backend/src-tauri/clarity.properties:677-758.
Key (clarity.cloudsync.) |
Field | Code default |
|---|---|---|
local_to_host_cloudsync |
bool auto-start switch | false |
source_url |
source instance base URL | https://127.0.0.1:3030 |
source_user |
source login email | blank (properties ships admin) |
source_pass |
source password | blank (properties ships admin) |
remote_url |
remote instance base URL | blank (properties ships an example VM IP) |
remote_user |
remote login email | blank (properties ships admin) |
remote_pass |
remote password | blank (properties ships admin) |
lookback_minutes |
trailing window size (also sets cycle cadence) | 30 |
start_time_ms |
explicit backfill start | None/blank |
end_time_ms |
explicit backfill end | None/blank |
tag_list |
explicit tags to sync | blank = auto-discover |
collection_ids |
explicit collections to sync | blank = all |
accept_invalid_certs |
skip TLS cert validation | true |
http_timeout_secs |
per-request timeout | 30 |
tag_batch_size |
tags per chunk | 100000 |
NOTE: The code defaults (blank user / blank remote URL) differ from the example values checked into
clarity.properties. The properties-file values are what ship on disk. Setting bothstart_time_msandend_time_msswitches the task into one-shot backfill mode.
singleton_gate / is_leader integration and is not registered with the HA SingletonGate. On an HA cluster it would start on every node where local_to_host_cloudsync is true, not only the leader.now - lookback_minutes rather than from where the previous run left off.create_collection is not idempotent — re-creating an already-present collection regenerates a UUID (clarity:backend/src-tauri/src/cloudsync/client.rs:146-184).PUT /config is not persisted — a full-replace config change survives only until process restart.The repo-root clarity:ingest.py is a separate standalone load-test / data-simulation script (class BufferedIngestSimulator) that posts synthetic sine-wave data to /ingest/1/1; it is not part of the CloudSync production path, though it exercises the same /exactapi/login and /exactapi/collection endpoints.
Last updated: 2026-07-17 from commit 6800acc