Source: clarity:SDK/python/clarity_sdk.py
The Clarity Python SDK is the primary integration path for Python applications and data pipelines connecting to a Clarity historian.
Two classes: ClarityDecoder (binary format decoding) and ClarityClient (high-level HTTP client).
The SDK now ships as a wheel. pyproject.toml (setuptools, py-modules = ["clarity_sdk"], deps requests/urllib3) defines the build; build_wheel.sh produces dist/clarity_sdk-1.0.0-py3-none-any.whl. clarity:SDK/python/pyproject.toml:1-16, clarity:SDK/python/build_wheel.sh:1-17
As of
a18d35cthe decoders use numpy (imported at module top,clarity_sdk.py:17).to_dataframeimports numpy lazily as well. > TODO-VERIFY:numpyis not listed inpyproject.toml's dependencies — confirm whether the wheel declares it (or relies on it being present transitively via pandas/the bundled venv).
The built wheel is vendored into the backend's Python services bundle (backend/src-tauri/Assets/python_services/extra_modules/) and installed into the g-adk venv at setup time via extra_modules_req.txt (uv pip install --no-cache --reinstall …). See Python Services. (SDK/python/dist/ is now git-ignored.)
Static methods for decoding binary responses from the fast query endpoints.
_ArrayBackedDict (internal)New in c41a03b: both decoders return an _ArrayBackedDict — a plain dict subclass that additionally carries the raw numpy arrays from the decode in a hidden _clarity_arrays attribute ({tag: (ts_uint64_array, val_float32_array)}). It behaves identically to a regular dict (isinstance/equality/iteration/JSON); the attribute exists only so to_dataframe/query_dataframe can build DataFrames without re-boxing millions of points. clarity:SDK/python/clarity_sdk.py:31-44
f64 wire support (
c9fceb2). Both decoders now branch on the payloadformat_version: the point dtype is split into_POINT_DTYPE_V1(<u8ts +<f4value, 12 B/pt) and_POINT_DTYPE_V2(<u8ts +<f8value, 16 B/pt).decode_fast_query_binaryaccepts basic v1 (f32) and v2 (f64);decode_fast_query_optimisedaccepts optimised v2 (f32) and v3 (f64).query_dataframenow emits float64 columns (preserving 0.001 resolution for large totalisers).clarity:SDK/python/clarity_sdk.py:28-33
decode_fast_query_binary(data, kairos=False)Decodes the basic binary format (v1 = f32, v2 = f64) from /exactapi/fast_query_binary. The header reserved field is read as flags (bit 0 = contains_gaps → NaN gap buckets become None). clarity:SDK/python/clarity_sdk.py:52-131
Binary layout:
Header: [num_tags: u32][format_version: u32 = 1][reserved: u32]
For each tag:
[tag_name_len: u16][tag_name: utf8]
[num_points: u32]
For each point:
[timestamp: u64][value: f32]
Returns: Dict[str, List[Tuple[int, float]]] — tag name → list of (timestamp, value).
If kairos=True: Kairos format ({"queries": [...]}) with timestamps converted to milliseconds.
Points are decoded with a single np.frombuffer(data, dtype=_POINT_DTYPE, count=num_points, offset=…) call, where _POINT_DTYPE is a structured dtype (<u8 timestamp + <f4 value, 12 bytes/point). As of c41a03b, boxing uses list(zip(ts_arr.tolist(), val_arr.tolist())) (two flat .tolist() calls + zip — per the source comment ~30% faster than structured arr.tolist(), identical (int, float) tuples), and the raw arrays are retained on the result via _clarity_arrays (see _ArrayBackedDict above).
NOTE: Performance optimization only — the mapping content is unchanged.
clarity:SDK/python/clarity_sdk.py:48-108
to_dataframe(raw) (static)Converts decode_fast_query_binary / query_binary output (Dict[tag_id, List[(timestamp_ms, value)]]) into a wide pandas DataFrame: a time column (ms int) plus one float64 column per tag id, sorted by time; tags with no data become all-NaN columns. Raises ImportError if pandas is absent (imported lazily).
Fast path (new in c41a03b): when the input still carries _clarity_arrays (i.e. it came straight from a binary decode), each pd.Series is built directly from the raw numpy arrays (val.astype(float64) indexed by ts.astype(int64)), skipping the np.array(points) re-parse; the boxed-list path remains as fallback (length-mismatch or plain dicts). clarity:SDK/python/clarity_sdk.py:231-290
decode_fast_query_optimised(data, kairos=False)Decodes the optimised shared-timestamp format (v2 = f32, v3 = f64) from /exactapi/fast_query_optimised. clarity:SDK/python/clarity_sdk.py:135-210
Binary layout:
Header: [num_timestamps: u32][num_tags: u32][format_version: u32 = 2][reserved: u32]
Shared timestamps: [timestamp: u64] × num_timestamps
For each tag:
[tag_name_len: u16][tag_name: utf8]
Validity bitmap: ceil(num_timestamps / 8) bytes
Non-null values: [value: f32] for each set bit in bitmap
Returns: Dict[str, List[Tuple[int, Optional[float]]]] — sparse values, None for missing.
Decoding is vectorised with numpy: shared timestamps via a zero-copy np.frombuffer('<u8') view; the validity bitmap via np.unpackbits(..., bitorder='little'); values placed into an np.full(n, np.nan, np.float32) array masked by the bitmap. NaN becomes None on the .tolist() round-trip. New in c41a03b: the shared timestamps are boxed once (ts_list, reused for every tag, clarity_sdk.py:145); dense tags (valid_count == num_timestamps) take a zero-copy value view and skip the NaN scatter entirely (clarity_sdk.py:166-171); the result is an _ArrayBackedDict carrying the raw arrays.
NOTE: Performance optimization only — the mapping content is unchanged.
clarity:SDK/python/clarity_sdk.py:112-190
High-level HTTP client. Requires requests (now imported at module top alongside requests.adapters.HTTPAdapter, clarity:SDK/python/clarity_sdk.py:17-19).
c41a03b)The constructor creates one requests.Session per client with an HTTPAdapter(pool_connections=16, pool_maxsize=32) mounted on both http:// and https://, session.verify = False (self-signed certs). Every HTTP call in the client now goes through self._session, so TCP + TLS handshakes are paid once and connections are re-used (keep-alive) across requests. clarity:SDK/python/clarity_sdk.py:384-390
client = ClarityClient(
base_url="https://localhost:3030",
token="your_jwt_token",
# Optional: mapped_tags collection coordinates
mapped_tags_org="MAPPED_TAGS_ORG",
mapped_tags_site="MAPPED_TAGS_SITE",
mapped_tags_unit="MAPPED_TAGS_UNIT",
mapped_tags_grid="MAPPED_TAGS_GRID",
mapped_tags_interval_ms=60000,
)
clarity:SDK/python/clarity_sdk.py:349-399
| Parameter | Type | Default | Description |
|---|---|---|---|
base_url |
str |
required | Base URL of the Clarity API (e.g. https://localhost:3030) |
token |
str |
required | Bearer JWT token used in Authorization header |
mapped_tags_org |
str |
"MAPPED_TAGS_ORG" |
organization coordinate of the auto-managed mapped_tags collection |
mapped_tags_site |
str |
"MAPPED_TAGS_SITE" |
site coordinate |
mapped_tags_unit |
str |
"MAPPED_TAGS_UNIT" |
unit coordinate |
mapped_tags_grid |
str |
"MAPPED_TAGS_GRID" |
grid coordinate |
mapped_tags_interval_ms |
int |
60000 |
Sampling interval (ms) used when auto-creating the mapped_tags collection |
clarity:SDK/python/clarity_sdk.py:349-383
ClarityClient.login(base_url, email, password, **kwargs) (classmethod)Convenience constructor that authenticates and returns a ready-to-use client. POSTs {email, password} to {base_url}/exactapi/login (TLS verification disabled — verify=False), reads the token from token or access_token in the response, and returns cls(base_url, token=token, **kwargs). Raises RuntimeError if login succeeds but no token is present; HTTP errors propagate via raise_for_status(). clarity:SDK/python/clarity_sdk.py:401-429
query_binary(start, end, ..., tags=None, mapped_tags=None, pipeline=None, kairos=False)/exactapi/fast_query_binary (format v1) and returns decoded binary (Dict[tag_id, List[(timestamp_ms, value)]])start/end are now milliseconds (changed from Unix seconds)pipeline (if provided) is forwarded to the binary endpoint in the payload and applied server-side before encoding — it no longer reroutes to the JSON /exactapi/fast_query endpointtags (plain list) or mapped_tags (spec dicts)clarity:SDK/python/clarity_sdk.py:734-805
query_binary_parallel(start, end, queries, kairos=False, max_workers=8)Fetches multiple independent tag groups for the same time range concurrently via a ThreadPoolExecutor (pool size min(len(queries), max_workers)). queries is a list of dicts each with tags (required) and an optional pipeline; each is sent through query_binary. Returns a list of decoded results in the same order as queries (each Dict[tag_id, List[(timestamp_ms, value)]]). clarity:SDK/python/clarity_sdk.py:806-852
query_fast(start, end, organization=None, site=None, unit=None, grid=None, tags=None, mapped_tags=None, pipeline=None)Queries the JSON /exactapi/fast_query endpoint directly (vs. the binary endpoints). Pass exactly one of tags or mapped_tags (else ValueError); mapped_tags are resolved to tag IDs via a filtered GET /exactapi/tag_mappings lookup (see mapped_tags Feature — one spec can resolve to multiple tags). pipeline is an optional aggregation dict keyed by tag ID.
Changed in 67ac68c (PR #260, commit "query_fast dumb bug fix"). The method now returns the server's decoded JSON verbatim (return response.json()). The previous client-side post-processing that rebuilt each tag's rows as (int(ts), float(val)) tuples was removed (commented out). The result shape is therefore whatever /exactapi/fast_query emits — Dict[tag_id, List[[timestamp_ms, value]]] with rows as JSON arrays (not Python tuples) and values left un-coerced. clarity:SDK/python/clarity_sdk.py:853-922
TODO-VERIFY: the method's return-type annotation (
-> Dict[str, List[Tuple[int, float]]]) and docstringReturns:line still describe the old tuple-coerced shape and were not updated to match the raw-JSON return (withinclarity_sdk.py:853-922).
query_optimised(start, end, ..., tags=None, mapped_tags=None, pipeline=None, kairos=False)/exactapi/fast_query_optimised (format v2 + bitmap) — clarity:SDK/python/clarity_sdk.py:923-1017c41a03b: if pipeline is provided, the request goes to the binary Format-1 endpoint /exactapi/fast_query_binary (the optimised endpoint ignores pipeline server-side; the JSON /exactapi/fast_query fallback used before was much slower). The response is decoded with decode_fast_query_binary and rows are re-boxed as lists ({tag: [[timestamp_ms, value], ...]}) to preserve the exact shape the old JSON path produced. clarity:SDK/python/clarity_sdk.py:980-1009clarity:SDK/python/clarity_sdk.py:879-973
query_dataframe(tags, start, end)Convenience wrapper over query_binary that returns a wide pandas DataFrame:
timestamp_ms (millisecond epoch int)timestamp: the index materialized as a column (renamed from index)float64 column per tag ID (empty column logged as a warning if a tag has no data)Raises ImportError if pandas is not installed (imported lazily inside the method).
Fast path: when the query_binary result still carries _clarity_arrays, each column is built straight from the raw arrays as a float64 pd.Series (contiguous writable copy of the value array, int64 timestamp index) — no per-point boxing round-trip; the list path remains as fallback. clarity:SDK/python/clarity_sdk.py:1045-1110
TODO-VERIFY:
query_dataframe's docstring still saysstart/endare Unix seconds, but it forwards them verbatim toquery_binary, whose docstring says milliseconds (both withinclarity_sdk.py:734-805, 1018-1099). The two docstrings disagree; confirm the intended unit.
Three endpoints available, all accepting the same three payload shapes:
| Method | Endpoint | Durability | Target |
|---|---|---|---|
write() |
/exactapi/write |
fsync | General-purpose |
write_fast() |
/exactapi/write_fast |
mmap, no fsync | <100ms, 100k+ tags |
write_buffered() |
/exactapi/write_buffered |
queued flush | <1ms per request |
Payload shapes (pass exactly one):
tags: Dict[str, List[[timestamp, value]]] — plain tag dictdata: List[Dict] — bulk batch with shared timestampsmapped_tags: List[Dict] — tag-spec dicts each containing metricName + datawrite_buffered additionally supports single-point shape: tag + timestamp + optional value.
NOTE: Unlike the JS SDK, the Python SDK does not implement HA shadow dual-write — writes go only to the primary endpoint (no
shadow_writemirroring for zero-loss failover). See JS SDK.
clarity:SDK/python/clarity_sdk.py:1165-1404
NOTE: The Python SDK docstring for
write_bufferedstates "background flush every 50ms" (withinclarity:SDK/python/clarity_sdk.py:1308-1404). The JS SDK doc comment says the same. The Rust backend's actual flush interval is 20ms — the SDK docstrings are not authoritative for this value. See Storage Engine.
_context_fields (private) omits any argument that is None using if x is not None checks — fields with None values are silently excluded from the JSON request body.
clarity:SDK/python/clarity_sdk.py:723-733
pipeline is an optional dict keyed by tag ID. Each value is a list of aggregation step dicts, each with op (string) and optional bucket (milliseconds; default 60000 if not specified). Ops are evaluated server-side (fast_query and fast_query_binary both apply the pipeline before encoding).
clarity:backend/src-tauri/src/api/aggregator.rs:8-34, 198
op string |
Extra parameters | Description |
|---|---|---|
"mean" / "avg" |
— | Arithmetic mean per bucket |
"sum" |
— | Sum per bucket |
"min" |
— | Minimum per bucket |
"max" |
— | Maximum per bucket |
"count" |
— | Count of non-null values per bucket |
"first" |
— | First value in bucket |
"last" |
— | Last value in bucket |
"dev" |
— | Standard deviation per bucket |
"diff" |
— | Difference between consecutive values |
"gaps" |
— | Gap-fill resampling |
"percentile" |
percentile (float) |
Nth percentile per bucket |
"histogram" |
percentile (float) |
Histogram percentile per bucket |
"rate" |
unit (string, e.g. "second") |
Rate of change per time unit |
"scale" |
factor (float) |
Multiply all values by factor |
"div" |
divisor (float) |
Divide all values by divisor |
"filter" |
filter_op ("LT" | "LTE" | "GT" | "GTE" | "EQUAL"), threshold (float) |
Retain only values matching the condition |
"sampler" |
unit (string) |
Placeholder — currently treated identically to "rate" |
NOTE:
"sampler"is an explicit placeholder in the Rust source (aggregator.rs:177-179: "Placeholder: treat like Rate for now")."save_as"variant exists in the enum but is not implemented (aggregator.rs:181-184).
Verified from clarity:backend/src-tauri/src/api/aggregator.rs — enum uses #[serde(tag = "op", rename_all = "lowercase")]. As of 45a686e the parameter sub-enums (trim, filter, and score's order/boundary) are also lowercase-canonical; the UPPERCASE spellings in the tables below remain valid as aliases:
op string |
Additional fields | Notes |
|---|---|---|
"leastsquares" |
none | Linear regression fit over the time window |
"trim" |
trim: "FIRST" | "LAST" | "BOTH" |
Remove first, last, or both data points |
"saveas" |
metric_name (string), tags (object), ttl (int, optional), add_saved_from (bool, optional) |
Placeholder — saving not implemented in pipeline (aggregator.rs:181-184) |
"score" |
order: "ASCENDING" | "DESCENDING", thresholds: list of {value, boundary: "SUPERIOR"\|"INFERIOR"} |
Maps values to a score index based on threshold buckets |
Additional variants not shown in tables above:
op string |
Additional fields | Notes |
|---|---|---|
"avg" |
bucket (ms, optional) |
Alias for "mean" |
"dev" |
bucket (ms, optional) |
Standard deviation per bucket |
"gaps" |
bucket (ms, required), requires query start/end |
Fill time gaps with null values |
"diff" |
none | Sequential differences (v2 - v1) |
"sampler" |
unit (string) |
Placeholder — treated identically to "rate" |
clarity:SDK/python/clarity_sdk.py:355-1404
| Error type | Condition |
|---|---|
ValueError |
Invalid argument combination (e.g. both tags and mapped_tags supplied, or neither; missing metricName in mapped_tags spec; missing data key on write spec) |
requests.exceptions.HTTPError |
HTTP 4xx / 5xx response — raised by response.raise_for_status() in every HTTP call |
Allows callers to reference tags by semantic spec rather than by raw tag ID.
Spec format: {"metricName": "temperature", "customField": "value", ...}
On read (changed in 45a686e): each spec is sent as a where filter via GET /exactapi/tag_mappings?filter={"where": spec} (_find_tag_mappings), and every matching row's generatedDataTagId becomes a tag name in the query. Any field you omit from the spec is unconstrained, so a single spec can resolve to many tags — e.g. {"metricName": "temperature", "chemical": "H2O"} fetches every zone/system variant sharing that metric+chemical in one call. A spec matching nothing resolves to zero tags; unlike the write path, a read never creates a tag_mappings row. (This replaces the former find-or-create POST, which resolved each spec to exactly one row.)
On write: spec + "data": [[timestamp, value], ...] → resolves ID via find-or-create POST (_post_tag_mapping, exactly one row) → ensures collection exists (create or append) → writes using ID.
Collection auto-management:
GET /exactapi/collection to check if mapped-tags collection existsPOST /exactapi/create_collection with all resolved tag IDsPOST /exactapi/update_collection to append any new tag IDsclarity:SDK/python/clarity_sdk.py:444-528 — _post_tag_mapping (write, find-or-create) at :444-466, _find_tag_mappings (read filter) at :468-492, _resolve_mapped_tags_for_fetch at :494-528.
Last updated: 2026-07-17 from commit 6800acc