Clarity manages Python-based connector and ML services as child processes within the Tauri app. Two modules are involved: python_proxy.rs (HTTP forwarding) and service_manager.rs (lifecycle) — python_runner.rs was deleted in c41a03b (see Python Runner — removed). The Python source is shipped encrypted and verified against an embedded hash manifest before launch — see Build-time obfuscation & integrity.
python_proxy.rsSource: clarity:backend/src-tauri/src/api/python_proxy.rs
Forwards incoming warp HTTP routes to FastAPI or Flask backends running as child processes. Acts as a transparent reverse proxy — it re-issues the request to the Python process's local port and streams the response back.
The proxy now talks to Python services over plain HTTP on loopback (http://127.0.0.1:{port}), not HTTPS. The previous danger_accept_invalid_certs(true) setting was removed — Python services speak plain HTTP, so there is no TLS handshake on the loopback leg. The shared reqwest::Client also sets pool_idle_timeout(60s) so reqwest never reuses a pooled connection that waitress has already closed (waitress closes idle connections at ~120s). clarity:backend/src-tauri/src/api/python_proxy.rs:123-205
handle_proxy instruments every proxied request and emits a structured timing log line on completion:
[PythonProxy] <method> <path> → <service> | status=<code> | resolve=<µs> upstream=<ms> body=<ms> total=<ms> | req=<bytes> resp=<bytes>
resolve — route-registry lookup time (microseconds)upstream — TCP connect + send + wait for response headers (no TLS on the loopback leg); pooled connections skip the connectbody — time to read the response bodytotal — end-to-end proxy timeOn an upstream send failure the proxy logs a FAIL line (with total/resolve timing) and returns an HTTP 502 JSON response rather than a warp rejection (a rejection would fall through to the static-asset catch-all and yield a misleading 405).
clarity:backend/src-tauri/src/api/python_proxy.rs:147-258
service_manager.rsSource: clarity:backend/src-tauri/src/python_services/service_manager.rs
Implements PythonServicesRootConfig which manages the set of Python services:
clarity:backend/src-tauri/src/python_services/service_manager.rs:30-55
data_dir parameter (new in c41a03b)start_python_services / init_python_services / setup_python_services all gained a data_dir parameter (wired from main.rs:5013-5018), driven by the NSIS installer switching to perMachine (Program Files is read-only for standard users):
logs_dir = data_dir.join(&config.logs_directory) (was the install dir).data_dir/python_work/<service.name> (was the exe dir), avoiding EACCES under Program Files.CLARITY_LOG_DIR (services write their own log file — piped stderr capture is unreliable) and PDF_OUTPUT_DIR (defaults to <work_dir>/output; a value in python_services_config.json wins).clarity:backend/src-tauri/src/python_services/service_manager.rs (commit a2a577f)
Per-service config fields and defaults:
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | required | Service name |
startup_file |
string | required | Entry point script/binary |
port |
u16 | required | Port assigned to this service |
log_file |
string | required | Log file name |
env_vars |
map | {} |
Service-specific env overrides |
enable_health_check |
bool | false |
Enable HTTP health polling |
health_check_url |
string? | null |
URL polled when health check enabled |
health_check_interval |
u64 (secs) | 30 |
Polling interval in seconds |
restart_policy |
string | "always" |
Restart policy string |
max_restarts |
u32 | 5 |
Max restart attempts |
auto_start |
bool | true |
Start automatically at app launch |
routes |
list | [] |
URL patterns this service handles (FastAPI {param} and Flask <param> styles) |
Root-level config fields (apply to all services):
| Field | Default | Description |
|---|---|---|
python_runtime |
"python_services/python_runtime" |
Directory holding the real CPython runtime (python.exe on Windows, bin/python on Unix) |
venv_dir |
"python_services/g-adk" |
The uv-created virtual environment directory |
global_env_vars |
{} |
Env vars injected into every service |
clarity:backend/src-tauri/src/python_services/service_manager.rs:72-94
Platform note: Windows only.
On Windows, start_python_services prefers <base>/<python_runtime>/python.exe (the real CPython binary) over the venv's Scripts\python.exe. uv generates Scripts\python.exe as a trampoline shim with the build machine's Python path hard-coded inside; that path does not exist on end-user machines, producing uv trampoline failed / entity not found (os error 2) on launch. If python_runtime\python.exe is absent it falls back to the venv shim, then to the Unix path.
clarity:backend/src-tauri/src/python_services/service_manager.rs:246-270
To keep the venv's packages visible to the runtime interpreter, the manager injects the venv's Lib\site-packages onto PYTHONPATH (prepended to any existing value) in global_env_vars.
clarity:backend/src-tauri/src/python_services/service_manager.rs:291-312
patch_pyvenv_cfg (which rewrites the venv's pyvenv.cfg home = line to point at the bundled runtime) was hardened: it aborts with an error if the runtime's Python binary is missing (incomplete python_runtime/), strips the \\?\ extended-length prefix from the canonicalized Windows path, preserves the original file's line endings (CRLF vs LF), and aborts if no home = line is found.
clarity:backend/src-tauri/src/python_services/service_manager.rs:560-630
When a service sets enable_health_check=true but leaves health_check_url empty/unset, the manager now logs a warning and disables the health check for that service instead of polling an empty URL.
clarity:backend/src-tauri/src/python_services/service_manager.rs:454-465
Before launching a service, start_python_services calls integrity::verify_service_dir(svc_dir), which SHA-256-hashes every .py and .dat file in the service directory and compares each against the build-time manifest. A mismatch (tampered module stub or .dat) rejects the service — it is not spawned and failed is incremented. Verifying the whole directory (not just the entry stub) closes the gap where a tampered imported module would still execute. clarity:backend/src-tauri/src/python_services/service_manager.rs:485-495 — see Build-time obfuscation & integrity.
Platform note: Windows only.
kill_process_on_port only kills a PID it can confirm is a Python process. As of c41a03b the confirmation uses the sysinfo crate with a single-PID refresh (no subprocess), replacing the previous three-tier wmic.exe → PowerShell → wmi-crate cascade (wmic is gone on current Windows 11, and every console child flashed a window under the GUI-subsystem build); remaining subprocess spawns use .hide_console(). clarity:backend/src-tauri/src/python_services/service_manager.rs
python_runner.rs (REMOVED in c41a03b)python_runner.rs and its single Tauri command run_python_binary(binary_name, function, args) (spawn a bundled binary, write [function, args] JSON to stdin, return stdout) were deleted in commit 32c6c65; no references remain in the source tree. The papa_agent_app (agent layer) binary is launched and supervised through the ProcessManager path instead: its path resolves in get_paths() (clarity:backend/src-tauri/src/main.rs:2005) and it is spawned via setup_python_services (main.rs:5013-5018), recognized by filename in the supervisor (clarity:backend/src-tauri/src/process_manager/supervisor.rs:572).
Still true: tauri.conf.json:bundle.resources was narrowed in a18d35c from Assets/**/* to ["Assets/python_services/**/*", "Assets/scripts/**/*"] — seed_data/ is compiled into the binary via include_str! and no longer ships as a runtime resource. clarity:backend/src-tauri/tauri.conf.json:26-31
Source: clarity:backend/src-tauri/Assets/python_services/python_services_config.json
Services loaded at startup from python_services_config.json. The routes list defines which HTTP paths are forwarded to each service via python_proxy_routes().
| Service name | Port | Routes | Notes |
|---|---|---|---|
data-calculation-b |
17001 | — | MQTT-driven calc engine; auto_start=true, health check disabled |
overview-kpi-boxes |
6061 | /dataflow/ems/kpiparams, /dataflow/ems/load-share-donut/linewise, /dataflow/ems/line-consumption-trend, /dataflow/ems/tags/summary, /dataflow/ems/tags/summary/download, /dataflow/ems/meters/group-config |
KPI/EMS frontend data; WAITRESS_THREADS=16 |
log-flood-test |
6099 | — | Test/diagnostic service |
totalizer-calc-b |
9002 | — | Totalizer calculation, health check disabled |
test-flask-api |
6019 | /test/sensordata/* |
Test sensor data endpoints (vibration, orbit/waterfall/FFT plots, DCS, templates) |
reports-to-pdf-python |
17000 | /report/download, /report/download1, /report/generate-test, /report/builder-generate, /report/generate1, /pulse-ui-v2-report/#/<reportType>, /pulse-reports-builder/generate |
PDF/report generation (submodule: reports-to-pdf-python; previously reports-to-pdf) |
report-scheduler |
18000 | — (no routes) |
New in c41a03b: drives reports-to-pdf-python on a schedule read from schedule.xlsx. Env: SCHEDULER_FILE_PATH=schedule.xlsx, SCHEDULER_TIMEZONE=Asia/Kolkata, SCHEDULER_TRIGGER_OFFSET_MINUTES=10, SCHEDULER_STATE_PATH=schedule.csv.state.json, SCHEDULER_FILE_REFRESH_MINUTES=5, MISSED_JOB_GRACE_MINUTES=60, REPORT_API_URL=http://localhost:17000. Entry: python_services/reports-to-pdf-python/scheduler.py. |
elog-group-data-report-api |
9003 | /elog/group/data/health, /elog/group/data, /elog/summary/download, /elog/group/data/deviation-report, /elog/group/data/downloadcsv |
Elog group data reports (new in f14ebda; see note below) |
meta-ui-upload-service |
5100 | /meta/upload, /meta/update-eqpmeta, /meta/standard-queries, /meta/reset-instances |
Metadata UI upload; WAITRESS_THREADS=1 (single-threaded). Three routes added in c41a03b (commit a227156) |
data-api |
17002 | /sensordata/spcplot, /sensordata/elog/datapoints/query, /sensordata/elog/datapoints |
SPC plot data + elog datapoints query/list (submodule data-api). /sensordata/elog/datapoints/query route added in e021b09; /sensordata/elog/datapoints added in dab6060. |
profiles-service |
17005 | — (no routes) |
Submodule profiles-service (port-clarity branch). auto_start=true. No declared proxy routes. Port moved 17002 → 17005 in e021b09. |
equipment-status-v2 |
17004 | — (no routes) |
Submodule equipment-status-v2 (new in 57c553a). MQTT-driven equipment status; env Q_PORT=1883, BROKER_ADDRESS=127.0.0.1. auto_start=true, health check disabled (enable_health_check=false). |
util-services |
17008 | — (no routes) |
Submodule util-services (new in 57c553a). auto_start=true. No env vars, no declared proxy routes. |
NOTE:
meta-ui-upload-service(port 5100,/meta/upload) is still present in the config — an earlier wiki revision incorrectly stateddata-api"replaced" it. Both coexist.
NOTE: The previously-flagged port collision between
data-apiandprofiles-service(both on 17002) was resolved ine021b09—profiles-servicemoved to17005.clarity:backend/src-tauri/Assets/python_services/python_services_config.json:159,176
TODO-VERIFY: The elog routes (
/elog/group/data/*) are also registered as Rust handlers inelog.rs(see Processing API). Determine which implementation takes precedence — depends on route registration order inmain.rs. Theelog-group-data-report-apiPython service was added in commit f14ebda with message "elog download compatable with latest clarity", suggesting the Python version may have been updated to match current API behavior.
clarity:backend/src-tauri/Assets/python_services/python_services_config.json:1-213
waitress was added to requirements.txt; services honour a WAITRESS_THREADS env var (e.g. overview-kpi-boxes=16, meta-ui-upload-service=1). In c41a03b, kaleido==0.2.1 → kaleido>=1.0.0 and setup_venv.ps1 now bundles Chrome for Testing (plotly_get_chrome -y staged into Assets/python_services/chrome-win64/, skipped if chrome.exe already present) for reliable plotly/CDP PDF rendering. clarity:backend/src-tauri/Assets/python_services/requirements.txt, setup_venv.ps1setup_venv.sh / setup_venv.ps1 install an extra requirements file extra_modules_req.txt (uv pip install --no-cache --reinstall …) after the main requirements. It contains the vendored Clarity Python SDK wheel extra_modules/clarity_sdk-1.0.0-py3-none-any.whl, refreshed from SDK/python/dist/ by extra_modules/get_whl.sh. clarity:backend/src-tauri/Assets/python_services/setup_venv.sh:71-86python_services.sh init/updates each service submodule (checking out branch port-clarity) and clears stale __pycache__ dirs before each submodule update. As of 57c553a the SUBMODULES/URLS arrays and the per-service checkout blocks also cover util-services and equipment-status-v2. clarity:backend/src-tauri/python_services.shThe shipped Python service code is encrypted at build time and verified against an embedded hash manifest before each service is spawned. Both steps were moved into Rust (build.rs) in a18d35c — the previous Python obfuscator (Assets/python_services/obfuscate.py, 303 lines) was deleted.
build.rs main() runs three steps in order: run_obfuscation() (creates .obf_key if missing), embed_service_key() (bakes the key into the binary), then generate_python_manifest(). clarity:backend/src-tauri/build.rs:30-36
New build dependencies: sha2, flate2, rand, serde_json ([build-dependencies]); wmi (Windows runtime dep, used by fingerprinting + port-kill detection). clarity:backend/src-tauri/Cargo.toml:12-16,100-103, clarity:backend/Cargo.toml:39-41
run_obfuscation)For each service in python_services_config.json, every .py (except __init__.py, kept verbatim) is encrypted to a sibling .dat and replaced by a small loader stub. The original sources are moved to src-tauri/.python_sources/<svc>/ (never shipped). clarity:backend/src-tauri/build.rs:316-475
zlib(best) → XOR(32-byte key) → base64-std. (The deleted Python version used marshal + zlib + XOR + base85; the Rust version uses compile() + zlib + XOR + base64 so it carries no Python-bytecode-version dependency.)_SK env var (injected by the binary at spawn time), decrypt the .dat, and exec it. The entry stub also installs a MetaPathFinder so sibling modules load from their .dat files; non-entry modules get a simpler stub. Stubs begin with the marker comment # obf-rs.# obf-rs is skipped..obf_key is a 32-byte random file at src-tauri/.obf_key; embed_service_key() reads it and bakes it into the binary as SERVICE_DECRYPTION_KEY. The binary injects it as _SK (hex) when spawning each service.Build-time toggles:
| Env var | Effect |
|---|---|
CLARITY_RESTORE=1 |
restore_python_sources() moves .python_sources/* back into Assets/python_services/, then sets CLARITY_NO_OBFUSCATE=1 so the restored files are not re-encrypted. For editing service sources. clarity:backend/src-tauri/build.rs:5-11,151 |
CLARITY_NO_OBFUSCATE=1 |
Skip obfuscation entirely. clarity:backend/src-tauri/build.rs:316-319 |
generate_python_manifest)Hashes every .py and .dat under Assets/python_services/ (skipping python_runtime, g-adk, gtk_libs, extra_modules, __pycache__, .git, and obfuscate.py) with SHA-256 and emits OUT_DIR/python_manifest.rs — a static PYTHON_HASHES: &[(&str, [u8; 32])] keyed by the Assets/python_services/… relative path. clarity:backend/src-tauri/build.rs:478-525
integrity/mod.rs)The new integrity module include!s the generated manifest. verify_python_file(path) normalizes the path to its Assets/python_services/… key, looks up the expected hash, re-hashes the file, and returns Err on mismatch or if the file is absent from the manifest. verify_service_dir(dir) recurses and verifies every .py/.dat. When PYTHON_HASHES is empty (dev build with no Assets), both are no-ops. clarity:backend/src-tauri/src/integrity/mod.rs:1-64. Called by start_python_services before each spawn (see Integrity verification before spawn).
Last updated: 2026-07-11 from clarity@c41a03b