Module introspect 

Mesh-topology introspection types and attrs.

This module owns the typed internal model used by mesh-admin and the TUI: mesh-topology attr keys, typed attrs views, NodeRef, and the domain NodePayload / NodeProperties / FailureInfo values derived from hyperactor::introspect::IntrospectResult.

These keys are published by HostMeshAgent, ProcAgent, and MeshAdminAgent to describe mesh topology (hosts, procs, root). Actor-runtime keys (status, actor_type, messages_processed, etc.) are declared in hyperactor::introspect.

The HTTP wire representations live in dto. That submodule owns the curl-friendly JSON contract, schema/OpenAPI generation, and boundary invariants for string-encoded references and timestamps. This module keeps the internal typed invariants.

These invariants govern the introspection model and derived payloads exposed by mesh-admin; lower-level runtime accounting invariants remain owned by the runtime modules that produce those values.

See hyperactor::introspect for naming convention, invariant labels, and the IntrospectAttr meta-attribute pattern.

Mesh key invariants (MK-*)

• MK-1 (metadata completeness): Every mesh-topology introspection key must carry @meta(INTROSPECT = ...) with non-empty name and desc.
• MK-2 (short-name uniqueness): Covered by test_introspect_short_names_are_globally_unique in hyperactor::introspect (cross-crate).

Proc debug stats invariants (PD-*)

These invariants govern the proc-debug introspection surface exposed by mesh-admin: proc attrs, typed proc views, and the proc-debug portion of NodeProperties::Proc.

They do not define proc runtime mechanics. The underlying per-actor queue-depth accounting invariants live in hyperactor::proc; this module owns the proc-level debug values derived from that runtime state.

• PD-1: actor_work_queue_depth_max <= actor_work_queue_depth_total.
• PD-2: process_rss_bytes and process_vm_size_bytes are None on non-Linux or read failure. Never fabricated.
• PD-3: All debug fields default to zero/None for backward compatibility. Old procs that haven’t published yet produce a valid ProcDebugStats::default().
• PD-4: Queue depth aggregation covers live actors only. Stopped/retained actor snapshots are excluded.
• PD-5: See hyperactor::proc module doc for the per-actor queue depth accounting invariants (PD-5a through PD-5e).

HTTP boundary invariants (HB-*)

These govern the HTTP DTO layer in dto.

• HB-1 (typed-internal, string-external): NodeRef, ActorAddr, ProcAddr, and SystemTime are typed Rust values internally. At the HTTP JSON boundary, dto::NodePayloadDto, dto::NodePropertiesDto, and dto::FailureInfoDto encode them as canonical strings.
• HB-2 (round-trip): The HTTP string forms round-trip through the internal typed parsers (NodeRef::from_str, ActorAddr::from_str, humantime::parse_rfc3339). Timestamps are formatted at millisecond precision; sub-millisecond values are truncated at the boundary.
• HB-3 (schema-honesty): Schema/OpenAPI are generated from the DTO types, so the published schema reflects the actual wire format rather than the internal domain representation.

Attrs invariants (IA-*)

These govern how IntrospectResult.attrs is built in hyperactor::introspect and how properties is derived via derive_properties.

• IA-1 (attrs-json): IntrospectResult.attrs is always a valid JSON object string.
• IA-2 (runtime-precedence): Runtime-owned introspection keys override any same-named keys in published attrs.
• IA-3 (status-shape): status_reason is present in attrs iff the status string carries a reason.
• IA-4 (failure-shape): failure_* attrs are present iff effective status is failed.
• IA-5 (payload-totality): Every IntrospectResult sets attrs – never omitted, never null.
• IA-6 (open-row-forward-compat): View decoders ignore unknown attrs keys; only required known keys and local invariants affect decoding outcome. Concretized by AV-3.

Attrs view invariants (AV-*)

These govern the typed view layer (*AttrsView structs).

• AV-1 (view-roundtrip): For each view V, V::from_attrs(&v.to_attrs()) == Ok(v) (modulo documented normalization/defaulting).
• AV-2 (required-key-strictness): from_attrs fails iff required keys for that view are missing.
• AV-3 (unknown-key-tolerance): Unknown attrs keys must not affect successful decode outcome. Concretization of IA-6.

Derive invariants (DP-*)

• DP-1 (derive-precedence): derive_properties dispatches on status first (DP-5), then node_type, then error_code, then unknown. This order is the canonical detection chain.
• DP-2 (derive-totality-on-parse-failure): derive_properties is total; malformed or incoherent attrs never panic and map to NodeProperties::Error with detail.
• DP-3 (derive-precedence-stability): derive_properties detection order is stable and explicit: status > node_type > error_code > unknown.
• DP-4 (error-on-decode-failure): Any view decode or invariant failure maps to a deterministic NodeProperties::Error with a malformed_* code family, without panic.
• DP-5 (actor-view classification safety): a payload carrying the core STATUS key always decodes as Actor. STATUS is set only by the blanket Actor builder (build_actor_attrs) and is core-owned, so the actor-attrs snapshot seam can neither remove nor override it (AS-2), and no non-actor payload (root/host/proc/error) carries it. Therefore an actor cannot inject node_type/error_code via the seam to spoof a different node kind.

Execution presentation (EX-*)

These govern the execution field on NodeProperties::Actor — an actor’s in-flight handler execution, reported through the generic actor-attrs snapshot seam (AS-* in hyperactor::introspect) and decoded from the EXECUTION attr in derive_properties.

• EX-1 (unsupported-vs-idle): execution: None means the actor does not report execution (no snapshot installed) – unsupported, not idle. A supported-but-idle actor is Some with active_count == 0.
• EX-2 (partial-detail, never absence): complete == false means the per-handler detail was momentarily unavailable on that read (e.g. a non-blocking tracker miss); active_count stays authoritative and the field stays Some – contention never collapses execution to None.
• EX-3 (observational, not transactional): active_count and active_handlers are independent point-in-time reads; active_count need not equal sum(active_handlers[*].active_count) on a given poll (cf. IO-3). Consumers must not derive one from the other.
• EX-4 (deterministic truncation): active_handlers is ordered oldest-first with a stable tie-break on name; truncated == true means it is a prefix of the N oldest while active_count remains the full total.
• EX-5 (post-mortem semantics): a terminated actor’s stored snapshot persists its last live_actor_payload, so execution reflects state as of termination. The producer drains in-flight entries on stop (try/finally), so a stopped actor reports active_count == 0.

Inbound ordering presentation (IO-*)

Mesh-admin presentation extension of the cross-crate IO-* family. The lower-level invariants IO-1 (tri-state absence), IO-2 (publish-time try_lock), and IO-3 (no arithmetic relation between queue_depth and reorder-buffer depth) live in hyperactor::introspect. The presentation layer below adds:

• IO-4 (snapshot_complete derivation): InboundOrdering.snapshot_complete == (skipped_session_count == 0). Mirrors OrderingSnapshot::is_complete() at the presentation layer. By IO-2, the aggregate lock makes this an availability check: true means every session was observed and false means none were.
• IO-5 (complete-snapshot exactness): When snapshot_complete == true, known_session_count == sessions.len() and every returned_* rollup is exact over that complete session set.
• IO-6 (unavailable-snapshot sentinel): When snapshot_complete == false, the aggregate sequencing lock was busy and no session state was observed. The current producer returns sessions == [], skipped_session_count == 1, and zero returned_* rollups; the legacy presentation derivation therefore sets known_session_count == 1. These are fixed unavailability sentinel values, not observations of one session with zero stalls. Consumers must retry rather than diagnose from them.
• IO-7 (live-actor exposure): For any actor built through Instance::new, /v1/{actor} exposes inbound_ordering: Some(...) – never None. None indicates either structural absence (test fixtures, hand-built InstanceCellState) or a regression in the publish path.

py-spy integration (PS-*)

• PS-1 (target locality): PySpyDump always targets std::process::id() of the handling ProcAgent process. No caller-supplied PID exists in the API.
• PS-2 (deterministic failure shape): Execution failures are classified into BinaryNotFound { searched } vs Failed { pid, binary, exit_code, stderr }, never collapsed.
• PS-3 (binary resolution order): Resolution order is exactly: PYSPY_BIN config attr (if non-empty) then "py-spy" on PATH. The attr is read via hyperactor_config::global::get_cloned; env var PYSPY_BIN feeds in through the config layer. If the first attempt is not found, the fallback attempt is required.
• PS-4 (structured JSON output): py-spy runs with --json; output is parsed into Vec<PySpyStackTrace>. A successful result’s capture_mode records whether it used native_all, native, or python_only. Parse failure maps to PySpyResult::Failed.
• PS-5 (subprocess timeout): try_exec bounds the py-spy subprocess inside the worker to MESH_ADMIN_PYSPY_TIMEOUT (default 10s). The budget is sized for --native --native-all which unwinds native stacks via libunwind — significantly slower than Python-only capture on loaded hosts. Each child leads its own process group. On timeout or cancellation the group is killed so descendants cannot retain inherited pipes; the direct-child reap has its own one-second bound. The worker returns Failed { stderr: "…timed out…" } on timeout.
• PS-6 (bridge timeout): The HTTP bridge uses a separate MESH_ADMIN_PYSPY_BRIDGE_TIMEOUT (default 13s), which must exceed MESH_ADMIN_PYSPY_TIMEOUT so the subprocess kill/reap and reply can arrive before the bridge declares gateway_timeout. Independent of MESH_ADMIN_SINGLE_HOST_TIMEOUT.
• PS-7 (non-blocking delegation): ProcAgent never awaits py-spy execution inline. On PySpyDump it spawns a child PySpyWorker, forwards the request, and returns immediately.
• PS-8 (worker lifecycle): Each PySpyWorker handles exactly one forwarded RunPySpyDump, replies directly to the forwarded OncePortRef, then self-terminates via cx.stop(). Clean exit, no supervision event.
• PS-9 (concurrent dumps): py-spy is spawn-per-request, so overlapping dumps on the same proc are allowed. Each worker runs independently.
• PS-10 (nonblocking retry): In nonblocking mode, try_exec retries up to 3 times with 100ms backoff on failure, because py-spy can segfault reading mutating process memory. Attempts and backoff share the MESH_ADMIN_PYSPY_TIMEOUT deadline; timeout cleanup has the separate one-second reap bound in PS-5.
• PS-11a (native-all-immediate-downgrade): If py-spy rejects --native-all with the recognized unsupported-flag signature (exit code 2, stderr mentions --native-all), try_exec retries immediately with native_all = false and native = true in the same outer attempt. This also preserves native capture for an actor caller that supplied native_all = true, native = false.
• PS-11b (native-all-no-retry-consumption): That downgrade retry does not consume an outer nonblocking retry slot (PS-10) and does not incur the 100ms inter-attempt backoff.
• PS-11c (native-all-downgrade-warning): A successful downgraded result includes pyspy::native_all_downgrade_warning(label), which names how the py-spy binary was resolved — PS-3 means the caller cannot otherwise tell which one ran. The warning survives failed downgraded outer retries and is attached to any eventual successful result, whose capture_mode is native.
• PS-11d (native-all-failure-passthrough): If the downgraded retry also fails, the failure flows through the normal nonblocking retry logic (PS-10) unchanged.
• PS-11e (native-all-sticky-downgrade): Once the unsupported-flag signature is detected, effective_opts.native_all remains false for all subsequent outer retries. The flag is not re-tested on later attempts.
• PS-12 (universal py-spy): Worker procs and the service proc can handle PySpyDump. Worker procs handle it via ProcAgent; the service proc handles it via HostAgent (same spawn-worker pattern). pyspy_bridge routes to a HostAgent only when its exact actor identity is registered with the mesh admin; all other procs route to proc_agent[0]. Procs lacking either agent (e.g. mesh-admin) fast-fail via PS-13.
• PS-13 (defensive probe): Before sending PySpyDump, pyspy_bridge probes the selected actor with an introspect query bounded by MESH_ADMIN_QUERY_CHILD_TIMEOUT (default 100ms). Three outcomes: (a) probe reply arrives — proceed with PySpyDump; (b) probe times out or recv closes — return not_found (actor absent/unreachable); (c) probe send itself fails — return internal_error (bridge-side infrastructure failure). Cases (b) and (c) fast-fail instead of waiting the full 13s MESH_ADMIN_PYSPY_BRIDGE_TIMEOUT.
• PS-14 (reachability-based capability): A proc supports py-spy iff its stable handler actor is reachable: the service proc requires a reachable host_agent; non-service procs require a reachable proc_agent[0]. PySpyWorker is transient per-request machinery (spawned on PySpyDump, stopped after replying) and is not part of the reachability contract.
• PS-15a (native-immediate-downgrade): Native frames matter — a proc stuck in a collective or an allocator shows nothing useful without them — but a partial dump beats no dump. If a capture that requested --native or --native-all returns Failed for any reason other than the PS-11a unsupported-flag signature — libunwind rejecting the target with UNW_EINVAL, say — try_exec drops both native flags and retries Python-only in the same outer attempt, and PS-15c says so in the result.
• PS-15b (native-no-retry-consumption): That downgrade retry does not consume an outer nonblocking retry slot (PS-10) and does not incur the 100ms inter-attempt backoff.
• PS-15c (native-downgrade-warning): A successful downgraded result has capture_mode = python_only and includes pyspy::native_downgrade_warning(label). The enum provides a queryable capture-mode signal, while the warning names both the py-spy that fell short and the remedy: point the PYSPY_BIN environment variable on the dumped proc at a build that can unwind the target. The warning survives failed Python-only outer retries and is attached to any eventual successful result.
• PS-15d (native-sticky-downgrade): Once native capture has failed, effective_opts.native and effective_opts.native_all remain false for all subsequent outer retries. Native is not re-tested on later attempts.

v1 contract notes:

• The current py-spy bridge expects a ProcAddr-form reference and rejects other forms as bad_request. This may be broadened in future versions.
• If worker.send() fails after the reply port has moved into RunPySpyDump, the caller receives no explicit PySpyResult::Failed — they observe a timeout. MailboxSenderError does not carry the unsent message, so the port is irrecoverable on this path.
• Contract change (D96756537 follow-up): PySpyResult::Ok replaced stack: String (raw py-spy text) with stack_traces: Vec<PySpyStackTrace> (structured JSON) and added warnings: Vec<String>. Clients reading the old stack field will see it absent; they must migrate to stack_traces.

py-spy profiling (PP-*)

Profile capture (py-spy record) is a separate contract from dump (py-spy dump). Types, messages, workers, and routes are independent — no shared state, no shared timeout budget.

• PP-1 (input validation): duration_s (u32) must be non-zero and at most MESH_ADMIN_PYSPY_MAX_PROFILE_DURATION. rate_hz must be 1..1000. Violations → 400 before any actor messaging.
• PP-2 (dynamic timeout cascade): Subprocess timeout = duration_s + 15s. Bridge timeout = subprocess + 5s. Computed per-request from validated opts, not static config.
• PP-3 (temp file lifecycle): py-spy record writes to a temp file; the worker reads it after successful exit and deletes via tempfile drop. On failure or timeout, stderr is captured. On timeout or cancellation, the child process group is killed; the direct-child reap has its own one-second bound. If the file is missing, empty, or unreadable after successful exit, the result is OutputMissing, OutputEmpty, or OutputReadFailure, not Ok.
• PP-4 (target locality): Inherits PS-1 — always targets std::process::id(), never a caller-supplied PID.
• PP-5 (separate worker): PySpyProfileWorker is a distinct actor from PySpyWorker. Profile durations block for seconds to minutes; isolation prevents starving dumps.
• PP-6 (wire projection): ProfileExecOutcome maps to PySpyProfileResult 1:1 via From. Every internal variant has an identically-named wire variant. The only shape change is TimedOut.timeout: Duration → TimedOut.timeout_s: u64.

Mesh-admin config (MA-*)

• MA-C1 (timeout config centralization): Mesh-admin timeout budgets are read from config attrs at call-time, with defaults in config.rs. No hardcoded timeout constants in mesh_admin.rs.

Modules

dto

HTTP boundary DTO types for mesh-admin introspection.

Structs

ActiveHandler

One handler with in-flight invocations, aggregated by name (EX-4).

ErrorAttrsView

Typed view over attrs for an error node.

Execution

An actor’s in-flight handler execution, reported through the generic actor-attrs snapshot seam (AS-*). Carried both as the EXECUTION attr value and as the NodeProperties::Actor.execution field; core hyperactor does not interpret it. See EX-* in module doc.

FailureInfo

Structured failure information for failed actors.

HostAttrsView

Typed view over attrs for a host node.

InboundOrdering

Mesh-admin presentation of inbound ordering state. Computed from the upstream hyperactor::ordering::OrderingSnapshot; rollup fields are derived at conversion time so consumers don’t have to iterate sessions for the common “is anything stalled?” question.

NodePayload

Uniform response for any node in the mesh topology.

ProcAttrsView

Typed view over attrs for a proc node.

ProcDebugStats

Proc-level debug/operational stats. Groups hosting-process memory (process-scoped) and actor queue pressure (proc-scoped) into one operational summary.

ProcessMemoryStats

Memory stats of the hosting OS process. Shared by host and proc introspection surfaces — both agents are authoritative for the OS process they run in.

RootAttrsView

Typed view over attrs for a root node.

Enums

NodeProperties

Node-specific metadata. Externally-tagged enum — the variant name is the discriminator (Root, Host, Proc, Actor, Error).

NodeRef

Typed reference to a node in the mesh-admin navigation tree.

NodeRefParseError

Error parsing a NodeRef from a string.

Statics

ACTOR_WORK_QUEUE_DEPTH_HIGH_WATER_MARK

Maximum proc-wide queue depth observed since startup (PD-6). Eventually consistent — concurrent readers may transiently observe total > high_water_mark. Retained evidence — driven from the runtime accounting path, not publish-time sampling.

ACTOR_WORK_QUEUE_DEPTH_MAX

Maximum current per-actor message queue depth across live actors at publish time. This is not a historical high-water mark.

ACTOR_WORK_QUEUE_DEPTH_TOTAL

Sum of per-actor message queue depths across live actors.

ADDR

Host network address (e.g. “10.0.0.1:8080”).

EXECUTION

In-flight handler execution for an actor.

FAILED_ACTOR_COUNT

Count of failed actors in a proc.

IS_POISONED

Whether this proc is refusing new spawns due to actor failures.

LAST_NONZERO_QUEUE_DEPTH_AGE_MS

How long ago proc-wide queue depth was last observed non-zero (PD-7). None means no counted actor work has traversed the queue accounting path since startup. Uses wall clock, so the age is best-effort telemetry and may not be strictly monotonic. Retained evidence — driven from the runtime accounting path.

NODE_TYPE

Topology role of this node: “root”, “host”, “proc”, “error”.

NUM_ACTORS

Number of actors in a proc.

NUM_HOSTS

Number of hosts in the mesh (root only).

NUM_PROCS

Number of procs on a host.

PROCESS_RSS_BYTES

RSS of the hosting OS process (bytes). None means the measurement was unavailable (for example non-Linux or procfs read/parse failure); values are never fabricated (PD-2).

PROCESS_VM_SIZE_BYTES

Virtual memory size of the hosting OS process (bytes). None means the measurement was unavailable (for example non-Linux or procfs read/parse failure); values are never fabricated (PD-2).

PROC_NAME

Human-readable proc name.

STARTED_AT

Timestamp when the mesh was started.

STARTED_BY

Username who started the mesh.

STOPPED_CHILDREN

References of stopped children (proc only).

STOPPED_RETENTION_CAP

Cap on stopped children retention.

SYSTEM_CHILDREN

References of system/infrastructure children.

Functions

derive_properties

Derive NodeProperties from a JSON-serialized attrs string.

to_node_payload

Convert an IntrospectResult to a presentation NodePayload. Lifts IntrospectRef → NodeRef and passes through typed timestamps.

to_node_payload_with

Convert an IntrospectResult to a NodePayload, overriding identity and parent for correct tree navigation.