Module introspect

Expand description

Introspection protocol for hyperactor actors.

Every actor has a dedicated introspect task that handles IntrospectMessage by reading InstanceCell state directly, without going through the actor’s message loop. This means:

Stuck actors can be introspected (the task runs independently).
Introspection does not perturb observed state (no Heisenberg).
Live status is reported accurately.

Infrastructure actors publish domain-specific metadata via publish_attrs(), which the introspect task reads for Entity-view queries. Non-addressable children (e.g., system procs) are resolved via a callback registered on InstanceCell.

Callers navigate topology by fetching an IntrospectResult and following its children references.

§Design Invariants

The introspection subsystem maintains twelve invariants (S1–S12). Each is documented at the code site that enforces it.

S1. Introspection must not depend on actor responsiveness – a wedged actor can still be introspected (runtime task, not actor loop).
S2. Introspection must not perturb observed state – reading InstanceCell never sets last_message_handler to IntrospectMessage.
S3. Sender routing is unchanged – senders target the same control PortId across processes.
S4. IntrospectMessage never produces a WorkCell – pre-registration via bind_control_port gives the introspect port its own channel, independent of the actor’s work queue.
S5. Replies never use PanickingMailboxSender – the introspect task replies via Mailbox::serialize_and_send_once.
S6. View semantics are stable – Actor view uses live structural state + supervision children; Entity view uses published properties + domain children.
S7. QueryChild must work without actor handlers – system procs are resolved via a per-actor callback on InstanceCell.
S8. Published properties are constrained – actors cannot publish Root or Error payloads (only Host and Proc variants).
S9. Port binding is single source of truth – the introspect port is bound exactly once via bind_handler_port() in Instance::new().
S10. Introspect receiver lifecycle – created in Instance::new(), spawned in start(), dropped in child_instance().
S11. Terminated snapshots do not keep actors resolvable – store_terminated_snapshot writes to the proc’s snapshot map, not the instances map. resolve_actor_ref checks terminal status independently and is unaffected by snapshot storage.
S12. Introspection must not impair actor liveness – introspection queries (including DashMap reads for actor enumeration) must not cause convoy starvation or scheduling delays that stall concurrent actor spawn/stop operations.

§Introspection key invariants (IK-*)

IK-1 (metadata completeness): Every actor-runtime introspection key must carry @meta(INTROSPECT = ...) with non-empty name and desc.
IK-2 (short-name uniqueness): No two introspection keys may share the same IntrospectAttr.name. Duplicates would break the FQ-to-short HTTP remap and schema output.

§Failure introspection invariants (FI-*)

The FailureInfo presentation type lives in hyperactor_mesh::introspect; these invariants are documented here because the enforcement sites are in hyperactor (proc.rs serve(), live_actor_payload).

FI-1 (event-before-status): All InstanceCell state that live_actor_payload reads must be written BEFORE change_status() transitions to terminal.
FI-2 (write-once): InstanceCellState::supervision_event is written at most once per actor lifetime.
FI-3 (failure attrs <-> status): Failure attrs are present iff status is "failed".
FI-4 (is_propagated <-> root_cause_actor): failure_is_propagated == true iff failure_root_cause_actor != this_actor_id.
FI-5 (is_poisoned <-> failed_actor_count): is_poisoned == true iff failed_actor_count > 0.
FI-6 (clean stop = no artifacts): When an actor stops cleanly, supervision_event is None, failure attrs are absent, and the actor does not contribute to failed_actor_count.
FI-7 (propagated-stopped-root-cause): When a failed actor’s supervision chain bottoms out in a Stopped child event, structured failure metadata must still name the stopped child as failure_root_cause_actor.
FI-8 (propagation-classification): failure_is_propagated is derived from root-cause actor identity; a parent that failed due to a child’s event must report failure_is_propagated == true.

§Attrs view invariants (AV-*)

These govern the typed view layer (ActorAttrsView). The full AV-* / DP-* family is documented in hyperactor_mesh::introspect; the subset relevant to this crate:

AV-1 (view-roundtrip): For each view V, V::from_attrs(&v.to_attrs()) == Ok(v).
AV-2 (required-key-strictness): from_attrs fails iff required keys for that view are missing.
AV-3 (unknown-key-tolerance): Unknown attrs keys must not affect successful decode outcome.

§Inbound ordering exposure invariants (IO-*)

IO-1 (inbound-ordering tri-state semantics): ActorAttrsView::inbound_ordering carries three meaningful states that consumers (DTO, TUI, agents) MUST distinguish:
- None – no snapshot callback was installed. In current code this means structural absence: an InstanceCellState not built through Instance::new (hand-built test fixtures, or any future code path that bypasses the constructor). Live actors built via Instance::new always install Some, and terminated-actor payloads – which still go through live_actor_payload(&cell) while the cell exists – inherit that Some.
- Some({enabled: false, ...}) – ordered path exists but reorder buffering is disabled; sessions is empty regardless of traffic. Messages bypass receiver-local sequencing.
- Some({enabled: true, ...}) – buffering active; sessions is meaningful.
None is NOT equivalent to Some({enabled: false, ...}).
IO-2 (inbound-ordering reflects publish-time state): When present, the snapshot is computed at build_actor_attrs invocation time via the sequenced receiver’s snapshot handle. last_released_seq etc. are point-in-time. Sessions held by a concurrent receive show up in skipped_session_count (never silently omitted); is_complete() reports the all-clear.
IO-3 (queue-depth and inbound-ordering are independent diagnostics, no arithmetic contract):
- ACTOR_QUEUE_DEPTH (per PD-5a/PD-5b in proc.rs): accepted handler work not yet dequeued by the actor loop.
- INBOUND_ORDERING.sessions[*].buffered_count: messages held by receiver-local sequencing waiting for a seq gap to fill.
These are two independent point-in-time diagnostics. No arithmetic or ordering relationship between them is part of the API contract; the accounting paths are free to change. Consumers must not derive one from the other.

§Actor-attrs snapshot invariants (AS-*)

These govern the generic per-actor introspection-attrs snapshot seam: an actor may install a Fn() -> Attrs callback via Instance::set_attrs_snapshot, which the introspect task invokes in build_actor_attrs and merges into the Actor view. The seam is the runtime-agnostic transport for actor-supplied introspection data (e.g. a Python actor reporting in-flight handler execution); core interprets none of it.

AS-1 (snapshot-opacity): actor-supplied attrs are merged into the Actor view verbatim. Core neither interprets nor validates the keys – any key the actor sets is transported as-is. This is what keeps the seam runtime-agnostic (no Rust-vs-Python knowledge, no “endpoint” concept in core).
AS-2 (core-precedence): on a key collision, the core/runtime key wins over the actor-supplied value. This is the Actor-view analog of IA-2, which governs the published-attrs path; the two sources of actor-supplied keys (published attrs, snapshot seam) are distinct, but both yield to core keys.
AS-3 (snapshot-non-fatal): an absent, empty, or panicking snapshot degrades to the core-only Actor view. The callback is catch_unwind-guarded and the merge falls back to the core attrs, so a bad snapshot can never empty or invalidate attrs (with IA-1, IA-5).

Structs§

ActorAttrsView: Typed view over attrs for an actor node.
FailureAttrs: Structured failure fields decoded from FAILURE_* attrs.
IntrospectResult: Internal introspection result. Carries attrs as a JSON string. The mesh layer constructs the API-facing NodePayload (with properties) from this via derive_properties.
RecordedEvent: Structured tracing event from the actor-local flight recorder.

Enums§

AttrsViewError: Error from decoding an Attrs bag into a typed view.
IntrospectMessage: Introspection query sent to any actor.
IntrospectRef: Typed reference to an introspectable entity.
IntrospectRefParseError: Error returned when parsing an IntrospectRef.
IntrospectView: Context for introspection query - what aspect of the actor to describe.

Statics§

ACTOR_QUEUE_DEPTH: Accepted handler work not yet dequeued by the actor loop (per proc.rs PD-5a/PD-5b). Independent of INBOUND_ORDERING: no arithmetic or ordering relationship between the two is part of the API contract. See IO-3.
ACTOR_TYPE: Fully-qualified actor type name.
CHILDREN: Child references for tree navigation. Published by infrastructure actors (HostMeshAgent, ProcAgent) so the Entity view can return children without parsing mesh-layer keys.
CREATED_AT: Timestamp when this actor was created.
ERROR_CODE: Machine-readable error code for error nodes.
ERROR_MESSAGE: Human-readable error message for error nodes.
FAILURE_ERROR_MESSAGE: Failure error message.
FAILURE_IS_PROPAGATED: Whether the failure was propagated from a child.
FAILURE_OCCURRED_AT: Timestamp when failure occurred.
FAILURE_ROOT_CAUSE_ACTOR: Actor that caused the failure (root cause).
FAILURE_ROOT_CAUSE_NAME: Name of root cause actor.
FLIGHT_RECORDER: Flight recorder JSON (recent trace events).
INBOUND_ORDERING: Per-session reorder state from the sequenced receiver snapshot. sessions[*].buffered_count reports messages held by receiver-local sequencing waiting for a seq gap to fill. Independent diagnostic from queue_depth; no arithmetic contract – see IO-3.
INSTANCE_ID: Stable per-instance identifier (Uuid::now_v7) assigned at Instance::new.
IS_SYSTEM: Whether this actor is infrastructure/system.
LAST_HANDLER: Name of the last message handler invoked.
MESSAGES_PROCESSED: Number of messages processed by this actor.
STATUS: Actor lifecycle status: “running”, “stopped”, “failed”.
STATUS_REASON: Reason for stop/failure (absent when running).
TOTAL_PROCESSING_TIME_US: Total CPU time in message handlers (microseconds).

Functions§

format_timestamp: Format a SystemTime as an ISO 8601 timestamp with millisecond precision.
live_actor_payload: Build an IntrospectResult from live InstanceCell state.