Module pickle 

Pickling support for Monarch.

This module pickles Python objects for actor messages, collecting tensor engine references and out-of-band mesh references during serialization so they are restored together on decode.

Out-of-band mesh-reference invariants

Mesh references (proc/host/actor meshes) are not pickled inline. They ride out of band in a refs table carried next to the payload bytes, leaving a pop_mesh_reference sentinel in the pickle stream. Two invariants keep the table and its payload from ever drifting apart:

REFS-1 (ref-aware decode). Any payload that can carry out-of-band refs must be decoded only through a ref-aware path: PicklingState::from_parts and PicklingState::unpickle, as wrapped by PythonMessage::decode and PythonResponseMessage::decode. A bare pickle.loads/cloudpickle.loads on the payload bytes drops the table, so a sentinel later pops with no active state and raises “No active pickling state”. The raw payload bytes are deliberately not exposed on PythonMessage (there is no .message getter), so a bare decode is not even expressible from Python.

REFS-2 (refs preserved through intermediates). Any intermediate that relays a payload – PythonResponseMessage and the ValueOverlay behind a .call() valuemesh – must carry its refs beside the bytes, all the way to the decode site. Dropping refs at a relay boundary reintroduces the REFS-1 failure downstream. Refs are therefore part of PythonResponseMessage equality: two runs with identical bytes but different refs must not coalesce.

These invariants have one sound exception. A table entry and a pop_mesh_reference sentinel are emitted together during pickling (1:1), so an empty refs table means the payload carries no sentinels: a bare decode of it pops nothing and is sound. The .call() valuemesh collector (collect_valuemesh in endpoint.rs) relies on this to preserve D96180139’s lazy unpickle: it decodes a ref-empty batch lazily via PyValueMesh::build_from_parts (value_mesh.rs), which resolves on access outside the ref-aware path, and only a ref-carrying batch eagerly via build_from_objects. That lazy build is the sole bare decode of a payload that could carry refs; see collect_valuemesh for the gate itself.

Structs

PendingMessage

A message whose reserved mesh slots must be filled before it can be sent.

PicklingState

Python-visible wrapper for the result of a pickling operation.

PicklingStateInner

Inner data for a completed pickling operation.

Functions

pickle

Pickle an object with support for pending pickles and tensor engine references.

pickle_to_part

Pickle a Python object and return the serialized data as a Part.

push_mesh_reference_if_active

Push a mesh reference to the active pickling state if mesh-reference collection is active in this context.

reduce_shared

Reduce a PyShared for pickling.

register_python_bindings

Register the pickle Python bindings into the given module.

reserve_mesh_reference_if_active

Reserve a slot for a pending mesh whose *MeshRef is not available yet, registering handle for a sender-side fill once the mesh resolves.