Telemetry persistence#
Save telemetry to query a job after it stops. Use the same Monarch build for collection and offline reading; schemas are stored in the Arrow IPC data.
Configure your job#
Configure your existing job before calling state():
from monarch.job import TelemetryConfig
job.enable_telemetry(
TelemetryConfig(persist_dir="/tmp/monarch-telemetry")
)
state = job.state(cached_path=None)
print(f"Apply ID: {job.apply_id}")
Python settings#
|
Default |
Effect |
|---|---|---|
|
|
Persistence directory for collectors; |
|
|
Overrides the client collector’s directory; otherwise it uses |
|
|
Live in-memory retention only; it does not delete persisted data. |
Choose storage that survives worker allocation cleanup for remote jobs. If the
client cannot access the workers’ mount, set client_persist_dir to a writable
client directory. Files on a worker’s local /tmp must be copied before that
storage is discarded.
Rust writer settings#
These PersistConfig defaults are internal to the Rust writer. They are
not exposed as Python or CLI options; adjusting them requires a code change
and rebuilding Monarch.
Setting |
Default |
Effect |
|---|---|---|
|
|
Interval for merging pending batches and flushing file buffers. |
|
|
Interval for syncing written data to storage. |
|
|
File size that triggers rotation to the next segment. |
|
|
Queue capacity; additional sends can drop batches when full. |
|
|
Pending row count that triggers an immediate write for that table. |
Flush and sync intervals are scheduling intervals, not maximum-loss guarantees. Fixed format limits are 16 MiB per frame payload and 256 bytes per table name. Oversized batches are split by rows; a row that cannot fit is dropped.
Locate and query files#
Files are stored under:
PERSIST_DIR/APPLY_ID/HOSTNAME-PID/000000.frames
PID identifies the collector process. Pass the apply-ID directory containing
those collector directories to the CLI. Replace APPLY_ID below with the value
printed by your job:
MONARCH_LOG_ROOT="/tmp/monarch-telemetry/APPLY_ID"
monarch query --dir "$MONARCH_LOG_ROOT" 'SHOW TABLES'
monarch query --dir "$MONARCH_LOG_ROOT" \
'SELECT COUNT(*) AS spans FROM spans'
Repeat --dir to combine separate client and worker roots. Include each
collector’s files once to avoid duplicate rows. The offline reader loads the
saved data into a fresh store without live retention or worker connections.
Open the dashboard#
monarch dashboard open "$MONARCH_LOG_ROOT" --port 0
--port defaults to 8265; 0 selects an available port. Multiple roots can
be passed as positional paths. Open the printed URL and keep the command
running while viewing the dashboard. Ctrl+C stops it; the files can be reopened
later with the matching Monarch build.
Shutdown and recovery#
Use ordinary job shutdown, such as job.kill() in a finally block, and allow
cleanup to finish before exporting files or releasing storage. Collectors drain
pending batches, write an empty __end marker, and sync the files. Fixed close
wait limits are 5 s for the writer and 10 s for the sidecar’s client close RPC.
A timeout or storage error can leave the log unclean.
clean: readback recovered the collector’s final__endmarker.unclean: no final marker was recovered, including after an unexpected stop or while the collector is still running.torn: bytes skipped from the first incomplete or undecodable frame to the end of that segment. Earlier complete frames remain queryable.
A process can die between complete frames, leaving an unclean log with zero torn bytes. Queued, buffered, or unsynced data may be missing after a crash. Recovery requires the files to remain accessible. The clean flag describes collector log closure; capture is best effort and dropped-batch counts are not stored in the files.