OpenTelemetry#
Monarch can export its metrics, logs, and spans over OTLP/HTTP to an OpenTelemetry Collector or any backend that accepts OTLP. Built-in telemetry is exported with no code changes. OTLP export is independent of job-local distributed telemetry, and the two can run on the same job.
What Monarch exports#
Signal |
Contents |
Notes |
|---|---|---|
Metrics |
Built-in actor, mailbox, and channel metrics, plus instruments created with |
Cumulative temporality, pushed every |
Logs |
Events recorded with Rust |
A log emitted inside a span carries that span’s trace and span IDs |
Spans |
Rust |
Python spans are named after the name passed to |
Some noisy internal targets are filtered out of logs and spans.
Enable export#
Set OTEL_EXPORTER_OTLP_ENDPOINT to the collector’s base URL in the
environment of the client and of every worker host:
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
Monarch appends the standard signal paths and sends OTLP/HTTP with protobuf
payloads to /v1/metrics, /v1/logs, and /v1/traces. Export is off when the
variable is unset. The collector must define a pipeline for each signal it
receives; see the example collector configuration in
OpenTelemetry and Grafana.
Configuration#
Set these values before telemetry initializes in each client and worker process. Later changes do not reconfigure existing exporters, metric readers, or tracing sinks.
Setting |
Default |
Effect |
|---|---|---|
|
Unset |
Collector base URL; enables metric, log, and span export |
|
Unset |
Extra request headers, such as authentication tokens |
|
Unset |
Timeout for each export request |
|
Unset |
Extra resource attributes, as comma-separated |
|
|
How often metrics are collected and pushed |
|
|
Set to |
|
|
Set to |
Resource attributes#
Every exported metric, log, and span carries these resource attributes:
Attribute |
Value |
|---|---|
|
|
|
ID shared by the processes of one Monarch run |
|
User running the process |
|
Host name |
|
Execution environment detected by Monarch, such as |
|
Client trace ID, when one is set |
Attributes from OTEL_RESOURCE_ATTRIBUTES are added to these. They do not
replace the attributes above, and OTEL_SERVICE_NAME does not change
service.name.
Delivery and limitations#
Export is best-effort. A slow or unreachable collector drops telemetry rather than failing the job.
Logs and spans are batched and sent in the background. The tracing dispatcher flushes its sinks periodically and drains queued events and flushes again when it shuts down. Process exit does not guarantee that dispatcher shutdown runs; abrupt termination such as
SIGABRTbypasses it. A process that exits immediately after emitting telemetry can lose its final batch.A span is exported when it ends. A span that never closes is not exported.
Python log records do not carry the trace context of an enclosing Python
span().