Metrics#

Monarch records built-in counters, gauges, and histograms with OpenTelemetry (OTel). The same instruments can feed job-local distributed telemetry, an external OpenTelemetry Protocol (OTLP) endpoint, or both. Start with the observability overview if you are choosing among Monarch’s diagnostic surfaces.

Recording model#

One process-global OTel meter provider owns metric instruments and their readers. Libraries request a meter and record values without knowing which backends are active. Threads and libraries in the same Rust process share this provider. A different process, binary, or language runtime has its own provider and exports independently.

Each instrument has a name, scope, unit, and type. The instrumentation code that records a point also selects its attribute set. OTel aggregates points by their complete identity, including attributes, so high-cardinality attributes create more time series and more rows at each collection interval.

Independent periodic readers collect the same instruments for each enabled export path. OTEL_METRIC_EXPORT_INTERVAL controls their interval and defaults to one second. This provides low-latency visibility for short-lived jobs but produces more export traffic than typical monitoring intervals. Increase the interval for long-running jobs or high-cardinality instruments to reduce application and backend load. A slow or unavailable destination does not require different instrumentation for the other destinations.

Record custom metrics#

Use Monarch’s process-global meter to create OpenTelemetry instruments. The same instrument feeds every enabled export path:

from monarch.actor import get_meter

requests = get_meter().create_counter("example.requests")
requests.add(1, {"operation": "predict", "outcome": "success"})

Create instruments once and reuse them. Use attributes only for bounded dimensions such as operation and outcome; request IDs and other unbounded values create a new time series for each value.

Metric identity and context#

An exported metric point combines context from several levels. Distributed telemetry preserves those levels separately:

OTel concept	Set by	Meaning	Distributed telemetry representation
Resource attributes	Meter-provider initialization	Process-wide service and runtime identity shared by every metric, such as `service.name`	`resource_attributes_json`
Instrumentation scope	The library requesting a meter	The library or module that owns the instrument	`scope_name`
Instrument	The instrument builder	The metric name, unit, and aggregation type	`name`, `unit`, and the selected metric table
Point attributes	Each record operation	Dimensions for one measurement, such as operation or outcome	`attributes_json`

Point attributes are part of a metric series identity. Monarch’s built-in instrumentation selects these attributes. If instrumentation code records the same instrument with {"outcome": "success"} and {"outcome": "failure"}, OTel creates two independently aggregated series and therefore up to two rows per collection interval. Instrumentation authors should use attributes for bounded dimensions, not request IDs, timestamps, or other values that grow without limit.

Resource attributes describe the process rather than an individual measurement. Startup configuration selects supported values such as service.name before the process-global provider is built. These attributes are then copied into every distributed metric row and cannot change per measurement. Instrumentation scope is also separate from point attributes: it identifies the library that created the instrument and disambiguates same-named instruments from different libraries. When grouping or joining metric rows, treat resource attributes, scope, instrument metadata, and point attributes as the complete context.

Distributed telemetry currently stores only the instrumentation scope name. OTel scope version, schema URL, and scope attributes are not included in the metric tables.

Choose an export path#

Path	Enablement	Destination	Primary use
Distributed telemetry	`job.enable_telemetry(...)`	Job-local SQL tables	Debugging and analysis within the current job
OTLP	`OTEL_EXPORTER_OTLP_ENDPOINT`	OpenTelemetry Collector over OTLP/HTTP	External aggregation, monitoring, and visualization

You can enable both paths. They share instruments and resource attributes but collect and export independently.

Distributed telemetry path#

OTel instruments
    └─> periodic reader
          └─> Unix Domain Socket (UDS) metric exporter
                └─> host-local telemetry sidecar
                      └─> DataFusion SQL tables

Calling job.enable_telemetry(...) activates the process-global Unix-socket sink. On each collection, the UDS exporter encodes non-empty aggregation buffers as Arrow batches and sends them to the host-local telemetry sidecar. The sidecar exposes three physical tables:

Table	Aggregation
`metric_gauges`	Gauge points
`metric_sums`	Counter and up-down-counter points
`metric_histograms`	Explicit histogram points and buckets

The UDS exporter uses delta temporality, so each row describes one collection interval. Each unique metric name, scope, and attribute set produces its own row. Numeric values remain in separate f64, i64, and u64 columns rather than being converted to one common type.

This path is job-local, in memory, and best-effort. Empty aggregation buffers produce no frame. A full producer queue, serialization failure, oversized frame, or socket failure drops the affected frame rather than blocking the application. Activating the socket starts delivery for future intervals; it does not replay metrics collected while the sink was inactive. Metric tables follow TelemetryConfig.retention_secs.

See Distributed Telemetry for complete table contents and SQL examples.

OTLP path#

OTel instruments
    └─> periodic reader
          └─> OTLP metric exporter
                └─> OpenTelemetry Collector
                      └─> external metrics backend

In a build with OTLP support, setting OTEL_EXPORTER_OTLP_ENDPOINT enables an independent metric reader. It sends collections to the configured OpenTelemetry Collector over OTLP/HTTP. The collector controls routing, processing, and delivery to external metrics systems.

See OpenTelemetry and Grafana for a Kubernetes collector, Prometheus, and Grafana example.

Configuration#

Setting	Default	Effect
`ENABLE_OTEL_METRICS`	`true`	Enables Monarch OTel metric collection
`OTEL_METRIC_EXPORT_INTERVAL`	`1s`	Sets the periodic reader interval
`OTEL_EXPORTER_OTLP_ENDPOINT`	Unset	Enables OTLP metric export when configured
`OTEL_SERVICE_NAME`	`unknown_service`	Sets the service name resource attribute
`TelemetryConfig.retention_secs`	`3600`	Retains distributed metric rows for this many seconds; `0` disables retention