Skip to Content
Telemetry & cost

Telemetry & cost

Live

DDK answers two questions about every governed call: what happened? and who pays for it? It does so with OpenTelemetry GenAI spans, per-run correlation IDs, cost-attribution tags, and routing and resilience signals.

OpenTelemetry GenAI spans

Every governed call produces a span following the OpenTelemetry GenAI semantic conventions — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — plus attributes for the governance layer that generic instrumentation cannot know about:

donkey.policy.decision      = allow | refuse
donkey.policy.type          = pii_detected | token_budget | injection | …
donkey.budget.remaining     = 18450
donkey.correlation_id       = …
donkey.cost.team            = support
donkey.cost.project         = triage-v2
donkey.cost.env             = prod
donkey.cost.enduser.id      = user-42
donkey.routing.type         = ModelBased          # how the gateway routed
donkey.routing.fallback     = false               # did it fail over?
gen_ai.response.model       = gpt-5.1             # the model that actually served
donkey.cache.status         = hit                 # semantic-cache outcome (on a cached proxy)
donkey.cache.score          = 0.9518              # similarity score, hit only
donkey.usage.cached_tokens        = 512   # omitted when the provider reports none
donkey.usage.cache_write_tokens   = 128
donkey.usage.reasoning_tokens     = 96

The three donkey.usage.* counts carry the cost-relevant detail tokens the semantic conventions have no pinned key for — cached / cache-write prompt tokens and reasoning-model thinking tokens. They are read from the response usage block’s detail sub-objects and are omitted, never 0, when the provider reports no detail counts. When the response passes through the SDK’s shared HTTP client, the same counts are exposed per-call on donkey.last_call.

A refused request still produces a span, with donkey.policy.decision=refuse and otel.status_code=ERROR. A streaming response produces exactly one span, with token counts filled in at stream end.

Export goes over OTLP to wherever you already send spans. Nothing in the emit path is Anypoint-specific, so if your team already runs Grafana, Elastic, Honeycomb, New Relic, Datadog, Dynatrace, Langfuse or Phoenix, “policy refusals per hour by type” and “tokens per ticket” show up in the dashboard you already have. See Send spans to your observability backend.

Zero-config export

Set the standard OpenTelemetry endpoint env var and spans flow — there is no SDK-specific variable.

pip install "donkey-kit[otel]"
export OTEL_EXPORTER_OTLP_ENDPOINT=https://langfuse.acme.internal
export OTEL_SERVICE_NAME=support-triage       # standard OTel var, honoured for free
python -m my_app                              # spans flow, refused calls included

Donkey.from_env() reads OTEL_EXPORTER_OTLP_ENDPOINT (or the traces-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) and, when one is set, installs an OTLP exporter behind a batch processor. The network flush runs on that background thread, off your request path, keeping instrumentation overhead under 1 ms per call. The [otel] extra ships the http/protobuf exporter; OTEL_EXPORTER_OTLP_PROTOCOL=grpc is honoured only if you also install opentelemetry-exporter-otlp-proto-grpc.

With no endpoint set, the export path is inert and silent — no exporter is built, nothing connects, nothing is printed. If your process already configures its own OpenTelemetry provider (say via opentelemetry-instrument), DDK rides it rather than replacing it, so your spans flow through the pipeline you already set up.

Opt out of telemetry entirely with a single flag:

export DONKEY_TELEMETRY=false      # or telemetry = false in .donkey-kit.toml

Send spans to your observability backend

Any backend that accepts OTLP works with the same three standard variables: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS for the vendor’s credentials, and OTEL_SERVICE_NAME. There is no DDK-specific setting and no code change between backends. Pick yours:

Grafana Cloud stores traces in Tempo. In the Grafana Cloud portal, open your stack and choose Configure on the OpenTelemetry tile. It generates the endpoint for your region and a token, already base64-encoded as instanceID:token.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<base64 instanceID:token>"
export OTEL_SERVICE_NAME="support-triage"

Grafana’s own note for Python: write the space after Basic as %20. Find the spans in Explore → Tempo or in Application Observability. For production, Grafana recommends sending through Grafana Alloy (its OpenTelemetry Collector) rather than straight from the app.

Docs: Send data to the Grafana Cloud OTLP endpoint 

A few rules hold for every backend:

  • Base endpoint vs traces endpoint. OTEL_EXPORTER_OTLP_ENDPOINT is a base URL: the exporter appends /v1/traces. If you set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead, give the full URL including /v1/traces, or the backend answers 404.
  • Encode spaces in headers. Header values such as Basic <token> or ApiKey <key> should be written with %20 in place of the space; separate several headers with commas.
  • Protocol. DDK exports http/protobuf, which every backend above accepts. For gRPC, set OTEL_EXPORTER_OTLP_PROTOCOL=grpc and install opentelemetry-exporter-otlp-proto-grpc.
  • In production, consider a Collector. Sending to an OpenTelemetry Collector  (or a vendor distribution such as Grafana Alloy) lets you batch, retry, redact and fan out to several backends without touching the app.
  • Prompts stay private by default. None of these backends receives prompt or completion text unless you opt in; see Message content stays off spans by default.

Semantic-convention stability and sinks

The GenAI conventions are still Development status upstream, so attribute names can change. DDK pins the semconv version and dual-emits: gen_ai.* at the pinned version, plus a stable donkey.* namespace under this project’s control. Your dashboards do not break when upstream renames something.

DDK exports standard OTLP and leaves the sink to you. Whether Anypoint Monitoring or Agent Visualizer ingests OTLP GenAI spans is not publicly documented, so don’t rely on spans appearing there.

Message content stays off spans by default

Spans carry metadata only — model, token counts, policy decision, cost tags, correlation id. They do not carry prompt or completion text.

Spans are emitted upstream of the gateway’s PII masking. The Omni Gateway masks sensitive content in its logs; a DDK span is created inside your process, before the request reaches the gateway. Putting message text on the span would re-export the very content the platform masks — straight to whatever OTLP collector you have wired up. So capturing content is opt-in, and opting in is you assuming that obligation.

Turn it on only when your collector is a trusted sink and you have accepted responsibility for the content that lands there:

# kwarg, or DONKEY_TELEMETRY_CAPTURE_CONTENT=true, or
# telemetry_capture_content = true in .donkey-kit.toml
donkey = Donkey.from_env(telemetry_capture_content=True)

telemetry_capture_content resolves along the standard precedence (kwarg → env → .donkey-kit.toml → default) and defaults to False. When enabled, content is emitted under the pinned semconv attribute names — gen_ai.prompt and gen_ai.completion — and no others. When off, those attributes never reach a span, and the allowlist that builds every span drops any content-shaped attribute a call site hands it, so there is no accidental path for message text to leak.

Routing & resilience

The gateway can fail over between providers when one degrades (“Enhanced Resilience for Intelligent Routing”). It reports what it did with each request on the response — which provider and model served it, how it routed, and whether that was a fallback. DDK reads those signals off the shared transport, so you get them with no framework required — the raw donkey.llm.client() path benefits just as the deep adapters do.

Every governed call through the SDK’s shared HTTP client exposes them on donkey.last_call, beside the usage and identity fields:

donkey = Donkey.from_env()
await donkey.openai().responses.create(model="gpt-5.1", input="…")
 
r = donkey.last_call
r.requested_model    # "gpt-5.1"  — what you asked for
r.served_model       # "gpt-5.1"  — or a substitute after failover
r.served_provider    # "openai"
r.routing_type       # "ModelBased" — or "Semantic" on a semantic-routing proxy
r.fallback           # False      — True if the gateway failed over
r.substituted        # False      — served_model != requested_model
r.matched_topic      # None       — the topic a Semantic proxy matched (else None)
r.routing_score      # None       — that match's similarity score (else None)

On a semantic-routing proxy (routing_type == "Semantic"), the gateway also reports why it picked a provider: the topic your prompt matched and the similarity score behind that match. Those land on r.matched_topic / r.routing_score. On a model-based proxy both are None — the gateway emits no semantic header, and DDK never fabricates a value for a signal it did not observe.

They also land on the span (gen_ai.response.model, donkey.routing.type, donkey.routing.fallback, and — on a semantic route — donkey.routing.matched_topic / donkey.routing.score) — the most useful thing to have on hand when latency spikes: it tells an operator whether a slow call was routed normally or recovered from a degraded provider.

For the complete list of last_call fields — observability status, gateway identity, routing, and token usage — see the last_call field reference.

The same record also carries the call’s identity and usage: request_id (the upstream provider’s id), api_instance_id and environment_id (which gateway instance served it), and the token counts including cached and reasoning tokens. The gateway identity example walks through every field.

When last_call is unavailable

donkey.last_call is populated only when the governed response passes through the SDK’s shared httpx client. Four connection_kwargs()-only adapters route outside that response path: ADK and CrewAI send requests through LiteLLM, while LlamaIndex and Microsoft Agent Framework receive only default_headers. That static snapshot excludes the correlation ID bound later by donkey.run(id=...), so those two adapters also do not propagate the run’s correlation ID.

When every adapter resolved on a Donkey is one of those four, a cold read reports the limitation explicitly. For a Donkey that resolved only ADK:

r = donkey.last_call
r.status       # LastCallStatus.UNAVAILABLE
r.available    # False
r.surface      # "adk"

This is different from UNOBSERVED, which means the current context has not yet received a governed response. On an unavailable surface the SDK cannot observe any response-derived last_call field, including gateway identity, routing, fallback, and usage. If multiple non-observing adapters were resolved, surface lists their names.

Two behaviours worth knowing

The SDK never double-retries a fallback. DDK retries 502/503/504 with backoff, but if the gateway already failed over internally, a 503 it marked as a fallback is not retried again — a second recovery layer stacked on a working first one just multiplies latency against an outage the gateway already handled.

Opt in to model determinism. A silent substitution is surfaced passively on last_call.substituted by default. When a substitution is not acceptable — your evaluation, cost model and token assumptions are all pinned to one model — opt into a hard error:

donkey = Donkey.from_env(on_model_substitution="raise")
# raises ModelSubstituted when the served model differs from the requested one

on_model_substitution resolves along the standard precedence (kwarg → env DONKEY_ON_MODEL_SUBSTITUTION → .donkey-kit.toml → default) and defaults to "off".

On a model-based routing proxy you address models as provider/model (for example openai/gpt-5-mini), and the gateway reports the served model without the prefix (gpt-5-mini), with the provider in its own header. A prefix that names the served provider is not counted as a difference, so asking for openai/gpt-5-mini and being served gpt-5-mini by openai is not a substitution. The same model served by a different provider still is.

Semantic caching & semantic routing Live

Omni Gateway can answer a request from its semantic cache (a similar prompt was answered before, so no provider round-trip and no token cost) and can route semantically (pick the model by matching the prompt to a topic). Both happen at the gateway. DDK caches nothing and computes no embeddings itself; it lets you steer the gateway’s cache per request — skip it, not store a response, or override the TTL or similarity threshold via donkey.cache(...) — and reports what happened:

  • the cache outcome (hit, miss, bypass, no-store) and similarity score on donkey.last_call.cache_status / .cache_score and the span (donkey.cache.status / donkey.cache.score);
  • the matched routing topic and its score, beside the routing fields above.

A cache hit is a verbatim replay with no provider round-trip, so it never advances the token budget — the replayed usage is not fresh spend.

TypeScript parity Roadmap

The same signals will surface as donkey.lastRouting.servedModel / .fallback in the TypeScript SDK.

Correlation IDs

Set a per-run id once, and every call inside the block carries it — on the wire, on every span, and on every exception — with nothing threaded through your framework state.

async with donkey.run(id=ticket.id):
    await triage_agent.run(ticket)

donkey.run(id=…) binds two ids:

  • a run id → the X-Correlation-Id request header → the span’s donkey.correlation_id → DonkeyError.correlation_id. Shared by every call in the block, so a client-side log line joins to the gateway’s own record.
  • a fresh per-call id → the X-Donkey-Request-Id request header → DonkeyError.call_id. Unique per logical request, stable across that request’s retries, so one call is pinpointable within a run.

The gateway reads the inbound X-Correlation-Id and echoes it verbatim on the response, so the request and response x-correlation-id are the same value. X-Donkey-Request-Id is a client-owned per-call id the gateway does not consume. Both header names are overridable (correlation_header / call_id_header) for a gateway that expects different ones.

Propagation is contextvar-based, so it reaches through framework nodes (every LangGraph node, for instance) without threading an argument through every function, and concurrent runs never leak into each other. It works with or without OpenTelemetry installed. donkey.run(...) is a dual sync/async context manager (plain with works too); nested blocks rebind then restore.

The decorator form

When a whole function should be one run, @donkey.governed is the decorator equivalent of wrapping its body in donkey.run():

@donkey.governed(team="support")
async def handle_ticket(ticket):
    await triage_agent.run(ticket)

Each call opens its own run — a fresh run/correlation id — and binds the optional per-run cost tags, the OTel span, and typed refusals, exactly the scope donkey.run() establishes. It wraps both sync and async callables and is usable bare (@donkey.governed) or parametrised. There is deliberately no id=: pinning one id across every call would collapse unrelated runs into a single correlation, so when you need a specific id, use donkey.run(id=…) directly.

Cost-attribution tags

A small, fixed set of tags — team, project, env, enduser.id — set once and emitted on every call, both as request headers and as donkey.cost.* span attributes:

donkey = Donkey.from_env(team="support", project="triage-v2", env="prod")
 
async with donkey.run(id=ticket.id, enduser_id=agent_user.id):
    await triage_agent.run(ticket)

The tags resolve along the standard precedence — Donkey.from_env(team=…) kwargs, then DONKEY_COST_* env vars, then a [donkey.cost] table in .donkey-kit.toml. Per-run overrides layer on top: donkey.run(team=…, project=…, env=…, enduser_id=…) wins per field for its block and the rest fall back to the configured tags. The key set is fixed — an unknown dimension is a configuration error, never a silently-dropped header. Values are validated — fixed keys, bounded length — so nobody stuffs a JSON blob into a header.

The Anypoint LLM Gateway does not ingest cost tags from request headers — it meters cost from token usage per API instance and consuming client application. The authoritative carrier is the donkey.cost.* OTel span attribute. The X-Anypoint-Cost-* request headers are a convention nothing currently reads; their names are overridable (cost_*_header) for a gateway that does read one.

The question this answers

Finance asks what the support agent cost last month versus the HR bot. Without tags, both agents share one client_id and there is no way to split the bill. With tags it is a group-by.

For compliance — “prove the HR bot’s answer to user X on date Y went through the content-safety policy” — the correlation ID on the log line joins to the gateway record, and the span carries enduser.id and donkey.policy.type. An EU AI Act Article 12 log request becomes one query, not an investigation.

Last updated on
Policy handshakeLocal simulator