Skip to main content

Overview

Stream production conversations straight from your agent using the standard OpenTelemetry SDK, with no UserTrace SDK and no transcript uploads. Point an OTLP exporter at /v1/traces, add your API key header, and follow the OTel GenAI semantic conventions. Conversations are evaluated automatically as they close, and the results show up in Production Results.
If you already run OpenTelemetry, you need two things: a second exporter pointed at UserTrace, and gen_ai.conversation.id on your spans. Everything else is convention you may already emit.

Before you start

Choose which evals run on ingested conversations. In the dashboard, open the Agent Setup page and scroll to the bottom, to the Live traces section. Until you do, conversations ingest and close cleanly but nothing is scored. Two settings there:

Sending spans

Accepts application/json and application/x-protobuf, both of which stock OTel exporters send unmodified, and Content-Encoding: gzip. Returns 200 with an empty OTLP ExportTraceServiceResponse, which is what exporters expect to parse. Any other content type is 415.
In OTLP/JSON, traceId and spanId are hex strings, a deliberate deviation from the usual protobuf-to-JSON mapping, which would base64 them. Stock exporters get this right; only hand-built payloads need care.

What we read

Span ids and timestamps come free from OTLP; you never set them yourself.
Spans without gen_ai.operation.name are ignored. You can point an existing TracerProvider at UserTrace and your HTTP, database and framework spans pass through harmlessly.

Your own metadata

Any attribute outside the gen_ai.* namespace, such as journey.id, tenant, region or service.name, is captured as a searchable tag on the conversation. There’s no UserTrace-specific convention here; they’re ordinary OTel attributes, on the span or on the resource.

Retries are safe

Rows are keyed on the span id, so re-sending a batch inserts nothing. This matters because OTLP exporters retry on their own. At-least-once delivery is the norm, not an edge case.

Closing a conversation

Closing is what queues evaluation.
Make this a plain HTTP call, not a span. It must not depend on the trace pipeline having flushed.
Results appear under GET /api/prod-evaluations/sessions/?run=Live traces once they finish, typically within seconds. Conversations are grouped into a daily run, so a day’s traffic is one row in the dashboard rather than thousands.
A conversation left idle for 24 hours is closed automatically, through the same path so a crashed client doesn’t strand its data. Calling /end yourself just makes results arrive sooner.

Recovering unscored conversations

Conversations that arrive before any evals are configured close cleanly but are never scored, and because the dashboard renders sessions through their run, they’re invisible until they are. The Live traces section of the Agent Setup page shows the count and offers to score them. This is a dashboard action, not an API one. Scoring uses your saved eval configuration, so the backlog is scored with the eval set you’ve committed to and nothing else. With none configured, the button tells you to pick evals first. Each conversation joins the run for its own date, so a backlog can produce several. A backlog spanning a week produces one run per day rather than one run mislabelled as today’s.

Errors