# Supported Agents
Source: https://docs.getusertrace.com/agent-support
Complete overview of AI agent types and architectures supported by UserTrace
## Agent Types
UserTrace supports comprehensive testing for all major AI agent architectures and deployment models.
### Voice Agents
**Phone System Integration**
• Traditional phone number systems
• Inbound and outbound call handling
• PSTN and VoIP compatibility
• Multi-line concurrent support
**Real-Time Voice Communication**
• WebRTC integration
• In-app voice features
• Browser-based calling
• Mobile app voice systems
### Conversational Agents
**Written Conversation Systems**
• Customer support chatbots
• Virtual assistants
• WhatsApp Business agents
• Slack workspace bots
• Web chat widgets
• Messaging platform bots
**Direct LLM Testing**
• System prompt evaluation
• Single-turn responses
• Prompt optimization
• A/B prompt testing
## Agent Architectures
### RAG Agents
**Retrieval-Augmented Generation Systems**
• Knowledge base integration
• Document retrieval testing
• Context accuracy validation
• Source attribution verification
### Tool-Using Agents
**Function Calling & API Integration**
• External API interactions
• Database query execution
• Third-party service integration
• Multi-step workflow validation
### Multi-Modal Agents
**Cross-Format Communication**
• Voice + text combinations
• Image understanding capabilities
• Document processing agents
• Video analysis systems
## Framework Compatibility
### Standard Frameworks
**Supported Providers:**
**Vapi** for full conversation flow testing, **Retell** for real-time voice evaluation, **Livekit** for WebRTC and telephony testing, **Pipecat** for pipeline-based voice agents, and **Custom SIP** for direct SIP integration.
**Supported Integrations:**
**OpenAI** for GPT models with function calling, **Anthropic** for Claude with tool use, **LangChain** for multi-agent workflows, **LlamaIndex** for RAG and knowledge systems, and **Custom APIs** for any Chat Completions format.
### Supported Agents
**Telephony & Web-based Voice**
• Phone system integration
• WebRTC real-time communication
• VoIP and PSTN compatibility
**Text-based Conversations**
• Customer support chatbots
• WhatsApp Business agents
• Slack workspace bots
• Web chat widgets
**Direct LLM Testing**
• System prompt evaluation
• Single-turn responses
• A/B prompt testing
**Bring Your Own Agent**
• Chat completions API compatible
• Sandbox environment testing
• Proprietary model integration
## Getting Started
Choose your integration path based on your agent type:
Test prompts directly with LLM models
Test custom chat agents through API endpoints
Test WhatsApp Business agents
Test Slack workspace bots
**Not sure which category fits your agent?** Contact our support team at [support@getusertrace.com](mailto:support@getusertrace.com) for personalized integration guidance.
# Authentication
Source: https://docs.getusertrace.com/api-reference/authentication
API keys for programmatic access to the UserTrace API
## Overview
The UserTrace API uses long-lived **API keys** for programmatic access from a script, a
CI job, or your agent's production runtime, where a short-lived dashboard session is
impractical.
A key grants **exactly its owner's access**: the same organization, the same scoped
environments. Creating a key never widens what someone can see.
## Base URL
```
https://api.getusertrace.com
```
## Creating a key
Keys are created in the dashboard.
The **API key** panel is the last section on the page.
Give it a name that says where it runs, such as `CI reporting` or `prod-agent`, so
you know what you're revoking later.
The full key is shown **exactly once**. Only a hash is stored, so it cannot be
recovered afterwards.
If you lose a key, you cannot look it up. Revoke it and create another.
The panel lists every key with its prefix, when it was created, when it was last used, and
its status. **Revoke** takes effect on the very next request.
**Maximum 10 active keys.** Revoked keys stay in the list as history and don't count
toward the limit.
**"Last used" lags by up to an hour.** It's written at most once per hour to keep the
write off the request path, and stays empty until the key's first use, so a key you
just used may still show `—`.
## Using a key
Pass it in the `x-usertrace-api-key` header. No `Authorization` header is needed.
```bash theme={null}
curl -X GET "https://api.getusertrace.com/api/simulations/sessions/?limit=20" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json"
```
## What a key can access
Keys work only on endpoints that explicitly opt in:
| Endpoint | Purpose |
| ----------------------------------------- | ---------------------------------------------------------------------------------- |
| `GET /api/simulations/sessions/` | [Simulation results](/api-reference/endpoint/simulation-sessions) |
| `GET /api/prod-evaluations/sessions/` | [Production results](/api-reference/endpoint/production-sessions) |
| `POST /v1/traces` | [Live trace ingestion](/api-reference/endpoint/live-traces) |
| `POST /v1/sessions/{conversation_id}/end` | [Close a conversation](/api-reference/endpoint/live-traces#closing-a-conversation) |
Every other endpoint rejects a key with `401`, including the API-key management
endpoints themselves, so a leaked key cannot mint more keys.
## Rate limits
**120 requests per minute, per key.** Exceeding it returns `429` with a `Retry-After`
header giving the seconds until the window resets:
```json theme={null}
{ "message": "API key rate limit exceeded" }
```
The window is fixed rather than sliding, so a burst spanning a minute boundary can briefly
reach twice the limit. Dashboard-authenticated requests are not rate limited.
## Errors
| Response | Meaning |
| -------------------------------------------------- | ---------------------------------------------------------- |
| `401 {"message": "Invalid API key"}` | Unknown, revoked, or the owner's access no longer resolves |
| `401 {"message": "Authorization header required"}` | The `x-usertrace-api-key` header is missing |
| `401` on any other endpoint | That endpoint is not one of the four above |
| `429 {"message": "API key rate limit exceeded"}` | 120/min per key exceeded; see `Retry-After` |
# Live Traces (OTLP)
Source: https://docs.getusertrace.com/api-reference/endpoint/live-traces
Send production conversations with a standard OpenTelemetry exporter
## Overview
Stream production conversations straight from your agent using the **standard
OpenTelemetry SDK**, with no UserTrace SDK and no transcript uploads. Point an OTLP exporter at
`/v1/traces`, add your API key header, and follow the
[OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
Conversations are evaluated automatically as they close, and the results show up in
[Production Results](/api-reference/endpoint/production-sessions).
If you already run OpenTelemetry, you need two things: a second exporter pointed at
UserTrace, and `gen_ai.conversation.id` on your spans. Everything else is convention you
may already emit.
## Before you start
Choose which evals run on ingested conversations. In the dashboard, open the **Agent
Setup** page and scroll to the bottom, to the **Live traces** section. Until you do,
conversations ingest and close cleanly but nothing is scored.
Two settings there:
| Setting | Meaning |
| ------------ | -------------------------------------------------------------------------------------------------------------------- |
| **Evals** | Which evals score every ingested conversation. **At most 5**, since each one is a model call on *every* conversation |
| **Run name** | Base name for the daily run conversations are grouped under. Defaults to `Live traces` |
## Sending spans
```
POST /v1/traces
```
```http theme={null}
x-usertrace-api-key: ut_live_YOUR_KEY_HERE
Content-Type: application/json
```
Accepts `application/json` and `application/x-protobuf`, both of which stock OTel
exporters send unmodified, and `Content-Encoding: gzip`. Returns `200` with an empty OTLP
`ExportTraceServiceResponse`, which is what exporters expect to parse. Any other content
type is `415`.
```python Python theme={null}
# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
import json, uuid
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider(resource=Resource.create({
"service.name": "support-agent",
"deployment.environment.name": "production",
}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
endpoint="https://api.getusertrace.com/v1/traces",
headers={"x-usertrace-api-key": "ut_live_YOUR_KEY_HERE"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")
conversation_id = f"conv_{uuid.uuid4().hex[:8]}"
history = []
def record_turn(user_text, assistant_text, usage):
"""One turn = one span. Start it before the LLM call, end it after."""
history.append({"role": "user", "parts": [{"type": "text", "content": user_text}]})
with tracer.start_as_current_span("chat gpt-4o") as span:
span.set_attribute("gen_ai.conversation.id", conversation_id)
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.request.model", "gpt-4o")
span.set_attribute("gen_ai.response.model", usage["model"])
span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"])
span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"])
span.set_attribute("gen_ai.input.messages", json.dumps(history))
answer = {"role": "assistant",
"parts": [{"type": "text", "content": assistant_text}],
"finish_reason": "stop"}
span.set_attribute("gen_ai.output.messages", json.dumps([answer]))
history.append(answer)
```
```typescript Node theme={null}
// npm i @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http \
// @opentelemetry/api @opentelemetry/resources
import { trace } from '@opentelemetry/api';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';
const sdk = new NodeSDK({
resource: resourceFromAttributes({
'service.name': 'support-agent',
'deployment.environment.name': 'production',
}),
traceExporter: new OTLPTraceExporter({
url: 'https://api.getusertrace.com/v1/traces',
headers: { 'x-usertrace-api-key': 'ut_live_YOUR_KEY_HERE' },
}),
});
sdk.start();
const tracer = trace.getTracer('support-agent');
const conversationId = `conv_${crypto.randomUUID().slice(0, 8)}`;
const history: unknown[] = [];
// One turn = one span. Start it before the LLM call, end it after.
export async function recordTurn(userText: string, call: () => Promise<{
text: string; model: string; inputTokens: number; outputTokens: number;
}>) {
history.push({ role: 'user', parts: [{ type: 'text', content: userText }] });
return tracer.startActiveSpan('chat gpt-4o', async (span) => {
span.setAttributes({
'gen_ai.conversation.id': conversationId,
'gen_ai.operation.name': 'chat',
'gen_ai.request.model': 'gpt-4o',
'gen_ai.input.messages': JSON.stringify(history),
});
const res = await call();
const answer = {
role: 'assistant',
parts: [{ type: 'text', content: res.text }],
finish_reason: 'stop',
};
history.push(answer);
span.setAttributes({
'gen_ai.response.model': res.model,
'gen_ai.usage.input_tokens': res.inputTokens,
'gen_ai.usage.output_tokens': res.outputTokens,
'gen_ai.output.messages': JSON.stringify([answer]),
});
span.end();
return res;
});
}
```
```bash curl theme={null}
curl -X POST "https://api.getusertrace.com/v1/traces" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{
"resourceSpans": [{
"resource": {"attributes": [
{"key": "service.name", "value": {"stringValue": "support-agent"}}
]},
"scopeSpans": [{
"spans": [{
"traceId": "4b1e9a7c3d2f48619ae05c7b1f83d240",
"spanId": "a1b2c3d4e5f60718",
"name": "chat gpt-4o",
"startTimeUnixNano": "1789034400000000000",
"endTimeUnixNano": "1789034402150000000",
"attributes": [
{"key": "gen_ai.conversation.id", "value": {"stringValue": "conv_8f14c2"}},
{"key": "gen_ai.operation.name", "value": {"stringValue": "chat"}},
{"key": "gen_ai.input.messages", "value": {"stringValue":
"[{\"role\":\"user\",\"parts\":[{\"type\":\"text\",\"content\":\"Where is my order A-4471?\"}]}]"}},
{"key": "gen_ai.output.messages", "value": {"stringValue":
"[{\"role\":\"assistant\",\"parts\":[{\"type\":\"text\",\"content\":\"It shipped on 8 Sep.\"}],\"finish_reason\":\"stop\"}]"}}
]
}]
}]
}]
}'
```
In OTLP/JSON, `traceId` and `spanId` are **hex strings**, a deliberate deviation from
the usual protobuf-to-JSON mapping, which would base64 them. Stock exporters get this
right; only hand-built payloads need care.
## What we read
| Attribute | Purpose |
| ------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `gen_ai.conversation.id` | Groups turns into one conversation. **Required**, or send the `X-Conversation-Id` header |
| `gen_ai.operation.name` | `chat` = a turn, `execute_tool` = a tool call, `invoke_agent` = a turn wrapper. **Required** |
| `gen_ai.input.messages` / `gen_ai.output.messages` | The turn's content |
| `gen_ai.request.model` / `gen_ai.response.model` | Model attribution |
| `gen_ai.usage.input_tokens` / `output_tokens` | Token counts |
| `gen_ai.tool.name` / `.call.arguments` / `.call.result` | Tool call detail |
| span `status` | A non-OK status marks the turn failed |
Span ids and timestamps come free from OTLP; you never set them yourself.
**Spans without `gen_ai.operation.name` are ignored.** You can point an existing
TracerProvider at UserTrace and your HTTP, database and framework spans pass through
harmlessly.
## Your own metadata
Any attribute outside the `gen_ai.*` namespace, such as `journey.id`, `tenant`,
`region` or `service.name`, is captured as a tag on the conversation. There's no
UserTrace-specific convention here; they're ordinary OTel attributes, on the span or on
the resource.
Tags come back as `tags` on each session in
[Production Results](/api-reference/endpoint/production-sessions), and you can filter by
them:
```
GET /api/prod-evaluations/sessions/?tag=tenant:acme&tag=journey.id:returns
```
| Rule | Detail |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Where to set them | On the resource for values fixed per process (`service.name`, `region`); on the span for per-conversation values (`tenant`, `journey.id`). Any span in the conversation may carry them |
| Same key twice in one batch | The resource attribute wins over a span attribute; otherwise the first span wins |
| Across batches | A later batch overwrites the value, so the latest one sent is kept |
| Limits | 50 tags per request, including resource attributes your SDK adds on its own such as `telemetry.sdk.language`. String values over 512 characters are truncated |
## Retries are safe
Rows are keyed on the span id, so re-sending a batch inserts nothing. This matters because
OTLP exporters retry on their own. At-least-once delivery is the norm, not an edge case.
## Closing a conversation
```
POST /v1/sessions/{conversation_id}/end
```
Closing is what queues evaluation.
```bash theme={null}
curl -X POST "https://api.getusertrace.com/v1/sessions/conv_8f14c2/end" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE"
```
```json theme={null}
{
"conversation_id": "conv_8f14c2",
"status": "evaluating",
"run": "Live traces 2026-09-15",
"evaluations_queued": 2
}
```
Make this a plain HTTP call, not a span, so closing doesn't depend on the trace
pipeline.
**Flush your spans before calling `/end`.** Exporters send spans in batches, every few
seconds by default, and `/end` returns `404` for a conversation UserTrace hasn't
received yet. Call `provider.force_flush()` in Python or `await provider.forceFlush()`
in Node first.
Results appear under
`GET /api/prod-evaluations/sessions/?run=Live traces` once they finish, typically within
seconds. Conversations are grouped into a **daily** run, so a day's traffic is one row in
the dashboard rather than thousands.
| Behaviour | Detail |
| -------------------- | -------------------------------------------------------------------------------------------------------------------- |
| Idempotent | A second call returns `"already_closed": true` and queues nothing |
| Unknown conversation | `404` |
| No evals configured | Closes cleanly with `evaluations_queued: 0`; traces are kept |
| Deleted eval | Skipped and logged, never queued. The conversation would otherwise wait forever for an outcome that can never arrive |
**A conversation left idle for 24 hours is closed automatically**, through the same path
so a crashed client doesn't strand its data. Calling `/end` yourself just makes
results arrive sooner.
## Recovering unscored conversations
Conversations that arrive before any evals are configured close cleanly but are never
scored, and because the dashboard renders sessions through their run, they're invisible
until they are. The **Live traces** section of the **Agent Setup** page shows the count
and offers to score them.
This is a dashboard action, not an API one.
Scoring uses your **saved** eval configuration, so the backlog is scored with the eval set
you've committed to and nothing else. With none configured, the button tells you to pick
evals first.
Each conversation joins the run for **its own** date, so a backlog can produce several. A
backlog spanning a week produces one run per day rather than one run mislabelled as
today's.
## Errors
| Response | Meaning |
| ------------------------ | ---------------------------------------------------------------------------- |
| `400 {"message": "..."}` | No conversation id. Set `gen_ai.conversation.id` or send `X-Conversation-Id` |
| `413` | Payload over 10 MB. Lower your exporter's batch size |
| `415` | Content type is neither `application/json` nor `application/x-protobuf` |
| `429` | 120 requests per minute per key exceeded; see `Retry-After` |
# Production Results
Source: https://docs.getusertrace.com/api-reference/endpoint/production-sessions
Fetch production evaluation sessions and their eval outcomes
## Overview
The production counterpart of [Simulation Results](/api-reference/endpoint/simulation-sessions).
Returns production evaluation sessions with their eval outcomes, newest first, whether
those conversations arrived as uploaded transcripts or as
[live traces](/api-reference/endpoint/live-traces).
No evaluation id required.
## Endpoint
```
GET /api/prod-evaluations/sessions/
```
## Authentication
```http theme={null}
x-usertrace-api-key: ut_live_YOUR_KEY_HERE
```
See [Authentication](/api-reference/authentication).
## Query parameters
All filters are optional.
Substring match on the **evaluation run's** name, e.g. `Compliance V3`.
Case-insensitive.
Keep only sessions scored by the named **eval**, e.g. `Tone check`. Substring match,
case-insensitive.
Return only sessions with at least one failing eval, showing only the failing outcomes.
Keep only sessions carrying this tag, written `key:value`, e.g. `tenant:acme` or
`journey.id:returns`. Tags are the attributes you send with
[live traces](/api-reference/endpoint/live-traces#your-own-metadata). Exact,
case-sensitive match. Repeat the parameter to require several: every one must match.
ISO 8601 timestamp. Sessions started at or after this moment.
ISO 8601 timestamp. Sessions started at or before this moment.
Include the conversation turns for each session.
Number of sessions to skip.
Sessions per page. Capped at 500.
**`run` vs `eval`**: a *run* is one batch job that scored N evals over M conversations
("Compliance V3"). An *eval* is a single scoring criterion ("Tone check"). `run` is the
production analogue of `simulation` on the simulations endpoint.
## Example request
```bash theme={null}
curl -X GET "https://api.getusertrace.com/api/prod-evaluations/sessions/?run=Compliance&failed_only=true" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json"
```
Failing conversations for one tenant on one journey:
```bash theme={null}
curl -G "https://api.getusertrace.com/api/prod-evaluations/sessions/" \
--data-urlencode "tag=tenant:acme" \
--data-urlencode "tag=journey.id:returns" \
--data-urlencode "failed_only=true" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE"
```
## Response
The sessions on this page.
Deep link to the session in the web app.
Name of the evaluation run this session belongs to.
**Your own** identifier for the conversation: the session id from an uploaded
transcript, or the `gen_ai.conversation.id` you sent with live traces. Use it to
correlate results back to your own logs.
The attributes you sent with [live traces](/api-reference/endpoint/live-traces#your-own-metadata),
e.g. `{"tenant": "acme", "journey.id": "returns"}`. Values keep their OTLP type
(string, number, boolean). `{}` when none were sent.
Session status, e.g. `completed`.
ISO 8601 timestamp.
ISO 8601 timestamp.
`total_evaluations`, `passed_count`, `failed_count`, `pass_rate` for the **whole**
session.
One entry per eval that scored this session: `eval_name`, `eval_category`,
`eval_type`, `outcome`, `reasoning`.
Conversation turns: `session_sequence`, `user_input`, `agent_response`. Present
only with `include_traces=true`.
Sessions matching the filters, across all pages.
```json theme={null}
{
"items": [
{
"url": "https://app.getusertrace.com/production-eval/ca0a2090-...?sessionId=b3e046d3-...",
"run": "Compliance V3",
"source_session_id": "814b9f0898",
"tags": {
"service.name": "support-agent",
"tenant": "acme",
"journey.id": "returns"
},
"status": "completed",
"started_at": "2026-07-25T12:30:56.263968+00:00",
"ended_at": "2026-07-25T12:31:26.072613+00:00",
"outcome_summary": {
"total_evaluations": 1,
"passed_count": 1,
"failed_count": 0,
"pass_rate": 1.0
},
"eval_outcomes": [
{
"eval_name": "Compliance V3: Bias, Hallucination, Policy Handling",
"eval_category": "compliance",
"eval_type": "session_level",
"outcome": "pass",
"reasoning": "No policy-restricted advice was offered..."
}
]
}
],
"total": 666,
"skip": 0,
"limit": 50
}
```
**Production sessions have no agent, scenario or persona.** Those concepts exist only
for simulations, so the keys are absent rather than `null`. Use
[Simulation Results](/api-reference/endpoint/simulation-sessions) for simulation runs.
## What's excluded
**Sessions with no eval outcomes don't appear.** An evaluation creates one session per
uploaded document, and the transcript parser then fans out into child sessions that carry
all the outcomes, leaving the parent as a bookkeeping row with nothing to report. `total`
counts only sessions that actually have results.
## Filtering by tag
Tags match as **text**, so number and boolean attributes are filtered by their string
form: an attribute sent as the integer `3` matches `tag=retries:3`, and `true` matches
`tag=beta:true`.
Everything before the **first** colon is the key, so values may contain colons:
`tag=ref:urn:acme:42` matches key `ref`, value `urn:acme:42`. A `tag` without a colon
returns `400`.
## Failures only
With `failed_only=true`, `outcome_summary` still describes the **whole** session, so
`len(eval_outcomes)` equals `outcome_summary.failed_count` and is deliberately smaller
than `total_evaluations`.
# Simulation Results
Source: https://docs.getusertrace.com/api-reference/endpoint/simulation-sessions
Fetch simulation sessions and their eval outcomes
## Overview
Returns simulation sessions with their eval outcomes, newest first. Filter by the agent
that was tested and a date range. No simulation id is required, so you can pull results
without knowing anything about how a run was set up in the dashboard.
Every item deep-links back to the session in the web app.
## Endpoint
```
GET /api/simulations/sessions/
```
## Authentication
```http theme={null}
x-usertrace-api-key: ut_live_YOUR_KEY_HERE
```
See [Authentication](/api-reference/authentication).
## Query parameters
All filters are optional. Omit them all and you get every session the key's owner can see.
Substring match on the agent's **name or URL**. Case-insensitive.
ISO 8601 timestamp. Sessions started at or after this moment.
ISO 8601 timestamp. Sessions started at or before this moment.
Keep only sessions scored by the named eval, e.g. `Tone check`. Substring match,
case-insensitive.
Return only sessions with at least one failing eval, showing only the failing outcomes.
Include the conversation turns for each session.
Number of sessions to skip.
Sessions per page. Capped at 500.
## Example request
```bash theme={null}
curl -X GET "https://api.getusertrace.com/api/simulations/sessions/?agent=Support%20Bot&started_after=2026-09-01T00:00:00Z" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json"
```
## Response
The sessions on this page.
Deep link to the session in the web app. `null` in the rare case the simulation row
was deleted. The session and its outcomes are still reported.
Name of the simulation run this session belongs to.
Name of the agent (environment configuration) that was tested.
The scenario the simulated user was following.
The persona the simulated user adopted.
Session status, e.g. `completed`.
ISO 8601 timestamp.
ISO 8601 timestamp.
`total_evaluations`, `passed_count`, `failed_count`, `pass_rate` for the **whole**
session.
One entry per eval that scored this session: `eval_name`, `eval_category`,
`eval_type`, `outcome`, `reasoning`.
Conversation turns: `session_sequence`, `user_input`, `user_reason`,
`agent_response`. Present only with `include_traces=true`.
Sessions matching the filters, across all pages.
```json theme={null}
{
"items": [
{
"url": "https://app.getusertrace.com/simulation/3d9c49fc-...?sessionId=e076939a-...",
"simulation": "Nightly regression",
"agent": "Support Bot v4",
"scenario": "Customer disputes a duplicate charge",
"persona": "Frustrated customer",
"status": "completed",
"started_at": "2026-09-11T09:20:55.967514+00:00",
"ended_at": "2026-09-11T09:22:58.845944+00:00",
"outcome_summary": {
"total_evaluations": 1,
"passed_count": 1,
"failed_count": 0,
"pass_rate": 1.0
},
"eval_outcomes": [
{
"eval_name": "Loop Prevention and Conversation Coherence",
"eval_category": "Safety",
"eval_type": "session_level",
"outcome": "pass",
"reasoning": "The agent acknowledged the dispute and moved to resolution..."
}
]
}
],
"total": 769,
"skip": 0,
"limit": 50
}
```
`traces` is omitted entirely unless `include_traces=true`. It is an absent key rather
than an empty list, which would read as "this session had no turns".
## Failures only
`failed_only=true` keeps sessions with at least one failing eval and publishes only the
failing outcomes, which is the shape you want for a CI gate or an alerting job.
```bash theme={null}
curl -X GET "https://api.getusertrace.com/api/simulations/sessions/?failed_only=true&agent=Support%20Bot" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json"
```
`total` then counts failing sessions, so pagination walks failures rather than sessions.
Each session's `outcome_summary` still describes the **whole** session. So
`len(eval_outcomes)` equals `outcome_summary.failed_count` and is deliberately smaller
than `total_evaluations`. Don't compute a pass rate from the array length.
## Pagination
Walk pages with `skip` and `limit` until `skip + len(items) >= total`:
```bash theme={null}
curl -X GET "https://api.getusertrace.com/api/simulations/sessions/?skip=50&limit=50" \
-H "x-usertrace-api-key: ut_live_YOUR_KEY_HERE"
```
# Core Concepts
Source: https://docs.getusertrace.com/core-concepts
Understanding UserTrace's testing framework components
## Overview
Understanding UserTrace's testing framework requires grasping four key components that work together to create realistic agent evaluations.
## Scenario
A **Scenario** defines the testing situation you want to evaluate. Think of it as the "what if" question you want answered.
**Example:** *"A 30-year-old migrant delivery worker based in Bengaluru, under financial stress"*
This simple description gets expanded into a comprehensive test case that includes:
* Contextual background and situation details
* Expected user goals and pain points
* Linked persona for behavioral modeling
* Connected evaluation criteria for success measurement
## Persona
A **Persona** brings your scenario to life by defining how the simulated user thinks, behaves, and communicates.
Taking our delivery worker example, the persona would detail:
* **Communication style**: Direct, time-conscious, uses local language mix
* **Daily goals**: Maximize deliveries, minimize downtime, manage expenses
* **Emotional state**: Stressed about finances, frustrated with app issues
* **Knowledge level**: Tech-savvy but impatient with complex processes
* **Decision-making patterns**: Quick decisions, price-sensitive choices
## Evals (Evaluations)
**Evals** define your pass/fail criteria - the specific metrics that determine if your agent handled the scenario successfully.
Examples include:
* Did the agent resolve the delivery issue within 2 minutes?
* Was the tone empathetic when discussing financial concerns?
* Did the agent offer appropriate solutions for the user's location?
* Was sensitive information handled securely?
## Simulation
A **Simulation** executes your test by running the scenario with its linked persona and evaluations.
**Scaling Example:**
* 1 Scenario: "Financial stress delivery worker"
* 4 Personas: Different ages, locations, stress levels, communication styles
* 2 Sessions each: Morning rush vs evening fatigue states
* **Result**: 8 total conversations automatically tested against your evaluation criteria
Each conversation provides detailed insights into how your agent performs across different user types and situations.
## How It Works
**The simple truth: You only need to select the scenarios you want to test.**
When you choose a scenario in UserTrace, the appropriate personas and evaluations are automatically selected and linked for you. This means:
* **No manual persona creation** - We've already built realistic user personas that match your scenarios
* **No evaluation setup** - Relevant success criteria are pre-configured for each scenario type
* **No complex configuration** - Just pick your scenarios and run your tests
## Workflow
1. **Browse** available scenarios in your UserTrace dashboard
2. **Select** the scenarios that match your testing needs
3. **Run** simulations - personas and evals are automatically applied
4. **Review** results with detailed conversation analysis
This automated approach ensures you get comprehensive testing without the complexity of manual setup, while still providing the depth and realism your agent testing requires.
# Overview
Source: https://docs.getusertrace.com/index
Test and improve your AI agents through realistic simulations and comprehensive evaluation
## Getting Started
Test your AI agents with realistic user simulations in minutes.
Connect your agent and create test scenarios in three simple steps.
## Supported Agent Types
UserTrace works with all major AI agent architectures and platforms.
Test telephony-based, WebRTC, and voice platform integrations (Vapi, Retell, LiveKit, Pipecat).
Evaluate custom APIs, WhatsApp Business, Slack bots, and prompt-based agents.
Direct integration with popular voice and chat platforms.
Pull results programmatically and stream live production traces over OTLP.
## How It Works
Create realistic testing scenarios with our four-component framework.
Define realistic testing situations based on your users' actual needs and contexts.
Simulate authentic user behavior patterns, communication styles, and decision-making.
Set pass/fail criteria to measure agent performance and quality metrics.
Execute tests and get detailed insights on agent performance and user satisfaction.
## Key Benefits
Why leading AI teams choose UserTrace for agent evaluation.
Test with authentic user personas and real-world scenarios, not generic test cases.
Support for voice, chat, API-based, and platform-integrated agents.
Connect your agent and start testing in minutes with minimal setup.
Get detailed conversation analysis and specific improvement recommendations.
## Trusted by Leading AI Teams
Learn how Grammarly, Adaface, and Virtual Internship improve their AI agents.
# Custom Chat Agent
Source: https://docs.getusertrace.com/quickstarts/custom-chat-agent
Test custom chat agents through API endpoints
Test your custom chat agents by hosting them in a sandbox environment. UserTrace will interact with your API using the OpenAI Chat Completions format.
## API Requirements
Your sandbox must expose an API endpoint compatible with OpenAI's Chat Completions format, plus a health check endpoint.
### 1. Chat Completions Endpoint
**Endpoint:** `POST /v1/chat/completions`
**Request Format:**
```json theme={null}
{
"messages": [
{
"role": "system",
"content": "You are a helpful customer service agent."
},
{
"role": "user",
"content": "I need help with my order #12345"
}
],
"user_id": "sim_user_abc123",
"session_id": "session_xyz789",
"metadata": {
"scenario_id": "financial_stress_delivery",
"persona": "delivery_worker_bengaluru",
"time_of_day": "morning_rush"
}
}
```
**Response Format:**
```json theme={null}
{
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "I'd be happy to help you with your order. Let me look up order #12345 for you."
},
"finish_reason": "stop"
}
],
"session_id": "session_xyz789",
"chat_id": "chat_456def",
"metadata": {
"agent_confidence": 0.95,
"response_time_ms": 1200,
"tools_used": ["order_lookup"]
},
"evaluation_metadata": {
"order_found": true,
"tone": "empathetic",
"resolution_step": 1,
"customer_acknowledged": true
}
}
```
### 2. Health Check Endpoint
**Endpoint:** `GET /health`
**Request:**
```bash theme={null}
curl -X GET https://your-sandbox.com/health
```
**Response Format:**
```json theme={null}
{
"status": "healthy",
"timestamp": "2024-01-21T10:30:00Z",
"version": "1.0.0",
"uptime": 3600
}
```
## Advanced Configuration
### User Management (Optional)
If your agent requires user-specific contexts or session management, we can create users at the start of each simulation session.
**Endpoint:** `POST /users`
**Request Format:**
```json theme={null}
{
"user_id": "sim_user_abc123",
"metadata": {
"scenario_id": "financial_stress_delivery",
"persona": "delivery_worker_bengaluru",
"session_type": "morning_rush"
}
}
```
**Response Format:**
```json theme={null}
{
"user_id": "sim_user_abc123",
"created_at": "2024-01-21T10:30:00Z",
"status": "active"
}
```
### Optional Parameters
**user\_id** and **session\_id** are optional in requests. If your agent doesn't require session management, you can omit them.
**metadata** is optional and can contain any scenario or context information your agent needs.
**Request with minimal parameters:**
```json theme={null}
{
"messages": [
{
"role": "user",
"content": "Hello, I need help"
}
]
}
```
**Request with full parameters:**
```json theme={null}
{
"messages": [
{
"role": "system",
"content": "You are a helpful customer service agent."
},
{
"role": "user",
"content": "I need help with my order #12345"
}
],
"user_id": "sim_user_abc123",
"session_id": "session_xyz789",
"metadata": {
"scenario_id": "financial_stress_delivery",
"persona": "delivery_worker_bengaluru",
"stress_level": "high",
"location": "bengaluru"
}
}
```
## Implementation Guide
### 1. Set Up Your Sandbox
Deploy your chat agent to a publicly accessible endpoint with HTTPS support.
**Required Headers:**
* `Content-Type: application/json`
* `Authorization: Bearer ` (best practice for security)
### 2. Test Your Endpoints
**Chat Completions Test:**
```bash theme={null}
curl -X POST https://your-sandbox.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"messages": [
{"role": "user", "content": "Hello, test message"}
],
"user_id": "test_user_123",
"session_id": "test_session_456"
}'
```
**Health Check Test:**
```bash theme={null}
curl -X GET https://your-sandbox.com/health
```
### 3. Configure in UserTrace
1. **Add your sandbox URL** in the UserTrace dashboard
2. **Set authentication** if required (API keys, tokens)
3. **Enable user management** if your agent needs user contexts
4. **Configure metadata passing** for state-dependent scenarios
## Error Handling
Your API should return appropriate HTTP status codes:
**400 Bad Request:**
```json theme={null}
{
"error": {
"message": "Invalid request format",
"type": "invalid_request_error",
"code": "bad_request"
}
}
```
**500 Internal Server Error:**
```json theme={null}
{
"error": {
"message": "Internal server error",
"type": "internal_error",
"code": "server_error"
}
}
```
**429 Rate Limit:**
```json theme={null}
{
"error": {
"message": "Rate limit exceeded",
"type": "rate_limit_error",
"code": "rate_limit_exceeded"
}
}
```
## Best Practices
**Response Optimization**
• Target \< 5 second response times
• Support 100 RPS for parallel simulations
• Implement proper error handling
• Add request/response logging
**API Security**
• Use HTTPS endpoints only
• Implement rate limiting
• Validate all input data
## Tool Calling Support
For agents that use function calling, the simulated user will send simple requests without tool definitions. Your agent must handle all tool calling internally - execute the necessary functions, get the results, and include the complete sequence of tool calls and responses in your response. The evaluator will use this information to evaluate your agent's tool usage and decision-making.
**Request from Simulated User:**
```json theme={null}
{
"messages": [
{
"role": "user",
"content": "What's the status of my order #12345?"
}
],
"user_id": "sim_user_abc123",
"session_id": "session_xyz789"
}
```
**Response with Tool Call and Results:**
```json theme={null}
{
"choices": [
{
"message": {
"role": "assistant",
"content": "I found your order! Order #12345 is currently being prepared and will be shipped within 2 hours.",
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "get_order_status",
"arguments": "{\"order_id\": \"12345\"}",
"result": {
"order_id": "12345",
"status": "preparing",
"estimated_ship_time": "2 hours",
"tracking_number": null
}
}
}
]
}
}
],
"session_id": "session_xyz789",
"chat_id": "chat_456def",
"evaluation_metadata": {
"tool_called": "get_order_status",
"tool_success": true,
"response_included_status": true,
"customer_informed": true
}
}
```
## Common Integration Patterns
* **RAG Systems**: Pass document context in system messages
* **Multi-Agent**: Route requests based on message content
* **Memory Integration**: Use user metadata for conversation history
* **External APIs**: Include tool calls for third-party integrations
**Need help with integration?** Our team can assist with custom requirements and complex architectures. Contact [support@getusertrace.com](mailto:support@getusertrace.com).
**Next Steps**: Once configured, [create test scenarios](/scenarios-generation) and [run simulations](/running-simulations) to validate your agent.
# Prompt-Based Testing
Source: https://docs.getusertrace.com/quickstarts/prompt-based
Test your prompts directly with LLM models
Test your system prompts across different models and scenarios without building a full agent. Perfect for prompt optimization and A/B testing.
## Getting Started
### 1. Configure Your Model
Choose your preferred LLM provider and model:
**Available Models:**
• GPT-4o, GPT-4o mini
• GPT-4 Turbo, GPT-4
• GPT-3.5 Turbo
**Available Models:**
• Claude 3.5 Sonnet
• Claude 3 Opus, Claude 3 Sonnet
• Claude 3 Haiku
**Available Models:**
• Gemini 1.5 Pro
• Gemini 1.5 Flash
• Gemini 1.0 Pro
### 2. Input Your System Prompt
Enter your complete system prompt in the text area. This is what will be used to guide the model's behavior during testing.
**Example:**
```
You are a helpful customer service agent for a delivery company.
Your goal is to resolve customer issues quickly and empathetically.
Guidelines:
1. Always ask for the order number first
2. Acknowledge the customer's frustration
3. Provide clear next steps
4. Offer compensation when appropriate
Keep responses under 100 words and maintain a professional tone.
```
**Remove Temporary Variables**: Make sure to remove any placeholder text like `{customer_name}` or `{order_id}` from your prompt. Use static examples instead.
### 3. Configure Model Parameters
**Temperature (0.0 - 1.0)**
* **0.0-0.3**: Consistent, predictable responses
* **0.4-0.7**: Balanced creativity and consistency
* **0.8-1.0**: More creative and varied responses
**Response Format**
* **Text**: Standard text responses
* **JSON**: Structured JSON output (specify schema in prompt)
### 4. Set Evaluation Context
This is crucial for accurate testing. Add any information from your prompt that the evaluator needs to know:
**What to Include:**
* Key guidelines or rules from your prompt
* Expected response format or structure
* Specific goals or success criteria
* Any constraints or limitations
**Example Evaluation Context:**
```
The agent should:
- Always ask for order number first
- Acknowledge customer frustration
- Keep responses under 100 words
- Maintain professional tone
- Offer compensation when issues warrant it
Success criteria:
- Issue resolution within 3 exchanges
- Customer satisfaction maintained
- Company policies followed
```
**Why This Matters**: The simulated user and evaluator don't see your system prompt during testing. The evaluation context ensures they understand what behavior to expect and how to measure success.
### 5. Advanced Features
**Tool Calls & Function Calling**
If your prompt involves tool calls or function calling, use our [Custom Chat Agent](/quickstarts/custom-chat-agent) setup instead, which provides full control over tool definitions and execution.
## Best Practices
**Clear Instructions**
• Use specific, actionable guidelines
• Include examples of good responses
• Define success criteria explicitly
**Effective Testing**
• Start with edge cases
• Test across different user personas
• Compare performance across models
## Example Workflow
1. **Input** your system prompt
2. **Select** OpenAI GPT-4o with temperature 0.3
3. **Add** evaluation context about expected behavior
4. **Choose** scenarios like "frustrated delivery customer"
5. **Run** simulations across multiple personas
6. **Analyze** results and iterate on your prompt
## Common Use Cases
* **Customer Service**: Testing support agent responses
* **Content Creation**: Evaluating writing assistant prompts
* **Educational**: Testing tutoring or explanation prompts
* **Classification**: Testing categorization and tagging prompts
* **Summarization**: Testing document or conversation summaries
**Ready to test more complex scenarios?** Check out our [Custom Chat Agent](/quickstarts/custom-chat-agent) guide for tool calling, multi-turn conversations, and custom API integrations.
# Slack Chat Agent
Source: https://docs.getusertrace.com/quickstarts/slack-chat-agent
Test Slack workspace bots with realistic user scenarios
Test your Slack bot by connecting your Slack workspace to UserTrace. Our simulated users will interact with your bot directly through Slack messages.
## Getting Started
### 1. Connect Your Slack Bot
In the UserTrace dashboard, navigate to **Agent Setup** and select **Slack Integration**.
**Required Information:**
* Slack Bot Token or App credentials
* Workspace URL or ID
**Example:**
```
Bot Token: xoxb-your-bot-token
Workspace: yourcompany.slack.com
```
### 2. Set Evaluation Context
Define your pass and fail criteria for what constitutes successful interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Responds within 30 seconds
- Uses appropriate workplace tone
- Provides relevant information or solutions
- Follows company guidelines
- Handles requests efficiently
Fail Criteria:
- Response time exceeds 60 seconds
- Uses inappropriate or unprofessional tone
- Provides incorrect or unhelpful responses
- Violates company policies
- Fails to understand user intent
```
## Important Simulation Details
### Sequential Testing
**One at a Time**: All simulations run sequentially (one after another) from the same test user to avoid message conflicts and ensure clean conversation threads.
**How it works:**
* Simulation 1 completes entirely before Simulation 2 starts
* Each simulation appears as a separate conversation thread or DM
* No message overlap or confusion between different test scenarios
### Test User Management
**Our Test User:**
All simulations will come from a designated UserTrace test user account. This user will:
* Send messages as different personas
* Complete full conversation scenarios
* Reset between simulations for clean testing
**Your Setup:**
Ensure your Slack bot can receive messages from new workspace members and respond appropriately.
## Best Practice: Use Test Workspace
**Recommended Approach**: Use a test Slack workspace instead of your production workspace to avoid any concerns about triggering real business actions during testing.
If you must use your production Slack workspace, ensure you prevent the test user from triggering real business actions:
**Actions to Disable:**
* **Meeting bookings** - Don't schedule real meetings
* **Task assignments** - Don't create actual work items
## Testing Process
### Inbound Bot Testing
For bots that respond to incoming messages:
1. Our test user sends the first message to your Slack bot
2. Your bot responds based on the scenario
3. The conversation continues until completion
4. Results are analyzed and scored
### Outbound Bot Testing
For bots that initiate conversations (proactive messaging):
1. UserTrace sends a webhook request to your system
2. Your bot initiates the conversation by sending the first message
3. Our simulated user responds according to the scenario
4. The conversation continues until completion
5. Results are analyzed and scored
**Webhook Format for Outbound Bots:**
```json theme={null}
{
"trigger": "start_simulation",
"scenario_id": "daily_standup_reminder",
"target_user": "@test-user",
"metadata": {
"persona": "busy_developer",
"context": "sprint_deadline_pressure"
}
}
```
### 2. Between Simulations
* 30-second delay between different scenarios
* New conversation thread starts fresh
* Previous context is cleared
### 3. View Results
Each simulation provides:
* Complete conversation transcript
* Response time analytics
* Success/failure against your criteria
* Persona-specific behavior analysis
## Best Practices
**Natural Conversation Flow**
• Respond within 30 seconds for best results
• Handle message bursts gracefully
• Use threading for complex discussions
**Platform Integration**
• Use interactive buttons and menus
• Support file sharing and attachments
• Handle mentions and reactions appropriately
## Common Scenarios
**Workplace Productivity Examples:**
* "Schedule a meeting for tomorrow at 2 PM"
* "What's the status of the Q4 project?"
* "I need help with the expense report process"
* "Can you remind me about my upcoming deadlines?"
**HR and Support Examples:**
* "How do I request time off?"
* "Where can I find the employee handbook?"
* "I'm having trouble with my laptop"
* "Who should I contact about benefits?"
## Troubleshooting
**Common Issues:**
* **Messages not delivering**: Check Slack app permissions and OAuth scopes
* **Slow responses**: Optimize your webhook response time
* **Test actions triggering**: Verify you're using test workspace or proper safeguards
* **Thread mixing**: Ensure proper conversation threading
**Need help with Slack setup?** Our team can assist with bot configuration and testing best practices. Contact [support@getusertrace.com](mailto:support@getusertrace.com).
**Next Steps**: [Create test scenarios](/scenarios-generation) for your Slack bot and [run simulations](/running-simulations) to validate performance.
# WhatsApp Chat Agent
Source: https://docs.getusertrace.com/quickstarts/whatsapp-chat-agent
Test WhatsApp Business agents with realistic user scenarios
Test your WhatsApp Business agent by connecting your WhatsApp number to UserTrace. Our simulated users will interact with your agent directly through WhatsApp messages.
## Getting Started
### 1. Connect Your WhatsApp Number
In the UserTrace dashboard, navigate to **Agent Setup** and select **WhatsApp Integration**.
**Required Information:**
* WhatsApp Business number (with country code)
**Example:**
```
WhatsApp Number: +1-555-123-4567
```
### 2. Set Evaluation Context
Define your pass and fail criteria for what constitutes successful interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Responds within 30 seconds
- Uses friendly, conversational tone
- Asks for order number when relevant
- Provides clear solutions or next steps
- Resolves customer issues effectively
Fail Criteria:
- Response time exceeds 60 seconds
- Uses inappropriate or unprofessional tone
- Fails to gather necessary information
- Provides incorrect or unhelpful responses
- Escalates unnecessarily
```
## Important Simulation Details
### Sequential Testing
**One at a Time**: All simulations run sequentially (one after another) from the same phone number to avoid message conflicts and ensure clean conversation threads.
**How it works:**
* Simulation 1 completes entirely before Simulation 2 starts
* Each simulation appears as a separate conversation thread
* No message overlap or confusion between different test scenarios
### Test Number Management
**Our Test Number:**
All simulations will come from a designated UserTrace phone number. This number will:
* Send messages as different personas
* Complete full conversation scenarios
* Reset between simulations for clean testing
**Your Setup:**
Ensure your WhatsApp Business account can receive messages from new numbers without restrictions.
## Best Practice: Use Sandbox Environment
**Recommended Approach**: Use your WhatsApp sandbox number instead of your production number to avoid any concerns about triggering real business actions during testing.
If you must use your production WhatsApp number, ensure you prevent the test number from triggering real business actions:
**Actions to Disable:**
* **Booking confirmations** - Don't create real reservations
* **Payment processing** - Don't charge actual money
## Best Practices
**Natural Conversation Flow**
• Respond within 30 seconds for best results
• Use typing indicators if available\
• Handle message bursts gracefully
**WhatsApp Features**
• Support media messages (images, voice)
• Handle quick reply buttons
• Use structured messages when appropriate
## Testing Process
### Inbound Agent Testing
For agents that respond to incoming messages:
1. Our test number sends the first message to your WhatsApp
2. Your agent responds based on the scenario
3. The conversation continues until completion
4. Results are analyzed and scored
### Outbound Agent Testing
For agents that initiate conversations (outbound messaging):
1. UserTrace sends a webhook request to your system
2. Your agent initiates the conversation by sending the first message
3. Our simulated user responds according to the scenario
4. The conversation continues until completion
5. Results are analyzed and scored
**Webhook Format for Outbound Agents:**
```json theme={null}
{
"trigger": "start_simulation",
"scenario_id": "delivery_followup",
"target_number": "+1-555-TEST-NUM",
"metadata": {
"persona": "busy_customer",
"context": "late_delivery_followup"
}
}
```
### 2. Between Simulations
* 30-second delay between different scenarios
* New conversation thread starts fresh
* Previous context is cleared
### 3. View Results
Each simulation provides:
* Complete conversation transcript
* Response time analytics
* Success/failure against your criteria
* Persona-specific behavior analysis
## Common Scenarios
**Customer Service Examples:**
* "My order is late, where is it?"
* "I want to cancel my subscription"
* "The food was cold when delivered"
* "I was charged twice for my order"
**Sales Examples:**
* "What's your most popular dish?"
* "Do you deliver to my area?"
* "I have food allergies, what can I eat?"
* "What are your hours today?"
## Troubleshooting
**Common Issues:**
* **Messages not delivering**: Check WhatsApp Business API status
* **Slow responses**: Optimize your webhook response time
* **Test actions triggering**: Verify blacklist implementation
* **Conversation mixing**: Ensure proper thread management
**Need help with WhatsApp setup?** Our team can assist with Business API configuration and testing best practices. Contact [support@getusertrace.com](mailto:support@getusertrace.com).
**Next Steps**: [Create test scenarios](/scenarios-generation) for your WhatsApp agent and [run simulations](/running-simulations) to validate performance.
# Running Simulations
Source: https://docs.getusertrace.com/running-simulations
Start and manage realistic AI agent testing simulations
## Starting Your Simulation
Launch comprehensive testing sessions with your configured agents and scenarios.
## Simulation Setup
Choose from your configured voice or conversational agents
• Select generated scenarios
• Linked personas and evals auto-populate
• Optionally add specific personas or evaluations
Click "Start Simulation" to begin real-time testing
## During Simulation
• **Real-time monitoring** of agent interactions
• **Live performance metrics** and evaluation scores\
• **Conversation flow tracking** across multiple scenarios
• **Issue detection** as problems occur
## Simulation Results
Once complete, review comprehensive analytics:
• **Performance insights** on response quality and conversation flow
• **Issue detection** highlighting conversation breakdowns and safety concerns
• **User experience analysis** showing friction points and satisfaction metrics\
• **Actionable recommendations** for improving agent capabilities
This page will be expanded with detailed simulation management features and monitoring capabilities.
# Creating Scenarios
Source: https://docs.getusertrace.com/scenarios-generation
Generate realistic test scenarios with AI-powered scenario, evaluation, and persona creation
## AI-Powered Scenario Generation
UserTrace automatically creates comprehensive test scenarios from simple text descriptions or document uploads.
## Input Methods
**Simple Natural Language Input**
Describe your target user in plain text:
*"30-year-old professional dealing with work stress and seeking career guidance"*
UserTrace generates everything automatically.
**Batch Scenario Import**
Upload a document containing multiple scenarios you want to test your agent against.
Perfect for existing test cases or comprehensive scenario libraries.
## Advanced Settings
Fine-tune your scenario generation with advanced configuration options.
**Control Evaluation Criteria Precision**
• **Lenient**: Broader acceptance criteria, focuses on general success\
• **Moderate**: Balanced evaluation with reasonable expectations\
• **Strict**: Precise criteria with high standards for agent performance
**Edge Cases and Failure Conditions**
• **Basic**: Core happy path scenarios\
• **Comprehensive**: Includes common edge cases\
• **Extensive**: Maximum coverage with rare failure conditions and stress tests
## Generation Process
### 1. Preview & Review
After initial generation, UserTrace shows a preview of:
• **Scenarios** - Realistic interaction flows\
• **Evaluations** - Success criteria and metrics\
• **Personas** - Detailed user profiles with motivations and characteristics
### 2. Iterate with Feedback
Not satisfied with the initial generation? Provide feedback and regenerate:
• "Make scenarios more challenging"
• "Focus on angry customers"\
• "Add technical troubleshooting cases"
UserTrace refines the generation based on your input.
### 3. Deep Research Mode
Once you approve the preview, UserTrace enters **deep research mode**:
Analyzes your website and knowledge base to understand your agent's role and capabilities.
Reviews any production logs you've uploaded to understand real user patterns and pain points.
Incorporates feedback from past simulations to improve scenario relevance and quality.
Builds a personalized knowledge graph tailored to your specific agent and use cases.
**Processing Time**: Deep research mode typically takes a few minutes to complete. You'll receive a notification when your personalized knowledge graph is ready for simulation.
## What You Get
The final output includes:
Complete conversation flows with realistic user interactions, emotional states, and contextual background.
Multi-dimensional evaluation criteria covering functionality, safety, user satisfaction, and domain-specific metrics.
Detailed user profiles with demographics, motivations, communication styles, and behavioral patterns.
## Next Steps
With your scenarios generated, you're ready to:
1. **Review** the detailed scenarios, evals, and personas
2. **Start simulations** using your configured agents
3. **Analyze results** and iterate on improvements
Ready to run your first simulation? Head to [Running Simulations](/running-simulations) to launch realistic testing sessions.
# LiveKit Integration
Source: https://docs.getusertrace.com/voice/livekit
Test voice agents built with LiveKit real-time infrastructure
Test your LiveKit-powered voice agents by connecting them to UserTrace. Our simulated users will interact with your LiveKit-based agents through WebRTC connections for real-time voice conversations.
## Getting Started
### 1. Connect Your LiveKit Agent
In the UserTrace dashboard, navigate to **Agent Setup** and select **LiveKit Integration**.
**Required Information:**
* LiveKit Server URL (WSS endpoint)
* API Key and Secret
* Room name or agent identifier
**Example:**
```
LiveKit Server: wss://your-project.livekit.cloud
API Key: APIxxxxxxxxxxxxx
Room Name: usertrace-test-room
```
### 2. Set Evaluation Context
Define your pass and fail criteria for voice interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Establishes WebRTC connection within 5 seconds
- Maintains stable audio throughout conversation
- Responds within 3 seconds of user input
- Provides relevant and accurate responses
- Handles network issues gracefully
Fail Criteria:
- Connection fails or takes too long to establish
- Audio dropouts or quality degradation
- Response delays exceeding 8 seconds
- Irrelevant or incorrect responses
- Unexpected disconnections
```
## Testing Process
### Real-Time Voice Testing
LiveKit testing follows a WebRTC-based approach:
1. UserTrace connects to your LiveKit room
2. WebRTC connection is established with your agent
3. Real-time audio streams are initiated
4. Simulated user interacts based on scenario
5. Conversation quality and agent responses are monitored
6. Results are analyzed including technical metrics
**Connection Flow:**
1. **Room Join**: UserTrace joins specified LiveKit room
2. **Agent Discovery**: Identifies your voice agent participant
3. **Audio Setup**: Establishes bidirectional audio streams
4. **Conversation Start**: Begins scenario-based interaction
5. **Quality Monitoring**: Tracks audio quality and latency
6. **Natural Completion**: Ends when scenario objectives are met
## LiveKit-Specific Features
### Real-Time Infrastructure
**WebRTC Quality:**
* Ultra-low latency voice communication
* Adaptive bitrate for varying network conditions
* Echo cancellation and noise suppression
* Network resilience and recovery
**Room Management:**
* Multi-participant support
* Dynamic participant joining/leaving
* Room metadata and configuration
* Custom room layouts and settings
### Agent Architecture
**Server-Side Agents:**
* Python/Node.js agent implementations
* Real-time audio processing
* AI model integration
* Custom business logic
**Client-Side Integrations:**
* Web browser compatibility
* Mobile app support
* Desktop application integration
* Custom client implementations
## Best Practices
**Real-Time Optimization**
• Target \< 200ms end-to-end latency
• Use LiveKit's adaptive streaming
• Optimize AI model inference time
• Monitor connection quality metrics
**Connection Stability**
• Implement reconnection logic
• Handle network interruptions
• Use LiveKit's connection events
• Monitor participant status
## Implementation Examples
### Server Agent Setup
**Python Agent Example:**
```python theme={null}
import asyncio
from livekit import rtc
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
async def entrypoint(ctx: JobContext):
initial_ctx = llm.ChatContext().append(
role="system",
text="You are a helpful voice assistant. Respond naturally and concisely."
)
await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
# Handle voice interactions
# Process audio, generate responses, etc.
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```
**Room Configuration:**
```javascript theme={null}
const room = new Room({
adaptiveStream: true,
dynacast: true,
publishDefaults: {
audioPreset: AudioPreset.SPEECH,
},
});
await room.connect(livekit_url, token);
```
### Authentication
**Token Generation:**
```python theme={null}
from livekit import api
# Generate access token
token = api.AccessToken(api_key, api_secret) \
.with_identity("voice_agent") \
.with_name("UserTrace Test Agent") \
.with_grants(api.VideoGrants(
room_join=True,
room="usertrace-test-room"
)).to_jwt()
```
## Common Scenarios
**Interactive Voice Assistants:**
* Personal assistant interactions
* Smart home control
* Task management
* Information retrieval
**Customer Support:**
* Real-time help desk
* Technical troubleshooting
* Product information
* Issue resolution
**Collaborative Applications:**
* Team meetings with AI participants
* Educational tutoring sessions
* Brainstorming facilitation
* Multi-user voice experiences
## Advanced Features
### AI Integration
**Speech-to-Text:**
* Real-time transcription
* Multiple language support
* Custom vocabulary
* Confidence scoring
**Text-to-Speech:**
* Natural voice synthesis
* Voice cloning capabilities
* Emotion and tone control
* Multi-speaker support
**LLM Integration:**
* Streaming AI responses
* Context preservation
* Function calling
* Multi-modal processing
### Quality Monitoring
**Audio Metrics:**
```javascript theme={null}
// Monitor connection quality
room.on('connectionQualityChanged', (quality, participant) => {
console.log(`Connection quality: ${quality} for ${participant.identity}`);
});
// Track audio levels
room.localParticipant.on('audioLevelChanged', (level) => {
console.log(`Local audio level: ${level}`);
});
```
## Troubleshooting
**Common Issues:**
**Connection Problems:**
* **WebRTC connection fails**: Check firewall settings and TURN servers
* **Audio not flowing**: Verify microphone permissions and audio tracks
* **High latency**: Optimize server location and network routing
* **Frequent disconnections**: Monitor network stability and implement reconnection
**Agent Issues:**
* **Agent not responding**: Check LiveKit agent deployment and logs
* **Poor audio quality**: Verify audio processing pipeline and network bandwidth
* **Delayed responses**: Optimize AI model inference and processing time
* **Memory leaks**: Monitor resource usage in long-running sessions
## Development Setup
### Local Testing
**Docker Setup:**
```yaml theme={null}
version: '3.8'
services:
livekit:
image: livekit/livekit-server:latest
ports:
- "7880:7880"
- "7881:7881"
volumes:
- ./livekit.yaml:/livekit.yaml
command: --config /livekit.yaml
```
**Configuration File:**
```yaml theme={null}
port: 7880
rtc:
tcp_port: 7881
port_range_start: 50000
port_range_end: 60000
keys:
APIxxxxxxxxxxxxx: secretxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```
**Need help with LiveKit integration?** Check the [LiveKit documentation](https://docs.livekit.io) or contact [support@getusertrace.com](mailto:support@getusertrace.com).
# Pipecat Integration
Source: https://docs.getusertrace.com/voice/pipecat
Test voice agents built with Pipecat AI framework
Test your Pipecat-powered voice agents by connecting them to UserTrace. Our simulated users will interact with your Pipecat-based agents through real-time voice conversations.
## Getting Started
### 1. Connect Your Pipecat Agent
In the UserTrace dashboard, navigate to **Agent Setup** and select **Pipecat Integration**.
**Required Information:**
* Pipecat agent endpoint URL
* Authentication credentials (if required)
* Transport method (WebRTC, WebSocket, or SIP)
**Example:**
```
Agent Endpoint: wss://your-agent.com/pipecat
Transport: WebRTC
Authentication: API Key
```
### 2. Set Evaluation Context
Define your pass and fail criteria for voice interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Establishes connection within 3 seconds
- Maintains real-time conversation flow
- Processes interruptions naturally
- Provides contextually relevant responses
- Handles audio quality gracefully
Fail Criteria:
- Connection timeouts or failures
- Unnatural or robotic responses
- Poor handling of interruptions
- Off-topic or irrelevant responses
- Audio artifacts or quality issues
```
## Testing Process
### Real-Time Pipeline Testing
Pipecat testing focuses on the real-time processing pipeline:
1. UserTrace connects to your Pipecat agent endpoint
2. Real-time audio processing pipeline is established
3. Simulated user begins conversation based on scenario
4. Pipecat processes audio through its pipeline stages
5. Agent responses are generated and delivered
6. Conversation quality and pipeline performance are monitored
**Pipeline Flow:**
1. **Audio Input**: User speech captured and processed
2. **Speech-to-Text**: Real-time transcription
3. **LLM Processing**: Context understanding and response generation
4. **Text-to-Speech**: Natural voice synthesis
5. **Audio Output**: Delivered to user
6. **Quality Metrics**: Latency and accuracy tracking
## Pipecat-Specific Features
### Pipeline Architecture
**Modular Components:**
* Audio input/output processors
* Speech recognition services
* LLM integrations
* Text-to-speech engines
* Transport mechanisms
**Real-Time Processing:**
* Streaming audio processing
* Low-latency pipeline execution
* Interrupt handling
* Context preservation
### Transport Options
**WebRTC Transport:**
* Browser-based connections
* Ultra-low latency
* Built-in echo cancellation
* Network adaptation
**WebSocket Transport:**
* Simple integration
* Custom audio protocols
* Server-to-server communication
* Scalable architecture
**SIP Integration:**
* Traditional telephony systems
* PBX compatibility
* Carrier-grade reliability
* Standards compliance
## Best Practices
**Performance Tuning**
• Minimize processing latency
• Optimize buffer sizes
• Use appropriate audio codecs
• Monitor pipeline bottlenecks
**Sound Processing**
• Configure noise suppression
• Implement echo cancellation
• Handle variable audio quality
• Test with different microphones
## Implementation Examples
### Basic Pipecat Agent
**Python Implementation:**
```python theme={null}
import asyncio
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask
from pipecat.services.openai import OpenAILLMService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.transports.network.websocket import WebsocketTransport
async def run_agent():
# Configure services
stt_service = DeepgramSTTService(api_key="your-deepgram-key")
llm_service = OpenAILLMService(api_key="your-openai-key")
tts_service = ElevenLabsTTSService(api_key="your-elevenlabs-key")
# Set up transport
transport = WebsocketTransport(
host="localhost",
port=8765,
params={"access_token": "your-token"}
)
# Create pipeline
pipeline = Pipeline([
transport.input_processor(),
stt_service,
llm_service,
tts_service,
transport.output_processor()
])
# Run the task
task = PipelineTask(pipeline)
await task.run()
if __name__ == "__main__":
asyncio.run(run_agent())
```
### Advanced Configuration
**Custom Pipeline:**
```python theme={null}
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.audio.audio_buffer import AudioBuffer
# Configure advanced pipeline with custom processors
pipeline = Pipeline([
# Audio preprocessing
AudioBuffer(sample_rate=16000),
# Speech recognition
DeepgramSTTService(
api_key="your-key",
model="nova-2",
language="en-US"
),
# Context management
OpenAILLMContext(
messages=[
{"role": "system", "content": "You are a helpful assistant."}
]
),
# LLM processing
OpenAILLMService(
api_key="your-key",
model="gpt-4",
max_tokens=150
),
# Voice synthesis
ElevenLabsTTSService(
api_key="your-key",
voice_id="your-voice-id"
)
])
```
## Common Scenarios
**Conversational AI:**
* Personal assistants
* Customer service bots
* Educational tutors
* Healthcare assistants
**Real-Time Applications:**
* Live translation services
* Meeting assistants
* Voice-controlled systems
* Interactive voice response (IVR)
**Multi-Modal Experiences:**
* Video conferencing bots
* Smart home interfaces
* Automotive assistants
* Gaming characters
## Advanced Features
### Interrupt Handling
**Barge-in Support:**
```python theme={null}
from pipecat.processors.interruption import InterruptionHandler
interruption_handler = InterruptionHandler(
enable_barge_in=True,
sensitivity=0.7,
timeout_seconds=1.0
)
```
### Context Management
**Conversation Memory:**
```python theme={null}
from pipecat.processors.context import ConversationContext
context = ConversationContext(
max_turns=10,
context_window=4000,
preserve_system_message=True
)
```
### Custom Processors
**Audio Processing:**
```python theme={null}
from pipecat.processors.base import AudioProcessor
class CustomAudioProcessor(AudioProcessor):
async def process_audio(self, audio_data):
# Custom audio processing logic
processed_audio = self.apply_filters(audio_data)
return processed_audio
```
## Troubleshooting
**Common Issues:**
**Pipeline Problems:**
* **High latency**: Optimize processor order and buffer sizes
* **Audio dropouts**: Check network stability and audio codec settings
* **Memory issues**: Monitor processor memory usage and cleanup
* **Context loss**: Verify context management configuration
**Service Integration:**
* **API errors**: Validate service credentials and rate limits
* **Model failures**: Test with different AI models and configurations
* **Transport issues**: Check network connectivity and protocol settings
* **Audio quality**: Verify codec compatibility and audio processing
## Development Setup
### Local Development
**Docker Compose:**
```yaml theme={null}
version: '3.8'
services:
pipecat-agent:
build: .
ports:
- "8765:8765"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- DEEPGRAM_API_KEY=${DEEPGRAM_API_KEY}
- ELEVENLABS_API_KEY=${ELEVENLABS_API_KEY}
volumes:
- ./config:/app/config
```
**Environment Configuration:**
```bash theme={null}
export OPENAI_API_KEY="your-openai-key"
export DEEPGRAM_API_KEY="your-deepgram-key"
export ELEVENLABS_API_KEY="your-elevenlabs-key"
export TRANSPORT_HOST="0.0.0.0"
export TRANSPORT_PORT="8765"
```
### Testing Pipeline
**Unit Testing:**
```python theme={null}
import pytest
from pipecat.pipeline.pipeline import Pipeline
@pytest.mark.asyncio
async def test_pipeline_latency():
pipeline = create_test_pipeline()
start_time = time.time()
# Test audio processing
result = await pipeline.process(test_audio)
latency = time.time() - start_time
assert latency < 0.5 # Target < 500ms
```
**Need help with Pipecat setup?** Check the [Pipecat documentation](https://docs.pipecat.ai) or contact [support@getusertrace.com](mailto:support@getusertrace.com).
# Retell AI Integration
Source: https://docs.getusertrace.com/voice/retell
Test voice agents built with Retell AI platform
Test your Retell AI-powered voice agents by connecting them to UserTrace. Our simulated users will interact with your Retell agents through realistic voice conversations.
## Getting Started
### 1. Connect Your Retell Agent
In the UserTrace dashboard, navigate to **Agent Setup** and select **Retell AI Integration**.
**Required Information:**
* Retell Agent ID
* Retell API Key
**Example:**
```
Retell Agent ID: agent_1234567890abcdef
```
### 2. Set Evaluation Context
Define your pass and fail criteria for voice interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Natural conversation flow
- Quick response times (< 2 seconds)
- Accurate understanding of user requests
- Professional and friendly tone
- Successful task completion
Fail Criteria:
- Robotic or unnatural speech
- Long response delays (> 5 seconds)
- Misunderstanding simple requests
- Inappropriate tone or language
- Failing to complete requested tasks
```
## Testing Process
### Inbound Agent Testing
For Retell agents that handle incoming calls:
1. Our test number calls your Retell agent
2. Retell processes the conversation using your agent configuration
3. The conversation follows the defined scenario
4. Results are analyzed and scored
### Outbound Agent Testing
For Retell agents that make outgoing calls:
1. UserTrace triggers your Retell agent via API
2. Your agent calls our designated test number (5-minute window)
3. Our simulated user responds according to the scenario
4. The conversation continues until completion
5. Results are analyzed and scored
**Key Requirements:**
* **Call our number when simulation starts** (5-minute window)
* **Use the exact Retell agent** you configured in setup
* **Ensure your agent is deployed** and accessible
## Retell-Specific Features
### Voice Configuration
**Voice Models:**
* Test different Retell voice options
* Verify voice consistency across calls
* Check pronunciation and clarity
**Language Support:**
* Test multilingual capabilities
* Verify accent and dialect handling
* Check language switching functionality
### LLM Integration
**Custom Prompts:**
* Test your agent's custom system prompts
* Verify behavior consistency
* Check edge case handling
**Function Calling:**
* Test API integrations through voice commands
* Verify parameter extraction accuracy
* Ensure proper error handling
### Real-Time Features
**Interruption Handling:**
* Test natural conversation interruptions
* Verify smooth resumption of speech
* Check context preservation
**Live Transcription:**
* Monitor real-time transcription accuracy
* Test with various accents and speaking speeds
* Verify noise handling capabilities
## Best Practices
**Response Optimization**
• Target \< 2 second response times
• Use Retell's streaming features
• Optimize LLM prompt length
• Monitor latency metrics
**Natural Interactions**
• Design clear conversation flows
• Handle ambiguous inputs gracefully
• Use appropriate filler words and pauses
• Test various speaking styles
## Common Scenarios
**Customer Service:**
* Support ticket resolution
* Order status inquiries
* Account management
* Technical troubleshooting
**Sales & Marketing:**
* Lead qualification calls
* Product demonstrations
* Follow-up conversations
* Appointment setting
**Healthcare & Wellness:**
* Appointment scheduling
* Symptom assessment
* Medication reminders
* Health surveys
## Troubleshooting
**Common Issues:**
**Retell Agent Issues:**
* **Agent not responding**: Check agent deployment status in Retell dashboard
* **Poor voice quality**: Verify voice model settings and network connection
* **Function calls failing**: Validate API endpoints and authentication
* **Inconsistent responses**: Review and optimize system prompts
**Call Connectivity:**
* **Calls not connecting**: Ensure phone number is properly configured
* **Audio issues**: Check Retell's audio settings and network quality
* **Dropped calls**: Monitor Retell's connection stability
* **Latency problems**: Optimize LLM response generation
## Advanced Configuration
### Custom Integrations
**Webhook Configuration:**
```json theme={null}
{
"webhook_url": "https://your-domain.com/retell-webhook",
"events": ["call_started", "call_ended", "function_called"],
"authentication": {
"type": "bearer_token",
"token": "your-webhook-secret"
}
}
```
**API Integration Example:**
```javascript theme={null}
// Trigger outbound call via Retell API
const response = await fetch('https://api.retellai.com/call', {
method: 'POST',
headers: {
'Authorization': 'Bearer ' + RETELL_API_KEY,
'Content-Type': 'application/json'
},
body: JSON.stringify({
agent_id: 'agent_1234567890abcdef',
to_number: '+1-555-TEST-NUM',
from_number: '+1-555-YOUR-NUM'
})
});
```
**Need help with Retell AI setup?** Check the [Retell AI documentation](https://docs.retellai.com) or contact [support@getusertrace.com](mailto:support@getusertrace.com).
# Telephony-Based Voice Agents
Source: https://docs.getusertrace.com/voice/telephony-based
Test voice agents that work over traditional phone systems
Test your voice agents that operate through traditional phone systems (PSTN, VoIP). UserTrace supports testing for both incoming and outgoing voice agents.
**Supported Regions**: We currently accept phone numbers from the United States, Singapore, and India.
## Getting Started
### 1. Connect Your Phone System
In the UserTrace dashboard, navigate to **Agent Setup** and select **Telephony Integration**.
**Required Information:**
* Phone number (with country code)
* Agent type (Incoming or Outgoing)
**Example:**
```
Phone Number: +1-555-123-4567
Agent Type: Incoming Agent
Region: United States
```
### 2. Set Evaluation Context
Define your pass and fail criteria for voice interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Answers within 3 rings
- Clear audio quality and speech
- Understands customer requests accurately
- Provides helpful solutions
- Professional and friendly tone
- Completes calls efficiently
Fail Criteria:
- No answer or long wait times
- Poor audio quality or unclear speech
- Misunderstands customer requests
- Provides incorrect information
- Unprofessional or inappropriate tone
- Call drops or technical issues
```
## Agent Types
### Incoming Voice Agents
For agents that answer incoming calls (traditional customer service):
**How It Works:**
1. You provide your agent's phone number
2. Our simulated user calls your number when simulation starts
3. Your agent answers and handles the conversation
4. The call continues based on the scenario
5. Results are analyzed after call completion
**Setup Requirements:**
* Ensure your phone system can accept calls from our test numbers
* Configure your agent to handle the expected call volume
* Test that your IVR (if any) allows reaching the agent quickly
### Outgoing Voice Agents
For agents that make outbound calls (sales, follow-ups, reminders):
**How It Works:**
1. You start a simulation in the UserTrace dashboard
2. Your agent calls our designated test number (shown in dashboard)
3. Our simulated user answers as the target persona
4. The conversation proceeds based on the scenario
5. Results are analyzed after call completion
**Key Requirements:**
* **Call our number when simulation starts** (5-minute window)
* **Call from the exact number** you registered in agent setup
* **We will only answer calls** from your registered number
* Have your agent ready to make the call within the time window
**Dashboard Information:**
```
Test Number to Call: +1-555-TEST-999
Your Registered Number: +1-555-123-4567
Status: Ready to Receive Call
```
## Advanced Setup: Webhook Triggers
For automated outgoing agents, we can send a webhook to trigger your system to make the call:
**Webhook Format:**
```json theme={null}
{
"trigger": "start_outbound_call",
"scenario_id": "sales_followup",
"target_number": "+1-555-TEST-999",
"call_from": "+1-555-123-4567",
"metadata": {
"persona": "interested_prospect",
"context": "product_demo_followup",
"expected_duration": "5-10 minutes"
}
}
```
**Your Response:**
Your system should acknowledge the webhook and initiate the call within the 5-minute window:
```json theme={null}
{
"status": "accepted",
"estimated_call_time": "30 seconds",
"call_id": "your-internal-call-id"
}
```
## Best Practices
**Audio & Connection**
• Ensure clear audio quality
• Test your phone system beforehand
• Have backup numbers ready
• Monitor call completion rates
**Timing Requirements**
• Answer incoming calls within 3 rings
• Make outgoing calls within 5-minute window
• Handle natural conversation pauses
• Don't rush through responses
## Testing Process
### For Incoming Agents
1. **Start Simulation**: Click "Run Test" in UserTrace dashboard
2. **Incoming Call**: Our simulated user calls your number
3. **Answer Promptly**: Your agent should answer within 3 rings
4. **Handle Conversation**: Respond based on your training/prompts
5. **Natural Completion**: Let the conversation end naturally
6. **Review Results**: Analyze call transcript and scoring
### For Outgoing Agents
1. **Start Simulation**: Click "Run Test" in UserTrace dashboard
2. **Call Instructions**: Dashboard shows the number to call
3. **Make the Call**: Your agent calls within 5-minute window
4. **We Answer**: Our simulated user picks up as the target persona
5. **Handle Conversation**: Your agent leads the conversation
6. **Natural Completion**: Complete the call objective
7. **Review Results**: Analyze call performance and outcomes
## Common Scenarios
**Customer Service (Incoming):**
* "I need help with my order"
* "My service isn't working properly"
* "I want to cancel my subscription"
* "Can you help me with billing?"
**Sales & Outreach (Outgoing):**
* Follow-up after product demo
* Appointment scheduling calls
* Customer satisfaction surveys
* Payment reminder calls
## Troubleshooting
**Common Issues:**
**Incoming Agents:**
* **No answer**: Check if your phone system is configured correctly
* **Busy signal**: Ensure sufficient line capacity
* **Poor quality**: Test your telecom provider's connection
**Outgoing Agents:**
* **Call not connecting**: Verify you're calling the exact test number
* **Call rejected**: Ensure you're calling from your registered number
* **No answer from us**: Check if you're calling within the 5-minute window
## Regional Setup
**Phone Format**: +1-XXX-XXX-XXXX
**Example**: +1-555-123-4567
**Phone Format**: +65-XXXX-XXXX
**Example**: +65-6123-4567
**Phone Format**: +91-XXXXX-XXXXX
**Example**: +91-98765-43210
**Need help with telephony setup?** Our team can assist with phone system configuration and regional requirements. Contact [support@getusertrace.com](mailto:support@getusertrace.com).
# Vapi Integration
Source: https://docs.getusertrace.com/voice/vapi
Test voice agents built with the Vapi platform
Test your Vapi-powered voice agents by connecting them to UserTrace. Our simulated users will interact with your Vapi assistants through realistic voice conversations.
## Getting Started
### 1. Connect Your Vapi Assistant
In the UserTrace dashboard, navigate to **Agent Setup** and select **Vapi Integration**.
**Required Information:**
* Vapi Assistant ID
* Vapi API Token
**Example:**
```
Vapi Assistant ID: 01234567-89ab-cdef-0123-456789abcdef
```
### 2. Set Evaluation Context
Define your pass and fail criteria for voice interactions:
**Example Pass/Fail Criteria:**
```
Pass Criteria:
- Responds within 3 seconds of user input
- Uses clear, natural speech patterns
- Correctly understands user intent
- Provides accurate information
- Maintains conversational flow
Fail Criteria:
- Response time exceeds 10 seconds
- Speech is unclear or robotic
- Misunderstands simple requests
- Provides incorrect information
- Abruptly ends conversations
```
## Testing Process
### Inbound Agent Testing
For Vapi assistants that answer incoming calls:
1. Our test number calls your Vapi assistant
2. Vapi processes the conversation according to your assistant configuration
3. The conversation continues based on the scenario
4. Results are analyzed and scored
### Outbound Agent Testing
For Vapi assistants that make outgoing calls:
1. UserTrace triggers your Vapi assistant to initiate a call
2. Your assistant calls our designated test number (5-minute window)
3. Our simulated user responds according to the scenario
4. The conversation continues until completion
5. Results are analyzed and scored
**Key Requirements:**
* **Call our number when simulation starts** (5-minute window)
* **Use the exact Vapi assistant** you configured in setup
* **Ensure your assistant is active** and properly configured
## Vapi-Specific Features
### Assistant Configuration
**Voice Settings:**
* Leverage Vapi's voice synthesis options
* Configure speech speed and tone
* Set up custom pronunciation rules
**Function Calling:**
* Test your custom functions through voice commands
* Verify function parameters are correctly extracted
* Ensure proper error handling for failed functions
**Knowledge Base Integration:**
* Test retrieval of information from your knowledge base
* Verify accurate responses to domain-specific questions
* Check handling of out-of-scope queries
### Advanced Features
**Interruption Handling:**
* Test how your assistant handles user interruptions
* Verify smooth conversation flow resumption
* Check for appropriate acknowledgment of interruptions
**Transfer and Escalation:**
* Test handoff scenarios to human agents
* Verify proper context preservation during transfers
* Check escalation triggers work correctly
## Best Practices
**Optimize Speech**
• Use Vapi's voice tuning features
• Test different voice models
• Optimize for your use case
• Monitor speech clarity scores
**Timing Requirements**
• Target \< 3 seconds response time
• Handle processing delays gracefully
• Use Vapi's streaming features
• Implement proper timeout handling
## Common Scenarios
**Customer Support:**
* Technical troubleshooting calls
* Account assistance requests
* Product information inquiries
* Complaint resolution
**Sales & Lead Qualification:**
* Initial prospect screening
* Product demonstrations
* Quote discussions
* Follow-up conversations
**Appointment Scheduling:**
* Booking confirmations
* Rescheduling requests
* Reminder calls
* Cancellation handling
## Troubleshooting
**Common Issues:**
**Vapi Connection:**
* **Assistant not responding**: Verify Vapi assistant is active and published
* **Poor audio quality**: Check Vapi voice model settings
* **Function calls failing**: Validate function schemas and endpoints
* **Knowledge base errors**: Verify knowledge base is properly indexed
**Call Flow:**
* **Calls not connecting**: Ensure phone number is correctly configured in Vapi
* **Unexpected disconnections**: Check Vapi's call duration limits
* **Transfer failures**: Verify transfer phone numbers and settings
**Need help with Vapi setup?** Check the [Vapi documentation](https://docs.vapi.ai) or contact [support@getusertrace.com](mailto:support@getusertrace.com).
# WebRTC-Based Voice Agents
Source: https://docs.getusertrace.com/voice/webrtc-based
Test custom voice agents through WebRTC API endpoints
Test your custom voice agents by hosting them in a sandbox environment with WebRTC support. UserTrace will interact with your voice agent using WebRTC protocols for real-time voice communication.
## API Requirements
Your sandbox must expose a WebRTC-compatible endpoint, plus a health check endpoint.
### 1. WebRTC Voice Endpoint
**Endpoint:** `WebSocket: wss://your-domain.com/voice`
**Connection Handshake:**
```json theme={null}
{
"type": "start_call",
"user_id": "sim_user_abc123",
"session_id": "session_xyz789",
"metadata": {
"scenario_id": "customer_support",
"persona": "frustrated_customer",
"audio_config": {
"codec": "opus",
"sampleRate": 48000
}
}
}
```
**Response Format:**
```json theme={null}
{
"type": "call_ready",
"session_id": "session_xyz789",
"call_id": "call_456def",
"webrtc_offer": {
"type": "offer",
"sdp": "v=0\r\no=- 123456789 123456789 IN IP4..."
},
"metadata": {
"agent_ready": true,
"audio_codec": "opus",
"estimated_setup_time": "2s"
},
"evaluation_metadata": {
"connection_quality": "excellent",
"setup_success": true,
"audio_path_established": true
}
}
```
### 2. Health Check Endpoint
**Endpoint:** `GET /health`
**Request:**
```bash theme={null}
curl -X GET https://your-sandbox.com/health
```
**Response Format:**
```json theme={null}
{
"status": "healthy",
"timestamp": "2024-01-21T10:30:00Z",
"version": "1.0.0",
"webrtc_status": "operational"
}
```
## Advanced Configuration
### Optional Parameters
**user\_id** and **session\_id** are optional in requests. If your agent doesn't require session management, you can omit them.
**metadata** is optional and can contain any scenario or context information your agent needs.
**Request with minimal parameters:**
```json theme={null}
{
"type": "start_call",
"audio_config": {
"codec": "opus"
}
}
```
**Request with full parameters:**
```json theme={null}
{
"type": "start_call",
"user_id": "sim_user_abc123",
"session_id": "session_xyz789",
"metadata": {
"scenario_id": "customer_support",
"persona": "frustrated_customer",
"context": {
"issue_type": "billing",
"urgency": "high",
"customer_tier": "premium"
}
}
}
```
## Implementation Guide
### 1. Set Up Your Sandbox
Deploy your voice agent to a publicly accessible WebSocket endpoint with WSS (secure WebSocket) support.
**Required Headers:**
* `Sec-WebSocket-Protocol: voice-agent`
* `Authorization: Bearer ` (best practice for security)
### 2. Test Your Endpoints
**WebRTC Connection Test:**
```bash theme={null}
curl --include \
--no-buffer \
--header "Connection: Upgrade" \
--header "Upgrade: websocket" \
--header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
--header "Sec-WebSocket-Version: 13" \
--header "Sec-WebSocket-Protocol: voice-agent" \
wss://your-sandbox.com/voice
```
**Health Check Test:**
```bash theme={null}
curl -X GET https://your-sandbox.com/health
```
### 3. Configure in UserTrace
1. **Add your sandbox URL** in the UserTrace dashboard
2. **Set authentication** if required (API keys, tokens)
3. **Configure audio settings** (codec preferences, quality settings)
4. **Test connection** using the built-in connectivity checker
## Best Practices
**Response Optimization**
• Target \< 200ms audio latency
• Support 100 concurrent connections
• Implement proper error handling
• Add connection quality monitoring
**WebRTC Security**
• Use WSS (secure WebSocket) only
• Implement DTLS for media encryption
• Validate all signaling messages
• Use secure TURN servers
## Error Handling
Your API should return appropriate WebSocket close codes and error messages:
**Connection Errors:**
```json theme={null}
{
"type": "error",
"error": {
"code": "connection_failed",
"message": "Failed to establish WebRTC connection",
"details": "ICE connectivity check failed"
}
}
```
**Audio Errors:**
```json theme={null}
{
"type": "error",
"error": {
"code": "audio_error",
"message": "Audio stream initialization failed",
"details": "Unsupported codec: G.722"
}
}
```
## Voice Processing Support
For agents that handle voice conversations, your response should include both the voice interaction results and conversation metadata:
**Voice Response Example:**
```json theme={null}
{
"type": "conversation_update",
"session_id": "session_xyz789",
"call_id": "call_456def",
"transcript": {
"user": "Hi, I need help with my billing issue",
"agent": "I'd be happy to help you with your billing. Can you provide your account number?"
},
"metadata": {
"audio_quality": "excellent",
"response_time_ms": 1800,
"confidence_score": 0.94
},
"evaluation_metadata": {
"understood_intent": true,
"appropriate_response": true,
"tone": "professional",
"next_action": "gather_account_info"
}
}
```
## Common Integration Patterns
* **Speech Recognition**: Real-time transcription with confidence scores
* **Voice Synthesis**: Text-to-speech with natural voice generation
* **Intent Detection**: Understanding user requests from voice input
* **Context Management**: Maintaining conversation state across turns
* **Quality Monitoring**: Real-time audio quality and connection metrics
## Troubleshooting
**Common Issues:**
* **Connection not establishing**: Check WebSocket endpoint and SSL certificates
* **Audio not flowing**: Verify WebRTC signaling and ICE connectivity
* **Poor audio quality**: Optimize codec settings and network configuration
* **High latency**: Review server location and network routing
**Need help with WebRTC implementation?** Our team can assist with technical setup and optimization. Contact [support@getusertrace.com](mailto:support@getusertrace.com).