> ## Documentation Index > Fetch the complete documentation index at: https://docs.support.onclarity.com/llms.txt > Use this file to discover all available pages before exploring further. # Test agents with evals > Run a suite of simulated calls against one agent version and read the judge's verdict on every test case. An **eval** tests an agent with simulated calls. A synthetic caller plays each scenario of a **suite** against the agent, and a **judge** model reads every transcript and scores it against your **test cases**. One run of a suite against one agent version is an **eval submission**. | Piece | What it holds | Endpoint | | - | - | - | | Judge | The model, temperature and prompt that grade calls. Reusable across suites. | `/v1/projects/{projectId}/eval-judges` | | Suite | Scenarios (what the caller says and wants) and the test cases to check. | `/v1/projects/{projectId}/eval-suites` | | Scenario | A caller persona and goal, with its own test cases. Authored inside a suite. | `/v1/projects/{projectId}/eval-scenarios` (read only) | | Test case | One thing to check: when it applies, what passes, what fails. Authored inside a suite. | `/v1/projects/{projectId}/eval-test-cases` (read only) | | Submission | One run: a suite, a judge and one agent version. | `/v1/projects/{projectId}/agents/{agentId}/eval-submissions` | Eval calls are real conversations with `origin: "eval"`. Hide them from your call lists with `is_eval=false` (see [Conversations](/voice/conversations.md#filters)). ## Run an eval **1. Create a judge** Start from one of Clarity's judge prompts. `GET /eval-judges/templates` returns them with their `name`, `description` and `template`; `full` is the recommended one. ```bash theme={null} TEMPLATE=$(curl -s https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-judges/templates \ -H "Authorization: Bearer $CLARITY_API_KEY" \ | jq -r '.presets[] | select(.name == "full") | .template') jq -n --arg t "$TEMPLATE" '{ name: "Default judge", model: "gemini-2.5-flash", temperature: 0, prompt_template: $t }' | curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-judges \ -H "Authorization: Bearer $CLARITY_API_KEY" \ -H "Content-Type: application/json" \ -d @- ``` The response is `201` with `{ "judge": {...} }`. Keep `judge.id`. **2. Create a suite** ```bash theme={null} curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-suites \ -H "Authorization: Bearer $CLARITY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Appointment changes", "scenarios": [ { "name": "Move an appointment", "simulator_config": { "prompt": "You are Dana. You have a dental appointment on Tuesday at 9:00 and want to move it to Thursday morning. Give your name when asked.", "max_turns": 20 }, "test_cases": [ { "name": "Offers Thursday slots", "when_relevant": "The caller asks to move the appointment to Thursday.", "success_criteria": "The agent looks up availability and offers at least one Thursday morning slot.", "failure_criteria": "The agent offers no Thursday slot, or invents one without checking." } ] } ], "common_test_cases": [ { "name": "Polite close", "when_relevant": "The call reaches its end.", "success_criteria": "The agent confirms what was done before ending the call.", "failure_criteria": "The agent ends the call without confirming the outcome." } ] }' ``` The response is `201` with `{ "suite": {...} }`. Keep `suite.id`. **3. Create the submission** ```bash theme={null} curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/agents/$AGENT_ID/eval-submissions \ -H "Authorization: Bearer $CLARITY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Appointment changes 2026-10-22 14:00", "suite_id": "'"$SUITE_ID"'", "judge_id": "'"$JUDGE_ID"'", "runs_per_scenario": 3 }' ``` The response is `201` with `{ "submission": {...} }`. The submission is created **paused**: nothing runs yet. **4. Start it** ```bash theme={null} curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/agents/$AGENT_ID/eval-submissions/$SUBMISSION_ID/run \ -H "Authorization: Bearer $CLARITY_API_KEY" ``` The response is `202` with the submission, now `status: "active"`. Clarity places the calls and judges each one as it ends. **5. Wait for it to finish** ```bash theme={null} curl https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-submissions/$SUBMISSION_ID/status \ -H "Authorization: Bearer $CLARITY_API_KEY" ``` ```json theme={null} { "state": "in_progress", "status": "active", "error_message": null, "runs_completed": 2, "runs_total": 3, "judge_counts": { "pending": 1, "in_progress": 0, "done": 1, "error": 0, "error_final": 0, "unusable": 0 } } ``` The response also carries a `progress` breakdown, trimmed here. Poll every 15 to 30 seconds until `state` is `completed` or `failed`. ### Submission states `status` is what is stored: `paused` or `active`. `state` is the one to act on: | `state` | Meaning | | - | - | | `not_started` | Created, never run. Start it with `POST …/run`. | | `in_progress` | Calls are being placed or judged. | | `paused` | Paused part-way with no error. Resume with `POST …/run`. | | `failed` | Stopped by a failure (`error_message` says why), or finished with every call's judging in error. | | `completed` | Every call has run and every verdict is in. | A submission pauses itself when it finishes, so a `completed` submission reads `status: "paused"`. ## Read the results | What | Request | | - | - | | One report with totals, per-scenario and per-test-case pass rates, latency and the top failures | `GET /agents/{agentId}/eval-submissions/{id}/report` | | Every verdict with the judge's reasoning | `GET /agents/{agentId}/eval-submissions/{id}/verdicts` | | Per-call results: duration, turns, judging state | `GET /agents/{agentId}/eval-submissions/{id}/call-results` | Paths are under `/v1/projects/{projectId}`. If you only have the submission id, `GET /eval-submissions/{id}` and `GET /eval-submissions/{id}/verdicts` work without the agent id. ```json theme={null} { "verdicts": [ { "id": "6f0d2b8e-1c4a-4e9f-b7d3-2a5c8e1f0b96", "submission_id": "9a3e7c1b-2d4f-4a8e-b6c0-5f1d3e9a7b28", "call_id": "e3b1c9a0-7d24-4f8b-a6e5-2c9d0f1b8a37", "test_case_id": "1b7e4d2a-9c3f-4a6b-8e1d-0f5c2a7b9e43", "source": "scenario", "verdict": "passed", "reasoning": "The agent called the availability tool and offered Thursday at 9:30 and 11:00.", "error": null, "judged_at": "2026-10-22T14:06:12.000Z" } ] } ``` * `verdict` is `passed`, `failed`, `not_relevant` (the test case did not apply to this call), `not_sure`, or `error` (the judge could not score it; see `error`). * `source` is `scenario` for a scenario's own test case and `common` for one of the suite's common test cases. * Narrow the list with `?test_case_id=` or `?call_id=`. On a large run the unfiltered list is big. * `call_id` is the eval call. Its transcript and recording are in its conversation; see [Conversations](/voice/conversations.md). In the report, `pass_rate` counts `passed` out of `passed` + `failed`. `pass_rate_wide` also counts `not_sure` and `error` against the agent. ## Reference ### Judges **name** `string` *required* Up to 256 characters. **model** `string` *required* The judge model. It must be a model of the judge provider your Clarity deployment uses, for example `gemini-2.5-flash`. An unsupported model is refused with `422`. **prompt\_template** `string` *required* The grading prompt. It must contain the placeholders `{name}`, `{when_relevant}`, `{success_criteria}`, `{failure_criteria}` and `{conversation_text}`, or it is refused with `422`. Start from a template. **temperature** `number` (default: 0) From 0 to 2. **description** `string` Free text. ### Suites **name** `string` *required* Up to 256 characters. **scenarios** `array` *required* 1 to 20 entries. Each is either `{ "scenario_id": "…" }` to reuse an existing scenario, or a new scenario: * `name` (required), `description`. * `simulator_config.prompt` (required): the instructions for the synthetic caller, written to the caller. * `simulator_config.max_turns` (required): 1 to 100. * `simulator_config.close_instruction`: when the caller should hang up. Absent means the caller hangs up only as a last resort. * `test_cases` (required): 1 to 50 entries. * `silence_modes`: any of `off`, `always`, `flexible`. Whether the caller may stay silent while the agent works, for example during a slow lookup. The scenario is run under each mode you list, so two modes double its calls. * `field_overrides`: values for the agent's prompt variables during this scenario. **common\_test\_cases** `array` Up to 50 test cases checked on every scenario's calls. **field\_overrides** `object` Values for the agent's prompt variables on every call of the suite. A scenario's own value wins. When the agent declares which variables its prompt may use, a name it does not declare is refused when you create the submission. A test case is either `{ "test_case_id": "…" }` or a new one with `name`, `when_relevant`, `success_criteria` and `failure_criteria` (all required) and an optional `description`. Edit a suite with `PATCH /eval-suites/{id}` using the same fields. Scenarios and test cases are only created and changed through suites; the scenario and test-case endpoints are read-only. ### Submissions **name** `string` *required* Unique per agent. Reusing a name answers `409 submission_name_already_exists_for_agent`, so add a timestamp. **suite\_id** `string` *required* The suite to run. It must have at least one scenario. **judge\_id** `string` *required* The judge that grades the calls. **runs\_per\_scenario** `integer` (default: 1) 1 to 10. More runs give a steadier pass rate. **max\_parallel\_calls** `integer` (default: 2) 1 to 10 calls at the same time. **use\_draft** `boolean` (default: false) Test the agent's draft instead of its published version. **agent\_version\_id** `string` Test a specific version. Cannot be combined with `use_draft`. **extra\_common\_test\_case\_ids** `string[]` Up to 50 more test cases to check on every call of this run. **exclude\_common\_test\_case\_ids** `string[]` Up to 50 of the suite's common test cases to skip in this run. **judge\_concurrency** `integer` (default: 5) 1 to 50 calls judged at the same time. **judge\_max\_retries** `integer` (default: 3) 0 to 10. > **Synthetic caller settings** > > The platform picks the synthetic caller's speech and language models from the agent's language. You can override them with `simulator_audio_stack`, `simulator_llm`, `simulator_silence_mode` and `simulator_model_config`. Which values work depends on your deployment: a value it cannot run is refused with `422` rather than replaced. Leave them out unless Clarity support has told you which to use. The submission body is strict: an unknown field is refused with `422`. **Which version runs.** The version is fixed when you create the submission: the published version by default, the draft with `use_draft`, or the one you name with `agent_version_id`. Publishing while a run is in progress does not change it. An agent with no published version answers `409 no_published_version`; pass `use_draft: true` to test its draft. `use_draft` on an agent with no draft answers `409`. **Changing a run.** `PATCH …/eval-submissions/{id}` works only while the submission is paused. `POST …/{id}/pause` pauses an active run; `POST …/{id}/run` resumes it. `DELETE …/{id}` removes the submission. To run the same suite again, create a new submission. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.