Test agents with evals

An eval tests an agent with simulated calls. A synthetic caller plays each scenario of a suite against the agent, and a judge model reads every transcript and scores it against your test cases. One run of a suite against one agent version is an eval submission.

Piece	What it holds	Endpoint
Judge	The model, temperature and prompt that grade calls. Reusable across suites.	`/v1/projects/{projectId}/eval-judges`
Suite	Scenarios (what the caller says and wants) and the test cases to check.	`/v1/projects/{projectId}/eval-suites`
Scenario	A caller persona and goal, with its own test cases. Authored inside a suite.	`/v1/projects/{projectId}/eval-scenarios` (read only)
Test case	One thing to check: when it applies, what passes, what fails. Authored inside a suite.	`/v1/projects/{projectId}/eval-test-cases` (read only)
Submission	One run: a suite, a judge and one agent version.	`/v1/projects/{projectId}/agents/{agentId}/eval-submissions`

Eval calls are real conversations with origin: "eval". Hide them from your call lists with is_eval=false (see Conversations).

Run an eval

Create a judge

Start from one of Clarity’s judge prompts. GET /eval-judges/templates returns them with their name, description and template; full is the recommended one.

TEMPLATE=$(curl -s https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-judges/templates \
  -H "Authorization: Bearer $CLARITY_API_KEY" \
  | jq -r '.presets[] | select(.name == "full") | .template')

jq -n --arg t "$TEMPLATE" '{
  name: "Default judge",
  model: "gemini-2.5-flash",
  temperature: 0,
  prompt_template: $t
}' | curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-judges \
  -H "Authorization: Bearer $CLARITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d @-

The response is 201 with { "judge": {...} }. Keep judge.id.

Create a suite

curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-suites \
  -H "Authorization: Bearer $CLARITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Appointment changes",
    "scenarios": [
      {
        "name": "Move an appointment",
        "simulator_config": {
          "prompt": "You are Dana. You have a dental appointment on Tuesday at 9:00 and want to move it to Thursday morning. Give your name when asked.",
          "max_turns": 20
        },
        "test_cases": [
          {
            "name": "Offers Thursday slots",
            "when_relevant": "The caller asks to move the appointment to Thursday.",
            "success_criteria": "The agent looks up availability and offers at least one Thursday morning slot.",
            "failure_criteria": "The agent offers no Thursday slot, or invents one without checking."
          }
        ]
      }
    ],
    "common_test_cases": [
      {
        "name": "Polite close",
        "when_relevant": "The call reaches its end.",
        "success_criteria": "The agent confirms what was done before ending the call.",
        "failure_criteria": "The agent ends the call without confirming the outcome."
      }
    ]
  }'

The response is 201 with { "suite": {...} }. Keep suite.id.

Create the submission

curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/agents/$AGENT_ID/eval-submissions \
  -H "Authorization: Bearer $CLARITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Appointment changes 2026-10-22 14:00",
    "suite_id": "'"$SUITE_ID"'",
    "judge_id": "'"$JUDGE_ID"'",
    "runs_per_scenario": 3
  }'

The response is 201 with { "submission": {...} }. The submission is created paused: nothing runs yet.

Start it

curl -X POST https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/agents/$AGENT_ID/eval-submissions/$SUBMISSION_ID/run \
  -H "Authorization: Bearer $CLARITY_API_KEY"

The response is 202 with the submission, now status: "active". Clarity places the calls and judges each one as it ends.

Wait for it to finish

curl https://voice-api.onclarity.com/v1/projects/$PROJECT_ID/eval-submissions/$SUBMISSION_ID/status \
  -H "Authorization: Bearer $CLARITY_API_KEY"

{
  "state": "in_progress",
  "status": "active",
  "error_message": null,
  "runs_completed": 2,
  "runs_total": 3,
  "judge_counts": { "pending": 1, "in_progress": 0, "done": 1, "error": 0, "error_final": 0, "unusable": 0 }
}

The response also carries a progress breakdown, trimmed here. Poll every 15 to 30 seconds until state is completed or failed.

Submission states

status is what is stored: paused or active. state is the one to act on:

`state`	Meaning
`not_started`	Created, never run. Start it with `POST …/run`.
`in_progress`	Calls are being placed or judged.
`paused`	Paused part-way with no error. Resume with `POST …/run`.
`failed`	Stopped by a failure (`error_message` says why), or finished with every call’s judging in error.
`completed`	Every call has run and every verdict is in.

A submission pauses itself when it finishes, so a completed submission reads status: "paused".

Read the results

What	Request
One report with totals, per-scenario and per-test-case pass rates, latency and the top failures	`GET /agents/{agentId}/eval-submissions/{id}/report`
Every verdict with the judge’s reasoning	`GET /agents/{agentId}/eval-submissions/{id}/verdicts`
Per-call results: duration, turns, judging state	`GET /agents/{agentId}/eval-submissions/{id}/call-results`

Paths are under /v1/projects/{projectId}. If you only have the submission id, GET /eval-submissions/{id} and GET /eval-submissions/{id}/verdicts work without the agent id.

{
  "verdicts": [
    {
      "id": "6f0d2b8e-1c4a-4e9f-b7d3-2a5c8e1f0b96",
      "submission_id": "9a3e7c1b-2d4f-4a8e-b6c0-5f1d3e9a7b28",
      "call_id": "e3b1c9a0-7d24-4f8b-a6e5-2c9d0f1b8a37",
      "test_case_id": "1b7e4d2a-9c3f-4a6b-8e1d-0f5c2a7b9e43",
      "source": "scenario",
      "verdict": "passed",
      "reasoning": "The agent called the availability tool and offered Thursday at 9:30 and 11:00.",
      "error": null,
      "judged_at": "2026-10-22T14:06:12.000Z"
    }
  ]
}

verdict is passed, failed, not_relevant (the test case did not apply to this call), not_sure, or error (the judge could not score it; see error).
source is scenario for a scenario’s own test case and common for one of the suite’s common test cases.
Narrow the list with ?test_case_id= or ?call_id=. On a large run the unfiltered list is big.
call_id is the eval call. Its transcript and recording are in its conversation; see Conversations.

In the report, pass_rate counts passed out of passed + failed. pass_rate_wide also counts not_sure and error against the agent.

Reference

Judges

string

required

Up to 256 characters.

string

required

The judge model. It must be a model of the judge provider your Clarity deployment uses, for example gemini-2.5-flash. An unsupported model is refused with 422.

string

required

The grading prompt. It must contain the placeholders {name}, {when_relevant}, {success_criteria}, {failure_criteria} and {conversation_text}, or it is refused with 422. Start from a template.

number

default:"0"

From 0 to 2.

string

Free text.

Suites

string

required

Up to 256 characters.

array

required

1 to 20 entries. Each is either { "scenario_id": "…" } to reuse an existing scenario, or a new scenario:

name (required), description.
simulator_config.prompt (required): the instructions for the synthetic caller, written to the caller.
simulator_config.max_turns (required): 1 to 100.
simulator_config.close_instruction: when the caller should hang up. Absent means the caller hangs up only as a last resort.
test_cases (required): 1 to 50 entries.
silence_modes: any of off, always, flexible. Whether the caller may stay silent while the agent works, for example during a slow lookup. The scenario is run under each mode you list, so two modes double its calls.
field_overrides: values for the agent’s prompt variables during this scenario.

array

Up to 50 test cases checked on every scenario’s calls.

object

Values for the agent’s prompt variables on every call of the suite. A scenario’s own value wins. When the agent declares which variables its prompt may use, a name it does not declare is refused when you create the submission.

A test case is either { "test_case_id": "…" } or a new one with name, when_relevant, success_criteria and failure_criteria (all required) and an optional description. Edit a suite with PATCH /eval-suites/{id} using the same fields. Scenarios and test cases are only created and changed through suites; the scenario and test-case endpoints are read-only.

Submissions

string

required

Unique per agent. Reusing a name answers 409 submission_name_already_exists_for_agent, so add a timestamp.

string

required

The suite to run. It must have at least one scenario.

string

required

The judge that grades the calls.

integer

default:"1"

1 to 10. More runs give a steadier pass rate.

integer

default:"2"

1 to 10 calls at the same time.

boolean

default:"false"

Test the agent’s draft instead of its published version.

string

Test a specific version. Cannot be combined with use_draft.

string[]

Up to 50 more test cases to check on every call of this run.

string[]

Up to 50 of the suite’s common test cases to skip in this run.

integer

default:"5"

1 to 50 calls judged at the same time.

integer

default:"3"

0 to 10.

Synthetic caller settings

The platform picks the synthetic caller’s speech and language models from the agent’s language. You can override them with simulator_audio_stack, simulator_llm, simulator_silence_mode and simulator_model_config. Which values work depends on your deployment: a value it cannot run is refused with 422 rather than replaced. Leave them out unless Clarity support has told you which to use.

The submission body is strict: an unknown field is refused with 422. Which version runs. The version is fixed when you create the submission: the published version by default, the draft with use_draft, or the one you name with agent_version_id. Publishing while a run is in progress does not change it. An agent with no published version answers 409 no_published_version; pass use_draft: true to test its draft. use_draft on an agent with no draft answers 409. Changing a run. PATCH …/eval-submissions/{id} works only while the submission is paused. POST …/{id}/pause pauses an active run; POST …/{id}/run resumes it. DELETE …/{id} removes the submission. To run the same suite again, create a new submission.

Get started

Voice AI

Support and chat

SDKs

Run an eval

Submission states

Read the results

Reference

Judges

Suites

Submissions

​Run an eval

​Submission states

​Read the results

​Reference

​Judges

​Suites

​Submissions

Run an eval

Submission states

Read the results

Reference

Judges

Suites

Submissions