> ## Documentation Index > Fetch the complete documentation index at: https://docs.support.onclarity.com/llms.txt > Use this file to discover all available pages before exploring further. # Create an eval run > Creates an eval run that calls this agent with the suite's simulated callers and grades each call with the chosen judge. The run is created **paused** and places no calls until you start it with `POST …/eval-submissions/{id}/run`. By default it tests the agent's current published version; the version is fixed when the run is created, so publishing later does not change a run in progress. ## OpenAPI ````yaml /api-reference/voice-openapi.json post /v1/projects/{projectId}/agents/{agentId}/eval-submissions openapi: 3.0.3 info: title: Clarity Voice API version: '1.0' description: Voice agents, outbound calls, conversations, campaigns, and evals. servers: - url: https://voice-api.onclarity.com security: - organizationApiKey: [] tags: - name: Projects description: >- The projects your organization owns. Every other resource lives in a project. - name: Agents description: Create, configure, version and publish voice agents. - name: Outbound calls description: Have a published agent phone a number now. - name: Conversations description: >- One record per call session, with its calls, recordings, transcripts and artifacts. - name: Calls description: The calls of a project, with their status, direction, timing and outcome. - name: Campaigns description: Outbound calling campaigns that phone a list of numbers on a schedule. - name: Eval suites description: >- Reusable sets of eval scenarios and test cases to measure an agent against. - name: Eval judges description: The LLM judges that score eval calls against your test cases. - name: Eval scenarios description: The simulated-caller scenarios of your eval suites. - name: Eval test cases description: The checks a judge applies to each eval call. - name: Eval submissions description: Run an eval suite against an agent and read the verdicts. paths: /v1/projects/{projectId}/agents/{agentId}/eval-submissions: parameters: - $ref: '#/components/parameters/ProjectIdPath' - name: agentId in: path required: true schema: type: string description: Agent id post: tags: - Eval submissions summary: Create an eval run description: >- Creates an eval run that calls this agent with the suite's simulated callers and grades each call with the chosen judge. The run is created **paused** and places no calls until you start it with `POST …/eval-submissions/{id}/run`. By default it tests the agent's current published version; the version is fixed when the run is created, so publishing later does not change a run in progress. operationId: createAgentEvalSubmission requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalSubmissionCreateRequest' example: name: Booking regression 2026-10-22 14:00 suite_id: suite_01 judge_id: judge_01 runs_per_scenario: 2 max_parallel_calls: 2 responses: '201': description: The run was created, paused content: application/json: schema: $ref: '#/components/schemas/EvalSubmissionEnvelope' example: submission: id: sub_01J9 project_id: proj_123 created_by: null agent_id: agent_456 judge_id: judge_01 suite_id: suite_01 name: Booking regression 2026-10-22 14:00 description: null runs_per_scenario: 2 judge_concurrency: 5 judge_max_retries: 3 max_parallel_calls: 2 status: paused use_isolated_data: false simulator_audio_stack: hamsa simulator_llm: gemini simulator_silence_mode: 'off' simulator_model_config: null agent_version_id: ver_789 agent_version_number: 12 error_message: null created_at: '2026-10-22T14:00:00.000Z' updated_at: '2026-10-22T14:00:00.000Z' judge: id: judge_01 name: Default judge description: null model: gemini-3.5-flash prompt_template: … temperature: 0 suite: id: suite_01 name: Booking flows description: null scenario_ids: - scn_01 common_test_case_ids: - tc_7f3a scenarios: - id: scn_01 name: Caller books for tomorrow description: null simulator_config: prompt: You want to book a cleaning for tomorrow morning. max_turns: 20 test_case_ids: [] test_cases: [] common_test_cases: - id: tc_7f3a project_id: proj_123 name: Confirms the booking description: null when_relevant: The caller asked to book an appointment. success_criteria: >- The agent repeats the date and time back and the caller confirms. failure_criteria: >- The agent ends the call without confirming the date and time. created_by: null created_at: '2026-10-01T09:00:00.000Z' updated_at: '2026-10-01T09:00:00.000Z' runs_completed: 0 runs_total: 2 judge_counts: pending: 0 in_progress: 0 done: 0 error: 0 error_final: 0 unusable: 0 '400': description: >- The suite or one of its scenarios sets fields the agent does not declare, or the agent's architecture is no longer supported. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: >- eval suite uses field overrides not declared in agent.prompt_field_whitelist: "lead_source" (from suite "Booking flows"). Either add the keys under .system or .custom on the agent, or remove them from the suite/scenario field_overrides. offenders: - key: lead_source source: suite "Booking flows" '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': description: >- The project, agent, suite, judge, agent version or one of the listed test cases does not exist. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: Eval suite not found in this project suite_id: suite_01 '409': description: >- The suite has no scenarios; `use_draft` was set but the agent has no draft (`code: no_draft`); the agent has never been published and neither `use_draft` nor `agent_version_id` was given (`code: no_published_version`); or this agent already has a run with that name. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: >- This agent has no published version — run the eval on its draft explicitly (use_draft: true) or publish it first code: no_published_version '422': description: >- Validation failed; the simulated-caller settings are not available on your deployment; or the agent is a Voice Assist agent (`code: AGENT_TYPE_NOT_DISPATCHABLE`). content: application/json: schema: $ref: '#/components/schemas/ValidationError' example: error: use_draft and agent_version_id are mutually exclusive issues: - path: '' code: custom message: use_draft and agent_version_id are mutually exclusive components: parameters: ProjectIdPath: name: projectId in: path required: true schema: type: string description: Project id schemas: EvalSubmissionCreateRequest: type: object required: - name - suite_id - judge_id additionalProperties: false description: >- A run of one eval suite against this agent, graded by one judge. By default the run tests the agent's current published version; set `use_draft` to test the draft, or `agent_version_id` to test a specific version. `use_draft` and `agent_version_id` cannot be combined. properties: name: type: string minLength: 1 maxLength: 256 description: >- Unique per agent. Adding a date or time is the usual way to name repeated runs. description: type: string suite_id: type: string minLength: 1 description: The eval suite to run judge_id: type: string minLength: 1 description: The judge that grades each call runs_per_scenario: type: integer minimum: 1 maximum: 10 default: 1 description: How many calls to place for each scenario. judge_concurrency: type: integer minimum: 1 maximum: 50 default: 5 description: How many calls are graded at once. judge_max_retries: type: integer minimum: 0 maximum: 10 default: 3 description: Retries when a grading request fails. max_parallel_calls: type: integer minimum: 1 maximum: 10 default: 2 description: >- Most calls this run has in progress at once. A platform-wide limit also applies. extra_common_test_case_ids: type: array maxItems: 50 default: [] items: type: string minLength: 1 description: >- Test cases to check on every call of this run, in addition to the suite's own. exclude_common_test_case_ids: type: array maxItems: 50 default: [] items: type: string minLength: 1 description: Suite-wide test cases to leave out of this run. use_isolated_data: type: boolean default: false description: >- Give each call its own temporary copy of the Google Sheets the agent's tools use, so calls cannot change the real sheets or each other's data. Has no effect when no tool declares a sheet. simulator_audio_stack: type: string enum: - hamsa - humain - hamsa_selfhosted description: >- Speech recognition and synthesis used by the simulated caller. Omitted: the default for your deployment. A value your deployment cannot provide is refused with 422. simulator_llm: type: string enum: - selfhosted - gemini description: >- Language model behind the simulated caller. Omitted: the default for your deployment. A value your deployment cannot provide is refused with 422. simulator_silence_mode: type: string enum: - 'off' - always - flexible description: >- Whether the simulated caller may stay silent to wait for the agent: `off` (default) never, `always` whenever waiting is the right next step, `flexible` when a reasonable person would. simulator_model_config: $ref: '#/components/schemas/EvalSubmissionCallerModelConfig' use_draft: type: boolean default: false description: Test the agent's current draft instead of its published version. agent_version_id: type: string minLength: 1 description: Test this specific version of the agent (any status). EvalSubmissionEnvelope: type: object required: - submission properties: submission: $ref: '#/components/schemas/EvalSubmission' Error: type: object required: - error additionalProperties: true properties: error: type: string description: Error message code: type: string description: >- A stable machine-readable reason, present on the refusals a client may branch on. Among them: `PUBLISH_CAPABILITY_REQUIRED` (403, the credential may not change what is live), `AGENT_NOT_PUBLISHED` (409, the agent has no published version yet), `AGENT_TYPE_NOT_DISPATCHABLE` (422, a Voice Assist agent named where an agent must place or take a call; the body carries `agent_type` and `agent_id`), `NOT_APPLICABLE_TO_AGENT_TYPE` (422, a voice-only setting written on a Voice Assist agent), `AGENT_TYPE_BOUNDARY` (409), `AGENT_TYPE_NOT_SUPPORTED` (400), `VOICE_ASSISTANT_REQUIRED` (403, the organization is not licensed for Voice Assist), `ORGANIZATION_UNAVAILABLE` (503) and `REGION_PROFILE_CONFLICT` (503). ValidationError: type: object description: >- A validation failure. `error` is a human-readable summary; `issues` (request validation) or `details` (agent configuration validation) locate each problem. properties: error: type: string message: type: string issues: type: array items: type: object properties: path: type: string description: Dotted path of the offending field code: type: string message: type: string details: type: array items: {} EvalSubmissionCallerModelConfig: type: object description: >- Optional models for the simulated caller, one slot each for speech recognition, language model and speech synthesis. Each slot you include must name both a provider and a model; a slot you leave out keeps the default for the run. Choose values from the simulated-caller options of GET /v1/agents/chained-options. properties: asr: type: object required: - asr_provider - asr_model properties: asr_provider: type: string minLength: 1 asr_model: type: string minLength: 1 llm: type: object required: - llm_provider - llm_model properties: llm_provider: type: string minLength: 1 llm_model: type: string minLength: 1 tts: type: object required: - tts_provider - tts_model properties: tts_provider: type: string minLength: 1 tts_model: type: string minLength: 1 tts_voice: type: string minLength: 1 description: Catalog voice name tts_dialect: type: string minLength: 1 description: Catalog dialect code, e.g. `us` or `ksa` EvalSubmission: type: object description: >- An eval run: one suite run against one agent version and graded by one judge, with the suite, judge, scenarios and test cases it uses and its progress so far. required: - id - agent_id - suite_id - judge_id - name - status properties: id: type: string project_id: type: string created_by: type: string nullable: true agent_id: type: string judge_id: type: string suite_id: type: string name: type: string description: type: string nullable: true runs_per_scenario: type: integer judge_concurrency: type: integer judge_max_retries: type: integer max_parallel_calls: type: integer status: type: string enum: - active - paused description: >- `paused` runs place no calls. New runs start paused; start one with POST …/run. A run pauses itself when it finishes or when it cannot continue (see `error_message`). use_isolated_data: type: boolean simulator_audio_stack: type: string enum: - hamsa - humain - hamsa_selfhosted simulator_llm: type: string enum: - selfhosted - gemini simulator_silence_mode: type: string enum: - 'off' - always - flexible simulator_model_config: allOf: - $ref: '#/components/schemas/EvalSubmissionCallerModelConfig' nullable: true agent_version_id: type: string nullable: true description: >- The agent version this run tests. Null for old runs, or when the tested draft was discarded. agent_version_number: type: integer nullable: true error_message: type: string nullable: true description: >- Why the run paused itself, when it did. Null after a manual pause or a clean finish. created_at: type: string format: date-time nullable: true updated_at: type: string format: date-time nullable: true judge: type: object description: The judge, as stored. properties: id: type: string name: type: string description: type: string nullable: true model: type: string prompt_template: type: string temperature: type: number suite: type: object description: The suite, as stored. properties: id: type: string name: type: string description: type: string nullable: true scenario_ids: type: array items: type: string common_test_case_ids: type: array items: type: string scenarios: type: array description: The suite's scenarios in order, each with its own test cases. items: type: object properties: id: type: string name: type: string description: type: string nullable: true simulator_config: type: object test_case_ids: type: array items: type: string test_cases: type: array items: $ref: '#/components/schemas/EvalSubmissionTestCase' common_test_cases: type: array description: Test cases checked on every call of this run. items: $ref: '#/components/schemas/EvalSubmissionTestCase' runs_completed: type: integer description: Calls that have completed. runs_total: type: integer description: Calls the run will place in total. judge_counts: $ref: '#/components/schemas/EvalSubmissionJudgeCounts' EvalSubmissionTestCase: type: object description: A test case (a check the judge applies to a call). properties: id: type: string project_id: type: string name: type: string description: type: string nullable: true when_relevant: type: string success_criteria: type: string failure_criteria: type: string created_by: type: string nullable: true created_at: type: string format: date-time nullable: true updated_at: type: string format: date-time nullable: true EvalSubmissionJudgeCounts: type: object description: Calls of this run by grading state. properties: pending: type: integer in_progress: type: integer done: type: integer error: type: integer description: Grading failed; includes failures that will still be retried. error_final: type: integer description: Part of `error` that will not be retried. unusable: type: integer description: Part of `done` where every check came back as an error. responses: Unauthorized: description: No credential was sent, or the API key is invalid or revoked. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: Invalid or revoked token Forbidden: description: >- The credential carries no access to this project, or the operation needs administrative rights content: application/json: schema: $ref: '#/components/schemas/Error' securitySchemes: organizationApiKey: type: http scheme: bearer description: >- Your organization API key (it starts with `anek_org_`). The same key works on the Voice API and the Support API. See [Authentication](/authentication). ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.