> ## Documentation Index > Fetch the complete documentation index at: https://docs.support.onclarity.com/llms.txt > Use this file to discover all available pages before exploring further. # Create an eval suite > Creates a suite in one request. Each scenario and test case is either a reference to an existing one (`scenario_id` / `test_case_id`) or a new one written inline, which is created with the suite. Everything is created together or not at all. ## OpenAPI ````yaml /api-reference/voice-openapi.json post /v1/projects/{projectId}/eval-suites openapi: 3.0.3 info: title: Clarity Voice API version: '1.0' description: Voice agents, outbound calls, conversations, campaigns, and evals. servers: - url: https://voice-api.onclarity.com security: - organizationApiKey: [] tags: - name: Projects description: >- The projects your organization owns. Every other resource lives in a project. - name: Agents description: Create, configure, version and publish voice agents. - name: Outbound calls description: Have a published agent phone a number now. - name: Conversations description: >- One record per call session, with its calls, recordings, transcripts and artifacts. - name: Calls description: The calls of a project, with their status, direction, timing and outcome. - name: Campaigns description: Outbound calling campaigns that phone a list of numbers on a schedule. - name: Eval suites description: >- Reusable sets of eval scenarios and test cases to measure an agent against. - name: Eval judges description: The LLM judges that score eval calls against your test cases. - name: Eval scenarios description: The simulated-caller scenarios of your eval suites. - name: Eval test cases description: The checks a judge applies to each eval call. - name: Eval submissions description: Run an eval suite against an agent and read the verdicts. paths: /v1/projects/{projectId}/eval-suites: parameters: - $ref: '#/components/parameters/ProjectIdPath' post: tags: - Eval suites summary: Create an eval suite description: >- Creates a suite in one request. Each scenario and test case is either a reference to an existing one (`scenario_id` / `test_case_id`) or a new one written inline, which is created with the suite. Everything is created together or not at all. operationId: createEvalSuite requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalSuiteCreateRequest' example: name: Booking golden path description: Happy-path bookings and a reschedule scenarios: - name: Book a table for two simulator_config: prompt: >- You are Sara. You want a table for two tomorrow at 8 pm. Give your phone number when asked. max_turns: 20 test_cases: - name: Confirms the booking when_relevant: The caller asks to book a table. success_criteria: >- The agent repeats date, time and party size and confirms the booking. failure_criteria: The agent ends the call without confirming the details. - scenario_id: b2a4c6e8-0f1d-4e3b-9c5a-7e9f1b3d5c7a common_test_cases: - test_case_id: c3d5e7f9-1a2b-4c3d-8e4f-5a6b7c8d9e0f responses: '201': description: The created suite content: application/json: schema: type: object required: - suite properties: suite: $ref: '#/components/schemas/EvalSuite' example: suite: id: 7d1c2f8e-3b0a-4c55-9a1e-2f4b6c8d0e11 project_id: 4f3a9b2c-1d2e-4f5a-8b6c-7d8e9f0a1b2c name: Booking golden path description: Happy-path bookings and a reschedule scenario_ids: - a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d - b2a4c6e8-0f1d-4e3b-9c5a-7e9f1b3d5c7a common_test_case_ids: - c3d5e7f9-1a2b-4c3d-8e4f-5a6b7c8d9e0f sheet_refs: {} field_overrides: {} scenarios: [] common_test_cases: [] submissions_count: 0 created_by: org-token:42 created_at: '2026-10-22T10:00:00.000Z' updated_at: '2026-10-22T10:00:00.000Z' deleted_at: null '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': description: >- The project, or a referenced scenario or test case, does not exist in this project content: application/json: schema: $ref: '#/components/schemas/Error' example: error: One or more scenario_ids not found in this project missing_scenario_ids: - b2a4c6e8-0f1d-4e3b-9c5a-7e9f1b3d5c7a '409': description: A suite with this name already exists in the project content: application/json: schema: $ref: '#/components/schemas/Error' example: error: >- An eval suite named "Booking golden path" already exists in this project '422': $ref: '#/components/responses/ValidationFailed' components: parameters: ProjectIdPath: name: projectId in: path required: true schema: type: string description: Project id schemas: EvalSuiteCreateRequest: type: object required: - name - scenarios properties: name: type: string minLength: 1 maxLength: 256 description: Unique within the project. description: type: string nullable: true scenarios: type: array minItems: 1 maxItems: 20 items: $ref: '#/components/schemas/EvalScenarioEntry' common_test_cases: type: array maxItems: 50 default: [] items: $ref: '#/components/schemas/EvalTestCaseEntry' sheet_refs: type: object additionalProperties: type: string minLength: 1 description: Sheet replacements for eval calls; see `EvalSuite.sheet_refs`. field_overrides: type: object additionalProperties: type: string minLength: 1 description: >- Prompt field values for every call of the suite. Checked against the agent's overridable fields when a submission is created. EvalSuite: type: object description: >- A named, reusable set of scenarios plus test cases that apply to every scenario. An eval submission runs one suite against one agent. required: - id - project_id - name - scenario_ids - common_test_case_ids - scenarios - common_test_cases - submissions_count properties: id: type: string project_id: type: string name: type: string maxLength: 256 description: type: string nullable: true scenario_ids: type: array items: type: string common_test_case_ids: type: array items: type: string sheet_refs: type: object additionalProperties: type: string description: >- Replaces the agent's data sheets during this suite's eval calls, so evals do not read or write production data. Keys are the agent's sheet argument names (for example `ORDERS_SHEET_ID`), values the sheet ids to use instead. Empty means the agent's own sheets. field_overrides: type: object additionalProperties: type: string description: >- Values for the agent's prompt fields used in every call of this suite. scenarios: type: array items: $ref: '#/components/schemas/EvalScenario' common_test_cases: type: array description: Test cases applied to every scenario of the suite. items: $ref: '#/components/schemas/EvalTestCase' submissions_count: type: integer description: How many eval submissions have run this suite. created_by: type: string nullable: true created_at: type: string format: date-time nullable: true updated_at: type: string format: date-time nullable: true deleted_at: type: string format: date-time nullable: true description: >- Set when the suite was deleted. Deleted suites still resolve for the submissions that ran them. Error: type: object required: - error additionalProperties: true properties: error: type: string description: Error message code: type: string description: >- A stable machine-readable reason, present on the refusals a client may branch on. Among them: `PUBLISH_CAPABILITY_REQUIRED` (403, the credential may not change what is live), `AGENT_NOT_PUBLISHED` (409, the agent has no published version yet), `AGENT_TYPE_NOT_DISPATCHABLE` (422, a Voice Assist agent named where an agent must place or take a call; the body carries `agent_type` and `agent_id`), `NOT_APPLICABLE_TO_AGENT_TYPE` (422, a voice-only setting written on a Voice Assist agent), `AGENT_TYPE_BOUNDARY` (409), `AGENT_TYPE_NOT_SUPPORTED` (400), `VOICE_ASSISTANT_REQUIRED` (403, the organization is not licensed for Voice Assist), `ORGANIZATION_UNAVAILABLE` (503) and `REGION_PROFILE_CONFLICT` (503). EvalScenarioEntry: description: Either a reference to an existing scenario or a new scenario. oneOf: - $ref: '#/components/schemas/EvalScenarioRef' - $ref: '#/components/schemas/EvalScenarioInput' EvalTestCaseEntry: description: Either a reference to an existing test case or a new test case. oneOf: - $ref: '#/components/schemas/EvalTestCaseRef' - $ref: '#/components/schemas/EvalTestCaseInput' EvalScenario: type: object description: >- A simulated caller (persona and goal) together with the test cases its calls are judged against. required: - id - project_id - name - simulator_config - test_case_ids properties: id: type: string project_id: type: string name: type: string maxLength: 256 description: type: string nullable: true simulator_config: $ref: '#/components/schemas/EvalSimulatorConfig' test_case_ids: type: array items: type: string field_overrides: type: object additionalProperties: type: string description: >- Values for the agent's prompt fields used in this scenario's calls. They are merged over the suite's `field_overrides` (the scenario wins on the same key). Keys must be fields the agent allows to be overridden. silence_modes: type: array nullable: true items: $ref: '#/components/schemas/EvalSilenceMode' description: >- The silence behaviors this scenario runs under; each entry is run separately. `null` means not set: the submission's own silence setting applies. test_cases: type: array description: The scenario's test cases, in order. items: $ref: '#/components/schemas/EvalTestCase' created_by: type: string nullable: true created_at: type: string format: date-time nullable: true updated_at: type: string format: date-time nullable: true EvalTestCase: type: object description: >- One check the judge applies to an eval call: when it is relevant, what counts as success and what counts as failure. required: - id - project_id - name - when_relevant - success_criteria - failure_criteria properties: id: type: string project_id: type: string name: type: string maxLength: 256 description: type: string nullable: true when_relevant: type: string description: >- When this test case applies to a call. A call where it does not apply is judged `not_relevant`. success_criteria: type: string description: What the agent must do for the test case to pass. failure_criteria: type: string description: What makes the test case fail. created_by: type: string nullable: true created_at: type: string format: date-time nullable: true updated_at: type: string format: date-time nullable: true ValidationError: type: object description: >- A validation failure. `error` is a human-readable summary; `issues` (request validation) or `details` (agent configuration validation) locate each problem. properties: error: type: string message: type: string issues: type: array items: type: object properties: path: type: string description: Dotted path of the offending field code: type: string message: type: string details: type: array items: {} EvalScenarioRef: type: object description: An existing scenario of this project, reused by id. required: - scenario_id properties: scenario_id: type: string minLength: 1 EvalScenarioInput: type: object description: A new scenario, created together with the suite. required: - name - simulator_config - test_cases properties: name: type: string minLength: 1 maxLength: 256 description: type: string nullable: true simulator_config: type: object required: - prompt - max_turns properties: prompt: type: string minLength: 1 max_turns: type: integer minimum: 1 maximum: 100 close_instruction: type: string maxLength: 4000 nullable: true description: Blank or `null` uses the platform default. test_cases: type: array minItems: 1 maxItems: 50 items: $ref: '#/components/schemas/EvalTestCaseEntry' field_overrides: type: object additionalProperties: type: string minLength: 1 silence_modes: type: array nullable: true minItems: 1 maxItems: 3 uniqueItems: true items: $ref: '#/components/schemas/EvalSilenceMode' description: Omitted or `null` leaves it unset; an empty array is refused. EvalTestCaseRef: type: object description: An existing test case of this project, reused by id. required: - test_case_id properties: test_case_id: type: string minLength: 1 EvalTestCaseInput: type: object description: A new test case, created together with the suite. required: - name - when_relevant - success_criteria - failure_criteria properties: name: type: string minLength: 1 maxLength: 256 description: type: string nullable: true when_relevant: type: string minLength: 1 success_criteria: type: string minLength: 1 failure_criteria: type: string minLength: 1 EvalSimulatorConfig: type: object description: How the simulated caller behaves in this scenario. required: - prompt - max_turns properties: prompt: type: string description: >- Instructions for the simulated caller (who they are, what they want, how they behave). max_turns: type: integer minimum: 1 maximum: 100 description: The most caller turns before the call is ended. close_instruction: type: string maxLength: 4000 description: >- Optional instruction for how the simulated caller ends the call. Omitted means the platform default: the agent closes the call and the caller hangs up only as a last resort. additionalProperties: true EvalSilenceMode: type: string enum: - 'off' - always - flexible description: >- How the simulated caller handles silences: `off` never stays silent, `always` and `flexible` test how the agent copes with a quiet caller. responses: Unauthorized: description: No credential was sent, or the API key is invalid or revoked. content: application/json: schema: $ref: '#/components/schemas/Error' example: error: Invalid or revoked token Forbidden: description: >- The credential carries no access to this project, or the operation needs administrative rights content: application/json: schema: $ref: '#/components/schemas/Error' ValidationFailed: description: >- The request body or query string failed validation. `error` lists each problem prefixed with the field it concerns; `issues` carries the same information per field. content: application/json: schema: $ref: '#/components/schemas/ValidationError' example: error: 'name: String must contain at least 1 character(s)' issues: - path: name code: too_small message: String must contain at least 1 character(s) minimum: 1 securitySchemes: organizationApiKey: type: http scheme: bearer description: >- Your organization API key (it starts with `anek_org_`). The same key works on the Voice API and the Support API. See [Authentication](/authentication). ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.