Create an evaluation suite
POST /v1/prompt-eval-suites
| Property | Value |
|---|---|
| Operation | create_prompt_eval_suite |
| Group | Prompt evaluations and experiments |
| Scope | prompts:write |
| Route class | control |
| Idempotency | Idempotency-Key accepted: a retry with the same key replays the first answer |
| Audit | Audited prompts.prompt_eval_suite.create (mutation) |
| Released in | 1.55.0 |
| Phase 0 host | Served (the hosted api pins 1.65.0) |
Creates an evaluation suite of one Agent at revision 1 with no scenario (1.55.0; prompt-manager.md section 9.1). An Agent the Organization lacks is prompt_eval_agent_unknown, and a judge or simulator the Provider registry or the Compliance profile refuses is provider_not_allowed, each a detail of 422 validation_failed.
Request
Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.
curl -X POST "$VOCAL_API/v1/prompt-eval-suites" \
-H "Authorization: Bearer $VOCAL_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{ "agent_id": "agt_...", "name": "<name>" }'const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites`, {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
'Idempotency-Key': crypto.randomUUID(),
'Content-Type': 'application/json',
},
body: JSON.stringify({
"agent_id": "agt_...",
"name": "<name>"
}),
});
console.log(response.status, await response.text());import os
import uuid
import requests
response = requests.post(
f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites",
headers={
"Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
"Idempotency-Key": str(uuid.uuid4()),
},
json={
"agent_id": "agt_...",
"name": "<name>",
},
)
print(response.status_code, response.text)Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
idempotency-key | header | string | no | A client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response. |
Request body
Content type application/json, required: PromptEvalSuiteCreateRequest.
| Field | Type | Required | Description |
|---|---|---|---|
agent_id | AgentId | yes | |
description | string | no | For reviewers; never sent to a model. |
gate | PromptEvalGateInput | no | The publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. None when omitted. |
judge | StageRefInput | no | The model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
name | string | yes | A display name. |
repetitions | integer | no | 5 when omitted (a target). |
simulator | StageRefInput | no | The model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
Responses
| Status | Content type | Shape | Error codes |
|---|---|---|---|
201 | application/json | PromptEvalSuite | |
400 | application/json | ErrorEnvelope | invalid_request |
401 | application/json | ErrorEnvelope | unauthenticated |
403 | application/json | ErrorEnvelope | forbidden, org_suspended |
409 | application/json | ErrorEnvelope | idempotency_in_progress, idempotency_key_reused |
422 | application/json | ErrorEnvelope | validation_failed |
429 | application/json | ErrorEnvelope | rate_limited |
500 | application/json | ErrorEnvelope | internal_error |
Fields of the 201 answer
The shape is PromptEvalSuite.
| Field | Type | Required | Description |
|---|---|---|---|
agent_id | AgentId | yes | The Agent whose versions the suite evaluates. |
archived_at | Timestamp or null | yes | When DELETE archived the suite, or null. An archived suite takes no write and no run (409 prompt_eval_suite_archived); its scenarios and runs stay readable. |
calibration | PromptEvalJudgeCalibration or null | no | When the suite's judge was last calibrated (1.63.0): the latest completed calibration, null before the first. Absent from an api before 1.63.0, which a reader treats as null. |
created_at | Timestamp | yes | |
description | string or null | yes | For reviewers; never sent to a model. |
gate | PromptEvalGate or null | no | The publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. Null for none; absent from an api before 1.63.0, which a reader treats as null. |
id | PromptEvalSuiteId | yes | |
judge | StageRef or null | yes | The model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
name | string | yes | A display name. |
object | "prompt_eval_suite" | yes | |
organization_id | OrganizationId | yes | |
repetitions | integer | yes | How many times a run repeats each scenario, because model replies vary. |
revision | integer | yes | Counts writes to the suite; returned as the ETag and required in If-Match. |
scenario_count | integer | yes | |
simulator | StageRef or null | yes | The model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
updated_at | Timestamp | yes |