VocalDocscontracts 1.65.0

Create an evaluation suite

POST /v1/prompt-eval-suites

PropertyValue
Operationcreate_prompt_eval_suite
GroupPrompt evaluations and experiments
Scopeprompts:write
Route classcontrol
IdempotencyIdempotency-Key accepted: a retry with the same key replays the first answer
AuditAudited prompts.prompt_eval_suite.create (mutation)
Released in1.55.0
Phase 0 hostServed (the hosted api pins 1.65.0)

Creates an evaluation suite of one Agent at revision 1 with no scenario (1.55.0; prompt-manager.md section 9.1). An Agent the Organization lacks is prompt_eval_agent_unknown, and a judge or simulator the Provider registry or the Compliance profile refuses is provider_not_allowed, each a detail of 422 validation_failed.

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl -X POST "$VOCAL_API/v1/prompt-eval-suites" \
  -H "Authorization: Bearer $VOCAL_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{ "agent_id": "agt_...", "name": "<name>" }'

Parameters

NameInTypeRequiredDescription
idempotency-keyheaderstringnoA client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response.

Request body

Content type application/json, required: PromptEvalSuiteCreateRequest.

FieldTypeRequiredDescription
agent_idAgentIdyes
descriptionstringnoFor reviewers; never sent to a model.
gatePromptEvalGateInputnoThe publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. None when omitted.
judgeStageRefInputnoThe model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required.
namestringyesA display name.
repetitionsintegerno5 when omitted (a target).
simulatorStageRefInputnoThe model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required.

Responses

StatusContent typeShapeError codes
201application/jsonPromptEvalSuite
400application/jsonErrorEnvelopeinvalid_request
401application/jsonErrorEnvelopeunauthenticated
403application/jsonErrorEnvelopeforbidden, org_suspended
409application/jsonErrorEnvelopeidempotency_in_progress, idempotency_key_reused
422application/jsonErrorEnvelopevalidation_failed
429application/jsonErrorEnveloperate_limited
500application/jsonErrorEnvelopeinternal_error

Fields of the 201 answer

The shape is PromptEvalSuite.

FieldTypeRequiredDescription
agent_idAgentIdyesThe Agent whose versions the suite evaluates.
archived_atTimestamp or nullyesWhen DELETE archived the suite, or null. An archived suite takes no write and no run (409 prompt_eval_suite_archived); its scenarios and runs stay readable.
calibrationPromptEvalJudgeCalibration or nullnoWhen the suite's judge was last calibrated (1.63.0): the latest completed calibration, null before the first. Absent from an api before 1.63.0, which a reader treats as null.
created_atTimestampyes
descriptionstring or nullyesFor reviewers; never sent to a model.
gatePromptEvalGate or nullnoThe publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. Null for none; absent from an api before 1.63.0, which a reader treats as null.
idPromptEvalSuiteIdyes
judgeStageRef or nullyesThe model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required.
namestringyesA display name.
object"prompt_eval_suite"yes
organization_idOrganizationIdyes
repetitionsintegeryesHow many times a run repeats each scenario, because model replies vary.
revisionintegeryesCounts writes to the suite; returned as the ETag and required in If-Match.
scenario_countintegeryes
simulatorStageRef or nullyesThe model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required.
updated_atTimestampyes