Add a scenario to an evaluation suite
POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/scenarios
| Property | Value |
|---|---|
| Operation | create_prompt_eval_scenario |
| Group | Prompt evaluations and experiments |
| Scope | prompts:write |
| Route class | control |
| Idempotency | Idempotency-Key accepted: a retry with the same key replays the first answer |
| Audit | Audited prompts.prompt_eval_scenario.create (mutation) |
| Released in | 1.55.0 |
| Phase 0 host | Served (the hosted api pins 1.65.0) |
Adds a simulated scenario with its rubric, or a scripted conversation test with its expectations, at revision 1 (1.55.0). Every issue is a detail of 422 validation_failed (PromptEvalScenarioIssueCode). The answer carries caller null, as every write does.
Request
Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.
curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../scenarios" \
-H "Authorization: Bearer $VOCAL_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{
"kind": "simulated",
"name": "<name>",
"caller": {
"type": "simulated",
"persona": "<persona>",
"goal": "<goal>"
}
}'const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites/pes_.../scenarios`, {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
'Idempotency-Key': crypto.randomUUID(),
'Content-Type': 'application/json',
},
body: JSON.stringify({
"kind": "simulated",
"name": "<name>",
"caller": {
"type": "simulated",
"persona": "<persona>",
"goal": "<goal>"
}
}),
});
console.log(response.status, await response.text());import os
import uuid
import requests
response = requests.post(
f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites/pes_.../scenarios",
headers={
"Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
"Idempotency-Key": str(uuid.uuid4()),
},
json={
"kind": "simulated",
"name": "<name>",
"caller": {
"type": "simulated",
"persona": "<persona>",
"goal": "<goal>",
},
},
)
print(response.status_code, response.text)Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
prompt_eval_suite_id | path | PromptEvalSuiteId | yes | The id of an evaluation suite: pes_ and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log. |
idempotency-key | header | string | no | A client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response. |
Request body
Content type application/json, required: PromptEvalScenarioCreateRequest.
| Field | Type | Required | Description |
|---|---|---|---|
caller | PromptEvalCallerInput | yes | Its type equals kind. |
expectations | array of ConversationTestExpectationInput | no | Required, at least one, when scripted. |
initial_slots | PromptEvalInitialSlots | no | The starting slot values (1.63.0); none when omitted. |
kind | PromptEvalScenarioKind | yes | |
max_turns | integer | no | 20 when omitted (a target). |
name | string | yes | A display name. |
rubric | array of PromptEvalRubricItemInput | no | Required, at least one, when simulated. |
tags | array of PromptEvalTag | no | |
tool_mocks | array of PromptEvalToolMockInput | no | |
variables | PromptEvalVariables | no |
Responses
| Status | Content type | Shape | Error codes |
|---|---|---|---|
201 | application/json | PromptEvalScenario | |
400 | application/json | ErrorEnvelope | invalid_request |
401 | application/json | ErrorEnvelope | unauthenticated |
403 | application/json | ErrorEnvelope | forbidden, org_suspended |
404 | application/json | ErrorEnvelope | not_found |
409 | application/json | ErrorEnvelope | prompt_eval_suite_archived, idempotency_in_progress, idempotency_key_reused |
422 | application/json | ErrorEnvelope | validation_failed |
429 | application/json | ErrorEnvelope | rate_limited |
500 | application/json | ErrorEnvelope | internal_error |
Fields of the 201 answer
The shape is PromptEvalScenario.
| Field | Type | Required | Description |
|---|---|---|---|
caller | PromptEvalCaller or null | yes | The caller, the content class text: null unless the read names include=text, and null in the answer of a write. |
created_at | Timestamp | yes | |
expectations | array of ConversationTestExpectation | yes | The expectations of a scripted scenario; empty otherwise. |
id | PromptEvalScenarioId | yes | |
initial_slots | PromptEvalInitialSlots | no | The starting slot values (1.63.0); absent from an api before 1.63.0, which a reader treats as none. |
kind | PromptEvalScenarioKind | yes | |
max_turns | integer | yes | The most Turns a Call of the scenario takes before it is ended. |
name | string | yes | A display name. |
object | "prompt_eval_scenario" | yes | |
revision | integer | yes | Counts writes; returned as the ETag and required in If-Match. |
rubric | array of PromptEvalRubricItem | yes | The rubric of a simulated scenario; empty otherwise. |
suite_id | PromptEvalSuiteId | yes | |
tags | array of PromptEvalTag | yes | |
tool_mocks | array of PromptEvalToolMock | yes | |
updated_at | Timestamp | yes | |
variables | PromptEvalVariables | yes |