VocalDocscontracts 1.65.0

Add a scenario to an evaluation suite

POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/scenarios

PropertyValue
Operationcreate_prompt_eval_scenario
GroupPrompt evaluations and experiments
Scopeprompts:write
Route classcontrol
IdempotencyIdempotency-Key accepted: a retry with the same key replays the first answer
AuditAudited prompts.prompt_eval_scenario.create (mutation)
Released in1.55.0
Phase 0 hostServed (the hosted api pins 1.65.0)

Adds a simulated scenario with its rubric, or a scripted conversation test with its expectations, at revision 1 (1.55.0). Every issue is a detail of 422 validation_failed (PromptEvalScenarioIssueCode). The answer carries caller null, as every write does.

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../scenarios" \
  -H "Authorization: Bearer $VOCAL_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{
    "kind": "simulated",
    "name": "<name>",
    "caller": {
      "type": "simulated",
      "persona": "<persona>",
      "goal": "<goal>"
    }
  }'

Parameters

NameInTypeRequiredDescription
prompt_eval_suite_idpathPromptEvalSuiteIdyesThe id of an evaluation suite: pes_ and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log.
idempotency-keyheaderstringnoA client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response.

Request body

Content type application/json, required: PromptEvalScenarioCreateRequest.

FieldTypeRequiredDescription
callerPromptEvalCallerInputyesIts type equals kind.
expectationsarray of ConversationTestExpectationInputnoRequired, at least one, when scripted.
initial_slotsPromptEvalInitialSlotsnoThe starting slot values (1.63.0); none when omitted.
kindPromptEvalScenarioKindyes
max_turnsintegerno20 when omitted (a target).
namestringyesA display name.
rubricarray of PromptEvalRubricItemInputnoRequired, at least one, when simulated.
tagsarray of PromptEvalTagno
tool_mocksarray of PromptEvalToolMockInputno
variablesPromptEvalVariablesno

Responses

StatusContent typeShapeError codes
201application/jsonPromptEvalScenario
400application/jsonErrorEnvelopeinvalid_request
401application/jsonErrorEnvelopeunauthenticated
403application/jsonErrorEnvelopeforbidden, org_suspended
404application/jsonErrorEnvelopenot_found
409application/jsonErrorEnvelopeprompt_eval_suite_archived, idempotency_in_progress, idempotency_key_reused
422application/jsonErrorEnvelopevalidation_failed
429application/jsonErrorEnveloperate_limited
500application/jsonErrorEnvelopeinternal_error

Fields of the 201 answer

The shape is PromptEvalScenario.

FieldTypeRequiredDescription
callerPromptEvalCaller or nullyesThe caller, the content class text: null unless the read names include=text, and null in the answer of a write.
created_atTimestampyes
expectationsarray of ConversationTestExpectationyesThe expectations of a scripted scenario; empty otherwise.
idPromptEvalScenarioIdyes
initial_slotsPromptEvalInitialSlotsnoThe starting slot values (1.63.0); absent from an api before 1.63.0, which a reader treats as none.
kindPromptEvalScenarioKindyes
max_turnsintegeryesThe most Turns a Call of the scenario takes before it is ended.
namestringyesA display name.
object"prompt_eval_scenario"yes
revisionintegeryesCounts writes; returned as the ETag and required in If-Match.
rubricarray of PromptEvalRubricItemyesThe rubric of a simulated scenario; empty otherwise.
suite_idPromptEvalSuiteIdyes
tagsarray of PromptEvalTagyes
tool_mocksarray of PromptEvalToolMockyes
updated_atTimestampyes
variablesPromptEvalVariablesyes