PromptEvalSuiteCreateRequest
Creates an evaluation suite with no scenario at revision 1 (1.55.0). An Agent the Organization lacks is prompt_eval_agent_unknown, and a judge or simulator the Provider registry or the Compliance profile refuses is provider_not_allowed with the stage llm, each a detail of 422 validation_failed.
An object with 7 fields.
Fields
| Field | Type | Required | Description |
|---|---|---|---|
agent_id | AgentId | yes | |
description | string | no | For reviewers; never sent to a model. |
gate | PromptEvalGateInput | no | The publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. None when omitted. |
judge | StageRefInput | no | The model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
name | string | yes | A display name. |
repetitions | integer | no | 5 when omitted (a target). |
simulator | StageRefInput | no | The model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
Used by
The operations whose parameters, request body or responses name this shape directly.