PromptEvalSuite
An evaluation suite of one Agent (1.55.0, FR-prompts-3, FR-agents-17): its judge, the model that plays simulated callers and the repetitions of a run, and from 1.63.0 the publish gate it defines and the record of its judge's calibration. The scenarios, labels and calibrations are sub-resources; every run is a report, and from 1.63.0 it gates an Agent only once the Agent's owner names the suite.
An object with 16 fields.
Fields
| Field | Type | Required | Description |
|---|---|---|---|
agent_id | AgentId | yes | The Agent whose versions the suite evaluates. |
archived_at | Timestamp or null | yes | When DELETE archived the suite, or null. An archived suite takes no write and no run (409 prompt_eval_suite_archived); its scenarios and runs stay readable. |
calibration | PromptEvalJudgeCalibration or null | no | When the suite's judge was last calibrated (1.63.0): the latest completed calibration, null before the first. Absent from an api before 1.63.0, which a reader treats as null. |
created_at | Timestamp | yes | |
description | string or null | yes | For reviewers; never sent to a model. |
gate | PromptEvalGate or null | no | The publish gate the suite defines (1.63.0): the expectation types and must items that must hold, with the floor and margin of the judged items, which an Agent applies once its owner names the suite. Null for none; absent from an api before 1.63.0, which a reader treats as null. |
id | PromptEvalSuiteId | yes | |
judge | StageRef or null | yes | The model of the llm stage that reads each transcript of a simulated scenario with its rubric and returns a verdict per item, through a Vocal task template with structured output (section 9.1). Its results are advisory until it is checked against human labels, which 1.55.0 does not define and 1.63.0 does (calibration). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
name | string | yes | A display name. |
object | "prompt_eval_suite" | yes | |
organization_id | OrganizationId | yes | |
repetitions | integer | yes | How many times a run repeats each scenario, because model replies vary. |
revision | integer | yes | Counts writes to the suite; returned as the ETag and required in If-Match. |
scenario_count | integer | yes | |
simulator | StageRef or null | yes | The model of the llm stage that plays the simulated callers, through a Vocal task template (section 9.1). Null until set; a run of a simulated scenario then refuses with prompt_eval_judge_required. |
updated_at | Timestamp | yes |
Used by
The operations whose parameters, request body or responses name this shape directly.