Skip to the content

Read an evaluation run

GET /v1/prompt-eval-suites/{prompt_eval_suite_id}/runs/{run_id}

Property	Value
Operation	`get_prompt_eval_run`
Group	Prompt evaluations and experiments
Scope	`prompts:read`
Route class	`read`
Idempotency	No `Idempotency-Key`
Audit	Not audited: A metadata read; a run holds ids, counts and rates only.
Released in	1.55.0
Phase 0 host	Served (the hosted api pins 1.65.0)

Answers one run with its status, the per-arm pass rates with intervals and sample sizes, the deterministic facts and the cost lines marked evaluation (1.55.0); never a rationale or a transcript. Vocal reports intervals and never declares a winner.

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:read scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl "$VOCAL_API/v1/prompt-eval-suites/pes_.../runs/per_..." \
  -H "Authorization: Bearer $VOCAL_API_KEY"

const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites/pes_.../runs/per_...`, {
  headers: {
    Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
  },
});
console.log(response.status, await response.text());

import os

import requests

response = requests.get(
    f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites/pes_.../runs/per_...",
    headers={
        "Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
    },
)
print(response.status_code, response.text)

Parameters

Name	In	Type	Required	Description
`prompt_eval_suite_id`	path	PromptEvalSuiteId	yes	The id of an evaluation suite: `pes_` and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log.
`run_id`	path	PromptEvalRunId	yes	The id of an evaluation run or a conversation test run: `per_` and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log.

Responses

Status	Content type	Shape	Error codes
`200`	`application/json`	PromptEvalRun
`400`	`application/json`	ErrorEnvelope	`invalid_request`
`401`	`application/json`	ErrorEnvelope	`unauthenticated`
`403`	`application/json`	ErrorEnvelope	`forbidden`
`404`	`application/json`	ErrorEnvelope	`not_found`
`429`	`application/json`	ErrorEnvelope	`rate_limited`
`500`	`application/json`	ErrorEnvelope	`internal_error`

Fields of the `200` answer

The shape is PromptEvalRun.

Field	Type	Required	Description
`advisory`	boolean	yes	True when the judged results gate nothing, because the judge is not checked against human labels (section 9.1 rule 5); true for every run of 1.55.0. From 1.63.0 false only when the suite's latest calibration met its target with the run's judge (`promptEvalJudgeAdvisory`), and false for a run that judged nothing.
`agent_id`	AgentId	yes
`baseline`	PromptEvalRunTarget or null	yes
`by_family`	array or null	no	One entry per evaluated family and arm (1.63.0), once `completed`; null before. Absent from an api before 1.63.0.
`by_item`	array or null	no	One entry per scenario, rubric item and arm (1.63.0), once `completed`; null before. Absent from an api before 1.63.0.
`candidate`	PromptEvalRunTarget	yes
`cost_lines`	array of PromptEvalCostLine	yes
`created_at`	Timestamp	yes
`created_by`	ActorRef	yes
`error_code`	string or null	yes
`finished_at`	Timestamp or null	yes
`gate`	PromptEvalGateOutcome or null	no	How the run fares against its suite's publish gate (1.63.0), once `completed`; null for a suite with no gate and before completion. Absent from an api before 1.63.0.
`id`	PromptEvalRunId	yes
`job_id`	JobId	yes	The `prompt.evaluation` job that runs it.
`judge`	StageRef or null	yes	The judge the run used; null with none.
`model_families`	array of ModelFamily	no	The model families the run evaluated (1.63.0): the request's `model_families`, or the family of the chain's primary. Absent from an api before 1.63.0.
`object`	`"prompt_eval_run"`	yes
`repetitions`	integer	yes
`scenario_count`	integer	yes
`started_at`	Timestamp or null	yes
`status`	PromptEvalRunStatus	yes
`suite_id`	PromptEvalSuiteId	yes
`suite_revision`	integer	yes
`summary`	array or null	yes	One entry per arm, the candidate first, once `completed`; null before.