Run an evaluation suite

POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/runs

Runs the suite's scenarios, or those scenario_ids names, against the candidate and the baseline as a prompt.evaluation job, and answers 202 with the run queued, its job_id and its Location (1.55.0; prompt-manager.md section 9.1). Each scenario repeats as the suite says, as a test Call with purpose: "evaluation" on the headless Engine. A request whose suite revision, candidate, baseline and scenarios equal those of a run still queued or running answers that run. Until a dedicated evaluation quota exists the Calls count against the daily allowance of test Calls (429 test_daily_cap_reached); from 1.63.0 they count against both, the evaluation quota refusing with 429 evaluation_quota_reached, and a run with model_families runs each scenario once per family. Until the headless Engine runs evaluation Calls the request is refused with 422 capability_not_available and the detail evaluation_engine_not_available. The job emits job.succeeded or job.failed, and a completed run prompt_evaluation.completed.

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../runs" \
  -H "Authorization: Bearer $VOCAL_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{
    "candidate": {
      "agent_version": 1
    }
  }'

const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites/pes_.../runs`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
    'Idempotency-Key': crypto.randomUUID(),
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "candidate": {
      "agent_version": 1
    }
  }),
});
console.log(response.status, await response.text());

import os
import uuid

import requests

response = requests.post(
    f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites/pes_.../runs",
    headers={
        "Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={
        "candidate": {
            "agent_version": 1,
        },
    },
)
print(response.status_code, response.text)

Parameters

Request body

Content type application/json, required: PromptEvalRunCreateRequest.

Responses

Fields of the 202 answer

The shape is PromptEvalRun.