Calibrate a suite's judge

POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/calibrations

Creates a calibration of the suite's current judge and answers 201 with it (1.63.0; prompt-manager.md section 9.1 rule 5): from run_id, a sample drawn at random from the completed run's simulated verdicts that kept their transcript, labelling until every entry is labelled; from from_calibration_id, the labelled sample of an earlier calibration, queued at once as a prompt.calibration job. The job judges each labelled transcript again and compares the judge with the labels; a completed calibration becomes the suite's calibration record, and its judged results stop being advisory once a record meets the target (0.9 agreement on at least 30 must items, targets) with the judge that produced them. The judge's tokens count against the evaluation quota (429 evaluation_quota_reached). Every issue is a detail of 422 validation_failed (PromptEvalCalibrationIssueCode).

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../calibrations" \
  -H "Authorization: Bearer $VOCAL_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{}'

const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites/pes_.../calibrations`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
    'Idempotency-Key': crypto.randomUUID(),
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({}),
});
console.log(response.status, await response.text());

import os
import uuid

import requests

response = requests.post(
    f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites/pes_.../calibrations",
    headers={
        "Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={},
)
print(response.status_code, response.text)

Parameters

Request body

Content type application/json, required: PromptEvalCalibrationCreateRequest.

Responses

Fields of the 201 answer

The shape is PromptEvalCalibration.