Label a sampled verdict
POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/labels
| Property | Value |
|---|---|
| Operation | create_prompt_eval_label |
| Group | Prompt evaluations and experiments |
| Scope | prompts:write |
| Route class | control |
| Idempotency | Idempotency-Key accepted: a retry with the same key replays the first answer |
| Audit | Audited evaluations.prompt_eval_label.create (mutation) |
| Released in | 1.63.0 |
| Phase 0 host | Served (the hosted api pins 1.65.0) |
Records a person's verdict on every rubric item of one entry of a calibration that is labelling and answers 201 with the label (1.63.0; prompt-manager.md section 9.1 rule 5). A label is immutable and an entry takes one; the label that completes the sample queues the calibration's comparison as a prompt.calibration job. Every issue is a detail of 422 validation_failed (PromptEvalLabelIssueCode).
Request
Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.
curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../labels" \
-H "Authorization: Bearer $VOCAL_API_KEY" \
-H "Idempotency-Key: $(uuidgen)" \
-H "Content-Type: application/json" \
-d '{
"calibration_id": "pec_...",
"entry_index": 0,
"items": [
{
"key": "verifies_before_details",
"verdict": "pass"
}
]
}'const response = await fetch(`${process.env.VOCAL_API}/v1/prompt-eval-suites/pes_.../labels`, {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.VOCAL_API_KEY}`,
'Idempotency-Key': crypto.randomUUID(),
'Content-Type': 'application/json',
},
body: JSON.stringify({
"calibration_id": "pec_...",
"entry_index": 0,
"items": [
{
"key": "verifies_before_details",
"verdict": "pass"
}
]
}),
});
console.log(response.status, await response.text());import os
import uuid
import requests
response = requests.post(
f"{os.environ['VOCAL_API']}/v1/prompt-eval-suites/pes_.../labels",
headers={
"Authorization": f"Bearer {os.environ['VOCAL_API_KEY']}",
"Idempotency-Key": str(uuid.uuid4()),
},
json={
"calibration_id": "pec_...",
"entry_index": 0,
"items": [
{
"key": "verifies_before_details",
"verdict": "pass",
},
],
},
)
print(response.status_code, response.text)Parameters
| Name | In | Type | Required | Description |
|---|---|---|---|---|
prompt_eval_suite_id | path | PromptEvalSuiteId | yes | The id of an evaluation suite: pes_ and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log. |
idempotency-key | header | string | no | A client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response. |
Request body
Content type application/json, required: PromptEvalLabelCreateRequest.
| Field | Type | Required | Description |
|---|---|---|---|
calibration_id | PromptEvalCalibrationId | yes | |
entry_index | integer | yes | |
items | array of PromptEvalLabelItemInput | yes | Every rubric item of the entry's verdict, each once. |
Responses
| Status | Content type | Shape | Error codes |
|---|---|---|---|
201 | application/json | PromptEvalLabel | |
400 | application/json | ErrorEnvelope | invalid_request |
401 | application/json | ErrorEnvelope | unauthenticated |
403 | application/json | ErrorEnvelope | forbidden, org_suspended |
404 | application/json | ErrorEnvelope | not_found |
409 | application/json | ErrorEnvelope | prompt_eval_suite_archived, idempotency_in_progress, idempotency_key_reused |
422 | application/json | ErrorEnvelope | validation_failed |
429 | application/json | ErrorEnvelope | rate_limited |
500 | application/json | ErrorEnvelope | internal_error |
Fields of the 201 answer
The shape is PromptEvalLabel.
| Field | Type | Required | Description |
|---|---|---|---|
arm | PromptEvalArm | yes | |
calibration_id | PromptEvalCalibrationId | yes | |
call_id | CallId | yes | The evaluation Call whose transcript was labelled. |
created_at | Timestamp | yes | |
created_by | ActorRef | yes | |
entry_index | integer | yes | The sample entry it labels. |
id | PromptEvalLabelId | yes | |
items | array of PromptEvalLabelItem | yes | |
object | "prompt_eval_label" | yes | |
repetition | integer | yes | |
run_id | PromptEvalRunId | yes | |
scenario_id | PromptEvalScenarioId | yes | |
scenario_revision | integer | yes | |
suite_id | PromptEvalSuiteId | yes |