VocalDocscontracts 1.65.0

Label a sampled verdict

POST /v1/prompt-eval-suites/{prompt_eval_suite_id}/labels

PropertyValue
Operationcreate_prompt_eval_label
GroupPrompt evaluations and experiments
Scopeprompts:write
Route classcontrol
IdempotencyIdempotency-Key accepted: a retry with the same key replays the first answer
AuditAudited evaluations.prompt_eval_label.create (mutation)
Released in1.63.0
Phase 0 hostServed (the hosted api pins 1.65.0)

Records a person's verdict on every rubric item of one entry of a calibration that is labelling and answers 201 with the label (1.63.0; prompt-manager.md section 9.1 rule 5). A label is immutable and an entry takes one; the label that completes the sample queues the calibration's comparison as a prompt.calibration job. Every issue is a detail of 422 validation_failed (PromptEvalLabelIssueCode).

Request

Set VOCAL_API to the base URL of your API and VOCAL_API_KEY to an API key with the prompts:write scope (Authentication and keys). Replace each placeholder, such as agt_... or <name>, with your own value.

curl -X POST "$VOCAL_API/v1/prompt-eval-suites/pes_.../labels" \
  -H "Authorization: Bearer $VOCAL_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{
    "calibration_id": "pec_...",
    "entry_index": 0,
    "items": [
      {
        "key": "verifies_before_details",
        "verdict": "pass"
      }
    ]
  }'

Parameters

NameInTypeRequiredDescription
prompt_eval_suite_idpathPromptEvalSuiteIdyesThe id of an evaluation suite: pes_ and the row's UUIDv7 in 26 Crockford base32 characters. Opaque and safe to log.
idempotency-keyheaderstringnoA client-chosen key, kept 24 hours per Organization, environment and route. A replay returns the stored response.

Request body

Content type application/json, required: PromptEvalLabelCreateRequest.

FieldTypeRequiredDescription
calibration_idPromptEvalCalibrationIdyes
entry_indexintegeryes
itemsarray of PromptEvalLabelItemInputyesEvery rubric item of the entry's verdict, each once.

Responses

StatusContent typeShapeError codes
201application/jsonPromptEvalLabel
400application/jsonErrorEnvelopeinvalid_request
401application/jsonErrorEnvelopeunauthenticated
403application/jsonErrorEnvelopeforbidden, org_suspended
404application/jsonErrorEnvelopenot_found
409application/jsonErrorEnvelopeprompt_eval_suite_archived, idempotency_in_progress, idempotency_key_reused
422application/jsonErrorEnvelopevalidation_failed
429application/jsonErrorEnveloperate_limited
500application/jsonErrorEnvelopeinternal_error

Fields of the 201 answer

The shape is PromptEvalLabel.

FieldTypeRequiredDescription
armPromptEvalArmyes
calibration_idPromptEvalCalibrationIdyes
call_idCallIdyesThe evaluation Call whose transcript was labelled.
created_atTimestampyes
created_byActorRefyes
entry_indexintegeryesThe sample entry it labels.
idPromptEvalLabelIdyes
itemsarray of PromptEvalLabelItemyes
object"prompt_eval_label"yes
repetitionintegeryes
run_idPromptEvalRunIdyes
scenario_idPromptEvalScenarioIdyes
scenario_revisionintegeryes
suite_idPromptEvalSuiteIdyes