Skip to main content

Run one evaluation

Endpoint: POST /evaluate

Description
Runs a single evaluation: you pass a metric short name, optional model, optional reasoning level, and threshold, and the input the metric needs. The response echoes those fields and adds scores, costs, timing, and the reasoning level applied.

Which fields you must send depends on the metric’s scope:

  • Record-scoped metrics score prompt / input / context / output / golden_answer and must omit span_id and trace.
  • Span-scoped agentic metrics (today, Tool Selection Correctness) require span_id and trace.
  • Whole-trace agentic metrics require trace only and must omit span_id.

Agentic metrics must also omit the record fields. Payload shape errors return 400 before a backing run is created.

Optionally include external_id (string, max 255 characters) to attach your own stable identifier to the record. It is echoed in the response and stored on the underlying record for later correlation.

Sharing rules

The evaluation creates a backing run owned by you. With a project-scoped API key, that run is org-shared and assigned to the key's project automatically. With a private key (including an org member’s private key), the backing run stays personal; usage still bills the organization pool when you belong to an org — see Introduction — API key scoping.

Parameters

  • Body — application/json:
{
  "metric_shortname": "string",
  "model_slug": "string | null",
  "reasoning_level": "string | null",
  "threshold": "0–100 | null",
  "metric_args": "object | null",
  "external_id": "string | null",
  "prompt": "string | null",
  "input": "string | number | boolean | object | array | null",
  "context": "string | array | null",
  "output": "string | number | boolean | object | array | null",
  "golden_answer": "string | number | boolean | object | array | null",
  "span_id": "string | null",
  "trace": "array of objects | null"
}

Agentic metrics must omit record fields (prompt, input, context, output, golden_answer). A non-null value of any of them — including "" or [] where the type allows — returns 400 (Metric {shortname} scores a trace; omit {fields}.), where {fields} is a comma-separated list of the record fields you sent. Record metrics must omit span_id and trace. A non-null value of either — including "" or [] — returns 400 (Metric {shortname} scores a record; omit {fields}.), listing only the fields you sent.

  • Tool Selection Correctness (tool_select) is span-scoped. It requires span_id and trace. span_id identifies the OpenInference span to score (see span_id below). trace is the OpenInference span list that contains that span (sibling and child spans included). The scorer looks span_id up in that list. A mismatch returns 400 (span_id must match a span in the trace.).
  • Every other agentic metric (plan_adh, plan_qual, task_comp, traject_eff) scores a whole trace and requires trace: a non-empty array of OpenInference span objects. Sending span_id returns 400 (Metric {shortname} scores a trace; omit span_id.).

These payload **400**s are returned before a backing run is created.

span_id

  • Required for a span-scoped metric (tool_select), together with trace. Omitting span_id (null) returns 400 (span_id is required.). A value that is not a string returns 422. The value must be an 8-byte OTLP hex id (optional 0x / 0X prefix; surrounding whitespace is ignored for matching). An empty, whitespace-only, malformed, wrong-width, or all-zero id returns 400 (Span id '{value}' must be an 8-byte OTLP hex id. or Span id '{value}' must not be all zeros.).
  • Matching finds a span whose context.span_id or top-level span_id is a string that normalizes to the same hex (bare hex and 0x… compare equal). No match returns 400 (span_id must match a span in the trace.).
  • For whole-trace agentic metrics and record metrics, omit span_id.

trace

  • Required for every agentic metric. A missing required trace returns 400 (trace is required.). [], or an array that contains an empty object ([{}]), returns 400 (Trace must be a non-empty JSON array of span objects.). A value that is not a JSON array of objects (for example ["not-a-span"]) returns 422. For tool_select, the list must include the span named by span_id. For record metrics, omit trace.

threshold

  • Optional. Integer in 0–100. Omit it (or send null) to use the metric's catalog default. Values outside that range return 422.

model_slug

  • Optional. When omitted, the server uses gpt-5.6-luna with reasoning level none. That model (or the one you pass) is used for every LLM step in the evaluation. For Factfulness and Plan Quality, a higher-capability model than the default is recommended.

reasoning_level

  • Optional. Controls how much extended reasoning the evaluation model uses. Valid values depend on the chosen model_slug — they are model-specific strings, not a global enum (for example none, low, medium, high, xhigh on many recent GPT reasoning models; low, medium, high, max on many Claude thinking models).
  • When omitted or whitespace-only, the API uses the model's configured default when the model advertises reasoning levels; otherwise the effective level is null.
  • Supplying a level for a model that does not advertise reasoning levels returns 400 (Model '{slug}' does not support reasoning levels.).
  • An unsupported level returns 400 with the allowed values for that model (Invalid reasoning level '{level}' for model '{slug}'. Allowed values: [...]). Related 400 responses name the model when its default is missing or not allowed (Model '{slug}' has no default reasoning level configured. Allowed values: [...]; Model '{slug}' default reasoning level '{default}' is not in allowed values: [...]).
  • The response includes the reasoning level applied (string | null), which may differ from the value you sent.

Error responses​

  • 401, 402 — Authentication or insufficient balance; shared with other routes (Introduction — status codes).
  • 403 — API key scoping violation (stale/cross-org key, scoped key on a personal resource).
  • 404 — No metric with the given metric_shortname; Model with slug {slug} not found. when a provided model_slug is not a documented model slug (when model_slug is omitted, the default model is used).
  • 400 — Metric is not active; model slug is deactivated (The following models are not active: …); auto-generated backing-run alias ({timestamp}_{metric_shortname}) already exists for your account; invalid reasoning_level for the model (model does not support reasoning; level not in that model's allowed set; invalid model default — error message names the model and allowed values when applicable); agentic payload missing trace (trace is required.) or empty (Trace must be a non-empty JSON array of span objects.); span-scoped metric missing span_id (span_id is required.), with an invalid OTLP hex span_id, or with a span_id absent from trace (span_id must match a span in the trace.); span_id sent for a whole-trace agentic metric (Metric {shortname} scores a trace; omit span_id.); any record field set on an agentic metric (Metric {shortname} scores a trace; omit {fields}.); record-metric payload with span_id or trace (Metric {shortname} scores a record; omit {fields}.); these payload errors are returned before a backing run is created (see span_id and trace above); other conflicting-state errors.
  • 422 — Request body failed validation. A threshold outside 0–100, a span_id that is not a string, or a trace that is not an array of objects, is rejected here.
  • 500 — Error during evaluation or persistence.

Responses

  • 201 — JSON object with the shape below.

Example response (201)

{
  "id": 1542,
  "run_id": 991,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "metric_args": null,
  "is_success": true,
  "is_gte_threshold": true,
  "threshold": 70,
  "external_id": "row-42",
  "prompt": "What is 2+2?",
  "input": null,
  "context": null,
  "output": "4",
  "golden_answer": "4",
  "span_id": null,
  "trace": null,
  "started_at": "2026-04-01T09:10:03Z",
  "result": 100,
  "explanation": "Answer exactly matches the expected value.",
  "evaluation_cost": "0.02",
  "finished_at": "2026-04-01T09:10:04Z",
  "metric_shortname": "ans_corr",
  "eval_metadata": null
}
{
  "id": 0,
  "run_id": 0,
  "model_slug": "string",
  "reasoning_level": "string | null",
  "metric_args": "object | null",
  "is_success": "boolean | null",
  "is_gte_threshold": "boolean | null",
  "threshold": 0,
  "external_id": "string | null",
  "prompt": "string | null",
  "input": "string | number | boolean | object | array | null",
  "context": "string | array | null",
  "output": "string | number | boolean | object | array | null",
  "golden_answer": "string | number | boolean | object | array | null",
  "span_id": "string | null",
  "trace": "array of objects | null",
  "started_at": "date",
  "result": "number | null",
  "explanation": "string | null",
  "evaluation_cost": "string | null",
  "finished_at": "date | null",
  "metric_shortname": "string",
  "eval_metadata": "object | null"
}

span_id and trace on this 201 echo the payload that was scored for an agentic metric. For a span-scoped metric, span_id is the id as written on the matching span in trace (bare hex and 0x… compare equal when matching, so the echoed value may differ from what you sent only by prefix/casing/whitespace normalization). For record metrics both are null. Neither field is returned by GET /runs/{run_id}, and the API does not persist them for later inspect or re-run.

metric_args echoes the per-metric arguments sent for this evaluation. Keys are argument names declared by the metric; each value follows the type declared by the metric (string, boolean, integer, number, list, or object). See the metric's doc page for accepted argument names and types. Argument names are not validated: an unknown key (for example a typo like min_cout) is echoed back but ignored, and the metric uses its default for that argument. A missing required argument or a value of the wrong type returns the evaluation with is_success: false and the reason in explanation.

eval_metadata is a metric-specific JSON object with extra signals about how the score was produced (for example, intermediate computations or token usage). It is null when the metric does not emit metadata or when the evaluation failed before metadata could be collected.

evaluation_cost is the USD amount charged to your balance for this evaluation, or null when the metric reports no provider cost. Nothing is charged in that case. When the provider cost is greater than zero, the billed amount is at least 0.02.

curl

curl -X POST "https://api.aegisevals.ai/api/v1/evaluate" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{
    "metric_shortname": "ans_corr",
    "model_slug": "gpt-4o",
    "threshold": 70,
    "external_id": "row-42",
    "prompt": "What is 2+2?",
    "output": "4",
    "golden_answer": "4"
  }'