Step 5: Gate, interpret, and act on results

Getting a run back is not the end; reading it correctly is. This is how you turn scores into decisions.

Know whether your run is finished

Runs default to non-blocking (is_blocking false): the create call returns immediately with evaluations in a pending state (result and finished_at null) while a background process scores them. Poll Get run (GET /api/v1/runs/{run_id}) until finished_at is set. If you want the answer inline (as in CI), create the run blocking instead. Single evaluate is always synchronous.

Read the two-level result

At the run level, result is the mean score and is_gte_threshold compares it to the run threshold; aggregate_results breaks the average down per metric. At the evaluation level, each row has its own result, threshold, is_gte_threshold, is_success, and explanation. Gate on whichever level matches your policy — an overall average, or every critical metric individually.

Separate "failed to run" from "scored low"

is_success false means the metric did not produce a score (missing required field, judge error). Do not average these in as zeros or read them as passes; surface them as pipeline errors to fix. is_gte_threshold false on a successful evaluation is a genuine quality miss.

Use explanations to act, not just to record

When a gate trips, the explanation on the offending evaluations tells you why — off-topic drift, unsupported claim, wrong format. That is your debugging lead. Route it to whoever owns the prompt or the retrieval step.

Export and correlate

Download a run's full results as CSV (Download run, GET /api/v1/runs/{run_id}/download) for offline analysis or to attach to a release. Use the external_id you set in Step 1 to join each result back to the exact request in your own system.

Poll and gate on scores yourself

There is no score-based alert or webhook — the only built-in alert is a dollar-balance alert. If you want to be paged when quality drops, your polling or CI code raises that from the flags you read here.