Edge cases, limits, and things teams miss
Read this before you rely on the pipeline for real decisions. These are the sharp edges that can make a “green” run feel safe when it isn’t.
LLM-judge scores are non-deterministic
The same output can score a few points differently from run to run. Set thresholds with a margin, prefer structural/deterministic metrics for hard gates, and do not gate on a one-point difference.
is_success false is not a pass
A metric that errored or lacked its required fields produces no score. If your gating code only checks is_gte_threshold, a run full of failed evaluations can look like it did not fail — always treat is_success false as a hard error.
Missing required fields silently break a metric
If a metric needs golden_answer or context and your data map omits it, you get is_success false, not a low score. Validate your column_mappings and field coverage against each metric's requirements before a big run.
No built-in scheduler, sampling, gating webhook, or score alert
Recurring cadence, production sampling, deploy-blocking, and score-based paging are all things you implement around the API. The platform scores and stores; you orchestrate and enforce.
Non-blocking runs need polling
If you forget that is_blocking defaults to false, your automation will read a still-pending run as if it were done (null scores). Either run blocking or poll until finished_at is set.
Cost is real
Cost is USD, billed per evaluation that actually calls a judge model. A run is records × metrics, so a large dataset with a large LLM suite adds up fast — sample on your side, keep that suite lean, and watch the balance alert. Structural metrics charge nothing.
Data privacy
Sending outputs (and prompts, inputs, context) to a judge model means that content leaves your system. Do not send secrets you would not send to your LLM provider; use the security and PII metrics to catch leakage in your own outputs.
Project-scoped keys are a one-way narrowing
A project-scoped API key can only see and create resources in its project. If a CI key seems to lose data or cannot attach to a collection, check that the key's project matches the target — a mismatch fails closed, which is the safe behaviour but a common surprise.
Threshold defaults if you omit them
A run-level threshold left unset defaults to the average of the selected metrics' thresholds. That is convenient but implicit; set it explicitly when the gate matters.
Long fields are truncated in API read-backs
Record values are truncated in some responses for display; rely on external_id and your own store as the source of truth for full inputs and outputs, and download the CSV when you need complete data.
Agentic evaluation goes through traces
The single-evaluate API does not expose trace/span inputs today, and most agentic metrics are inactive. For agent workflows, ingest OTLP traces and evaluate through that lane rather than expecting the record-based endpoints to cover them.
Golden answers are only as good as you make them
Correctness- and recall-style metrics compare against golden_answer; a stale or sloppy reference set produces confidently wrong scores. Curate the ground truth with the same care as the code it gates.