Step 6: Monitor and iterate over time

Evaluation is a program, not a one-off. Once the loop runs automatically, use the accumulated results to keep it sharp.

Watch the trend, not just the last run

The dashboards show usage, per-metric and per-model average scores over time, above-threshold percentages, and score evolution for a collection. A single run tells you today; the trend tells you whether a change helped or a slow regression is setting in.

Investigate regressions with the same tools you baselined with

When a trend dips, open the failing runs, read the explanations, and reproduce with single evaluation on the specific outputs. Fix the prompt, the retrieval, or the model — then confirm the fix moves the score back above threshold.

Expand coverage deliberately

As you find failure modes the current suite misses, add the metric that would have caught them, calibrate its threshold against real data, and fold it into the golden dataset. Retire metrics that never move or never correlate with real quality.

Keep the judge configuration stable

Changing the judge model, reasoning level, or metric_args changes the numbers. When you must change them, re-baseline so you are not comparing across an invisible discontinuity.

Optionally use run insights

A completed run can carry an LLM-generated set of observations about its results — advisory context to speed up triage, not a gate.