Step 3: Baseline in development

Before you gate anything, learn what "normal" looks like. Thresholds set from intuition are usually wrong; set them from a baseline run over real examples.

Assemble a representative sample

Take a set of real (or realistic) requests and their outputs from your stored data — include the easy cases, the hard cases, and the ones that have burned you before.

Run the suite once, manually

Upload the sample as a dataset and start a dataset run (Create run from dataset, POST /api/v1/runs/dataset), or fire a custom run (Create custom run, POST /api/v1/runs/custom) with the rows inline. For a baseline you want the full picture, so run it blocking (is_blocking true) so the response comes back with scores already computed, or poll the run until it finishes (see Step 5).

Read the explanations, not just the scores

Every evaluation carries a written justification. Read the low scorers to confirm the metric is penalizing what you actually care about, and the high scorers to confirm it is not rewarding the wrong thing. This is where you catch a mis-mapped field or a metric that does not mean what its name suggests.

Calibrate thresholds to observed reality

Look at the distribution of scores for each metric on outputs you consider acceptable versus unacceptable, and set the threshold between them — with a margin, because LLM-judge scores vary run to run. Deterministic (structural) metrics do not have this variance; lean on them where you need a crisp line.

Freeze a golden dataset

Once the suite is calibrated, save the curated examples as a persistent dataset with its metrics and thresholds attached. This becomes the regression suite you re-run in automation — the fixed yardstick that a new model, prompt, or code change is measured against.

Spot-check with single evaluate while iterating

When you are tweaking one prompt or one metric, single evaluation (POST /api/v1/evaluate) for a single output gives you an immediate score and explanation without spinning up a whole run.