Skip to main content

Step 4: Automate evaluation

Manual runs prove the concept; automation is what actually protects quality. There are three automation lanes, and it matters which parts of each are done for you versus by you.

CI/CD regression gating​

Wire your golden dataset into the pipeline so every change is measured before it ships. This is the primary automated lane.

  1. On each push, pull request, or pre-deploy step, have your app produce fresh outputs for the golden inputs (this generation step is yours; Aegis evaluates outputs, it does not call your model for you).
  2. Send those outputs to Aegis as a custom run, or start a dataset run against a stored dataset, using your CI's project-scoped API key. Run it blocking (is_blocking true) so the pipeline waits for scores; a blocking run returns the aggregate result, per-metric aggregates, and every evaluation's is_gte_threshold.
  3. Gate the pipeline yourself on the returned flags: fail the build if the run's aggregate falls below its threshold, or if any critical metric's is_gte_threshold is false, or if any evaluation came back is_success false (a metric that errored is not a pass). Aegis gives you the numbers; the decision to block the deploy is code you write around the response. There is no built-in "reject the deploy" webhook.
  4. Treat is_success false as a hard failure of the check, distinct from a low score — it usually means missing data or a misconfigured metric, and it must not be read as a pass.

Production monitoring via trace ingestion​

For live agent and LLM workflows, export OpenTelemetry traces to Ingest traces (POST /api/v1/traces/ingest) continuously. This captures real behaviour (timings, tokens, cost, inputs/outputs, tool calls) as trace and span rows you can inspect and, for agentic workflows, evaluate. This lane is about observing production, not gating a deploy.

Scheduled/recurring evaluation​

Decide the cadence (nightly, hourly, per-release) and drive it from your own scheduler (a cron job, a CI schedule, or your orchestrator), because Aegis has no built-in scheduler for recurring runs. Each tick pulls the current slice of your data and posts a run exactly like the CI lane does.

Production sampling​

If you evaluate live traffic, sample on your side before sending because Aegis has no built-in sample-rate or tail-sampling control. Choose a strategy (a fixed percentage, all errors and a sample of successes, or all low-confidence responses) in your own code, then push only the sampled rows to a run or ingest only the sampled traces. This keeps cost bounded and is entirely under your control.

What is manual vs. automated​

  1. Manual, human-driven: curating the golden dataset, calibrating thresholds, reading explanations to diagnose regressions, deciding when to add or retire a metric, reviewing dashboards.
  2. Automated, machine-driven (by you, using the API): generating fresh outputs, posting runs on push or on a schedule, polling for completion, reading is_gte_threshold and is_success, and blocking or alerting on the result.
  3. Automated, done for you (by Aegis): scoring each record against each metric, computing thresholds and aggregates, writing explanations, storing results and traces, and rolling everything up on dashboards.