Skip to main content

Core concepts

Learn these six words first; the rest of the methodology is built on them.

  • Record: one row you want judged, the unit of data. Its fields are external_id, prompt, input, context, output, and golden_answer (the expected/reference answer). Metrics read some subset of these; you do not have to populate all of them.
  • Metric: one scoring function (for example ans_rel, faith, ctx_recall). It reads the fields it requires, returns a score on 0–100, and carries a default threshold. Metrics are grouped into categories (general, RAG, content generation, structural, security, safety, agentic).
  • Evaluation: the result of running one metric against one record. It holds result (the 0–100 score), threshold, is_gte_threshold (pass/fail), is_success (did it run), and a human-readable explanation.
  • Run: a batch job that produces many evaluations. A dataset run scores every record in a dataset against that dataset's metrics; a custom run scores inline rows you send in the request. A run has an aggregate result (mean score), aggregate_results (per-metric averages), and its own threshold.
  • Dataset: a named, reusable set of records. This is your test suite that lives in Aegis. You can store an optional default set of metrics and thresholds on it; a dataset run can reuse that default or send its own metrics for that run. A data collection is a folder that groups datasets, runs, and traces together.
  • Trace and span: OpenTelemetry telemetry from a running agent, ingested separately from the record/run path. This is the production/agentic monitoring lane, not the batch-evaluation lane.

Two flags​

Two flags decide everything downstream, and confusing them is the most common mistake:

  1. is_success: the metric executed and produced a score. False means the metric errored or the required fields were missing, so there is no score at all (result is null).
  2. is_gte_threshold: the score met or beat the threshold (score >= threshold). This is your pass/fail signal. It is only meaningful when is_success is true.

Access model​

  1. Everything you create is owned by an user and/or an organization, and can be scoped to a project. Org-shared data rolls up into org dashboards and billing; personal data does not.
  2. An API key is either private (personal visibility) or project-scoped (locked to one project, where it can only see and create resources in that project). Choose deliberately; a project key silently narrows what a script can touch.