Skip to main content

Step 2: Choose metrics and set thresholds

A metric is a question you are asking about each output. Choose the questions that match your workflow, and make sure your data actually carries the fields those questions need.

Start from the metric categories and pick what fits your system​

Aegis has the following metric categories:

  1. General (core answer quality): answer relevancy, faithfulness, answer correctness, summarization, factfulness, content consistency, prompt quality, entity faithfulness, plus configurable judges G-Eval and DAG.
  2. RAG (retrieval quality): context relevancy, context recall, context sufficiency, context faithfulness, context ranking precision, context waste. Use these when your workflow retrieves before it answers.
  3. Content generation: format alignment, format consistency, content-generation faithfulness. Use when you generate structured or templated content.
  4. Structural: deterministic, non-LLM checks: valid JSON/SQL/XML/Python, schema match, exact/numeric match, BLEU/ROUGE, counts and readability. These are cheap, fast, and reproducible — prefer them for hard gates wherever a rule can express the check.
  5. Security: prompt-injection, PII/PHI leakage and exfiltration, role hijack, system-data leakage.
  6. Safety: toxicity, bias, harmfulness, misuse, manipulation, role violation.
  7. Agentic: plan adherence/quality, tool selection, trajectory efficiency, evaluated over traces. Agentic metrics are largely inactive today, and the single-evaluate API does not yet expose trace/span inputs — drive agentic evaluation through trace ingestion.

See the metrics introduction for how scores are computed and how each metric page is organized.

Check each metric's required fields before you commit​

A metric only runs if the record carries what it needs. For example answer relevancy needs input-or-prompt plus output; answer correctness needs output plus golden_answer; context faithfulness needs context plus output; prompt quality needs prompt. If a required field is missing, that evaluation fails (is_success false) rather than scoring low, so a wrong data map looks like a broken metric.

Set a threshold per metric, on the same 0–100 scale as the score​

The threshold is your definition of "good enough" — the minimum score that counts as a pass (is_gte_threshold = score >= threshold). Each metric ships a sensible catalog default (many LLM judges default around 80; several security/safety checks default to a strict 100; BLEU/ROUGE default lower). Override per dataset, per run, per evaluation group, or per metric.

Tune metric_args where a metric exposes them​

Some metrics take arguments (for example a cap on how many claims or statements to extract). These change how the metric behaves; set them once and keep them fixed within a suite for comparability.

Keep the suite honest and small at first​

Pick the two or three metrics that actually correspond to how your users judge the output, get those thresholds right against real data, then expand. A large suite of poorly-thresholded metrics produces noise, not signal.