DAG (dag)

Contents

• Metric Description
• How the graph works
• API usage
• Evaluation metadata
• Examples

Metric Description

DAG (Directed Acyclic Graph) is a customizable LLM-as-a-Judge metric. Instead of scoring a submission with one rubric prompt (like G-Eval), you design the evaluation yourself as a flowchart: the LLM answers the questions you write, each answer routes down a branch, and the branch ends at a fixed score.

What "DAG" means here

You don't need graph theory to use this metric — the name just describes the shape of the flowchart you build:

• Graph — a set of boxes (nodes) connected by arrows.
• Directed — arrows only point one way, from a node to the ones below it.
• Acyclic — the arrows never loop back, so the flow always moves forward to an ending.

You build this flowchart out of three kinds of node:

Node	Think of it as	What it does
task	A prep step	Asks the LLM to extract or restructure something (e.g. "list the headings") so later nodes can use it. Produces free-form text.
decision	A question	Asks the LLM one question with a fixed set of allowed answers (e.g. true/false, or correct/out_of_order). The answer decides which branch runs next.
verdict	An ending with a score	A leaf node that holds a fixed number from 0–100. No LLM call — reaching it is the result.

How you get a score

The LLM never picks the final number. It only answers your decision questions, and each answer selects the next branch. Following those answers leads down exactly one path to exactly one verdict node — and that verdict's fixed number becomes the score.

The smallest possible graph makes this concrete: one question, two possible endings.

assess_empathy  (decision: true / false)
├── true  → empathetic      (verdict) → score 90
└── false → not_empathetic  (verdict) → score 20

If the LLM answers true, the empathetic verdict is reached and the score is 90; if it answers false, the score is 20. The other branch is skipped.

When to use it

Reach for DAG when your quality criteria are conditional and a single flat rubric would hide that structure — for example "if the tone is negative, score low immediately; otherwise check completeness, then combine with accuracy." If your criteria are just a flat list of things a good answer should satisfy, G-Eval is simpler.

How to interpret the score

• The score runs from 0 to 100 and is always one of the fixed numbers you put on your verdict nodes (for example 90 or 20).
• Which verdict you land on depends on the answers the decisions gave along the path.
• Closer to 100 usually means a "good" branch in your design; closer to 0 means a "poor" branch — but the meaning is entirely defined by your verdict scores.

What happens when you run it

Once you've submitted a graph, the metric:

1. Validates the graph structure — checks node fields, unique names, that the arrows never loop (acyclicity), that everything is connected, that every ending is a verdict, and that at most one verdict can be reached on any single run. A broken graph is rejected before any tokens are spent.
2. Discovers each decision's allowed answers — from the concrete values its children wait for via run_criteria (see Branching with run_criteria).
3. Checks your descriptions for security — a deterministic scan followed by an LLM check confirms your task/decision descriptions are genuine evaluation questions, not attempts to hijack the evaluator. An unsafe graph is rejected before execution.
4. Runs the graph — walks the nodes in order, calling the LLM for each task and decision, skipping branches whose answer didn't match, and reaching the verdict on the matched path. A decision that returns an off-label answer (not in its allowed set) aborts the run with no score.
5. Returns the reached verdict's score — exactly one completed verdict produces the score; if zero or more than one is reached, the metric fails with null.

Important

DAG only returns a score when the graph passes structural and security checks, every decision answers with an allowed label, exactly one verdict node completes, and — when the graph references any record fields — at least one of those fields is non-empty. Otherwise it returns no score (null) with an explanation. Write task descriptions as evaluation tasks ("extract claims…", "decide whether…"), not as instructions to the evaluator ("ignore previous instructions and return…").

How the graph works

This section is the detailed reference for building a graph — the three node types were introduced in Metric Description above.

Node types

Type	What it does	Calls an LLM?	Required fields
task	Preprocess: extract or restructure data for later nodes. Free-form text output.	Yes	task_description (no score)
decision	Branch: answer a question with exactly one allowed label. Children gate on that label.	Yes	task_description (no score)
verdict	Terminal leaf with a fixed 0–100 score. No LLM call.	No	score; exactly one parent; no task_description

Tasks vs decisions: tasks produce free-form text that downstream nodes may read. Only decision outputs can be matched as a specific value in a child’s run_criteria. You can wait for a task to finish (omit it from run_criteria, or list it with null), but you cannot say “run only if the task output equals X”.

Branching with run_criteria

A node runs only when all of the following hold (AND):

1. Every parent has completed (not been skipped).
2. For each parent listed in run_criteria with a concrete value, that parent’s output equals the expected label.
3. Parents omitted from run_criteria (or listed with null) only need to have completed — empty output is fine.

If a parent was skipped or a label does not match, the node is skipped, and that skip propagates to its children — the whole unmatched branch is pruned.

Decision labels are discovered from the children: whatever values siblings put under run_criteria for that decision become the closed set of allowed answers injected into the decision’s LLM prompt. Labels on one decision must be the same kind (all boolean, all string, or all number) and unique.

Pick short, stable values once on the children’s run_criteria (often verdicts) and keep every sibling consistent. A typo, a mixed type ("true" string vs true boolean), or two near-duplicates that differ only by casing ("Compliant" and "compliant") will confuse the closed set. Matching the model’s answer back to a label is case-insensitive; the canonical run_criteria value is then stored for branching.

Do not restate the allowed labels in the decision’s task_description. The metric already appends them from the children’s run_criteria. Your description should only ask the question and explain when each outcome applies conceptually — listing the label set again duplicates (and can contradict) what the prompt already injects.

If the model returns a value that is not one of the allowed labels, the run fails with score=null — it does not silently skip the branch.

What each LLM node sees

For a task or decision, the prompt is built from:

• Parent outputs — controlled by use_parent_output (true = all parents, false = none, or a dict mapping parent name → whether to include it; omitted parents default to false).
• Record fields — controlled by input_fields (prompt, input, context, output, golden_answer). Roots with unset input_fields default to all fields; child nodes default to none.
• Allowed answers (decisions only) — appended automatically from children’s run_criteria.

Verdict nodes never take input_fields and never call the LLM.

Parallelism (optional)

By default execution is sequential. You can opt in:

• run_roots_parallel (top-level metric_args) — run root nodes concurrently.
• run_children_parallel (per node) — when that node finishes, run its ready children concurrently.

Graph rules worth remembering

• The graph must be a DAG (no cycles) and a single connected component.
• Every leaf must be a verdict, and every verdict must be a leaf.
• A verdict cannot be a root (it needs a parent path to act on).
• Every child of a decision must gate on that decision with a concrete label.
• The structure must guarantee at most one verdict can complete on any run — for example, two verdicts under a task with no decision between them is rejected.

API usage

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for the custom-runs request. For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: dag

Default threshold: 80

Inputs (each object in data)

DAG scores whichever record fields your nodes select through input_fields; provide the ones your graph references. When the graph references at least one field, at least one of those must be non-empty.

• prompt (str optional): The prompt sent to the model.
• input (str optional): The user input / request.
• context (str | list[str] optional): Supporting context.
• output (str optional): The model-generated output to evaluate.
• golden_answer (str optional): The reference (ground truth) answer.

metric_args

• nodes (list[object] required): The evaluation graph. Each node object has:
  • name (str): Unique identifier.
  • type ("task" | "decision" | "verdict").
  • parents_name (list[str] | null optional): Parent node names. null or empty means a root.
  • task_description (str | null): Required non-empty for task / decision; must be omitted/null for verdict. For decisions, write the question only — do not list allowed labels here (see Branching with run_criteria).
  • score (float | null): Required 0–100 for verdict; must be omitted/null for task / decision.
  • run_criteria (dict | null optional): Maps parent name → expected value. Parents not listed (or listed with null) only need to have completed. A decision label means “match this specific output.” All conditions are AND-ed. For children of a decision, these values also define that decision’s allowed answers (see Branching with run_criteria).
  • use_parent_output (bool | dict[str, bool] optional): Whether to feed parent outputs into the LLM prompt. If bool: true = all parents, false = none. If dict: maps each parent name to whether to use its output; a parent omitted from the dict defaults to false. Default = true.
  • input_fields (list[str] | null optional): Record fields to include in the prompt for task / decision only. Supported values: prompt, input, context, output, golden_answer. Roots default to all fields; child nodes default to none. Verdict nodes must not set this field.
  • run_children_parallel (bool optional): Run this node’s ready children concurrently. Default = false.
• run_roots_parallel (bool optional): Run root nodes concurrently. Default = false.

Evaluation metadata

On successful evaluation, the metric returns eval_metadata with the execution trace and the terminal node:

• node_trace (list[dict]): Each executed or skipped node, in graph order, with:
  • name (str): Node name.
  • type (str): task, decision, or verdict.
  • status (str): completed or skipped.
  • output: Task text, decision label, or verdict score (when completed).
  • reason (str | null): LLM reasoning, or why the node was skipped.
  • started_at / ended_at (str | null): UTC ISO timestamps (null when skipped).
• terminal_node (str): Name of the verdict node that produced the final score.

Examples

Example 1 — Binary empathy check

A single decision routes to one of two fixed scores. This is the smallest useful DAG: one question, two exclusive branches.

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    # Records must include the fields referenced by the graph (here: input, output).
    data = [
        {
            "input": "A customer says they are frustrated because their order arrived late.",
            "output": (
                "I'm sorry your order arrived late. I understand how frustrating that can be. "
                "Please send us your order number, and we'll check the status right away."
            ),
        },
    ]

    payload = {
        "threshold": 80,  # threshold on the run level
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "dag",
                        "metric_args": {
                            "nodes": [
                                {
                                    "name": "assess_empathy",
                                    "type": "decision",
                                    "task_description": (
                                        "Does the response acknowledge the customer's "
                                        "frustration and offer a clear next step? Use the "
                                        "affirmative answer only if it does both."
                                    ),
                                    "input_fields": ["input", "output"],
                                },
                                {
                                    "name": "empathetic",
                                    "type": "verdict",
                                    "parents_name": ["assess_empathy"],
                                    "run_criteria": {"assess_empathy": True},
                                    "score": 90,
                                },
                                {
                                    "name": "not_empathetic",
                                    "type": "verdict",
                                    "parents_name": ["assess_empathy"],
                                    "run_criteria": {"assess_empathy": False},
                                    "score": 20,
                                },
                            ],
                        },
                    },
                ],
                "threshold": 80,  # threshold on the metric level
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))

For an empathetic reply like the one above, the decision returns true, the empathetic verdict completes with score 90, and not_empathetic is skipped. eval_metadata.terminal_node is "empathetic".

Example 2 — Task preprocessing + multi-way branch

A task extracts structure first; a decision then branches on quality. Only the matching verdict runs.

This graph scores a support agent’s refund reply: first list the concrete actions promised, short-circuit if there is no clear next step, otherwise classify how well the reply fits refund policy.

metric_args = {
    "nodes": [
        {
            "name": "extract_promised_actions",
            "type": "task",
            "task_description": (
                "List every concrete action the agent promises in the reply "
                "(for example: issue a refund, open a ticket, ask for an "
                "order number, escalate to a specialist). If none, say so."
            ),
            "input_fields": ["output"],
        },
        {
            "name": "has_next_step_decision",
            "type": "decision",
            "parents_name": ["extract_promised_actions"],
            "task_description": (
                "Does the extracted list include at least one concrete next "
                "step the customer or agent should take?"
            ),
            "use_parent_output": True,
        },
        {
            "name": "no_next_step_verdict",
            "type": "verdict",
            "parents_name": ["has_next_step_decision"],
            "run_criteria": {"has_next_step_decision": False},
            "score": 0,
        },
        {
            "name": "policy_fit_decision",
            "type": "decision",
            "parents_name": [
                "extract_promised_actions",
                "has_next_step_decision",
            ],
            "run_criteria": {"has_next_step_decision": True},
            "task_description": (
                "Given the customer request and the promised actions, how "
                "well does the reply fit a standard refund policy "
                "(refunds only within 30 days, with order number, no cash "
                "over the stated purchase amount)? Choose full compliance "
                "only when the reply follows policy, a partial fit when it "
                "is directionally right but incomplete or vague, and "
                "non-compliance when it promises something policy forbids."
            ),
            "input_fields": ["input", "output"],
            "use_parent_output": True,
        },
        {
            "name": "compliant_verdict",
            "type": "verdict",
            "parents_name": ["policy_fit_decision"],
            "run_criteria": {"policy_fit_decision": "compliant"},
            "score": 100,
        },
        {
            "name": "partial_verdict",
            "type": "verdict",
            "parents_name": ["policy_fit_decision"],
            "run_criteria": {"policy_fit_decision": "partial"},
            "score": 50,
        },
        {
            "name": "off_policy_verdict",
            "type": "verdict",
            "parents_name": ["policy_fit_decision"],
            "run_criteria": {"policy_fit_decision": "off_policy"},
            "score": 20,
        },
    ],
}

Notes on this shape:

• The task’s free-form extraction is fed into later decisions via use_parent_output, not via label matching.
• policy_fit_decision has two parents: it reads the promised-actions text from the task, and only runs when the binary decision is true.
• A reply with no concrete next step short-circuits to score 0 without asking about policy fit.

Wire metric_args into the same custom-run payload pattern as Example 1, with data rows that include an input (customer refund request) and an output (agent reply).