Plan Adherence (plan_adh)
Contents
Metric Description
Plan adherence is a trace-level metric: it measures whether an agent followed the intended plan across a full OpenInference-format trace (an ordered list of spans). It scores procedure fidelity — coverage of the expected steps, whether work stayed on plan, and whether performed steps kept the intended order.
The score runs from 0 (the trajectory largely missed or ignored the plan) to 100 (the plan was followed). It combines three signals:
- Plan completeness — whether each applicable plan step was fully carried out, only partly carried out, or missed.
- Off-plan work — how much of the trajectory’s actionable work did not contribute to any plan step.
- Ordering — whether performed plan steps ran out of the intended sequence (parallel work issued in the same decision round is allowed).
Where the plan comes from
- If you pass
expected_planinmetric_args, that free-text procedure is the ground truth. - If you omit it (or leave it empty), the metric checks the trace’s system and user prompts for explicit instructions around plan execution.
- If neither source yields usable plan steps, evaluation exits with no score.
This metric does not score whether the executed procedure was a good plan for the task (that is Plan Quality), path efficiency, tool-argument quality, or whether the requested results were obtained. Pair it with Plan Quality when you care about an optimal procedure given the task and tools, with Trajectory Efficiency when path cost matters, and with Task Completion when you need whether the requested results were obtained.
How to interpret the score
- Closer to 100: applicable plan steps were largely completed, little work looked off-plan, and performed steps kept the intended order.
- Closer to 0: many plan steps were missed or only partial, a large share of work was off-plan, and/or performed steps ran out of order.
Evaluation fails when the payload is not a non-empty list of span objects, when expected_plan is provided but empty/invalid, when no plan steps can be resolved, when no system or user prompts are found (and no expected_plan was given), or when the trace has no actionable steps to score.
API usage
Prerequisites
After the environment variables are configured, the next step is to create a JSON payload. plan_adh scores a whole trace and is rejected on custom runs. An inline trace is scored with POST /evaluate: send trace and omit span_id. A stored trace is scored with POST /runs/trace.
Shortname: plan_adh
Default threshold: 80
Inputs (each object in data)
trace(list[object]required): An OpenInference-format trace — an ordered list of span dictionaries that share atrace_idand are linked byparent_id. The metric reads, among other fields:- Identity and nesting —
context.span_id,parent_id,openinference.span.kind. - LLM conversation and tool calls —
llm.input_messages.*,llm.output_messages.*.message.tool_calls.*(system/user prompts are used whenexpected_planis omitted). - Tool / retriever payloads —
tool.name,input.value/output.value, and related attributes.
- Identity and nesting —
For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.
metric_args
expected_plan(str, optional): Ground-truth plan as free text. When omitted or empty, the plan is extracted from the trace’s system and user prompts.
Evaluation metadata
On successful evaluation, the metric returns eval_metadata with:
plan_steps(dict[int, str]): The extracted plan — each step index mapped to its text.
When the score is below 100, it may also include:
incorrect_plan_steps(list[dict], optional): Plan steps that were not fully completed.incorrect_span_indexes(list[dict], optional): Trajectory steps executed off-plan.out_of_order_steps(list[dict], optional): Plan steps whose execution inverted the intended order.
Each item in incorrect_plan_steps has:
step(str): Plan step text.status(str): One ofpartialormissed.reason(str): Concise justification for that status.
Each item in incorrect_span_indexes has:
index(int): Step index in the processed trace.reason(str): Concise justification for the off-plan label.
Each item in out_of_order_steps has:
step(str): Plan step text.reason(str): Concise justification for the order inversion.
Example: inline trace
POST /evaluate. Send trace and omit span_id.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_EVALUATE_URL = f"{_BASE_URL}/evaluate"
def post_evaluate(payload: dict) -> requests.Response:
"""POST JSON payload to single evaluation; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_EVALUATE_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
# Minimal OpenInference-style trace (list of spans). Real traces include more.
trace = [
{
"name": "AgentRun",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x145e3c64b3b99a77",
},
"parent_id": None,
"start_time": "2026-07-08T11:07:57.720000Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "Trip Planner",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x083b5fc1127d658a",
},
"parent_id": "0x145e3c64b3b99a77",
"start_time": "2026-07-08T11:07:57.725590Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x64e5562a83068b87",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:57.729073Z",
"end_time": "2026-07-08T11:07:59.573356Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You plan trips by researching destinations, checking "
"flights, looking up weather, and suggesting activities."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Plan a short trip to Tokyo: research the destination, "
"check flights, look up weather, then suggest activities."
),
"llm.tools.0.tool.json_schema": {
"type": "function",
"function": {
"name": "search_web",
"parameters": {"type": "object"},
},
},
"llm.output_messages.0.message.tool_calls.0.tool_call.id": (
"call_research"
),
"llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
"search_web"
),
},
},
{
"name": "search_web",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x6c9868e6283f0ed1",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.574350Z",
"end_time": "2026-07-08T11:07:59.589819Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "TOOL",
"tool.name": "search_web",
"tool.call.id": "call_research",
"input.value": {"query": "Tokyo travel guide"},
"output.value": "Tokyo highlights: neighborhoods, transit...",
},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0xbbbbbbbbbbbbbbbb",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.610000Z",
"end_time": "2026-07-08T11:08:00.300000Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You plan trips by researching destinations, checking "
"flights, looking up weather, and suggesting activities."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Plan a short trip to Tokyo."
),
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.contents.0.message_content.type": (
"text"
),
"llm.output_messages.0.message.contents.0.message_content.text": (
"Here is a starter Tokyo plan based on research so far."
),
},
},
]
# Send trace and omit span_id.
payload = {
"metric_shortname": "plan_adh",
"threshold": 80,
"model_slug": "o4-mini",
"trace": trace,
# Omit expected_plan to extract the plan from
# system + user prompts in the trace instead.
"metric_args": {
"expected_plan": (
"For Tokyo: research the destination, "
"check flight options, look up the weather, "
"then suggest a few activities."
),
},
}
response = post_evaluate(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))
Example: stored trace
POST /runs/trace. trace_id is the integer id of the stored trace. Omit spans.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_TRACE_RUN_URL = f"{_BASE_URL}/runs/trace"
def post_trace_run(payload: dict) -> requests.Response:
"""POST JSON payload to a trace run; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_TRACE_RUN_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
payload = {
"trace_id": 42,
"threshold": 80,
"model_slug": "o4-mini",
"is_blocking": True,
"selected_metrics": {
"plan_adh": {
"threshold": 80,
# Omit expected_plan to extract the plan from
# system + user prompts in the trace instead.
"metric_args": {
"expected_plan": (
"For Tokyo: research the destination, "
"check flight options, look up the weather, "
"then suggest a few activities."
),
},
},
},
}
response = post_trace_run(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))