Plan Quality (plan_qual)
Contents
Metric Description
Plan quality is a trace-level metric: it measures whether an agent executed a high-quality plan across a full OpenInference-format trace (an ordered list of spans). It scores the trajectory against a generated gold plan — the procedure a capable agent should follow given the user task, the available tools, and system constraints — not against a user-supplied SOP.
The score runs from 0 (the trajectory largely missed or ignored the gold plan) to 100 (the gold plan was followed). It combines three signals:
- Plan completeness — whether each applicable gold-plan step was fully carried out, only partly carried out, or missed.
- Off-plan work — how much of the trajectory’s actionable work did not contribute to any gold-plan step.
- Ordering — whether performed gold-plan steps ran out of the intended sequence (parallel work issued in the same decision round is allowed).
Where the gold plan comes from
The metric generates the gold plan from the trace. It does not take expected_plan.
- The user prompt is what the user asked for. The gold plan is the best procedure for that request.
- Available tools are the tools the agent has. Gold-plan steps are built from that set, which is chosen in this order:
metric_args.available_tools, when provided. It is used as given and the trace is not searched for tools.- The tool catalogue advertised in the trace.
- When the trace advertises no catalogue (for example, Claude Agent SDK traces), a per-agent list of the tools and subagents each agent actually called. Only names are used, never arguments or results. This list only holds tools that were called, so the metric cannot tell that a needed tool was never called. Pass
available_toolswhen you need that.
- The system prompt supplies constraints (policies, required ids, forbidden actions).
If there is not enough context to generate gold-plan steps, evaluation exits with no score.
This metric does not score fidelity to an intended or extracted procedure (that is Plan Adherence), path efficiency, or whether the requested results were obtained. Pair it with Plan Adherence when you need the trajectory judged against expected_plan or instructions in the prompts, with Trajectory Efficiency when path cost matters, and with Task Completion when you need whether the requested results were obtained.
This metric uses your selected evaluation model for every LLM step. For optimal results, select a higher-capability model than the default.
How to interpret the score
- Closer to 100: applicable gold-plan steps were largely completed, little work looked off-plan, and performed steps kept the gold-plan order.
- Closer to 0: many gold-plan steps were missed or only partial, a large share of work was off-plan, and/or performed steps ran out of order.
Evaluation fails when the payload is not a non-empty list of span objects, when available_tools is not a list of non-empty strings or objects with a name, when no user prompt is found in the trace, when the prompts do not offer enough context to generate a gold plan, or when the trace has no actionable steps to score.
API usage
Prerequisites
After the environment variables are configured, the next step is to create a JSON payload. plan_qual scores a whole trace and is rejected on custom runs. An inline trace is scored with POST /evaluate: send trace and omit span_id. A stored trace is scored with POST /runs/trace.
Shortname: plan_qual
Default threshold: 80
Inputs (each object in data)
trace(list[object]required): An OpenInference-format trace — an ordered list of span dictionaries that share atrace_idand are linked byparent_id. The metric reads, among other fields:- Identity and nesting —
context.span_id,parent_id,openinference.span.kind. - LLM conversation and tool catalogue —
llm.input_messages.*(a non-empty user prompt is required),llm.tools.*(advertised catalogue used to generate the gold plan whenavailable_toolsis not provided),llm.output_messages.*.message.tool_calls.*. - Tool / retriever payloads —
tool.name,input.value/output.value, and related attributes.
- Identity and nesting —
For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.
metric_args
available_tools(object[] | string[], optional): Tool catalogue the gold plan is built from, used as given. Each item is a tool name or an object with aname(and optionally adescriptionandparameters). Use the exact names that appear in the trace (for examplemcp__trip__check_flights, notcheck_flights), and list subagents too, so gold-plan steps line up with the calls the agent made. When set, tools are not read from the trace. When omitted, see Where the gold plan comes from.
{
"metric": "plan_qual",
"metric_args": {
"available_tools": [
"mcp__trip__check_flights",
"mcp__trip__check_weather",
{
"name": "lodging_scout",
"description": "Subagent that runs mcp__trip__check_hotels.",
},
"mcp__trip__check_hotels",
],
},
}
Evaluation metadata
On successful evaluation, the metric returns eval_metadata with:
plan_steps(dict[int, str]): The generated gold plan — each step index mapped to its text.
When the score is below 100, it may also include:
incorrect_plan_steps(list[dict], optional): Gold-plan steps that were not fully completed.incorrect_span_indexes(list[dict], optional): Trajectory steps executed off-plan.out_of_order_steps(list[dict], optional): Gold-plan steps whose execution inverted the intended order.
Each item in incorrect_plan_steps has:
step(str): Gold-plan step text.status(str): One ofpartialormissed.reason(str): Concise justification for that status.
Each item in incorrect_span_indexes has:
index(int): Step index in the processed trace.reason(str): Concise justification for the off-plan label.
Each item in out_of_order_steps has:
step(str): Gold-plan step text.reason(str): Concise justification for the order inversion.
Example: inline trace
POST /evaluate. Send trace and omit span_id.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_EVALUATE_URL = f"{_BASE_URL}/evaluate"
def post_evaluate(payload: dict) -> requests.Response:
"""POST JSON payload to single evaluation; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_EVALUATE_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
# Minimal OpenInference-style trace (list of spans). Real traces include more.
trace = [
{
"name": "AgentRun",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x145e3c64b3b99a77",
},
"parent_id": None,
"start_time": "2026-07-08T11:07:57.720000Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "Trip Planner",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x083b5fc1127d658a",
},
"parent_id": "0x145e3c64b3b99a77",
"start_time": "2026-07-08T11:07:57.725590Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x64e5562a83068b87",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:57.729073Z",
"end_time": "2026-07-08T11:07:59.573356Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You plan trips by researching destinations, checking "
"flights, looking up weather, and suggesting activities."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Plan a short trip to Tokyo: research the destination, "
"check flights, look up weather, then suggest activities."
),
"llm.tools.0.tool.json_schema": {
"type": "function",
"function": {
"name": "search_web",
"parameters": {"type": "object"},
},
},
"llm.output_messages.0.message.tool_calls.0.tool_call.id": (
"call_research"
),
"llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
"search_web"
),
},
},
{
"name": "search_web",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x6c9868e6283f0ed1",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.574350Z",
"end_time": "2026-07-08T11:07:59.589819Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "TOOL",
"tool.name": "search_web",
"tool.call.id": "call_research",
"input.value": {"query": "Tokyo travel guide"},
"output.value": "Tokyo highlights: neighborhoods, transit...",
},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0xbbbbbbbbbbbbbbbb",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.610000Z",
"end_time": "2026-07-08T11:08:00.300000Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You plan trips by researching destinations, checking "
"flights, looking up weather, and suggesting activities."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Plan a short trip to Tokyo."
),
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.contents.0.message_content.type": (
"text"
),
"llm.output_messages.0.message.contents.0.message_content.text": (
"Here is a starter Tokyo plan based on research so far."
),
},
},
]
# Send trace and omit span_id.
payload = {
"metric_shortname": "plan_qual",
"threshold": 80,
"model_slug": "o4-mini",
"trace": trace,
}
response = post_evaluate(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))
Example: stored trace
POST /runs/trace. trace_id is the integer id of the stored trace. Omit spans.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_TRACE_RUN_URL = f"{_BASE_URL}/runs/trace"
def post_trace_run(payload: dict) -> requests.Response:
"""POST JSON payload to a trace run; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_TRACE_RUN_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
payload = {
"trace_id": 42,
"threshold": 80,
"model_slug": "o4-mini",
"is_blocking": True,
"selected_metrics": {
"plan_qual": {
"threshold": 80,
},
},
}
response = post_trace_run(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))