Trajectory Efficiency (traject_eff)
Contents
Metric Description
Trajectory efficiency is a trace-level metric: it measures whether an agent reached its goal efficiently across a full OpenInference-format trace (an ordered list of spans). It scores path efficiency — how much work, time, and tokens were needed relative to a more efficient way to achieve the same outcome from the steps the trace already contains.
The score runs from 0 (wasteful / poorly executed) to 100 (efficient). It is a weighted combination of three axes:
- Execution quality — whether decision steps and their tool (or close-out) actions were warranted and well-formed for the task.
- Duration efficiency — how the trace’s busy time compares to a more efficient structuring of the same work (for example, avoiding redundant decision rounds or better use of parallelism).
- Token efficiency — how much token usage came from steps that were actually needed, versus redundant, unused, or skippable work.
If execution quality cannot be computed, the score is based on duration and token efficiency alone.
What the metric takes into account (at a high level):
- Whether tool calls and decision steps were necessary, optional, incorrect, malformed, or harmful.
- Intermediate results that were produced but never used downstream.
- Decision rounds that become redundant once actions are structured more efficiently.
- Span duration (
start_time/end_time) and token usage on the trace.
The metric does not invent missing tool calls; it only reassesses work already present in the trace.
This metric does not score whether the requested results were obtained, and it does not enforce plan adherence or gold-plan quality. It ignores order-related instructions in the user or system prompts when a more efficient structure exists — for example, running independent steps in parallel instead of sequentially. Pair it with Plan Adherence when you need fidelity to the intended procedure from user or system instructions (or an expected_plan), with Plan Quality when you need the executed work judged against a generated gold plan, and with Task Completion when you need whether the requested results were obtained.
How to interpret the score
- Closer to 100: steps were largely warranted, and little duration or token spend looks redundant.
- Closer to 0: many steps were unnecessary, incorrect, or unused, and/or a large share of busy time and tokens was spent on work that could have been avoided or restructured.
Evaluation fails when the payload is not a non-empty list of span objects, when no processable spans remain, or when the trace has no usable duration or token usage — the metric needs both signals to score meaningfully.
API usage
Prerequisites
After the environment variables are configured, the next step is to create a JSON payload. traject_eff scores a whole trace and is rejected on custom runs. An inline trace is scored with POST /evaluate: send trace and omit span_id. A stored trace is scored with POST /runs/trace.
Shortname: traject_eff
Default threshold: 80
Inputs (request body for POST /evaluate; on POST /runs/trace the stored trace is used)
trace(list[object]required): An OpenInference-format trace — an ordered list of span dictionaries that share atrace_idand are linked byparent_id. The metric reads, among other fields:- Identity and nesting —
context.span_id,parent_id,openinference.span.kind,status.status_code. - Timing —
start_time/end_time(required for duration scoring). - Token usage — e.g.
llm.token_count.prompt/.completion/.totalwhen present (required for token scoring). - LLM conversation and tool calls —
llm.input_messages.*,llm.output_messages.*.message.tool_calls.*. - Tool / retriever payloads —
tool.name,input.value/output.value, and related attributes.
- Identity and nesting —
For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.
Evaluation metadata
On successful evaluation, the metric may return eval_metadata with only the findings that apply:
duration_waste_ratio(float): Fraction of busy duration estimated as waste (present when duration is suboptimal).tokens_waste_ratio(float): Fraction of tokens estimated as waste (present when token use is suboptimal).redundant_decision_indexes(list[int]): Decision-span indexes that become redundant under a more efficient structure.reparented_actions(dict[int, str]): Actions whose ideal parent decision differs from the one that issued them — each action index mapped to"{span}:{wave} -> {span}:{wave}".redundant_steps_due_to_unused_results(list[int], optional): Step indexes whose outputs were never consumed downstream.imperfect_steps(list[dict], optional): Decision targets that were not labeled fully correct.
Each item in imperfect_steps has:
index(int): Step index in the processed trace.label(str): One ofuseful_but_optional,not_required,indeterminate,incorrect,malformed_arguments, orharmful.reason(str): Concise justification for that label.
Example: inline trace
POST /evaluate. Send trace and omit span_id.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_EVALUATE_URL = f"{_BASE_URL}/evaluate"
def post_evaluate(payload: dict) -> requests.Response:
"""POST JSON payload to single evaluation; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_EVALUATE_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
# Minimal OpenInference-style trace (list of spans). Real traces include more.
trace = [
{
"name": "AgentRun",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x145e3c64b3b99a77",
},
"parent_id": None,
"start_time": "2026-07-08T11:07:57.720000Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "Bash Agent",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x083b5fc1127d658a",
},
"parent_id": "0x145e3c64b3b99a77",
"start_time": "2026-07-08T11:07:57.725590Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x64e5562a83068b87",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:57.729073Z",
"end_time": "2026-07-08T11:07:59.573356Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You complete tasks by running shell commands with the Bash tool."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Run two bash commands and return both counts."
),
"llm.tools.0.tool.json_schema": {
"type": "function",
"function": {
"name": "bash",
"parameters": {"type": "object"},
},
},
"llm.output_messages.0.message.tool_calls.0.tool_call.id": "call_a",
"llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
"bash"
),
"llm.output_messages.0.message.tool_calls.1.tool_call.id": "call_b",
"llm.output_messages.0.message.tool_calls.1.tool_call.function.name": (
"bash"
),
"llm.token_count.prompt": 113,
"llm.token_count.completion": 59,
"llm.token_count.total": 172,
},
},
{
"name": "bash",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x6c9868e6283f0ed1",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.574350Z",
"end_time": "2026-07-08T11:07:59.589819Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "TOOL",
"tool.name": "bash",
"tool.call.id": "call_a",
"input.value": {"command": "printf 'hello' | wc -c"},
"output.value": "5\n",
},
},
{
"name": "bash",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0xaaaaaaaaaaaaaaaa",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.590000Z",
"end_time": "2026-07-08T11:07:59.600000Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "TOOL",
"tool.name": "bash",
"tool.call.id": "call_b",
"input.value": {"command": "printf 'world' | wc -c"},
"output.value": "5\n",
},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0xbbbbbbbbbbbbbbbb",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.610000Z",
"end_time": "2026-07-08T11:08:00.300000Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You complete tasks by running shell commands with the Bash tool."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Run two bash commands and return both counts."
),
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.contents.0.message_content.type": "text",
"llm.output_messages.0.message.contents.0.message_content.text": "5\n5",
"llm.token_count.prompt": 150,
"llm.token_count.completion": 50,
"llm.token_count.total": 200,
},
},
]
# Send trace and omit span_id.
payload = {
"metric_shortname": "traject_eff",
"threshold": 80,
"model_slug": "o4-mini",
"trace": trace,
}
response = post_evaluate(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))
Example: stored trace
POST /runs/trace. trace_id is the integer id of the stored trace. Omit spans.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_TRACE_RUN_URL = f"{_BASE_URL}/runs/trace"
def post_trace_run(payload: dict) -> requests.Response:
"""POST JSON payload to a trace run; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_TRACE_RUN_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
payload = {
"trace_id": 42,
"threshold": 80,
"model_slug": "o4-mini",
"is_blocking": True,
"selected_metrics": {
"traject_eff": {
"threshold": 80,
},
},
}
response = post_trace_run(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))