Task Completion (task_comp)
Contents
Metric Description
Task completion is a trace-level metric: it measures whether an agent obtained the requested results on a full OpenInference-format trace. It scores outcomes, not procedure — not whether the SOP was followed (Plan Adherence), whether the plan was a good one (Plan Quality), or whether the path was cheap (Trajectory Efficiency).
The score runs from 0 (applicable outcomes were missed or blocked) to 100 (every applicable outcome is done). It is the mean credit of applicable outcomes: done 1, partial 0.5, missed 0, blocked 0. not_applicable (a conditional outcome whose condition was not met) is excluded from the average. If every outcome is not_applicable, the metric returns no score (null): there is nothing applicable to score. Extra work on the trajectory is ignored.
Where the criteria come from
- If you pass
definition_of_doneinmetric_args, that free-text success criteria is split into atomic outcomes. The value must be a non-empty string; an empty string,null, or a non-string fails evaluation. - If you omit the key, the metric generates atomic outcomes from the trace’s system and user prompts. Confirmation codes and similar proof artifacts are included only when the prompts asked for them.
- If neither source yields usable outcomes, evaluation exits with no score.
How it is judged
- Each outcome is scored from the agent’s final answer: the last LLM span in the trace, using the last non-empty assistant text on that span. If that last LLM span has no text (a tool-call-only decision turn), this pass is skipped.
- Outcomes the close-out cannot decide (
not_enough_data) or omits are scored from the compact trajectory, including tool/retriever outputs. That pass must pick a scoring status; it cannot returnnot_enough_data. - Outcomes the final answer can already decide (including
"Reserved!"when the definition of done did not require an id) are not re-judged on the trace.
This metric does not score plan quality, SOP adherence, tool-name correctness, or efficiency. Pair it with Plan Adherence when procedure fidelity matters, with Plan Quality when the executed work should be judged against a generated gold plan, and with Trajectory Efficiency when path cost matters.
How to interpret the score
- Closer to 100: applicable outcomes were largely obtained.
- Closer to 0: many applicable outcomes were missed, only partial, or blocked.
- No score (
null): every outcome was not applicable, so completion could not be assessed.
Evaluation fails when the payload is not a non-empty list of span objects, when definition_of_done is present but empty or not a string, when no system or user prompts are found (and definition_of_done was omitted), when no success criteria can be extracted or generated, when no outcome could be decided from the final answer and the trace has no actionable steps, or when every outcome is not applicable.
API usage
Prerequisites
After the environment variables are configured, the next step is to create a JSON payload. task_comp scores a whole trace and is rejected on custom runs. An inline trace is scored with POST /evaluate: send trace and omit span_id. A stored trace is scored with POST /runs/trace.
Shortname: task_comp
Default threshold: 80
Inputs (each object in data)
trace(list[object]required): An OpenInference-format trace — an ordered list of span dictionaries that share atrace_idand are linked byparent_id. The metric reads, among other fields:- Identity and nesting —
context.span_id,parent_id,openinference.span.kind. - LLM conversation —
llm.input_messages.*(a non-empty system or user prompt is required whendefinition_of_doneis omitted),llm.output_messages.*(final answer: last LLM span, last non-empty assistant text). - Tool / retriever payloads —
tool.name,input.value/output.value(used when the final answer is not enough to score an outcome).
- Identity and nesting —
For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.
metric_args
definition_of_done(str, optional): Ground-truth success criteria as free text. When the key is omitted, atomic outcomes are generated from the trace’s system and user prompts. When the key is present, it must be a non-empty string.
Evaluation metadata
On successful evaluation, the metric returns eval_metadata with:
outcomes(dict[int, str]): The atomic outcomes — each index mapped to its text.
When the score is present and below 100, it also includes:
incomplete_outcomes(list[dict]): Outcomes that are notdoneornot_applicable, each with:outcome(str): Outcome text.status(str): One ofpartial,missed, orblocked.reason(str): Concise justification for that status.
Example: inline trace
POST /evaluate. Send trace and omit span_id.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_EVALUATE_URL = f"{_BASE_URL}/evaluate"
def post_evaluate(payload: dict) -> requests.Response:
"""POST JSON payload to single evaluation; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_EVALUATE_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
# Minimal OpenInference-style trace (list of spans). Real traces include more.
trace = [
{
"name": "AgentRun",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x145e3c64b3b99a77",
},
"parent_id": None,
"start_time": "2026-07-08T11:07:57.720000Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "Table Host",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x083b5fc1127d658a",
},
"parent_id": "0x145e3c64b3b99a77",
"start_time": "2026-07-08T11:07:57.725590Z",
"end_time": "2026-07-08T11:08:00.332983Z",
"status": {"status_code": "OK"},
"attributes": {"openinference.span.kind": "AGENT"},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x64e5562a83068b87",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:57.729073Z",
"end_time": "2026-07-08T11:07:59.573356Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You are a dining assistant."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Reserve a table for 4 in Lisbon on 18 Nov, "
"prix fixe under $80."
),
"llm.tools.0.tool.json_schema": {
"type": "function",
"function": {
"name": "search_tables",
"parameters": {"type": "object"},
},
},
"llm.output_messages.0.message.tool_calls.0.tool_call.id": (
"call_search"
),
"llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
"search_tables"
),
},
},
{
"name": "search_tables",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0x6c9868e6283f0ed1",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.574350Z",
"end_time": "2026-07-08T11:07:59.589819Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "TOOL",
"tool.name": "search_tables",
"tool.call.id": "call_search",
"input.value": {
"city": "Lisbon",
"date": "18 Nov",
"party": 4,
},
"output.value": "tables: [{id: TB-7, price: 48}]",
},
},
{
"name": "response",
"context": {
"trace_id": "0x3c25552ecfec75f0dfa5baad5a544c4c",
"span_id": "0xbbbbbbbbbbbbbbbb",
},
"parent_id": "0x083b5fc1127d658a",
"start_time": "2026-07-08T11:07:59.610000Z",
"end_time": "2026-07-08T11:08:00.300000Z",
"status": {"status_code": "OK"},
"attributes": {
"openinference.span.kind": "LLM",
"llm.input_messages.0.message.role": "system",
"llm.input_messages.0.message.content": (
"You are a dining assistant."
),
"llm.input_messages.1.message.role": "user",
"llm.input_messages.1.message.content": (
"Reserve a table for 4 in Lisbon on 18 Nov, "
"prix fixe under $80."
),
"llm.output_messages.0.message.role": "assistant",
"llm.output_messages.0.message.contents.0.message_content.type": (
"text"
),
"llm.output_messages.0.message.contents.0.message_content.text": (
"I found a matching Lisbon table at $48. Reserved!"
),
},
},
]
# Send trace and omit span_id.
payload = {
"metric_shortname": "task_comp",
"threshold": 80,
"model_slug": "o4-mini",
"trace": trace,
# Omit definition_of_done to generate atomic outcomes
# from system + user prompts in the trace instead.
"metric_args": {
"definition_of_done": (
"A matching table for 4 in Lisbon on 18 Nov, "
"with prix fixe under $80, was found. "
"A reservation for that table exists."
),
},
}
response = post_evaluate(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))
Example: stored trace
POST /runs/trace. trace_id is the integer id of the stored trace. Omit spans.
import json
import os
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_TRACE_RUN_URL = f"{_BASE_URL}/runs/trace"
def post_trace_run(payload: dict) -> requests.Response:
"""POST JSON payload to a trace run; returns the raw response."""
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {_API_KEY}",
}
return requests.post(
_TRACE_RUN_URL,
headers=headers,
data=json.dumps(payload),
)
if __name__ == "__main__":
payload = {
"trace_id": 42,
"threshold": 80,
"model_slug": "o4-mini",
"is_blocking": True,
"selected_metrics": {
"task_comp": {
"threshold": 80,
# Omit definition_of_done to generate atomic outcomes
# from system + user prompts in the trace instead.
"metric_args": {
"definition_of_done": (
"A matching table for 4 in Lisbon on 18 Nov, "
"with prix fixe under $80, was found. "
"A reservation for that table exists."
),
},
},
},
}
response = post_trace_run(payload)
response.raise_for_status()
print(json.dumps(response.json(), indent=2))