Tool Selection Correctness (tool_select)

Contents

• Metric Description
• API usage
• Evaluation metadata

Metric Description

Tool selection correctness is a span-level metric: it measures whether an agent chose the right tools for its task on a single OpenInference-format LLM span. It scores tool selection only — not tool arguments, execution results, call ordering, or final-answer quality.

The score runs from 0 (poor selection) to 100 (correct selection). Scoring uses one of two modes, selected by metric_args:

1. Ground-truth mode (expected_tools provided): deterministic on exact tool-name match (call multiplicity counts). Score is 100 when the multiset of called tools equals the expected multiset; 0 when no tools were called but some were expected. Partial overlap yields an intermediate score, with an explanation that breaks down missing, under-called, extra, and duplicated tools.
2. LLM-judge mode (expected_tools omitted or empty): called tools are checked against available tools extracted from the span.

How to interpret the score

• Closer to 100: the agent called the expected tools (ground-truth mode), or the judge marked most selections as correct (LLM-judge mode).
• Closer to 0: expected tools were missed or extras/duplicates dominate (ground-truth), or most verdicts are not correct (LLM-judge).

Important

This metric judges which tools were selected, not whether arguments were valid, whether tool results were used well, or whether the requested results were obtained. Pair it with Task Completion when you need whether the requested results were obtained, and with other agentic or non-agentic metrics when those dimensions matter.

API usage

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for single evaluation. tool_select scores one span inside a trace, so it is not accepted on a dataset or custom run.

Shortname: tool_select

Default threshold: 100

Inputs on single evaluation

• span_id (string, required): The OpenInference context.span_id of the span to score. A missing or blank id returns 400 (span_id is required.).
• trace (array of objects, required): The OpenInference span list that contains that span. The metric reads the addressed span from this list:
  • Input messages from llm.input_messages.* (required; evaluation fails if none are found).
  • Tools called from llm.output_messages.*.message.tool_calls.* (may be empty if the agent called no tools).
  • Available tools from llm.tools.*.tool.json_schema (required in LLM-judge mode only).

span_id must equal context.span_id on one object in trace. A mismatch returns 400 (span_id must match a span in the trace.) and is not scored.

For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.

metric_args

• expected_tools (list[str], optional): Ground-truth tool names for multiset comparison. When provided and non-empty, ground-truth mode is used. When omitted or empty, the decision is based on available tools extracted from the span.
• available_tools (object[] | string[], optional): The catalogue used as given in LLM-judge mode. When omitted, tools are read from the span.

Evaluation metadata

On successful evaluation, when at least one tool is misaligned, the metric returns eval_metadata with:

• incorrect_tools (list[dict]). Each item has:
  • name (str): Tool name.
  • verdict (str): One of redundant, incorrect, or missing.
  • reason (str | null): Concise justification for that verdict.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_EVALUATE_URL = f"{_BASE_URL}/evaluate"


def post_evaluate(payload: dict) -> requests.Response:
    """POST JSON payload to single evaluation; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _EVALUATE_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    # Minimal OpenInference-style LLM span (attributes only; real spans include more).
    span = {
        "context": {"span_id": "0xabc"},
        "attributes": {
            "openinference.span.kind": "LLM",
            "llm.input_messages.0.message.role": "system",
            "llm.input_messages.0.message.content": "You are a weather expert.",
            "llm.input_messages.1.message.role": "user",
            "llm.input_messages.1.message.content": (
                "What's the weather in Paris right now? "
                "Then email the forecast to my manager."
            ),
            "llm.tools.0.tool.json_schema": json.dumps(
                {
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "description": "Get the current weather in a city.",
                        "parameters": {
                            "type": "object",
                            "properties": {"city": {"type": "string"}},
                        },
                    },
                }
            ),
            "llm.tools.1.tool.json_schema": json.dumps(
                {
                    "type": "function",
                    "function": {
                        "name": "send_email",
                        "description": "Send an email.",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "to": {"type": "string"},
                                "body": {"type": "string"},
                            },
                        },
                    },
                }
            ),
            "llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
                "get_weather"
            ),
            "llm.output_messages.0.message.tool_calls.1.tool_call.function.name": (
                "send_email"
            ),
        }
    }

    payload = {
        "metric_shortname": "tool_select",
        "threshold": 100,
        "model_slug": "o4-mini",
        "span_id": "0xabc",
        "trace": [span],
        # Omit expected_tools for comparison to available tools extracted from the span.
        "metric_args": {
            "expected_tools": ["get_weather", "send_email"],
        },
    }

    response = post_evaluate(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))