Tool Selection Correctness (tool_select)

Contents

• Metric Description
• API usage
• Evaluation metadata

Metric Description

Tool selection correctness is a span-level metric: it measures whether an agent chose the right tools for its task on a single OpenInference-format LLM span. It scores tool selection only — not tool arguments, execution results, call ordering, or final-answer quality.

The score runs from 0 (poor selection) to 100 (correct selection). Scoring uses one of two modes, selected by metric_args:

1. Ground-truth mode (expected_tools provided): deterministic on exact tool-name match (call multiplicity counts). Score is 100 when the multiset of called tools equals the expected multiset; 0 when no tools were called but some were expected. Partial overlap yields an intermediate score, with an explanation that breaks down missing, under-called, extra, and duplicated tools.
2. LLM-judge mode (expected_tools omitted or empty): called tools are checked against available tools extracted from the span.

How to interpret the score

• Closer to 100: the agent called the expected tools (ground-truth mode), or the judge marked most selections as correct (LLM-judge mode).
• Closer to 0: expected tools were missed or extras/duplicates dominate (ground-truth), or most verdicts are not correct (LLM-judge).

Important

This metric judges which tools were selected, not whether arguments were valid, whether tool results were used well, or whether the final answer is correct. Pair it with other agentic or non-agentic metrics when those dimensions matter.

API usage

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for the custom-runs request. For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: tool_select

Default threshold: 100

Inputs (each object in data)

• span (object required): An OpenInference-format LLM span (dictionary). The metric extracts:
  • Input messages from llm.input_messages.* (required; evaluation fails if none are found).
  • Tools called from llm.output_messages.*.message.tool_calls.* (may be empty if the agent called no tools).
  • Available tools from llm.tools.*.tool.json_schema (required in LLM-judge mode only).

For annotated span/trace examples and the attribute map Aegis reads, see OpenInference spans and traces.

metric_args

• expected_tools (list[str], optional): Ground-truth tool names for multiset comparison. When provided and non-empty, ground-truth mode is used. When omitted or empty, the decision is based on available tools extracted from the span.

Evaluation metadata

On successful evaluation, when at least one tool is misaligned, the metric returns eval_metadata with:

• incorrect_tools (list[dict]). Each item has:
  • name (str): Tool name.
  • verdict (str): One of redundant, incorrect, or missing.
  • reason (str | null): Concise justification for that verdict.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    # Minimal OpenInference-style LLM span (attributes only; real spans include more).
    span = {
        "attributes": {
            "openinference.span.kind": "LLM",
            "llm.input_messages.0.message.role": "system",
            "llm.input_messages.0.message.content": "You are a weather expert.",
            "llm.input_messages.1.message.role": "user",
            "llm.input_messages.1.message.content": (
                "What's the weather in Paris right now? "
                "Then email the forecast to my manager."
            ),
            "llm.tools.0.tool.json_schema": json.dumps(
                {
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "description": "Get the current weather in a city.",
                        "parameters": {
                            "type": "object",
                            "properties": {"city": {"type": "string"}},
                        },
                    },
                }
            ),
            "llm.tools.1.tool.json_schema": json.dumps(
                {
                    "type": "function",
                    "function": {
                        "name": "send_email",
                        "description": "Send an email.",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "to": {"type": "string"},
                                "body": {"type": "string"},
                            },
                        },
                    },
                }
            ),
            "llm.output_messages.0.message.tool_calls.0.tool_call.function.name": (
                "get_weather"
            ),
            "llm.output_messages.0.message.tool_calls.1.tool_call.function.name": (
                "send_email"
            ),
        }
    }

    data = [
        {
            "span": span,
        },
    ]

    payload = {
        "threshold": 100,
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "tool_select",
                        # Omit expected_tools for comparison to available tools extracted from the span.
                        "metric_args": {
                            "expected_tools": ["get_weather", "send_email"],
                        },
                    },
                ],
                "threshold": 100,
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))