Skip to main content

Word Count Match (word_ct_match)

Contents​

Metric description​

Word count match counts words in the output with options for delimiters, punctuation, minimum word length, and case handling. Optional min and max counts enforce length constraints.

How to interpret the score​

  • 100: the count satisfies configured bounds (or none set).
  • 0: fails bounds or could not be evaluated.

API usage​

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for the custom-runs request. For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: word_ct_match

Default threshold: 100

Structural metrics run without an LLM (deterministic checks). Your run may still include model_slug where the API expects it; scoring does not depend on it for this category.

Inputs (each object in data)

  • output (str, required): Text whose words are counted.

metric_args

  • min_count (number optional): Minimum number of words required.

  • max_count (number optional): Maximum number of words allowed.

  • exclude_punctuation (boolean optional): Exclude punctuation from token boundaries. Default: true.

  • min_word_length (number optional): Minimum characters for a token to count as a word. Default: 1.

  • case_sensitive (boolean optional): Preserve case during processing. Default: false.

  • custom_delimiters (string optional): Custom regex pattern for word delimiters.

Eval metadata​

Structural metrics do not populate eval_metadata; the field is omitted or ull on the result object.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    data = [
        {"output": "one two three four"}
    ]

    payload = {
        "threshold": 100,
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "word_ct_match",
                        "metric_args": {
                            "min_count": 1,
                            "max_count": 100,
                            "exclude_punctuation": True,
                        },
                    },
                ],
                "threshold": 100,
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))