Format Alignment (format_align)

Contents

• Metric Description
• API usage
• Evaluation metadata

Metric Description

Format alignment measures how well the model’s output follows formatting and style instructions—structure (JSON, Markdown, required sections), style (bullets vs paragraphs, concision), length or scope, tone, locale (dates, currency, spelling), audience, or brand-style constraints. It does not judge whether the substantive content is correct or grounded in a source; it focuses on how the answer is shaped relative to those instructions.

When defined, metric_args.instructions will serve as a target list to evaluate the output against. Otherwise, the list of format-related instructions will be extracted from your prompt and/or input. When extracting, each instruction is tagged with one of seven categories:

1. structure_format — Structure type of the output (JSON, Markdown, HTML, required fields or sections).
2. style_format — Writing style, organization, or presentation (bullets vs paragraphs, concision, formal language).
3. length_scope — Response length, detail level, or content scope (word limits, brief vs detailed).
4. tone_voice — Communication tone, personality, or emotional approach (friendly, professional, conversational).
5. locale_formatting — Dates, numbers, currency, units, language, or spelling variants (en-US vs en-GB).
6. audience_context — Target audience or contextual considerations (beginners, experts, children).
7. brand_guidelines — Brand voice, company guidelines, or organizational style requirements.

Important

When both prompt and input are provided, the metric treats them together as the instruction source. Pass only the fields that actually contain format or style rules. Include input when users can add extra formatting instructions there; omit unused fields so extraction stays focused and less noisy.

How to interpret the score

• Closer to 100: the output tends to satisfy most or all extracted (or supplied) formatting and style instructions.
• Closer to 0: many instructions are missed, or only weakly followed.

Important

High format alignment does not mean the answer is factually correct, safe, or faithful to retrieved context. Pair with factfulness, content generation faithfulness, or other metrics when those matter. For longer outputs, consider using format consistency to make sure the entire text follows the same patterns.

API usage

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for the custom-runs request. For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: format_align

Default threshold: 80

Inputs (each object in data)

• output (str, required): The model-generated text to evaluate.
• prompt (str, optional): System or developer instructions that may contain format/style rules. If both prompt and input are valid non-empty strings, they are combined before instruction extraction.
• input (str, optional): User message or task text that may contain format/style rules.

At least one of the following must be present so the metric can obtain instructions: a non-empty prompt, a non-empty input, or a non-empty instructions value under metric_args. If prompt and input are both missing or invalid and instructions is empty, evaluation cannot run.

metric_args

• instructions (str or list[str], optional): Explicit list of formatting/style instructions to check against output. If omitted or empty, instructions are extracted by an LLM from the combined prompt/input (or from whichever of prompt or input is available).
• categories (str or list[str], optional): Instruction categories to extract. Used only when instructions is omitted or empty; ignored when you supply instructions. If omitted, all categories are extracted. Supported values: structure_format, style_format, length_scope, tone_voice, locale_formatting, audience_context, brand_guidelines. Invalid values cause evaluation to fail with no score.

Evaluation metadata

On successful evaluation, the metric returns eval_metadata with structured details about instructions the output did not fully satisfy:

• unfollowed_instructions (list[dict]): One entry per instruction whose verdict was no or partially. Each object has:

  • instruction (str): The instruction that was not fully followed.
  • category (str | null): The extraction category (one of the seven listed above). null when you supplied instructions yourself (those are not categorized).
  • reason (str): A short explanation of what was not followed.

  If every instruction is followed, this list is empty.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    data = [
        {
            "prompt": "Respond in JSON with keys summary and bullets. Use British English.",
            "input": "Summarize the benefits of walking.",
            "output": '{"summary": "Walking improves health.", "bullets": ["Low impact", "Free"]}',
        },
    ]

    payload = {
        "threshold": 80,
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "format_align",
                        "metric_args": {
                            "instructions": ["Return JSON only", "Use British English"],
                        },
                    },
                ],
                "threshold": 80,
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))

To extract instructions from prompt / input instead of listing them, omit instructions. Optionally restrict extraction with categories, for example "metric_args": {"categories": ["structure_format", "locale_formatting"]}.