Skip to main content

Context Ranking Precision (ctx_rank_prec)

Contents​

Metric Description​

Context ranking precision measures how well the retrieved context chunks are ranked relative to their relevance for answering the original question (golden answer). It assesses whether the most relevant chunks appear higher in the ordering—not just whether relevant chunks exist. The metric blends (1) an LLM-as-a-Judge and (2) heuristic.

The score runs from 0 (poor ranking quality) to 100 (ideal ranking).

How to interpret the score​

  • Closer to 100: chunks are ranked in near-ideal order; the most relevant chunks appear first.
  • Closer to 0: ranking is poor; irrelevant or weakly relevant chunks are ranked higher than highly relevant ones.
Important

This metric evaluates the ranking quality of your RAG retrieval/reranking pipeline. It assumes the context has already been retrieved; it does not measure recall (whether all relevant documents were retrieved). Pair with context recall when you need to assess retrieval completeness.

API usage​

Prerequisites

The examples call POST /runs/custom. Set AEGIS_API_KEY to your API key and AEGIS_API_BASE_URL to the API base URL, including the /api/v1 prefix (for example https://api.aegisevals.ai/api/v1). For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: ctx_rank_prec

Default threshold: 80

Inputs (each object in data)

  • context (str or list required): The retrieved context chunks to evaluate. Can be a single string or a list of chunk strings. There is no separate input field.
  • golden_answer (str required): The question or query the chunks are judged against. The judge scores each chunk's relevance to this text, and embedding similarity is computed against it as well.

metric_args

  • ranks (list[number] optional): One rank per context chunk. When provided, chunks are reordered by these values before evaluation (lower rank first), for example to score a reranker's order. The numbers themselves are not returned. The list must have the same length as the context chunks. If omitted, chunks stay in the order given.

Evaluation metadata​

On successful evaluation, the metric returns eval_metadata to interpret ranking quality and per-chunk relevance notes:

  • non_relevant_chunks_reasons (list[dict]): Chunk-level explanations for why a chunk is not perfectly relevant. Each object has chunk_position (int, 1-based position in the evaluated chunk list, after any ranks reorder) and reason (str).
  • actual_ranks (list[int]): 0-based indexes of the evaluated chunks, listed in descending ideal-score order. These are the positions after any ranks reorder, not the rank numbers you sent.
  • ideal_ranks (list[int]): Always 0, 1, 2, …, parallel to actual_ranks.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    data = [
        {
            "golden_answer": "What features does the new laptop have?",
            "context": [
                "The new laptop model has a high-resolution Retina display.",
                "It includes a fast-charging battery with up to 12 hours of usage.",
                "Security features include fingerprint authentication and an encrypted SSD.",
                "Every purchase comes with a one-year warranty.",
                "Pineapples taste great on pizza.",
            ],
        }
    ]

    payload = {
        "threshold": 80,
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "ctx_rank_prec",
                        "metric_args": {"ranks": [1, 0, 2, 3, 4]},
                    },
                ],
                "threshold": 80,
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))