Create custom run

Endpoint: POST /runs/custom

Description Creates a run when you pass an evaluations array in the JSON body: each element defines metrics and rows to score.

Sharing rules

The new run is always owned by you (your account is the individual owner). Sharing isn't accepted directly on this endpoint — visibility is derived from the data collection (if any), from an optional project_id, or from a project-scoped API key. Runs created here are never org-only (org-only resources have no individual owner).

• Without data_collection_id:
  • Private (org_id = null, no project) when you use a private API key and omit project_id.
  • Org-shared and assigned to a project when you use a project-scoped API key (the key's project is applied and the run is org-shared automatically). A private key cannot org-share via project_id alone. See Introduction — API key scoping.
• With data_collection_id:
  • The collection is loaded via claim semantics. When claimable, the run is created shared or private according to the collection's org_id and inherits the collection's project. You must still be a member of that organization when the collection is org-shared. Unassigned collections can be claimed into the key's project; a collection already assigned to a different project returns 403.
  • Do not send project_id when linking to a collection — assign the collection to a project instead (400 if you try).

Use PUT /runs/{run_id} to share, unshare, attach, detach, assign a project, or rename a custom run after it has been created.

External IDs

Each object in evaluations[].data[] may include an optional external_id (string, max 255 characters). This is your stable handle for the row in your own systems (for example a ticket id or upstream row key). It is stored on the record and echoed on each matching evaluation as evaluations[].record.external_id. Omit it when you do not need correlation.

Parameters

• Body — application/json:

{
  "threshold": "integer | null",
  "model_slug": "string | null",
  "reasoning_level": "string | null",
  "is_blocking": false,
  "data_collection_id": "integer | null",
  "project_id": "integer | null",
  "alias": "string | null",
  "evaluations": [
    {
      "metrics": [
        "string  (metric shortname)",
        {
          "metric": "string  (metric shortname)",
          "metric_args": "object | null",
          "threshold": "integer | null",
          "model_slug": "string | null",
          "reasoning_level": "string | null"
        }
      ],
      "threshold": "integer | null",
      "model_slug": "string | null",
      "reasoning_level": "string | null",
      "data": [
        {
          "external_id": "string | null",
          "prompt": "string | null",
          "input": "string | number | boolean | object | array | null",
          "context": "string | array | null",
          "output": "string | number | boolean | object | array | null",
          "golden_answer": "string | number | boolean | object | array | null"
        }
      ]
    }
  ]
}

model_slug

• Optional run default. When omitted, the server uses gpt-5.6-luna with reasoning level none.

reasoning_level

• Optional at three levels: on the run, on each evaluation group (evaluations[]), and on each metric object (object form only). Whitespace-only values are treated as omitted.
• Model slug for each evaluation resolves with precedence: metric object → evaluation group → run.
• Reasoning level for each evaluation resolves as follows (the response includes the level applied to each evaluation; levels may differ when you use overrides):
  1. The metric object's reasoning_level, if set.
  2. If the metric object sets model_slug without reasoning_level → that model's configured default (run- and evaluation-group levels are not inherited).
  3. If the evaluation group sets model_slug → the evaluation group's reasoning_level, or that model's default when omitted (run-level is not inherited).
  4. If the evaluation group sets reasoning_level without a metric-level override → that value on the model used for the evaluation.
  5. Otherwise → the run-level reasoning_level, or the run model's default when omitted.
• Validation uses the same rules as Single Evaluation. The 201 response includes reasoning_level on the run and on each evaluation.

Each item in metrics is either a metric shortname (string) or an object with metric plus optional metric_args, threshold, model_slug, and reasoning_level. Use the object form when the metric accepts arguments (see the metric's doc page) or when you want to override threshold, model, or reasoning level for that metric only. Unknown argument names are rejected and missing required args (with no metric-level default) are rejected.

is_blocking

• false (default) — The run and evaluation records are saved, then evaluation work is started in the background; the 201 response returns right away. Results may still be missing in that payload — poll GET /runs/{run_id} until evaluations complete. Prefer this for large payloads.
• true — The API waits until every evaluation for this run has been executed. The response body normally includes filled-in scores, aggregate_results, and finished_at, and cost totals for the run are finalized before you receive 201.

Error responses

• 401, 402 — authentication or insufficient balance.
• 403 — API key scoping violation (project_id in the body conflicts with a project-scoped key, collection already assigned to a different project, scoped key on a personal resource).
• 422 — request validation failed.
• 400 — empty evaluations; unknown/inactive metric shortnames; duplicate alias; project_id sent with data_collection_id; invalid reasoning_level for an evaluation's effective model (same cases as Single Evaluation).
• 404 — one or more referenced model_slug values are not documented model slugs (run default, evaluation-group, and per-metric slugs are all checked — detail lists every invalid slug, for example The following models do not exist: ['slug-a', 'slug-b']); data_collection_id not found or not claimable; no metrics resolved.
• 500 — failure while creating the run.

Responses

• 201 — same run object shape as Get run.

Example response (201)

{
  "id": 992,
  "user": "analyst@acme.com",
  "author_email": "analyst@acme.com",
  "run_type": "Custom",
  "run_source": "API Call",
  "dataset": null,
  "data_collection": "Customer Support",
  "org_id": null,
  "number_of_metrics": 1,
  "result": 100,
  "threshold": 70,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "alias": "smoke-test",
  "aggregate_results": {
    "ans_corr": 100
  },
  "total_cost": "0.000400000000000",
  "started_at": "2026-04-01T09:15:01Z",
  "finished_at": "2026-04-01T09:15:03Z",
  "is_gte_threshold": true,
  "evaluations": []
}

Metric shortnames

Use these in metrics (or in object form metric).

Metric shortnames (by category)

Agentic

• tool_select

Content generation

• content_gen_faith
• format_align
• format_consist

General

• ans_corr
• ans_rel
• content_consist
• dag
• entity_faith
• fact
• faith
• g_eval
• prompt_qual
• summ

RAG

• ctx_faith
• ctx_rank_prec
• ctx_recall
• ctx_rel
• ctx_suff
• ctx_waste

Structural

• alpha_perc
• alphanum_perc
• bleu
• char_ct_match
• exact_match
• is_boolean
• is_date
• is_numeric
• is_string
• is_valid_json
• is_valid_python
• is_valid_sql
• is_valid_xml
• json_equal
• json_schema_match
• numeric_match
• par_ct_match
• rouge
• sent_ct_match
• text_readability
• word_ct_match
• xml_equal
• xml_schema_match

Safety

• bias
• harmfulness
• manipulation
• misuse
• role_viol
• toxic

Security

• ev_obf
• instr_integ_subv_att
• pii_phi
• pii_phi_exfil_att
• role_hijack
• sys_data_exfil_att
• sys_data_leak

curl

curl -X POST "https://api.aegisevals.ai/api/v1/runs/custom" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{"threshold":70,"model_slug":"gpt-4o","is_blocking":false,"alias":"smoke-test","evaluations":[{"metrics":["ans_corr"],"data":[{"external_id":"row-42","prompt":"What is 2+2?","output":"4","golden_answer":"4"}]}]}'

Examples

Several rows, one metric

{
  "threshold": 75,
  "model_slug": "gpt-4o",
  "is_blocking": false,
  "alias": "support-batch-2025-03-27",
  "evaluations": [
    {
      "metrics": ["ans_corr"],
      "threshold": 75,
      "model_slug": "gpt-4o",
      "data": [
        {
          "external_id": "ticket-1001",
          "prompt": "What is your refund policy for annual plans?",
          "output": "We refund unused months if you cancel within 14 days of renewal.",
          "golden_answer": "Annual plans are refundable for the unused portion within 14 days of the renewal charge."
        },
        {
          "external_id": "ticket-1002",
          "prompt": "How do I export my data?",
          "output": "Open Settings → Data → Export; you will get a CSV within a few minutes.",
          "golden_answer": "Use Settings → Data → Export to download a CSV of your workspace."
        }
      ]
    }
  ]
}

RAG: context + answer metrics

Pass retrieved context with the model output. Here ctx_faith and ctx_rel run on the same rows.

{
  "threshold": 70,
  "model_slug": "gpt-4o-mini",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": ["ctx_faith", "ctx_rel"],
      "threshold": 70,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "input": "When did the Acme Corp fiscal year end in 2024?",
          "context": "Acme Corp FY2024 ended on September 30, 2024. Revenue was $120M.",
          "output": "Acme’s 2024 fiscal year ended on September 30, 2024.",
          "golden_answer": null
        }
      ]
    }
  ]
}

Two evaluation blocks

Run one block with stricter threshold / different model than another (for example: cheap model for screening, stronger model for a smaller slice).

{
  "threshold": 80,
  "model_slug": "gpt-4o-mini",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": ["ans_rel"],
      "threshold": 60,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "prompt": "Summarize our SLA in one sentence.",
          "output": "We target 99.9% monthly uptime excluding scheduled maintenance."
        }
      ]
    },
    {
      "metrics": ["faith"],
      "threshold": 85,
      "model_slug": "gpt-4o",
      "data": [
        {
          "prompt": "What guarantees does the SLA provide?",
          "context": "SLA: 99.9% uptime; credits apply if below target.",
          "output": "The SLA promises 99.9% uptime and service credits if we miss it."
        }
      ]
    }
  ]
}

Mixed metrics list + metric_args

Use strings when no options are needed, and objects when a metric accepts metric_args (see that metric’s doc page).

{
  "threshold": 100,
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": [
        "exact_match",
        {
          "metric": "json_equal",
          "metric_args": {
            "ignore_extra_keys": true,
            "ignore_order": false
          }
        }
      ],
      "threshold": 100,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "output": "{\"status\":\"ok\",\"items\":[1,2]}",
          "golden_answer": "{\"items\":[1,2],\"status\":\"ok\"}"
        }
      ]
    }
  ]
}

Per-metric reasoning_level override

Set a run default, then override at the evaluation-group or per-metric level. See the reasoning_level precedence rules above.

{
  "threshold": 70,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": [
        "ans_rel",
        {
          "metric": "faith",
          "model_slug": "gpt-5.2",
          "reasoning_level": "high",
          "threshold": 80
        }
      ],
      "model_slug": "gpt-5.6-luna",
      "reasoning_level": "medium",
      "data": [
        {
          "prompt": "What guarantees does the SLA provide?",
          "context": "SLA: 99.9% uptime; credits apply if below target.",
          "output": "The SLA promises 99.9% uptime and service credits if we miss it."
        }
      ]
    }
  ]
}

Python

import json
import os
import requests
from dotenv import load_dotenv

load_dotenv(override=True)
API_KEY = os.environ["AEGIS_API_KEY"]
BASE = os.environ["AEGIS_API_BASE_URL"].rstrip("/")

payload = {
    "threshold": 75,
    "model_slug": "gpt-4o",
    "is_blocking": False,
    "alias": "python-example",
    "evaluations": [
        {
            "metrics": ["ans_corr"],
            "threshold": 75,
            "model_slug": "gpt-4o",
            "data": [
                {
                    "external_id": "geo-q-1",
                    "prompt": "Capital of France?",
                    "output": "Paris is the capital of France.",
                    "golden_answer": "Paris.",
                }
            ],
        }
    ],
}

r = requests.post(
    f"{BASE}/runs/custom",
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
    data=json.dumps(payload),
)
r.raise_for_status()
print(json.dumps(r.json(), indent=2))