Create custom run

Endpoint: POST /runs/custom

Description Creates a run when you pass an evaluations array in the JSON body: each element defines metrics and rows to score.

Sharing rules

The new run is always owned by you (your account is the individual owner). Sharing isn't accepted directly on this endpoint — visibility is derived from the data collection (if any), from an optional project_id, or from a project-scoped API key. Runs created here are never org-only (org-only resources have no individual owner).

• Without data_collection_id:
  • Private (org_id = null, no project) when you use a private API key and omit project_id.
  • Org-shared and assigned to a project when you use a project-scoped API key (the key's project is applied and the run is org-shared automatically). A private key cannot org-share via project_id alone. See Introduction — API key scoping.
• With data_collection_id:
  • The collection is loaded via claim semantics. When claimable, the run is created shared or private according to the collection's org_id and inherits the collection's project. You must still be a member of that organization when the collection is org-shared. Unassigned org collections can be claimed into the key's project; a collection already assigned to a different project returns 403. A project-scoped API key cannot use a private collection (org_id is null): the request returns 400 with detail Data collection must be shared with the organization before it can be added to a project.
  • Do not send project_id when linking to a collection — assign the collection to a project instead (400 if you try).

Use PUT /runs/{run_id} to share, unshare, attach, detach, assign a project, or rename a custom run after it has been created.

External IDs

Each object in evaluations[].data[] may include an optional external_id (string, max 255 characters). This is your stable handle for the row in your own systems (for example a ticket id or upstream row key). It is stored on the record and echoed on each matching evaluation as evaluations[].record.external_id. Omit it when you do not need correlation.

Parameters

• Body — application/json:

{
  "threshold": "0–100 | null",
  "model_slug": "string | null",
  "reasoning_level": "string | null",
  "is_blocking": false,
  "data_collection_id": "integer | null",
  "project_id": "integer | null",
  "alias": "string | null",
  "evaluations": [
    {
      "metrics": [
        "string  (metric shortname)",
        {
          "metric": "string  (metric shortname)",
          "metric_args": "object | null",
          "threshold": "0–100 | null",
          "model_slug": "string | null",
          "reasoning_level": "string | null"
        }
      ],
      "threshold": "0–100 | null",
      "model_slug": "string | null",
      "reasoning_level": "string | null",
      "data": [
        {
          "external_id": "string | null",
          "prompt": "string | null",
          "input": "string | number | boolean | object | array | null",
          "context": "string | array | null",
          "output": "string | number | boolean | object | array | null",
          "golden_answer": "string | number | boolean | object | array | null"
        }
      ]
    }
  ]
}

data_collection_id and project_id must be JSON integers. A digit string ("74"), a whole-number float (1.0), or a boolean (true / false) returns 422 before lookup. Both are optional: JSON null and omitting a field stay valid. threshold still accepts a digit string.

model_slug

• Optional run default. When omitted, the server uses gpt-5.6-luna with reasoning level none. That model (or the one you pass) is used for every LLM step in the evaluation. For Factfulness and Plan Quality, a higher-capability model than the default is recommended.

reasoning_level

• Optional at three levels: on the run, on each evaluation group (evaluations[]), and on each metric object (object form only). Whitespace-only values are treated as omitted.
• Model slug for each evaluation resolves with precedence: metric object → evaluation group → run.
• Reasoning level for each evaluation resolves as follows (the response includes the level applied to each evaluation; levels may differ when you use overrides):
  1. The metric object's reasoning_level, if set.
  2. If the metric object sets model_slug without reasoning_level → that model's configured default (run- and evaluation-group levels are not inherited).
  3. If the evaluation group sets model_slug → the evaluation group's reasoning_level, or that model's default when omitted (run-level is not inherited).
  4. If the evaluation group sets reasoning_level without a metric-level override → that value on the model used for the evaluation.
  5. Otherwise → the run-level reasoning_level, or the run model's default when omitted.
• Validation uses the same rules as Single Evaluation. The 201 response includes reasoning_level on the run and on each evaluation.

Each item in metrics is either a metric shortname (string) or an object with metric plus optional metric_args, threshold, model_slug, and reasoning_level. Metric identifiers must be shortnames — numeric metric ids are rejected. Use the object form when the metric accepts arguments (see the metric's doc page) or when you want to override threshold, model, or reasoning level for that metric only. metric_args names are not validated when the run is created: an unknown name (for example a typo like min_cout) is silently ignored and the metric uses its default for that argument, so check spelling against the metric's doc page. A missing required argument or a value of the wrong type does not reject the request either; that evaluation completes with is_success: false and the reason in explanation. Agentic metrics that score a trace or span cannot be used on custom runs.

threshold (run-level)

• Optional. Integer in 0–100. The same bound applies to each evaluation group's threshold and each metric object's threshold. When omitted (or null), the run threshold is 80. Values outside that range return 422. A digit string is still accepted for this field.
• Each evaluation uses the metric object's threshold if set, otherwise that evaluation group's threshold, otherwise the metric's catalog default.

is_blocking

• Must be a JSON boolean (true or false). A string ("yes", "true") or a number (1, 0) returns 422. Omitting the field is false.
• false (default) — The run and evaluation records are saved, then evaluation work is started in the background; the 201 response returns right away. Results may still be missing in that payload — poll GET /runs/{run_id} until evaluations complete. Prefer this for large payloads.
• true — The API waits until every evaluation for this run has been executed. The response body normally includes filled-in scores, aggregate_results, and finished_at, and cost totals for the run are finalized before you receive 201.

Error responses

• 401, 402 — authentication or insufficient balance.
• 403 — API key scoping violation (project_id in the body conflicts with a project-scoped key, collection already assigned to a different project).
• 422 — request validation failed. data_collection_id and project_id must be JSON integers. A digit string ("74"), a whole-number float (1.0), or a boolean (true / false) is rejected before lookup ("Input should be a valid integer"). JSON null and omitting either id are still accepted. is_blocking must be a JSON boolean: a string ("yes", "true") or a number (1, 0) is rejected. A run-level, evaluation-group, or per-metric threshold outside 0–100 is also rejected.
• 400 — empty evaluations; metric id used as a metric identifier (Invalid metric key '<id>'. Use a metric shortname, not a metric id.); unknown or inactive metric shortnames (validated up front — unavailable metrics no longer return a per-evaluation 404; messages list shortnames: The following metrics do not exist: …, The following metrics are not active: …); deactivated model_slug values (run default, evaluation-group, and per-metric slugs — The following models are not active: …); agentic metrics that score a trace or span (These metrics score a trace or a span and cannot be used on a dataset or custom run: <shortnames>.); duplicate alias; project_id sent with data_collection_id; a project-scoped key with a private data_collection_id (Data collection must be shared with the organization before it can be added to a project.); invalid reasoning_level for an evaluation's effective model (same cases as Single Evaluation).
• 404 — one or more referenced model_slug values are not documented model slugs (run default, evaluation-group, and per-metric slugs are all checked — detail lists every invalid slug, for example The following models do not exist: ['slug-a', 'slug-b']); data_collection_id not found or not claimable; no metrics resolved.
• 500 — failure while creating the run.

Responses

• 201 — same run object shape as Get run. The example is a private custom run you own, created with a private API key, so can_share is false. Flag meanings: Introduction — Permissions.

Example response (201)

{
  "id": 992,
  "user": "analyst@acme.com",
  "author_email": "analyst@acme.com",
  "run_type": "Custom",
  "run_source": "API Call",
  "trace_id": null,
  "dataset": null,
  "data_collection": "Customer Support",
  "org_id": null,
  "permissions": {
    "can_edit": true,
    "can_delete": true,
    "can_share": false,
    "can_assign_project": false,
    "can_move_collection": true
  },
  "number_of_metrics": 1,
  "result": 100,
  "threshold": 70,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "alias": "smoke-test",
  "aggregate_results": {
    "ans_corr": 100
  },
  "total_cost": "0.000400000000000",
  "started_at": "2026-04-01T09:15:01Z",
  "finished_at": "2026-04-01T09:15:03Z",
  "is_gte_threshold": true,
  "evaluations": []
}

Metric shortnames

Use these in metrics (or in object form metric).

Metric shortnames (by category)

Agentic

• plan_adh
• plan_qual
• task_comp
• tool_select
• traject_eff

Content generation

• content_gen_faith
• format_align
• format_consist

General

• ans_corr
• ans_rel
• content_consist
• dag
• entity_faith
• fact
• faith
• g_eval
• prompt_qual
• summ

RAG

• ctx_faith
• ctx_rank_prec
• ctx_recall
• ctx_rel
• ctx_suff
• ctx_waste

Structural

• alpha_perc
• alphanum_perc
• bleu
• char_ct_match
• exact_match
• is_boolean
• is_date
• is_numeric
• is_string
• is_valid_json
• is_valid_python
• is_valid_sql
• is_valid_xml
• json_equal
• json_schema_match
• numeric_match
• par_ct_match
• rouge
• sent_ct_match
• text_readability
• word_ct_match
• xml_equal
• xml_schema_match

Safety

• bias
• harmfulness
• manipulation
• misuse
• role_viol
• toxic

Security

• ev_obf
• instr_integ_subv_att
• pii_phi
• pii_phi_exfil_att
• role_hijack
• sys_data_exfil_att
• sys_data_leak

curl

curl -X POST "https://api.aegisevals.ai/api/v1/runs/custom" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{"threshold":70,"model_slug":"gpt-4o","is_blocking":false,"alias":"smoke-test","evaluations":[{"metrics":["ans_corr"],"data":[{"external_id":"row-42","prompt":"What is 2+2?","output":"4","golden_answer":"4"}]}]}'

Examples

Several rows, one metric

{
  "threshold": 75,
  "model_slug": "gpt-4o",
  "is_blocking": false,
  "alias": "support-batch-2025-03-27",
  "evaluations": [
    {
      "metrics": ["ans_corr"],
      "threshold": 75,
      "model_slug": "gpt-4o",
      "data": [
        {
          "external_id": "ticket-1001",
          "prompt": "What is your refund policy for annual plans?",
          "output": "We refund unused months if you cancel within 14 days of renewal.",
          "golden_answer": "Annual plans are refundable for the unused portion within 14 days of the renewal charge."
        },
        {
          "external_id": "ticket-1002",
          "prompt": "How do I export my data?",
          "output": "Open Settings → Data → Export; you will get a CSV within a few minutes.",
          "golden_answer": "Use Settings → Data → Export to download a CSV of your workspace."
        }
      ]
    }
  ]
}

RAG: context + answer metrics

Pass retrieved context with the model output. Here ctx_faith and ctx_rel run on the same rows.

{
  "threshold": 70,
  "model_slug": "gpt-4o-mini",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": ["ctx_faith", "ctx_rel"],
      "threshold": 70,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "input": "When did the Acme Corp fiscal year end in 2024?",
          "context": "Acme Corp FY2024 ended on September 30, 2024. Revenue was $120M.",
          "output": "Acme’s 2024 fiscal year ended on September 30, 2024.",
          "golden_answer": null
        }
      ]
    }
  ]
}

Two evaluation blocks

Run one block with stricter threshold / different model than another (for example: cheap model for screening, stronger model for a smaller slice).

{
  "threshold": 80,
  "model_slug": "gpt-4o-mini",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": ["ans_rel"],
      "threshold": 60,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "prompt": "Summarize our SLA in one sentence.",
          "output": "We target 99.9% monthly uptime excluding scheduled maintenance."
        }
      ]
    },
    {
      "metrics": ["faith"],
      "threshold": 85,
      "model_slug": "gpt-4o",
      "data": [
        {
          "prompt": "What guarantees does the SLA provide?",
          "context": "SLA: 99.9% uptime; credits apply if below target.",
          "output": "The SLA promises 99.9% uptime and service credits if we miss it."
        }
      ]
    }
  ]
}

Mixed metrics list + metric_args

Use strings when no options are needed, and objects when a metric accepts metric_args (see that metric’s doc page).

{
  "threshold": 100,
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": [
        "exact_match",
        {
          "metric": "json_equal",
          "metric_args": {
            "ignore_extra_keys": true,
            "ignore_order": false
          }
        }
      ],
      "threshold": 100,
      "model_slug": "gpt-4o-mini",
      "data": [
        {
          "output": "{\"status\":\"ok\",\"items\":[1,2]}",
          "golden_answer": "{\"items\":[1,2],\"status\":\"ok\"}"
        }
      ]
    }
  ]
}

Per-metric reasoning_level override

Set a run default, then override at the evaluation-group or per-metric level. See the reasoning_level precedence rules above.

{
  "threshold": 70,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "is_blocking": false,
  "evaluations": [
    {
      "metrics": [
        "ans_rel",
        {
          "metric": "faith",
          "model_slug": "gpt-5.2",
          "reasoning_level": "high",
          "threshold": 80
        }
      ],
      "model_slug": "gpt-5.6-luna",
      "reasoning_level": "medium",
      "data": [
        {
          "prompt": "What guarantees does the SLA provide?",
          "context": "SLA: 99.9% uptime; credits apply if below target.",
          "output": "The SLA promises 99.9% uptime and service credits if we miss it."
        }
      ]
    }
  ]
}

Python

import json
import os
import requests
from dotenv import load_dotenv

load_dotenv(override=True)
API_KEY = os.environ["AEGIS_API_KEY"]
BASE = os.environ["AEGIS_API_BASE_URL"].rstrip("/")

payload = {
    "threshold": 75,
    "model_slug": "gpt-4o",
    "is_blocking": False,
    "alias": "python-example",
    "evaluations": [
        {
            "metrics": ["ans_corr"],
            "threshold": 75,
            "model_slug": "gpt-4o",
            "data": [
                {
                    "external_id": "geo-q-1",
                    "prompt": "Capital of France?",
                    "output": "Paris is the capital of France.",
                    "golden_answer": "Paris.",
                }
            ],
        }
    ],
}

r = requests.post(
    f"{BASE}/runs/custom",
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
    data=json.dumps(payload),
)
r.raise_for_status()
print(json.dumps(r.json(), indent=2))