Skip to main content

Create run from a dataset

Endpoint: POST /runs/dataset

Description Starts a run using an existing dataset: the server takes the dataset’s records and a metric set for this run. Pass selected_metrics to choose metrics (and per-metric args) for the run — that map always wins and may differ from anything stored on the dataset. Omit selected_metrics to reuse the dataset’s stored default_selected_metrics (if any).

Per-metric arguments belong inside each selected_metrics entry (metric_args). There is no top-level metric_args on this request.

If dataset records include external_id, those values are carried through to the run — they appear on evaluations[].record.external_id in run responses and in the run CSV export when present. You do not pass external_id on this endpoint; it comes from the stored dataset rows.

Sharing rules

The new run is always owned by you (your account is the individual owner). Sharing isn't accepted directly on this endpoint — the run inherits visibility from the dataset (and its data collection). Runs created here are never org-only (org-only resources have no individual owner).

  • The run's data_collection_id is copied from the dataset.
  • The run's org_id is set in this order:
    1. The dataset's org_id, if any.
    2. Otherwise, the org_id of the dataset's data collection, if any.
    3. Otherwise, if the dataset belongs to a different user and you're in an organization, your own org_id (so the dataset owner can see the run via the same organization).
    4. Otherwise, when the key is not project-scoped, the run is created private. A project-scoped key with no org_id from items 1–3 creates the run with the key's organization org_id and assigns the run to the key's project. The dataset is not given that project.
  • When the run is org-shared, its project comes from your API key when the key is project-scoped (the run is assigned to the key's project). Otherwise it follows the dataset/collection project chain as today.
  • An unassigned dataset can be used to create the run. It is not claimed into the key's project. A dataset already assigned to a different project returns 403 ("This resource is assigned to a different project than this API key."). A dataset you cannot access returns 404.

Parameters

  • Body — application/json:
{
  "dataset_id": 1,
  "selected_metrics": {
    "ans_corr": { "threshold": 70, "metric_args": null }
  },
  "threshold": "0–100 | null",
  "model_slug": "string | null",
  "reasoning_level": "string | null",
  "is_blocking": false,
  "alias": "string | null"
}

dataset_id must be a JSON integer. A digit string ("74"), a whole-number float (1.0), or a boolean (true / false) returns 422 before lookup. threshold still accepts a digit string.

selected_metrics

  • Optional. When present, this map is the metric set for this run only (it replaces the dataset default for the run; it does not update the dataset). At least one entry is required when the field is sent — an empty map returns 422.
  • When omitted, the server uses the dataset’s stored default_selected_metrics. If the dataset has no default, the request returns 400.
  • Keys are metric shortnames ("ans_corr", "plan_adh"). Metric ids are rejected. Unknown shortnames return 400. The same shortname keys are stored on a dataset's default_selected_metrics and returned by GET.
  • Each value is { "threshold": 0–100 | null, "metric_args": object | null }. threshold is optional: omit it (or send null) to use that metric's catalog default. metric_args is optional (null or omitted means the metric's argument defaults). Those arguments are stored on each evaluation in this run. See the metric's doc page for accepted argument names and its default threshold.
  • Agentic metrics that score a trace or span (for example plan_adh, tool_select) cannot be used on dataset runs — the request returns 400 (These metrics score a trace or a span and cannot be used on a dataset or custom run: <shortnames>.).
  • Unknown or inactive shortnames return 400 (The following metrics do not exist: …, The following metrics are not active: …).

threshold (run-level)

  • Optional. Integer in 0–100. When omitted (or null), the run threshold is 80. Values outside that range return 422. A digit string is still accepted for this field.

model_slug

  • Optional. When omitted, the server uses gpt-5.6-luna with reasoning level none. That model (or the one you pass) is used for every LLM step in the evaluation. For Factfulness and Plan Quality, a higher-capability model than the default is recommended.

reasoning_level

  • Optional run-level setting applied to every evaluation in the run. There is no per-metric override on dataset runs — only this field applies.
  • Same resolution and validation rules as Single Evaluation (reasoning_level on POST /evaluate), evaluated against the run's model_slug (or default).
  • The 201 response includes reasoning_level on the run and on each item in evaluations[].

alias

  • Optional display name for the run (max 255 characters). Must be unique per account.
  • If omitted, the server generates one from the dataset name and creation timestamp.

is_blocking

  • Must be a JSON boolean (true or false). A string ("yes", "true") or a number (1, 0) returns 422. Omitting the field is false.
  • false (default)— The run and evaluation records are saved, then evaluation work is started in the background; the 201 response returns right away. Results may still be missing in that payload - poll GET /runs/{run_id} until evaluations complete. Prefer this for large datasets to avoid holding the HTTP request open.
  • true — The API waits until every evaluation for this run has been executed. The response body normally includes filled-in scores, aggregate_results, and finished_at, and cost totals for the run are finalized before you receive 201.

Error responses

  • 401, 402 — Auth or insufficient balance.
  • 403 — API key scoping violation (dataset already assigned to a different project than a project-scoped key).
  • 422 — Body/query validation failed. dataset_id must be a JSON integer: a digit string ("74"), a whole-number float (1.0), or a boolean (true / false) is rejected before lookup ("Input should be a valid integer"). An integer dataset_id that does not exist still returns 404. is_blocking must be a JSON boolean: a string ("yes", "true") or a number (1, 0) is rejected. Also includes a run-level or per-metric threshold outside 0–100, and selected_metrics cannot be empty when provided. Omit the field to use the dataset's default_selected_metrics, or include at least one metric.
  • 404 — model_slug is not a documented model slug; dataset not found or not claimable.
  • 400 — Cannot start a run without metrics. Dataset {id} has no default_selected_metrics and this request did not include selected_metrics. Either provide selected_metrics in this request, or set default_selected_metrics on the dataset first.; metric id used as a map key (Invalid metric key '<id>'. Use a metric shortname, not a metric id.); unknown or inactive metric shortnames (The following metrics do not exist: …, The following metrics are not active: …); deactivated model_slug (The following models are not active: …); invalid thresholds or metric_args on selected_metrics; agentic metrics that score a trace or span (These metrics score a trace or a span and cannot be used on a dataset or custom run: <shortnames>.); duplicate run alias (unique per user); invalid reasoning_level for the run's model (same cases as Single Evaluation).
  • 404 — No records on the dataset.
  • 500 — Failure creating the run; An error occurred while reading the dataset's default selected metrics. when the dataset's stored default_selected_metrics cannot be read.

Responses

Example response (201)

{
  "id": 991,
  "user": "analyst@acme.com",
  "author_email": "analyst@acme.com",
  "run_type": "Dataset",
  "run_source": "API Call",
  "trace_id": null,
  "dataset": "Support QA - March",
  "data_collection": "Customer Support",
  "org_id": null,
  "permissions": {
    "can_edit": true,
    "can_delete": true,
    "can_share": false,
    "can_assign_project": false,
    "can_move_collection": false
  },
  "number_of_metrics": 2,
  "result": 83.5,
  "threshold": 70,
  "model_slug": "gpt-5.6-luna",
  "reasoning_level": "none",
  "alias": "support-march",
  "aggregate_results": {
    "ans_corr": 86,
    "faith": 81
  },
  "total_cost": null,
  "started_at": "2026-04-01T09:12:00Z",
  "finished_at": null,
  "is_gte_threshold": null,
  "evaluations": []
}

curl

curl -X POST "https://api.aegisevals.ai/api/v1/runs/dataset" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id":1,"selected_metrics":{"ans_corr":{"threshold":70,"metric_args":null},"faith":{"threshold":80,"metric_args":null}},"threshold":70,"model_slug":"gpt-4o","is_blocking":false}'

Reuse the dataset default metrics

Omit selected_metrics when the dataset already stores a default_selected_metrics set. The run uses that map as-is.

curl -X POST "https://api.aegisevals.ai/api/v1/runs/dataset" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id":1,"threshold":70,"model_slug":"gpt-4o","is_blocking":false}'

Choose metrics and arguments for this run

Dataset 42 is scored with json_equal (arguments for this run) and exact_match (metric defaults). A later run of the same dataset can send a different selected_metrics map. exact_match omits threshold, so that evaluation uses the metric catalog default.

curl -X POST "https://api.aegisevals.ai/api/v1/runs/dataset" \
  -H "Authorization: Bearer sk_00000000000000000000000000000000" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": 42,
    "selected_metrics": {
      "json_equal": {
        "threshold": 70,
        "metric_args": { "ignore_extra_keys": true, "ignore_order": false }
      },
      "exact_match": { "metric_args": null }
    },
    "model_slug": "gpt-4o",
    "is_blocking": false
  }'