Skip to main content

Step 1: Model your data and get it into Aegis

You cannot evaluate what Aegis cannot see. The first real task is making your inputs, prompts, outputs, retrieved context, and reference answers available to the platform, either as a stored dataset you re-run, or as payloads you send at evaluation time.

Map your workflow onto the record fields​

For every example you want judged, decide which of these you can supply:

  1. prompt: the system/task prompt your app used.
  2. input: the end-user query or request.
  3. context: retrieved chunks for a RAG system (a string or a list of strings).
  4. output: the model's answer. This is the thing being judged and is required by almost every metric.
  5. golden_answer: the ground-truth/expected answer, when you have one. Needed for correctness- and recall-style metrics; there is no separate expected_output field, golden_answer is it.
  6. external_id: your own identifier for the row, so you can correlate a result back to the exact request, ticket, or test case in your own system. Populate this from day one; it is how you join Aegis results to your data later.

Get the data somewhere you can reach repeatedly​

Your inputs, outputs, prompts, and the rest must be captured and stored on your side first — in your database, a logging table, or exported files — so you can either (a) load them into Aegis once as a dataset, or (b) stream them to the API on demand. Both are valid; pick per use case.

Choose an ingestion path​

Aegis offers four, matched to different needs:

  1. CSV upload (Create dataset, POST /api/v1/datasets, multipart form): best for a curated, reusable test suite. You send the CSV file, a selected_metrics map (which metrics to run, keyed by numeric metric id, each with its own threshold and optional metric_args), an optional column_mappings object (mapping the Aegis fields external_id / prompt / input / context / output / golden_answer to your CSV's column names; headers are auto-detected if you omit it), and an optional name, description, project_id, and data_collection_id. Uploads are capped at 100 MB. Each CSV row becomes a record.
  2. Custom inline run (Create custom run, POST /api/v1/runs/custom): best for the development loop and CI, when you do not want to persist a dataset. You send evaluation groups, each naming its metrics (by shortname) and carrying a data array of rows with the same fields. No dataset is stored; Aegis creates ephemeral records and evaluates them.
  3. Single evaluate (Single evaluation, POST /api/v1/evaluate): one metric against one record, synchronously. Best for spot checks, debugging a metric's behaviour, or an inline guardrail on a single response. The score comes back in the response.
  4. Trace ingestion (Ingest traces, POST /api/v1/traces/ingest): the production/agentic lane. Point a standard OpenTelemetry OTLP HTTP exporter (protobuf or JSON, gzip allowed) at this endpoint with your API key as the bearer token. Spans are stored as trace and span rows for later inspection and agentic metrics; this is how you capture live agent behaviour rather than a curated test set. Argus (github.com/Strongbytes/argus) is a trace-ingestion library that helps with this process.

Decide recurrent vs. development-time access​

In development you will mostly hand-build small datasets or fire custom and single-evaluate calls. For anything recurring (nightly, per-release, per-deploy), you will pull the relevant slice of your stored data programmatically and push it through the API (see Step 4). The CSV path is for humans curating a suite; the API paths are for machines doing it on a schedule.

Organize as you grow​

Assign related datasets and runs to a data collection and/or a project so they roll up together on dashboards and stay visible to the right people. If you use a project-scoped API key, everything it creates is already locked to that project.