Skip to main content

Prerequisites and access setup

Do this once, before any evaluation work:

  1. Have an account, and (if you are a team) an organization and at least one project. Projects are how you keep one team's or one product's evaluations separate from another's.
  2. Create an API key from the UI menu. Decide its scope up front:
    • Private key (no project_id): personal visibility, good for a single developer's experiments.
    • Project-scoped key (project_id set): everything it creates is locked to that project. Use this for shared, team-wide, and CI keys so results land in the right project and roll up correctly.
  3. Store the key as a secret. All external API calls authenticate with an Authorization: Bearer sk_... header (the token is the literal sk_ prefix plus a 32-character secret). Never commit it; put it in your CI/CD secret store and your local environment.
  4. Fund your account. LLM-judge metrics cost money to run; the platform checks your USD balance before executing an evaluation and will refuse if it is insufficient. Set a balance alert (the only built-in alert is on dollar balance, not on scores) so a pipeline does not silently start failing because the balance ran out.
  5. Pick your judge model and reasoning level. LLM metrics are scored by a judge model (for example a GPT or Claude slug) at a chosen reasoning level. You can set these per run, per evaluation group, or per metric. Pick one combination and keep it fixed for a given suite so scores are comparable over time.
  6. Know where the API lives. All documented routes live under https://api.aegisevals.ai/api/v1. See the API introduction for authentication and payloads.