> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cruq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Compare models on a fixed suite of test cases and rank them on score, latency, and cost.

An agent pins one model per node. Evals answer the question that follows:
**would this agent be better, cheaper, or faster on a different model?**

You write a **suite** of test cases once, then run an **experiment** that
executes every case against every model you want to compare. The result is a
leaderboard ranked on quality, latency, and cost.

## How it fits together

* **Suite** - a named set of test cases. Belongs to one agent, or to the whole
  workspace so several agents can share it.
* **Case** - a prompt plus one or more **scorers** that decide whether the
  answer was good.
* **Variant** - one model under test, written as a fully-qualified
  `provider/model` and given a short label.
* **Experiment** - a suite x variants fan-out. Every (case, variant) pair is one
  real agent run.

Swapping the model is a rewrite of the agent's graph before it executes, so a
comparison exercises the same prompt, the same tools, and the same retrieval as
production. Nothing is stubbed.

<Note>
  An experiment pins the agent version at launch. Editing the agent while it runs
  does not change results already in flight, so a leaderboard always describes the
  graph that actually executed.
</Note>

## Scorers

Every scorer returns 0.0-1.0. Deterministic ones return exactly 0 or 1, and a
case's score is the weighted mean across its scorers.

| Type          | Checks                                                    | Required fields                        |
| ------------- | --------------------------------------------------------- | -------------------------------------- |
| `contains`    | The answer contains a substring.                          | `value` (optional `caseSensitive`)     |
| `regex`       | The answer matches a pattern.                             | `pattern`                              |
| `json_schema` | The answer parses as JSON and satisfies a schema.         | `schema`                               |
| `tool_call`   | The agent called a tool, optionally with given arguments. | `name` (optional `args`)               |
| `llm_judge`   | A separate model grades the answer against a rubric.      | `rubric` (optional `scale`, default 5) |

```json theme={null}
[
  { "type": "contains", "value": "total", "caseSensitive": false },
  { "type": "regex", "pattern": "\\$\\d+(\\.\\d{2})?" },
  { "type": "tool_call", "name": "get_portfolio" },
  { "type": "llm_judge", "rubric": "States a total USD value and breaks holdings down by percentage.", "scale": 5 }
]
```

`tool_call` matches against the run's persisted trace, and `args` is a subset
match - you can assert that a tool was called without pinning every argument.

An unknown scorer type records **no verdict** rather than zero. A scorer written
by a newer dashboard than the deployed worker will not quietly mark every model
as failing.

### Choosing a judge

`llm_judge` needs a `judge_model`, and it should be a strong model that is **not
one of the variants**. A model grading its own output is the most common way an
eval harness quietly flatters one model, so launching with the judge also under
test is rejected outright.

Judge spend is recorded separately from run cost. The cost column exists to
compare what each model costs to *serve*, and folding grading into it would
corrupt exactly that comparison.

## Running one

<Tabs>
  <Tab title="CLI">
    ```bash theme={null}
    # 1. A suite, attached to one agent
    cruqai evals create-suite "regression" --agent "Support bot"

    # 2. Cases. Scorers are JSON, inline or @file.
    cruqai evals add-case regression \
      --name "quotes the refund window" \
      --input "How long do I have to return something?" \
      --scorers '[{"type":"contains","value":"30 days"}]'

    # 3. Compare. --watch polls until it finishes.
    cruqai evals run "Support bot" --suite regression \
      --model fast=openrouter/openai/gpt-4o-mini \
      --model smart=openrouter/anthropic/claude-3.5-sonnet \
      --budget 2.00 --watch
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://app.cruq.ai/v1/evals/experiments \
      -H "Authorization: Bearer cruq_sk_..." \
      -H "Content-Type: application/json" \
      -d '{
        "agent": "Support bot",
        "suite": "regression",
        "name": "mini vs sonnet",
        "variants": [
          { "label": "fast",  "model": "openrouter/openai/gpt-4o-mini" },
          { "label": "smart", "model": "openrouter/anthropic/claude-3.5-sonnet" }
        ],
        "budget_usd": 2.00
      }'
    ```
  </Tab>

  <Tab title="Dashboard">
    Open the agent, expand its **Evals** section, add a suite and some cases, then
    **Compare models**. Progress and the leaderboard appear under **Evals** in the
    sidebar.
  </Tab>
</Tabs>

Models must be fully qualified - `openrouter/openai/gpt-4o-mini`, not
`gpt-4o-mini` - because the first segment selects the provider connection. See
[Models & providers](/docs/concepts/models).

## Reading the leaderboard

```
variant  model                                   score  n  failed  latency      cost
mini     openrouter/openai/gpt-4o-mini            0.90  1       0    820ms  <$0.0001
haiku    openrouter/anthropic/claude-3.5-haiku    0.40  1       1     2.4s   $0.0031
```

* **score** - mean case score, 0.0-1.0. `-` means nothing has been scored yet,
  which is not the same as scoring zero.
* **n** / **failed** - cells completed, and cells that errored.
* **latency** / **cost** - from the same run metering the rest of the platform
  uses, so they are directly comparable with production traffic.

Aggregates are computed on read, so a half-finished experiment reports on the
cells that have landed. You can watch the ranking form.

Every cell links to its run trace, so a bad score drills straight into the tool
calls that produced it.

## Cost control

A 5-model x 20-case comparison is 100 real agent runs. Three things bound it:

* **`budget_usd`** - once spend on the experiment passes it, the remaining cells
  are skipped, the experiment completes, and the reason is recorded on it.
* **Run quota** - eval runs are billable runs and are checked against your plan
  before any cell is created.
* **Cancel** - drops pending cells. Cells already mid-run are allowed to finish
  rather than being orphaned, so cancelling is not an instant stop.

Transient provider failures (429s, timeouts) are retried with backoff up to
three attempts. A cell that keeps failing is recorded as an error rather than
retried forever.

## Where eval runs go

Eval runs are written to `runs` with `source = "eval"` so they reuse the normal
trace viewer, and they are **excluded from AI Monitoring and `cruqai runs` by
default**. One comparison can add hundreds of rows, and burying real traffic
under them would make monitoring useless.

## CLI reference

| Command                                                        | Description                        |
| -------------------------------------------------------------- | ---------------------------------- |
| `cruqai evals suites [--agent A]`                              | List suites.                       |
| `cruqai evals create-suite <name> --agent A`                   | Create a suite (or `--workspace`). |
| `cruqai evals suite <ref>`                                     | Show a suite and its cases.        |
| `cruqai evals delete-suite <ref>`                              | Delete a suite and its cases.      |
| `cruqai evals add-case <suite> --name N --input T --scorers J` | Add a case.                        |
| `cruqai evals delete-case <id>`                                | Delete a case.                     |
| `cruqai evals run <agent> --suite S --model M --model M`       | Start a comparison.                |
| `cruqai evals list`                                            | List comparisons.                  |
| `cruqai evals results <id> [--matrix]`                         | Leaderboard, and every cell.       |
| `cruqai evals cancel <id>`                                     | Cancel a comparison.               |

With `--watch`, `cruqai evals run` polls until the comparison finishes and
**exits non-zero if any cell errored** - enough to gate a CI step on a
comparison that broke.