Skip to main content
An agent pins one model per node. Evals answer the question that follows: would this agent be better, cheaper, or faster on a different model? You write a suite of test cases once, then run an experiment that executes every case against every model you want to compare. The result is a leaderboard ranked on quality, latency, and cost.

​
How it fits together

  • Suite - a named set of test cases. Belongs to one agent, or to the whole workspace so several agents can share it.
  • Case - a prompt plus one or more scorers that decide whether the answer was good.
  • Variant - one model under test, written as a fully-qualified provider/model and given a short label.
  • Experiment - a suite x variants fan-out. Every (case, variant) pair is one real agent run.
Swapping the model is a rewrite of the agent’s graph before it executes, so a comparison exercises the same prompt, the same tools, and the same retrieval as production. Nothing is stubbed.
An experiment pins the agent version at launch. Editing the agent while it runs does not change results already in flight, so a leaderboard always describes the graph that actually executed.

​
Scorers

Every scorer returns 0.0-1.0. Deterministic ones return exactly 0 or 1, and a case’s score is the weighted mean across its scorers.
tool_call matches against the run’s persisted trace, and args is a subset match - you can assert that a tool was called without pinning every argument. An unknown scorer type records no verdict rather than zero. A scorer written by a newer dashboard than the deployed worker will not quietly mark every model as failing.

​
Choosing a judge

llm_judge needs a judge_model, and it should be a strong model that is not one of the variants. A model grading its own output is the most common way an eval harness quietly flatters one model, so launching with the judge also under test is rejected outright. Judge spend is recorded separately from run cost. The cost column exists to compare what each model costs to serve, and folding grading into it would corrupt exactly that comparison.

​
Running one

Models must be fully qualified - openrouter/openai/gpt-4o-mini, not gpt-4o-mini - because the first segment selects the provider connection. See Models & providers.

​
Reading the leaderboard

  • score - mean case score, 0.0-1.0. - means nothing has been scored yet, which is not the same as scoring zero.
  • n / failed - cells completed, and cells that errored.
  • latency / cost - from the same run metering the rest of the platform uses, so they are directly comparable with production traffic.
Aggregates are computed on read, so a half-finished experiment reports on the cells that have landed. You can watch the ranking form. Every cell links to its run trace, so a bad score drills straight into the tool calls that produced it.

​
Cost control

A 5-model x 20-case comparison is 100 real agent runs. Three things bound it:
  • budget_usd - once spend on the experiment passes it, the remaining cells are skipped, the experiment completes, and the reason is recorded on it.
  • Run quota - eval runs are billable runs and are checked against your plan before any cell is created.
  • Cancel - drops pending cells. Cells already mid-run are allowed to finish rather than being orphaned, so cancelling is not an instant stop.
Transient provider failures (429s, timeouts) are retried with backoff up to three attempts. A cell that keeps failing is recorded as an error rather than retried forever.

​
Where eval runs go

Eval runs are written to runs with source = "eval" so they reuse the normal trace viewer, and they are excluded from AI Monitoring and cruqai runs by default. One comparison can add hundreds of rows, and burying real traffic under them would make monitoring useless.

​
CLI reference

With --watch, cruqai evals run polls until the comparison finishes and exits non-zero if any cell errored - enough to gate a CI step on a comparison that broke.