Run an evaluation

Running an evaluation sends every test case through your agent on its real model, grades each answer, and spends money on your model keys. This page covers Run, the second step of the Evals tab, and the live run that follows it. To write or change test cases first, see Set up evals. To read what a finished run found, see Read results and fix failures.

​

Choose how answers are graded

Click Next: set up the run on Test cases, or Run in the step control. Set up the run opens with What’s tested: the agent, its framework and model, and how many steps, tools and test cases the run covers. Edits on this page save themselves. If you see “AI review is graded by …, the model your agent runs on.”, the reviewer is marking its own model’s work. Pick another model, or click ask Alpha to switch it.

​

Run each test case more than once

A model can answer the same question differently each time. One run per test case shows the agent can get it right; several show whether it does so reliably. Open Advanced and choose Run each test case: once, 2 times, 3 times or 5 times. Each extra run multiplies the agent runs and AI review calls, and the cost card reprices once the change saves itself. The evals Alpha writes run each test case once. Raise the count when a case passes one run and fails the next. Edit suite.yaml opens the setup file itself.

​

Check the cost and set a spending limit

The Estimated cost card prices the run as soon as the saved test cases are ready. It shows “About” and a total, then one line per part: Type a dollar amount into Spending limit, or leave it empty for none. No new test case starts once the next one would pass it. The limit is soft: cases already running finish, so the total can end slightly over. If the estimate is already above it, the card warns “Over your spending limit: some test cases will not run.” Alpha suggests one when the estimate is over a dollar.

​

Start the run

You can also ask the agent. It never starts a run on its own, only when you ask. Alpha will:

1. Say in one line what runs and about what it costs.
2. Start the run, which appears in the Evals tab like any other.
3. Post an Eval run started by Alpha card with Watch it in Evals.
4. Post a result card in the session when the run ends.

“Run again” and “resume it” work too. A resumed run posts Eval run resumed by Alpha.

​

Watch, pause or stop a run

Results shows Running and a line on what’s happening now: “Starting the run…”, then “Running test case 3 of 8”, or “Running 4 test cases at once · 2 of 8 done”. Test cases usually run four at a time; the agent sets one at a time when your agent’s tools touch something runs would clash over, such as a real inbox. Below are a progress bar, one tile per test case, and Passed so far, Spent so far, Time so far and Time left. These buttons are the session owner’s Owner only. A viewer of a shared session can watch the run but not control it. If the page says “Nothing new for a while; the runner may have stopped.”, click Resume.

​

Why a run can’t start

Run evaluation stays off until the eval is ready, and the cost card says what’s missing. Warnings that don’t block the run show as one line in the cost card, with Ask Alpha to fix it.

​

How a run ends

The run list on Results shows each run’s state beside its start time. Run those again appears on Results when nothing failed but some cases didn’t run. It returns to the Run step with only those cases chosen; Run every case puts the rest back. Cases that never reached grading count as not run, never as failures.

​

Next steps