Skip to main content
An eval runs your agent on a written set of test cases and grades every answer. Alpha writes the test cases and prices the run; you review them on the Evals tab, and nothing runs until you start it. To try a single input by hand while you build, use Try it instead.
The agent never starts a paid run on its own. It drafts, checks and prices the test cases, then stops. A run starts only when you click Run evaluation, or ask the agent for a run in plain words, such as “run it”.

​
How evals work

A test case is someone asking your agent something in a particular situation. The situation is a scenario and the someone is a persona. Alpha writes one test case for every scenario with every persona, so four scenarios and three personas give twelve test cases. For harbor-triage, a scenario might be a weather cancellation and a persona an upset customer. Each test case carries an input and the checks that grade the answer, of two kinds: A test case marked Must pass decides the whole run: “If this test case fails, the whole evaluation fails.” Checks marked Diagnostic feedback are reported in the results but never change whether the test case or the run passes. You pick the AI reviewer’s model on the next step, Run an evaluation.

​
Start setting up evals

Evals test an agent Alpha has built and proved in this session. There are four ways in, and each ends with the agent writing test cases in the transcript. The buttons first open a How evals work dialog: setting up is free, and every test case in a run calls your agent’s real model, which costs extra. Click Set up evals to send the request. The button reads “Asked the agent” for a moment, and the reply arrives in the transcript. Alpha will:
  1. Write the scenarios, personas and test cases into your agent’s test cases file, with your focus cases marked Must pass.
  2. Check the test cases and work out the cost of running them, without running your agent.
  3. Post an Eval proposal card in the transcript, then stop and wait for you.

​
Review the proposal

When the test cases are written, the agent posts an Eval proposal card tagged “not started”. It lists what the test cases cover and how big a run is: “12 cases × 1 trials = 12 agent runs per target.” A trial is one run of one test case; each runs once unless you ask for more. A target is one framework your agent was built for.
Running the evals costs real money. The card says so in its own words: “Costs real money. About $L to $H on your key.” The range runs from your agent’s share to the total with grading, and is an estimate, not a charge. Before posting the card, Alpha sends one tiny request to each model the run will call, a fraction of a cent on your key, to confirm the model exists and your account has credit.
Click Open eval setup to open the Evals tab on its Test cases step, with the proposed test cases chosen for the run. Not now folds the card to “Eval proposal; not now. Nothing ran.” The card says “press Start”; the button you press on the Evals tab is Run evaluation. A card that proposes only some test cases, for example after “run it with these new inputs”, names them: “Only harbor-refund-04: the other cases stay in the suite and do not run.” If the setup is already open, click Choose only these cases.

​
Shape the test cases

The Evals tab has three steps: Test cases, Run and Results. The first opens on a diagram of what the test cases cover: Scenarios, What’s being tested and Personas. The middle card names your agent, its scope and its steps. The scope reads “The whole team, every step from input to output” for an agent with several steps, such as harbor-triage, and “The whole agent, from input to output” for a one-step agent. Until the agent has written any test cases, the step shows a single Generate test cases button instead. Click a scenario or persona to filter the table to its test cases; click it again, or Show all, to clear the filter. A “draft” tag marks test cases nobody has reviewed yet. When you’re happy with them, tell the agent “these cases are right” and it marks them reviewed and locks them. If a locked test case changes later, the next run refuses to start.

​
Edit a test case

Click any row to open the test case in a side panel. Its tags at the top show the scenario, the persona, ”★ Must pass” and “Draft” where they apply. Edits stay unsaved until you click Save. Duplicate with new inputs asks Alpha to write another test case like this one, in the same scenario and persona with the same kind of checks. Remove takes this test case out. Like every edit here, it waits for Save.

​
Add more test cases and save

These buttons ask the agent to write test cases. They appear on the tab once it saves the file. Writing them takes a turn, not an eval run. Deleting a scenario or persona asks first, for example “Delete Refund request and its 3 test cases?”. Nothing is saved until you press Save, and Undo brings it back.
1

Save your changes

Once you’ve edited something, the footer reads “You have unsaved changes.” Click Save to write the test cases file, or Undo to drop every unsaved change.
2

Move on to the run

Click Next: set up the run. It saves anything still unsaved first, and stays off while there are no test cases. Continue with Run an evaluation.
If the agent changes the test cases file while you edit, a banner reads “The file changed in the sandbox while you edited”. Take the file drops your edits; Keep my edits keeps them, and saving replaces the agent’s change. If the file is broken, the tab reads “The test cases file doesn’t read”: click Ask Alpha to fix it. The Test cases and Run steps are Owner only. People you share the session with see only Results.

​
Next steps

Run an evaluation

Choose the AI reviewer’s model, set a spending limit and press Run evaluation.

Read results and fix failures

See why each test case failed and let Alpha propose one fix for all of them.

The Agent tab

The canvas carries your latest eval result on its flask button.

Build, try and test a support-triage agent

Follow harbor-triage from first prompt to a passing eval.