> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alpha.new/llms.txt
> Use this file to discover all available pages before exploring further.

# Build, try and test a support-triage agent

> Describe an agent that triages Harbor's inbox, try it on a real message, and prove it with an eval before you rely on it.

About 20 minutes

This guide builds an AI agent end to end: you describe it in one message, read it on the **Agent** tab, run it on a real customer email, then test it against a set of written test cases and fix what fails. If you want a feature inside an app that calls a model, rather than a standalone agent, describe the app instead; [Build an AI agent](https://docs.alpha.new/agents/build-an-agent.md) explains where the line falls.

By the end of this guide you will have `harbor-triage`, an agent that reads a message sent to Harbor Sailing School, sorts it, and drafts a reply, with an eval run that shows it passing every test case.

## Before you start

* A session you own. Start one from the [home screen](https://docs.alpha.new/sessions/home-screen.md), or use an existing one such as "Harbor bookings".
* An OpenAI key saved as a [secret](https://docs.alpha.new/secrets.md) named `OPENAI_API_KEY`. Add it under **Account menu → Global secrets** so every session gets it, or in the session's **Secrets** tab. The agent's live runs and the eval's AI reviewer both read it.
* The **Agent builder** skill switched on. It is on by default; see [Skills that ship with Alpha](https://docs.alpha.new/skills/built-in-skills.md).

> [!WARNING]
>
> Every live run spends money on your key. That covers the agent's own proof while it is built, each message you send in **Try it**, and every test case in an eval run. Writing test cases and working out the cost of a run are free. Nothing in an eval runs until you press **Run evaluation**.

## Build and test the agent

**1. Ask for the agent**

Paste this into the composer and press <kbd>Enter</kbd>. Say "agent" in words: "sort our inbox every morning" is built as app code instead.

Try this

```text theme={"system"}
Build an AI agent called harbor-triage for Harbor Sailing School.
It reads one customer message as text and returns:
the category (booking change, cancellation or weather, refund, question),
how urgent it is, a two-line note for staff, and a draft reply in a
friendly, plain voice. It must never promise a refund; staff decide those.
```

Alpha will:

1. Ask one [question card](https://docs.alpha.new/chat/plan-mode.md): the agent's shape first, then its inputs or framework if your message leaves them open.
2. Write the agent as a folder in `/workspace/harbor-triage` and draw it on the **Agent** tab before it is finished.
3. Prove it: check the code, run it on a stand-in model, then run it once live on your key.
4. Ask once whether you want a page around it. This guide declines.

Each question lists its recommended option first. This guide typed its own shape into **Something else…**: "Classify first, then write the staff note and the reply side by side, then one step that combines them." Click **Send answers**. **Skip — let the agent decide** leaves the choices to the agent.

**2. Read the agent on the canvas**

The **Agent** tab opens by itself each time the agent publishes. Read it top to bottom: the **Inputs** card is what the agent is given, each step card below it is one step with its model, instructions and **Returns**, and the **Answer** card lists the fields it hands back. Steps that run together sit in one outline with a `parallel · 2` chip.

Click any card to open its details beside the canvas, including **Defined in**, which opens `definition.yaml` and names the field the card comes from. To change the agent, ask in the composer, for example `Add a critic step that checks the reply`. See [The Agent tab](https://docs.alpha.new/agents/agent-tab.md).

**3. Try it on a real message**

Click **Try it** in the canvas toolbar. A chat opens beside the canvas, with one field for each input and a few **Start from a sample** cards the agent wrote. Pick one, or paste a real message such as `Hi, the forecast says 30 knots on Saturday. Is my 9am dinghy class still on? I can't do Sunday.` Then click **Run**.

Each message runs the whole agent once, live, with no memory of earlier messages. The dots on the canvas turn from blue to green as each step finishes. The answer card shows each field, with **JSON** and **Copy**; hover your message for **Edit and run again**. If a run fails, **Ask the builder to look into it** sends the failure to the agent. See [Try your agent](https://docs.alpha.new/agents/try-it.md).

**4. Set up evals and review the test cases**

Once the agent is proved live and published, the transcript shows a **Test this agent with evals?** card. Click **Set up evals**, read the **How evals work** dialog, and confirm with **Set up evals**. The canvas's flask button, **Not tested yet: set up evals**, does the same.

The agent writes test cases for free and posts an **Eval proposal** card with a cost range. Click **Open eval setup**. On the **Test cases** step, each test case is a persona asking something in a scenario, such as an upset customer asking for a refund. Click a row to edit **What the user asks** or **What it checks**, and star the ones that **Must pass**. Then click **Next: set up the run**. [Set up evals](https://docs.alpha.new/evals/set-up.md) explains the coverage grid.

**5. Run the evaluation with a spending limit**

The **Set up the run** step shows how each answer is graded: an AI reviewer scores it on the **Quality metrics** and explains each score, and **Format checks** check its shape in code, instantly and for free. Leave **AI reviewer model** as it is unless you have a reason to change it.

The **Estimated cost** card works out the price before anything runs. Type a cap into **Spending limit**: no new test case starts once the next one would pass it. Then click **Run evaluation**. The tab moves to **Results** and shows each test case as it runs. It keeps going if you close the tab; **Pause** and **Stop** are there if you need them. See [Run an evaluation](https://docs.alpha.new/evals/run.md).

**6. Fix a failure with Alpha and run again**

If the run fails, for example because a must-pass test case failed, the verdict reads **Not ready yet**. Its heading usually quotes what the first failed case expects, such as "Never promise a refund before staff review it." Click the case to read the agent's answer and the reviewer's reasoning.

Click **Fix with Alpha**. The agent reads the failed cases together and proposes one change, to the agent or to a test case that checks the wrong thing. It edits nothing until you say yes, and re-proves the agent after changing it. Then click **Run again**. When every test case passes, the verdict reads **Ready**, and **Approve as baseline** makes later runs compare against this one. See [Read results and fix failures](https://docs.alpha.new/evals/results.md).

> [!TIP]
>
> Your agent is built, tried on a real message and passing its eval. The canvas now carries the latest result on the answer step.

You described an agent, read it, ran it once by hand, then proved it against a written set of test cases and fixed what failed. Run the eval again after each change to the agent, so you can tell whether the change made it better or worse.

> [!NOTE]
>
> If Alpha builds app code instead of an agent and no **Agent** tab appears, the **Agent builder** skill may be off for your sessions. Open it on the [Skills page](https://docs.alpha.new/skills/overview.md) and switch on **Installed in your sessions**.

## Next steps

> **[Build an AI agent](https://docs.alpha.new/agents/build-an-agent.md)**
>
> The shapes an agent can take, the ten frameworks, and what each proof stage checks.

> **[Read results and fix failures](https://docs.alpha.new/evals/results.md)**
>
> Split grades, inconsistent cases, the detailed report and the shareable results page.

> **[Try your agent](https://docs.alpha.new/agents/try-it.md)**
>
> Typed fields, file inputs, choosing a framework per message, and earlier chats.

> **[Post a weekly summary to Slack](https://docs.alpha.new/guides/weekly-summary.md)**
>
> Put work on a timetable with a routine that runs while you're away.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.