# Eval-Driven Development

> Write the task-and-verifier pairs before the tool, prompt or agent, then let the numbers decide changes - the same discipline as tests-first, applied to components whose output is text.

- kind: guide
- category: design-patterns
- aliases: Eval-First, Agent Evals
- canonical: https://aipatterns.dev/guides/eval-driven-development
- inspired-by: https://www.anthropic.com/engineering/writing-tools-for-agents

## The problem it solves

Code has tests: run it, compare against expected, red or green. A tool description, a system prompt, or an agent loop has no `assert` - its output is prose and its failures are quiet: slightly worse routes, slightly more tokens, slightly more wrong turns. Teams then change prompts and tools by vibes, and quality drifts without anyone deciding to. Eval-driven development gives text-shaped components the thing tests give code: a fixed bar that a change must clear before it ships.

**The rule: if you cannot write a task and a verifier for a change, you are not ready to make it - and if you can, make the evals before the change.**

## The loop

1. **Name the tasks.** Write down what users or agents actually do with the component - "find the customer's recent orders", "route this ticket to billing" - as concrete inputs, not capabilities. Ten real tasks beat a hundred synthetic ones.
2. **Write a verifier per task.** The verifier answers "did this run do the job?" with as little judgment as possible: an exact answer, a schema check, a value bound (tokens, calls, latency), or a checklist another model applies against written criteria. If the verifier needs taste, the task is not defined well enough yet.
3. **Run the suite on the current version.** This is your baseline. Record pass rate plus the operational numbers: runtime, tool-call counts, token usage, errors by type.
4. **Make one change** - a tool description rewrite, a parameter rename, a consolidation. Change one thing; the suite tells you its price and its gain.
5. **Compare and keep or revert.** Improvement on the target tasks with no regression elsewhere keeps. Anything else reverts. The suite, not the reviewer's impression, decides.

## What to measure

- **Task success** - the pass rate over the verifier set, split by task type so you can see which tasks a change hurt.
- **Tool-call counts** per task - consolidation and description fixes show up here first.
- **Token usage** - the real cost of a prompt or schema change, per run.
- **Latency/runtime** - what the user experiences, not what the model averages.
- **Errors by type** - invalid arguments and wrong-tool picks point at descriptions; execution failures point at the tool.

A change that wins on quality but triples tool calls may still be right - but it should win because you saw both numbers, not because you saw neither.

## Task and verifier pairs

Anthropic's tool-writing guidance is blunt about the pairing: each evaluation prompt should come with a verifiable response or outcome, and the fastest way to improve an agent is to generate realistic task/verifier pairs and fix what they expose. Two properties make a pair good:

- **Realistic input**: drawn from real usage - a messy ticket, a half-specified request, a slug with a typo - not a clean example you invented with the answer in mind.
- **Verifiable outcome**: the expected entry slug, the exact structured answer, a bound ("under 2 tool calls, under 500 output tokens"), or a rubric another model applies against written criteria. "Looks good" verifies nothing.

Start each new tool, skill, or agent with three pairs - one typical, one edge, one refusal case. Grow the set only when a real failure shows you a gap; a large stale suite is worse than a small live one.

## How this platform runs its evals

The benchmark pages on this site run exactly this loop at the platform level: fixed tasks, agents with and without the MCP server, the same measurements, results published. Every tool-description change on the server should be able to name the eval numbers it moved - and when a change is proposed without numbers, the benchmark is where they come from.

## When not to bother

- One-off scripts and prototypes nobody will rerun. Evals pay for themselves across revisions; a thing with no next revision has no payer.
- Components with a trivially checkable output - a schema-validating parser already has its verifier; write the test, not an eval program.
- Exploratory work whose goal is to find out what the tasks even are. Explore first, then write the pairs - but write them before you polish.