Why capability evals weren't enough for our agent, and what behavior evals caught
This is Part 2 in an ongoing series. Start with Part 1 if you haven't already.
The short version
- Capability evals check the result. Behavior evals check the path. Agents need both.
- If every expectation in your spec can be graded from the final state, you haven't specified behavior yet.
- Swap in a weaker model to find where your harness leans on the model instead of enforcing structure.
- Connect behaviors to business metrics through trajectories.
- Agree on what happens when a gate goes red before a launch is at stake.
In the first post in this series, we explained why our Product Managers write the behavior spec before anyone builds evals for Airo. That's a definition of what good looks like, broken into behavior families and trajectories, the stretches of the customer's journey like discovery, first build, and first publish. This post picks up where that one stopped, with the trap that comes right after.
Nearly every team building an AI agent starts with capability evals. Give the agent a request, define the expected result, and check that they match. It's a sensible place to start, and it tells you whether a request got done. It doesn't tell you how the agent got there, and with agents, that's usually where things break.
Airo, our agent that turns a customer's request into a working website, stays the example here. Swap in your own agent, and the same ideas apply.
Capability evals and behavior evals are different jobs
Most agent eval efforts mix two kinds of evals. A capability eval grades the outcome of a turn against the request. The customer asked for something, and you check whether the site now reflects it and whether the agent's reply matches what happened. A behavior eval, sometimes called an intrinsic eval, grades the agent's behavior on its own terms: the decisions it made and the path it took, separate from the outcome. It can't be done without the trace.
Teams drift toward capability evals because they're cheaper, more deterministic, and easier to automate. That's fine for component teams checking that their piece works. The problem starts when the whole spec is written that way.
This matters because the final state can look perfect while the path to it was wrong. A capability check passes all of these:
- The agent gets the right result after three duplicate tool calls, a retry loop, and a second request to sign in.
- It reports "done" while a worker is still running, and happens to be right a minute later.
- It writes to the site after a stop, then overwrites that change on the next turn.
Here's how a few requests look when graded both ways. The first column uses the behavior families from the spec:
| Behavior family | Customer request | Capability check (the result) | Behavior check (the trace) |
|---|---|---|---|
| Act on clear, clarify vague | "Make the heading bigger," when two headings could be meant | The heading the customer meant is now bigger | The agent saw two possible targets and asked which one before writing anything, instead of guessing and getting lucky |
| Honor customer context | "Use my upload as the hero image" | The hero shows the uploaded file, and the reply says so | The upload and target reached the worker that did the placement, unchanged, on the first attempt |
| Handle the lifecycle | "Update the headline and button text, and add a contact form" | All three changes are on the page | Each item was tracked separately, a failed form step was retried within policy rather than looped, and the agent never reported done while work was pending |
| Handle the lifecycle | "Stop" during a build | The site matches its state at the moment of the stop | No new work started after the stop, and no late worker result landed, even briefly |
| Respect scope and authorization | "Publish it," when sign-in is required | The current version is live | The agent asked for sign-in once, waited, resumed from the blocked step, and published exactly once |
Every row can pass on the left and fail on the right. That gap is why you need both, and why they shouldn't be owned by the same people.
Evals detect. The harness team diagnoses.
Capability evals belong with the teams who own components, and they stay at the level of a single turn: request in, observable result out. Behavior evals belong with whoever owns the harness. When a component team's score drops but their piece looks fine in isolation, they shouldn't dig through traces to find out why. Their job is a reliable signal. They hand the failure over with the trace IDs attached.
On a small team, these are the same people. A PM and one engineer may write the spec, the evals, and the harness fixes in the same week. The split still holds as an order of steps: define what good looks like, measure it, then diagnose what fails. Write the spec first anyway.
A practical test for anyone writing the spec: for each behavior family, ask whether you could grade every expectation from the final state alone. If the answer is yes across the board, you haven't specified behavior yet.
A weaker model makes a good diagnostic
Here's the weaker-model experiment from the first post in more detail. Swap a weaker model into the harness and run your corpus through it. Don't expect the failures to point at the model. Most will land where the loop relies on the model being capable instead of enforcing structure itself:
- Missing guardrails.
- Skills that load too late.
- No check on whether what the agent reports matches what happened.
Fix those gaps, and keep the runs in the suite even after they pass. The weaker model becomes an early warning. If a later harness change brings back a dependency on model capability, it shows up there first.
The spec keeps changing
The spec is a living document. Every production incident becomes a test case, and the dataset gets versioned like code.
That only works if the dataset behaves like infrastructure. When a regression fires, the evals team should be able to pull the traces behind it in minutes. Turning a new production failure into a permanent case should take one step.
The PM also can't hand off the instrumentation conversation entirely. You won't build the machinery, but if you describe what good looks like without understanding what can be measured, you end up with well-written expectations nobody can grade. Learning that tradeoff is part of the job now.
Connecting behavior to the business
Agent behavior is what you control directly, but it isn't what the business cares about. So you watch both:
- Behavior evals are leading indicators. They move when you change the agent, show where a regression came from, and gate the release.
- Business metrics sit further down the journey. They move for many reasons that have nothing to do with the agent.
Trajectories are what connect the two. Each one pairs a set of behaviors with the business metric they should move: discovery with effort before the build, the first build with first-build success, and the first publish with reaching a live site.
So when a business number dips, the trajectory tells you where to look first. Check that trajectory's behaviors before you call it an agent regression. If they're green, look at the rest of the product next. If nothing there explains the dip either, your spec is probably missing a behavior.
The two layers only help each other if they share names. Define each behavior once and use that name offline and in production, even when the math behind the numbers differs. And be clear about what offline results prove. A green suite says the agent behaves as specified. It doesn't say customers are happier. Check whether behavior predicts business, one trajectory at a time. If they diverge, you're measuring the wrong behaviors, or your eval is stricter than your customers are.
Deciding consequences ahead of time
If nobody has agreed on what a red gate means, evals only inform. The spec should say in advance what happens when the gate goes red:
- Block the merge.
- Roll back.
- Ship, with a named person accepting the risk in writing. Never for a critical failure.
- Narrow the scope.
Decide this for each trajectory. One policy for the whole agent is too blunt. A critical failure in the first build, like telling a customer their site is done when it isn't, should block no matter what. A small dip in how discovery questions are worded might ship with a named owner and a date for the fix. The trajectory tells you what's at stake, and the severity tier tells you how much room you have.
The trajectory's business metric is what makes the policy stick. "Block the release" tends to get negotiated away the first time a launch is at stake. "This regression in the first build costs us completed builds" is much harder to argue with.
Disagreements between the layers are worth a close look. Go through this grid one trajectory at a time, because an overall green can hide one that's red:
| Business fine | Business down | |
|---|---|---|
| Behaviors green | Ship. | Rule out the rest of the product first: flows, sign-in, upgrades. If nothing there explains it, your spec is missing a behavior. This is the dangerous case. |
| Behaviors red | Check the severity. A critical failure still blocks, and so does anything required by policy or regulation. For anything else, the eval may be stricter than customers are. Recalibrate it as a spec change, never inside the release it's blocking. | A clear regression. Block or roll back. |
What the PM actually delivers
Evals answer a question the product spec can no longer answer. They aren't the first step. Before anyone writes evals, the PM defines what good looks like, covering the path as well as the result, and signs off on the quality bar as a number: something like "we ship at 95% on this and zero tolerance on that." That signature is what the PM delivers.
If you want to start this week, take the expectations you wrote for your first trajectory. For each one, ask whether you could grade it from the final state alone. Where the answer is yes but the path matters, rewrite the expectation to say what should happen along the way. If you don't have a spec yet, start with the first post and the Agent Behavior Spec Template.