The easiest way to evaluate an agent is to ask whether it produced the expected answer. In software, that can mean whether a patch matches a reference diff or whether a test suite turns green. These checks are useful. They are also only the beginning.

Real work has a longer tail. A change can pass the obvious test while increasing review burden, breaking an uncovered integration, consuming an unreasonable amount of compute, or leaving a task that no one else can resume. An agent that solves a benchmark by taking broad, poorly explained action may look capable until it is placed in a shared environment.

The practical question is not, "Did the model finish?" It is, "What did it take to finish, what did the system learn, and how much trust should the result earn?"

Final output is a narrow measurement

An agent can arrive at a correct patch through a fortunate guess. It can also arrive at an incorrect patch while producing a persuasive explanation. Neither possibility is visible when the only measure is a final answer.

For live systems, evaluation benefits from several complementary signals:

• Outcome: Did the requested behavior change in the intended way?
• Evidence quality: Did the task collect checks that actually support its claims?
• Scope: Did the agent make a focused change, or alter unrelated surfaces to force a result?
• Recovery: When an assumption failed, did it revise the plan and preserve the state?
• Review cost: Can another person understand the change without replaying the full task?
• Operational cost: Did the task stay within reasonable time, tool, and compute budgets?

No single score can faithfully represent all of this. The aim is not to manufacture a grand index. It is to notice failures that a simple pass rate hides.

Use the work that matters

Generic benchmarks are valuable for comparing models, but they rarely capture the conventions and risks of a particular organization. A productive evaluation set should include a small, evolving sample of real task shapes: a bug that crosses a frontend and API boundary, a dependency update with a compatibility condition, a visual regression at a narrow viewport, a change that requires a review before an external action.

The tasks should not be selected only because they are easy to score. They should represent places where the team would actually consider using an agent. For each task, the expected result can be defined in terms of behavior and evidence rather than an exact sequence of words or commands.

This approach has an additional benefit: it reveals where the infrastructure is weak. If agents repeatedly fail because they cannot identify the right environment, the problem may not be model reasoning alone. The workspace, tests, or task record may be missing a necessary signal. Evaluation can guide improvements to the system around the model.

Inspect the near misses

The most useful examples are often not clean successes or clean failures. They are near misses: a change that fixed the visible bug but removed an accessibility label; a task that found the right file but used a stale preview; a workflow that correctly prepared an external action but requested an approval too broad for the operation.

Near misses expose the relationship between capability and control. They tell us whether an agent tends to overreach, whether the runtime catches that overreach, and whether reviewers receive enough context to intervene. A mature evaluation practice collects these cases, turns the important ones into regression tasks, and updates the product rather than merely adjusting a prompt.

The system should learn without becoming opaque

Production experience should improve an agent system. A useful tool pattern can become a documented workflow. A recurring test failure can lead to better environment setup. A review that repeatedly catches the same missing evidence can change the task template.

But learning must remain visible and reversible. A model should not silently rewrite the policies that constrain it or mutate the shared instructions of other agents because one task appeared to succeed. Improvements to durable behavior need their own review path, provenance, and evaluation.

The result is a slower kind of compounding than a demo suggests, but a more valuable one. The organization becomes better at knowing which work to delegate, what evidence to request, and where the remaining uncertainty lies.

Agent evaluation is ultimately a study of systems, not only models. The measurement that matters is whether useful work becomes easier to complete, easier to review, and safer to maintain after the agent has participated.