Python

How to Evaluate AI Agents: Grade the World, Not the Transcript

An agent does not produce an answer you can compare with a reference; it produces a changed world and an account of the change. A harness that grades the world: state diffs, pass^k over five runs, failure shapes and cost per success, measured on Claude Haiku 5.5 and 4.5 against an LLM judge.