A successful agent run can conceal a failing system: the answer may be right even when the harness mishandled context, tools, permissions, or execution. This benchmark makes that gap measurable by scoring what the agent achieved separately from whether the surrounding harness behaved reliably.
Key Findings
- Across four harnesses, nine models, and 3,420 runs, harness macro scores range from 59.8% to 72.0%; the spread suggests harness design can materially affect reliability.
- In 31.1% of runs, the task succeeded despite at least one failed harness criterion. Task completion alone therefore overstates operational quality.
- The 95 mechanism-oriented tasks examine context, execution, governance, lifecycle, observability, tools, and verification. Native harness behavior is preserved to make comparisons more meaningful.
- Trajectory Tracing and Multi-source Evidence Acquisition connect verdicts to execution records, harness data, and environment state. Failure attribution uses explicit mechanism chains, providing an auditable path for review rather than treating a proposed link as proof of causation.
Who It's For
A useful reference for researchers and engineering teams comparing agent runtimes, diagnosing failures, or designing more trustworthy agent evaluations. Treat the aggregate scores as descriptive: the study keeps one selected run per model–harness–task cell, so it does not estimate repeat-run stability. Coverage is also uneven, with only one task for verification, which limits conclusions about that dimension.