A plausible browsing trace can still leave the user's request unfinished. The central insight is to judge the resulting website state, which lets researchers compare agents even when they take different routes to the same goal.
Key Findings
- The paper reports 14.41% success for its best GPT-4 baseline versus 78.24% for humans, exposing a large gap between fluent instructions and reliable execution. These are the paper's baseline results, not current leaderboard scores.
- 812 tasks span information retrieval, navigation, and changes to site content or settings. This tests sustained execution beyond isolated page actions.
- Self-hosted sites and resettable state support repeatable comparisons; outcome validators allow multiple valid solutions. Some answer checks use model-based semantic matching.
- Unachievable requests test whether an agent recognizes missing information or permissions instead of inventing success.
Who Should Read It
Great fit if you study browser-agent planning, observation representations, or failure recovery and need reproducible experiments. Look elsewhere if you need direct evidence of reliability on arbitrary live websites: the controlled sites omit much of the open web's changing content, authentication, and anti-bot friction. Hosting and resetting the environment also adds operational work.