Real desktop work mixes visual interaction and programmatic automation, yet benchmarks and environments often separate those modalities. RecreationWorld makes that mixture the evaluation primitive: agents must discover a running reference app and reimplement its behavior using any combination of GUI exploration, coding, and CLI, with hidden reference tests providing execution‑grounded rewards. This shifts the hard part from toy interfaces to coordinating perception, synthesis, and verification across real applications and OSes.
Key Findings
-
RecreationBench and the five‑platform harness (Ubuntu, macOS, Windows, Android, Web) let researchers generate reproducible, cross‑platform hybrid tasks that are automatically checkable via programmatic and visual assertions — so evaluations measure functional parity, not only surface actions.
-
Models trained on scaled, verified trajectories improve transfer to out‑of‑distribution coding and hybrid benchmarks, but still struggle: top systems verify static UI structure more reliably than dynamic interactions or computed outputs, and only a small fraction of tasks pass all programmatic tests end‑to‑end.
-
The framework surfaces concrete failure modes: agents often produce smaller, more monolithic implementations than references and fail to reliably orchestrate GUI steps with efficient CLI/code shortcuts; these gaps point to research directions in verification, modular synthesis, and multimodal planning.
Who it's for and tradeoffs
Great fit if you research or develop software agents that must blend visual perception with programmatic generation and need auditable, execution‑grounded evaluation across real OSes and apps. RecreationWorld is most useful when you can run sandboxed OS images and accept that building reliable verifiers and task references requires nontrivial engineering. Look elsewhere if you only need single‑modality benchmarks (pure CLI or pure GUI) or lightweight toy environments — RecreationWorld emphasizes realism and verifiability over minimal setup time.