Practical web development often forces engineers (or coding agents) to infer missing behavior from a working application rather than implement features from a spec. ProgramDistill tackles this gap by automatically factorizing interactive web apps into feature-level behaviors and turning those behaviors into replayable, reference-guided tasks that can verify agent implementations against a working reference.
Key Findings
- Large-scale, automated task synthesis: a mine-craft-patch pipeline discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention — meaning evaluation can scale beyond hand-curated feature lists.
- Behavior-first, implementation-agnostic evaluation: tests target observed input-output interactions (replays) rather than source structure, so agents can use any architecture or language but are judged by whether reconstructed behaviors match the reference.
- Difficulty and diagnosis at controlled granularity: full-application reconstruction yields wide variance (best agents hit ~49.2% and ~28.8% cumulative-workflow success), while partial restorations reveal steep drops in success as restoration depth increases (from 100%→64% and 96%→32%), exposing long-horizon and compositional failure modes.
- Useful for benchmarking and curriculum design: the dataset’s feature factorization lets researchers measure performance per feature granularity and construct curricula that progressively increase restoration depth.
Who it's for and trade-offs
Great fit if you need a behavioral, reproducible benchmark for coding agents or want to study how models infer and reimplement features from working software — particularly for diagnosing compositional and long-horizon failures in web apps. Look elsewhere if you need language- or implementation-specific tests (ProgramDistill deliberately evaluates behavioral parity) or if your target domain is non-interactive systems where UI-driven replay is irrelevant.