Most existing code-evaluation suites emphasize single-file fixes or short edits that can be overfit or validated by flawed tests. SWE-Bench ProMax reframes the problem around refactoring: coordinated, behavior-preserving changes across many files and languages that better reflect professional, long-horizon engineering work.
Key Findings
- SWE-Bench ProMax contains 170 human-curated refactoring instances drawn from real commits across seven languages (Python, Java, TypeScript, Go, C, C++, Rust). Instances average 11.4 modified files and 261.6 lines changed, substantially larger than prior benchmarks.
- Curation targets known benchmark failure modes: issue descriptions are rewritten from scratch for clarity, test suites are manually reviewed to remove overly narrow or overly broad checks, and trivial or single-file tasks are filtered out to preserve difficulty and realism.
- Empirical evaluation shows frontier models under common agent scaffolds achieve a peak resolve rate of 41.2%, confirming the benchmark is substantially unsaturated and stresses long-horizon coordination and behavior-preservation capabilities.
Who it's for and trade-offs
Great fit if you are evaluating or developing AI coding agents, autonomous refactoring tools, or benchmarks that measure cross-file, behavior-preserving edits across multiple programming languages. It highlights real-world failure modes (test quality, specification ambiguity, large-scope edits) that short bug-fix benchmarks often miss. Look elsewhere if you need micro-benchmarks focused on single-line fixes, unit-level algorithmic problems, or extremely high-volume automated scoring: ProMax prioritizes curation and realism over sheer instance count and may require longer evaluation cycles per instance.
