A benchmark can be difficult for a reason other than obscure facts: it can demand the kind of domain reasoning that even a highly educated researcher cannot easily borrow from the web. GPQA’s central contribution is to measure that gap under carefully tested conditions, making it useful for studying how people can supervise AI systems on questions beyond their own expertise.
Key Findings
- The collection contains 448 questions across biology, physics, and chemistry, written and checked by domain experts.
- Experts with or pursuing PhDs in the relevant fields reached 65% accuracy; the authors report 74% when identified expert mistakes are excluded. Skilled non-experts from other fields reached 34% after spending an average of 37 minutes per question with unrestricted web access.
- GPT-4 achieved 39% in the reported baseline, against 25% chance performance. This makes the benchmark informative for scalable oversight research, not just model ranking.
- GPQA Diamond is a stricter 198-question subset, selected for higher question quality and difficulty.
Who It’s For
A strong fit for researchers evaluating scientific reasoning, comparing model capabilities, or studying whether experts can reliably assess AI answers outside their specialties. Look elsewhere if you need a broad measure of general knowledge or a large, continuously refreshed question bank. “Google-proof” describes the tested conditions and validation process; it should not be treated as a guarantee against later data contamination or improved retrieval systems.
