GPQA Diamond
Definition
GPQA Diamond is a curated subset of a challenging scientific question benchmark.
GPQA Diamond contains in the cited version 198 multiple-choice questions from biology, physics, and chemistry. The subset is formed based on the answers of expert and non-expert examiners. It belongs to the GPQA dataset and is to be distinguished from the larger main dataset.
Simply explained
Like a particularly carefully selected exam folder. However, performing well in it does not automatically mean every task outside this folder has been solved.
Why it matters in practice
When comparing model results, it must be clear which dataset subset was used. The GPQA work distinguishes Extended, main dataset, and Diamond.
Example from business
Possible example: An AI team explicitly documents for a model comparison that it used the Diamond subset.
Limits and pitfalls
- The Diamond selection is not identical to the complete GPQA dataset. The selection considers expert answers and retrospectively traceable errors; it is not proof of universally objective questions or general professional suitability.
Origin & history
Rein and other authors submitted the cited GPQA initial version on November 20, 2023. The section on dataset splits also describes Diamond.
Use in mid-sized companies
- Possible applications include the classification of scientific model tests and checking whether published results concern the same dataset subset.
Frequently asked questions
Is Diamond the GPQA main dataset?
No; the work explicitly distinguishes both.
Which subjects does the cited work cover?
Biology, physics, and chemistry.
Sources
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark · Rein et al. (2023): GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Editorially reviewed on: