Testing models at the frontier of human knowledge: Humanity's Last Exam

Language-model evaluation has a saturation problem. On MMLU, the multi-discipline multiple-choice test that anchored capability claims for two years, the strongest systems now clear ninety per cent. A score near the ceiling carries almost no information, because everyone is already near it. In response, a collaboration between CAIS, a safety-research institute, and Scale AI built a benchmark whose whole purpose is to be too hard, and posted it to arXiv on 24 January 2025. They called it Humanity’s Last Exam.

The number that matters is small. Across 2,500 questions spanning dozens of subjects, from mathematics and physics to medicine, law and the humanities, the tested frontier models all landed below about fifteen per cent accuracy, and most well below ten. The authors, Long Phan and Dan Hendrycks among more than a thousand contributors, report that calibration is poor as well: the models answer with a confidence they have not earned. Human subject-matter experts, by contrast, operate near the ceiling. The gap is not a small margin; it is a different regime.

What is interesting is how the questions were made. Rather than scrape textbooks or mine past exams, the organizers put out an open call and collected roughly seventy thousand nominations from experts worldwide, then filtered them down to a verified set of 2,500. Each item has an unambiguous, easily graded answer, either multiple-choice or short-form, but is deliberately written so that a model cannot retrieve it from the training set or look it up quickly online. That last constraint is the design’s quiet point. Much of what reads as reasoning on older benchmarks is pattern completion over canonical material. A question that cannot be found on the internet forces the model to actually work.

The result is a benchmark that behaves like a stress test rather than a league table. No model is close to finishing it, and so the ordering at the bottom is less informative than the floor itself: a large and durable gap separates current systems from the expert frontier on closed-ended academic questions. The authors are careful not to overclaim. They describe HLE as a measurement instrument, not a capability verdict, and they note that multiple-choice format and expert-written items favour a particular kind of knowledge. A model could improve a great deal on this test without becoming broadly wiser, and it could fail this test while still doing useful work.

That caution is worth keeping. Hard benchmarks have a way of becoming the next target, and HLE will be no exception. Teams will tune for it, the questions will leak into training, and the scores will climb, gradually eroding the very difficulty that motivated the exercise. The long-run value of the paper is less the leaderboard than the method: crowdsourced, retrieval-resistant, expert-verified questions that resist contamination by design. As long as evaluation treats benchmark saturation as a recurring condition rather than a one-time fix, the field keeps a way to tell whether progress is real.

There is also a subtler argument hiding in the low scores. The models failing here are the same systems that summarize documents, write code and hold conversations that feel knowledgeable. Failing a test that cannot be looked up, and failing it with confidence, suggests that a large share of fluent performance is reconstruction rather than comprehension. Fluency supplies the texture of expertise; the expert question asks whether the expertise is present. On HLE, the answer for now is mostly no.

Read in that light, the benchmark is a useful corrective to a specific kind of optimism. The systems that look impressive on familiar tasks still cannot field a narrow expert question they have not seen, which is a concrete way of restating that fluency is not the same as understanding. The exam is hard on purpose, and the humility that comes from failing it is the point rather than an embarrassment.