Predicting model scale with fifty questions instead of ten trillion

Scaling laws are the quiet infrastructure of modern AI. Before anyone spends hundreds of millions of dollars training a frontier model, they train a ladder of smaller ones, measure how performance improves with compute, and extrapolate. The trouble is that measuring the ladder is itself enormous: thousands of checkpoints, tens of thousands of benchmark questions, repeated sampling over and over. A paper on arXiv submitted on 29 May 2026, by Sang Truong, Yuheng Tu, Rylan Schaeffer and Sanmi Koyejo at Stanford, argues that most of that measurement is wasted effort, and borrows a trick from the examination hall to cut it by more than 99 per cent.

The idea is to stop treating every model-question pair as its own number. Standard scaling estimation asks, for each checkpoint and each benchmark item, whether the model got it right, and fits a curve through all those points. That is a problem with as many parameters as there are models times questions. The Stanford team instead imports item response theory, the same framework that underpins standardized academic tests. In that world, a test has two unknowns: how able the student is, and how hard the question is. Estimate those two latent quantities, and the probability that a given student answers a given item correctly follows from them.

Applied to language models, a model is a student and a benchmark question is an item. The framework factors the estimation from a matrix of size models times questions down to the sum of the two, which collapses the parameter count by orders of magnitude. The authors extend the usual binary version of the theory, which only knows right from wrong, to handle the richer signals models actually produce: token probabilities during pretraining and pass rates when a model samples an answer many times. That extension, built on a Beta distribution, is what lets the method ingest the empirical probabilities rather than crude correct or incorrect labels.

The validation numbers are the point. For pretraining downstream scaling, the team used 6,612 model checkpoints and 37,682 questions across ten benchmarks. Once the framework is calibrated on existing responses, it reaches equally reliable scaling estimates using only 50 questions per benchmark, a reduction of 99.9 per cent, and matches or beats the decision accuracy of the brute-force approach. For test-time scaling, with 12 models and up to 2,500 samples per question, the same economy holds. Even better, the estimated model abilities transfer across benchmarks that share a goal, so a model’s strength on one task family can forecast its standing on another.

The appeal is practical. Under the old regime, a research lab that wanted to know how a new model would behave at scale had to run thousands of small models across tens of thousands of questions, a budget only the largest organizations could absorb. With 50 questions per benchmark, the same forecast becomes a weekend’s compute. That matters most for academia, where the cost of running a full scaling study has been prohibitive, but it also saves the large players money they currently spend confirming what a cheaper signal could already tell them.

The honest limits are that the method needs a one-time calibration on responses from existing models, so it is a framework for generalizing across the known rather than exploring the unknown. It also leans on the assumption that latent ability and question difficulty are the right way to carve up the problem, which is a bet about how these models work rather than a theorem about them.

Still, the result is a small revolution in how a field measures itself. For years, scaling has been treated as an expensive empirical exercise that only scale could afford. Once you ask which questions are actually informative, you can ask far fewer of them, and the answer does not get worse. Sometimes doing less work produces a better signal.