Teaching a model to grade its own homework

Alignment training usually depends on someone else. A reward model, trained on human preference pairs, tells a language model which of two answers is better, and reinforcement learning pushes the policy toward whatever scores highest. The reward model is the referee, the policy is the player, and both are ultimately trained by people. A paper posted to arXiv on 18 January 2024 by Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu and Jason Weston of Meta AI and New York University removes the referee. The model itself grades the answers. The answers it grades are the answers it produced. And the grading model and the policy model are, crucially, the same network.

That loop is worth slowing down to inspect, because at first glance it looks like a sleight of hand. The authors start from an already instruction-tuned model and prompt it as an evaluator: given a question, a candidate answer and a reference, it produces a judgement and a score. Those judgements are converted into preference pairs, one chosen response and one rejected one, and the next iteration of the model is fine-tuned on those pairs with iterative direct preference optimization. Then the whole cycle repeats. Three rounds in, starting from Llama 2 70B, the resulting model not only follows instructions better, it also produces higher-quality evaluations of its own outputs. The referee improves with practice, because the referee and the player are the same object.

The claim is not that the model trains itself from nothing. Human-written instruction data and a hand-built set of evaluation prompts frame the task, and the base evaluator is still a large model trained on human text. What is new is the removal of the external reward model as a fixed, separately trained quantity. In the standard recipe the reward model is a bottleneck: it must be labelled, fitted, tuned and frozen, and any bias it carries becomes a ceiling the policy cannot break through. If the reward signal comes from the same model that is being improved, that ceiling moves upward as the model improves.

The results make the loop worth taking seriously. On standard instruction benchmarks, the iterated model outperforms many existing systems, including larger or more heavily trained models that sat near the top of the leaderboard only a few months earlier. That is the part of the paper that makes people lean forward, because it suggests the most expensive part of alignment, collecting fresh human preference data for every round, may not scale as steeply as assumed. If a model can grade its own homework well enough, the marginal cost of another alignment round approaches the cost of generating and judging samples, not the cost of a new annotation campaign.

Read alongside the preference-optimization methods that deleted the separate reward model a year earlier, this work deletes the reward model as an external artefact altogether. The direction of travel over twelve months has been to shrink the pipeline: from reward models and reinforcement loops, to preference losses, to a single model that generates, judges and improves itself. Whether self-generated judgement keeps improving past three rounds, or collapses into flattery, is the open question the next year of work will answer. The loop itself is now on the table, and it is cheap enough to run that the answer will arrive quickly.