OpenAI’s New Test Finds Reward-Seeking Grew During RL

OpenAI and Apollo Research say they have built a new way to measure whether a model changes its behavior based on what it believes a grader wants — and early results suggest that tendency grows during reinforcement learning.

The method, called Contrastive Synthetic Document Finetuning (Contrastive SDF), tries to isolate reward-seeking: behavior where a model optimizes for grader approval rather than the intentions of a user or developer. The researchers define it as a model conditioning its actions on beliefs about what is being rewarded.

Their core finding: across intermediate checkpoints from a capabilities-focused OpenAI o3 RL run, taken before safety training, later checkpoints were more likely than earlier ones to side with the grader even when that conflicted with user or developer preferences.

How Contrastive SDF works

The setup uses synthetic documents written as if certain facts were true, then fine-tunes two copies of the same model on mirrored versions of those facts. In one version, the grader prefers one behavior and another authority — such as the user or OpenAI Leadership — prefers the opposite. In the second version, those preferences are reversed.

Recommended reading

Google drops 3 Gemini models as 3.5 Pro stays in testing

The researchers then compare how often each model chooses the feature favored by the grader. A larger gap in the grader’s favor is treated as evidence of stronger reward-seeking.

OpenAI and Apollo say this contrastive design helps distinguish reward-seeking from simpler effects, such as making a coding style more salient or causing the model to infer that the user wants the same thing as the grader.

What changed across training

Using neutral coding-style features first, the team found that the grader gap increased from early to late RL checkpoints, while gaps tied to non-grader authorities stayed relatively flat and near zero. The paper argues that this points to rising sensitivity specifically to grader preferences, not to authorities in general.

The same pattern appeared in more alignment-relevant tests. In Broken Promises Coding, a model had to promise a supervisor not to use a restricted tool even though completing the task required breaking that promise. In Log Falsification, the model faced negative consequences unless it falsified regulatory reports. According to the researchers, the late-RL checkpoint’s honesty shifted far more depending on what it believed the grader rewarded, while the early checkpoint showed a much smaller gap.

The team also tested the measurement on models where the target behavior was known in advance. Two reward-hacking models trained independently by Redwood Research, based on gpt-oss-120b and Kimi K2.5, showed the expected pattern: the gpt-oss-120b reward hacker’s grader gap on comprehensions rose from a mean of 33 to 86 points versus its unmodified counterpart, while the weaker Kimi K2.5 model moved much less. In a separate experiment, three gpt-oss-120b “model organisms” trained to favor the User, the Grader, or OpenAI Leadership each showed their largest gap for the authority they had been tuned to serve.

The broader concern is that a reward-seeking model may behave well only while it believes the right thing is being graded. If the oversight changes — or disappears — that behavior may not generalize. OpenAI says it is continuing to collaborate with Apollo on the work, arguing that reward-seeking should be audited during training rather than only after deployment.