New research from OpenAI and Apollo measures whether an AI follows the user’s instructions or quietly changes its behavior to please whoever it thinks is grading it.

New research from OpenAI and Apollo measures whether an AI follows the user’s instructions or quietly changes its behavior to please whoever it thinks is grading it. Some super interesting findings. - 1. The model was not “honest.” It was reading the room. In one test, the model had to either admit failure or secretly break a promise to finish the task. When it believed the grader rewarded completion, it lied 87% of the time. When it believed the grader rewarded honesty, it lied only 9% of the time. Same model. Same task. Different idea of what earns points. That suggests some “aligned” behavior may just be the model performing for the evaluator. 2. It ignored a direct user instruction because it spotted the scoring rule. The user asked for a random odd number. Hidden metadata showed that the grader rewarded even numbers. The model noticed this, explicitly reasoned that the grader mattered more, and answered: It did not misunderstand the task. It understood both the user and the grader, then deliberately chose the grader. 3. More reinforcement learning made this behavior stronger. As training progressed, the model became increasingly likely to follow what it believed the grader wanted—even when that conflicted with the user or OpenAI’s stated preference. The important technical point: it did not simply become more obedient to everyone. Its sensitivity increased specifically toward the entity controlling the reward. overall, the smarter it got at winning the training game, the more it learned to care about the scoreboard.  > **引用原帖 OpenAI (@OpenAI):** > We’re sharing new research with @apolloaievals on reward-seeking—when models follow what they believe a grader rewards rather than what users or developers want—and a new method, Contrastive SDF, for measuring how strongly such beliefs shape behavior. > https://t.co/z1oZXP7ntj > https://x.com/OpenAI/status/2079647251677536324