Understanding Reasoning from Pretraining to Post-Training

Can you figure out how much RL will actually buy you before you commit a month of GPU time? This paper says you can predict it based on the checkpoint you start from. The authors recreate the entire LLM pipeline in chess -- pretraining 5M-1B models on Lichess games , SFT on synthetic reasoning traces, then GRPO on puzzles with verifiable rewards -- and sweep 36 pretraining-RL combinations. Signal is mostly carried by two pretraining properties. The pretraining validation loss predicts the post-RL pass@1, with the fit tightening from |ρ|=0.93 to 0.99 as the reference RL compute increases; and the reward gained per 10× of RL compute increases with log pretraining tokens (Pearson r=+0.84). The same two patterns are seen in a 1B model trained on math text. The fitted frontier puts RL’s compute-optimal share at around 20% at 50M parameters, and 28% at 680M. With budget, the share increases but remains a minority. The mechanism is the part worth noticing. RL is not uniform sharpening: on the hardest puzzles it generates correct moves to almost non-existent levels after SFT, but makes already-preferred wrong moves many times more often. That is why they get pass@1 gains but not pass@16 gains, and it leads the work in a different direction: suppress wrong-mode amplification, rather than sharpen harder. – arxiv. org/abs/2607.16097 Title: "Understanding Reasoning from Pretraining to Post-Training"