One-sentence Summary

Researchers from New York University, Modal Labs, et al. use chess as a controlled testbed to study the pretraining-to-RL pipeline, establishing a scaling law where post-RL performance is predicted by pretraining loss and RL reward slopes improve linearly with pretraining tokens, and revealing that RL amplifies correct moves on easy puzzles while surfacing novel correct moves on hard ones, with findings extending to math.

Key Contributions

• A controlled chess testbed replicates the full LLM pipeline from pretraining through RL post-training, enabling systematic variation of pretraining choices and RL compute.
• A joint scaling law shows that post-RL performance level is well-predicted by pretraining loss, and the slope of RL reward improvement scales approximately linearly with pretraining data tokens.
• Mechanistic analysis reveals that RL amplifies correct moves already preferred by the SFT policy on easy puzzles, while on hard puzzles it surfaces correct moves that were nearly absent but sometimes reinforces incorrect ones, explaining why pass@1 gains do not consistently transfer to pass@16.

Introduction

The standard LLM training pipeline combines large-scale pretraining with post-training via reinforcement learning from verifiable rewards, but the optimal allocation of a fixed compute budget between these stages remains unclear. Prior work developed scaling laws for pretraining and for RL post-training separately, yet no quantitative characterization exists for how pretraining and RL interact, and competing views debate whether RL merely sharpens existing behaviors or composes new skills. The authors address these gaps by constructing a controlled chess testbed that mirrors the LLM training pipeline, enabling systematic sweeps over pretraining and RL compute. They derive a joint scaling law showing that pretraining loss predicts post-RL performance and that the RL improvement slope grows with pretraining data scale, and they analyze how RL differentially reshapes the policy across easy and hard problems.

Dataset

The authors construct three chess datasets from Lichess, all mutually disjoint at the board‑position level to prevent contamination.

• Pretraining corpus Source: Blitz and Rapid games played on Lichess during 2022. Scale: 54 B tokens. Usage: The scaling sweeps draws varying token budgets from this corpus to pretrain the base models.

• Post‑training puzzle set Source: 156 K quality‑filtered Lichess puzzles. Structure: Puzzles are grouped into five difficulty bins (B1 – B5, easiest to hardest). Filtering: Quality‑filtered, though specific criteria are not detailed in the main text. Usage: Used for post‑training (supervised fine‑tuning) after pretraining.

• Evaluation benchmark Source: 1 480 tactical puzzles drawn from the same source and difficulty bins. Curation: Balanced for theme diversity and solution length. Reporting: Because models rarely solve B5 puzzles, aggregate pass@k results are computed over B1 – B4; B5 is retained for difficulty‑stratified mechanism analysis.

The pretraining data is used for language modelling, the post‑training puzzles for instruction tuning with constructed reasoning traces, and the evaluation set for measuring puzzle‑solving accuracy across difficulty levels.

Method

The authors leverage chess as a controlled testbed to study reasoning capabilities, designing a training pipeline that mirrors the standard language model paradigm. The overall framework consists of three sequential stages: pretraining on human game data, supervised fine-tuning with synthetic reasoning traces, and reinforcement learning within a verifiable puzzle environment. As shown in the figure below, this pipeline systematically transitions the model from basic move prediction to complex multi-step planning.

Chess Representation and Pretraining To process chess games, the authors represent each game as an alternating sequence of player moves serialized into discrete tokens. Drawing on standard chess notations, each move is encoded using a four-token structure: $\langle piece \rangle$⟨piece⟩, $\langle source \rangle$⟨source⟩, $\langle destination \rangle$⟨destination⟩, and $\langle flag \rangle$⟨flag⟩, where the flag token denotes special actions like castling or checkmate. This formulation yields a compact vocabulary of size $|V|=81$∣V∣=81. During the pretraining stage, an autoregressive policy is trained on a large-scale corpus of human game trajectories using the standard next-token prediction objective, allowing the model to learn the distribution of plausible move sequences.

Supervised Fine-Tuning with Synthetic Reasoning Traces In the post-training phase, the model is trained to solve chess puzzles by generating its own reasoning traces before committing to a final move. The puzzle environment is formulated as a multi-step interactive decision process. Given an initial board state $s_0$s0​ and a ground-truth solution line, the model acts solely as the solver. At each step $t$t, the model observes the current state $s_t$st​ and proposes a candidate move $a_t$at​. The environment verifies $a_t$at​ against the ground truth. A mismatch immediately terminates the episode, while a correct move prompts the environment to append the opponent reply $o_t$ot​ to the context, transitioning the state to $s_{t+1}$st+1​.

To elicit in-context reasoning without relying on external search algorithms, the authors construct synthetic reasoning traces. They sample $K$K plausible game continuations from a pretrained proposal policy and merge them by their common prefixes into a tree structure rooted at $s_0$s0​. This tree is then serialized in depth-first order to form the reasoning trace $r$r: $r = \langle T \rangle \tilde{\tau}_1 \langle sep \rangle \tilde{\tau}_2 \langle sep \rangle \cdots \tilde{\tau}_m \langle /T \rangle$r=⟨T⟩τ~1​⟨sep⟩τ~2​⟨sep⟩⋯τ~m​⟨/T⟩ where $\tilde{\tau}_i$τ~i​ represents the root-to-leaf paths and $\langle T \rangle$⟨T⟩, $\langle /T \rangle$⟨/T⟩ are delimiting thinking tags. The model is subsequently trained on the concatenated sequence $w = (r, \tau^\star)$w=(r,τ⋆), where $\tau^\star$τ⋆ is the ground-truth solution continuation. During this training, the loss is masked for opponent-move tokens, ensuring the model is optimized only on the reasoning trace and its own moves.

Reinforcement Learning with Verifiable Rewards Building upon the supervised fine-tuned policy, the authors apply reinforcement learning to further optimize the model within the verifiable puzzle environment. They employ a strict binary outcome reward function: $R(\zeta, s_0) = \mathbf{1}[a_1 = a_1^\star, \ldots, a_H = a_H^\star]$R(ζ,s0​)=1[a1​=a1⋆​,…,aH​=aH⋆​] where $\zeta$ζ is the full trajectory comprising the reasoning trace and the executed move sequence, and $(a_1^\star, \ldots, a_H^\star)$(a1⋆​,…,aH⋆​) is the ground-truth solution line. The model receives a reward of 1 only if every executed move perfectly matches the ground truth, meaning a single error yields no reward. The policy is optimized using Group Relative Policy Optimization to refine its strategic planning and move selection capabilities.

Experiment

This study uses chess puzzles as a reasoning testbed to analyze how pretraining and reinforcement learning (RL) interact across model scales from 5M to 1B parameters. The key finding is a joint scaling law in which post-RL performance is predicted by pretraining loss, while the rate of improvement with RL compute depends on the number of pretraining tokens, leading to optimal compute allocations that shift toward RL as total budget grows. Mechanism analysis reveals that RL reshapes the move policy by amplifying correct modes and discovering some tail moves, but it also reinforces wrong modes on harder tasks, which limits pass@k gains; reasoning traces improve mainly through broader search rather than deeper planning. A qualitative transfer to math reasoning shows analogous patterns, suggesting that these scaling relationships extend beyond chess.