One-sentence Summary

Addressing training-inference mismatch in LLM reinforcement learning, researchers from Tianjin University and Alibaba introduce Monotonic Inference Policy Update (MIPU), a two-step framework that implements the Monotonic Inference Policy Improvement (MIPI) objective by constructing sampler-referenced candidate updates and selectively accepting synchronized candidates using an inference-side gap proxy, thereby enhancing reasoning performance and training stability.

Key Contributions

• The paper identifies the objective misalignment that updating the training policy does not guarantee improvement of the inference policy used at deployment, and formulates Monotonic Inference Policy Improvement (MIPI) as a new optimization objective to address this.
• It introduces Monotonic Inference Policy Update (MIPU), a two-step framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy.
• Experiments on two model scales under high mismatch (FP8-quantized rollout) show that MIPU improves average reasoning performance and training stability, and further analysis confirms the rationality of the candidate acceptance step.

Introduction

Reinforcement learning (RL) has become a key paradigm for post-training large language models, especially for reasoning tasks. Modern LLM RL pipelines decouple rollout generation on an inference engine from gradient computation on a training engine, which creates a training-inference mismatch: even with synchronized parameters, the training policy and the inference policy can assign different probabilities to the same trajectories due to differences in precision, decoding, and serving backends. Existing remedies address this mismatch by stabilizing the training-side optimization, but they offer no guarantee that an update that improves the training policy also yields a better inference policy, the one actually used for deployment. The authors redefine the problem from an objective-level perspective, proposing the Monotonic Inference Policy Improvement (MIPI) principle, which requires policies to be optimized for the inference engine’s performance. They then introduce a two-step framework, Monotonic Inference Policy Update (MIPU), that decouples policy improvement into a sampler-referenced update and an inference-gap-aware acceptance step, ensuring monotonic improvement for the inference policy. In experiments with FP8-quantized rollout, a high-mismatch setting, MIPU achieves stronger average performance and more stable training than existing approaches.

Dataset

The authors rely on a held-out validation set, denoted D_val, to decide whether to accept or roll back candidate training policies. The dataset description is sparse in the provided text, but the following points capture what is stated and implied about its construction and usage.

• Composition and sources

• The dataset consists of prompts x drawn from a validation split. No details about the original source (e.g., open‑source corpora, task‑specific collections) are included in the excerpt.

• The primary role of D_val is to produce completions y_i sampled from the current inference policy μ_{k+1}, enabling an online estimate of the post‑update inference gap.

• Key details

• A single subset, D_val, is mentioned. Its total size, number of prompts per evaluation step, and filtering rules (if any) are not specified.

• For each prompt, the inference engine generates multiple completions; the batch size and sampling temperature are not detailed here.

• The completions serve as the basis for group‑relative advantage computation under μ_{k+1}.

• How the data is used

• The validation set supports the acceptance test in Step 2: Inference‑Gap‑Aware Update Acceptance.

• For each prompt, the completions from μ_{k+1} yield group‑relative advantages Â_i^{μ_{k+1}}. These advantages are multiplied by importance weights ρ_i (Equation 9) to approximate the reverse‑form performance difference (Equation 8).

• The expectation over D_val produces the proxy \widehat{T}_{post}. A negative value above a tolerance -c signals that the training‑side improvement may not hold under the inference policy, triggering a rollback of both the trainer and the inference engine.

• The data is not used for training but exclusively as an online validation signal after each candidate update is synchronized to the inference engine.

• Processing details

• No cropping or deduplication is described. The only custom processing is the computation of length‑normalized importance weights ρ_i, which stabilizes the reverse‑form expectation by avoiding the high variance of full‑sequence importance sampling.

• Advantage calculation follows the same group‑relative scheme used during training (GRPO‑style), but applied to validation‑side completions.

• The procedure assumes that D_val is a representative sample of prompts where the inference gap can be reliably measured; however, no specific schema or metadata beyond the raw prompts is mentioned.

Method

The authors address the training-inference mismatch inherent in modern large language model reinforcement learning pipelines. Due to implementation differences in precision, decoding, or serving backends between the training and inference engines, the training policy $\pi$π and the inference policy $\mu$μ often assign different probabilities to the same trajectories. Consequently, optimizing the training-side objective does not guarantee an improvement in the deployed inference policy.

To resolve this objective misalignment, the authors propose Monotonic Inference Policy Improvement (MIPI), a principle that redefines the optimization target to align directly with the inference-policy transition. They formally decompose the inference-policy improvement into three distinct components: the post-update inference gap, the training-side update, and the pre-update inference gap. This decomposition is expressed as:

$J(\mu_{k+1}) - J(\mu_k) = \underbrace{J(\mu_{k+1}) - J(\pi_{k+1})}_{\text{post-update inference gap}} + \underbrace{J(\pi_{k+1}) - J(\pi_k)}_{\text{training-side update}} + \underbrace{J(\pi_k) - J(\mu_k)}_{\text{pre-update inference gap}}$J(μk+1​)−J(μk​)=post-update inference gapJ(μk+1​)−J(πk+1​)​​+training-side updateJ(πk+1​)−J(πk​)​​+pre-update inference gapJ(πk​)−J(μk​)​​

The authors realize the MIPI principle through a two-step mechanism called Monotonic Inference Policy Update (MIPU), which separates candidate construction from post-synchronization verification.

In the first step, the authors perform a sampler-referenced policy update to optimize the training-side update and pre-update inference gap terms. Standard algorithms like GRPO clip the probability ratio relative to the old training policy, which misaligns with the sampler generating the rollouts. To correct this, the authors factorize the full trainer-to-sampler ratio into a pre-update mismatch weight and a current update ratio. To prevent high variance caused by large pre-update discrepancies, they apply a truncated importance weight to the mismatch correction term while applying PPO-style clipping exclusively to the current update ratio. This yields a candidate training policy that is better aligned with the inference sampler.

In the second step, the authors implement inference-gap-aware update acceptance to validate the post-update inference gap. After synchronizing the candidate training policy to the inference engine, they evaluate whether the inference engine successfully realizes the proposed gains. Since the exact advantage under the candidate policy is unavailable, the authors derive a stable validation-based proxy using a reverse identity and length-normalized importance weights. This proxy serves as a risk signal. If the estimated gap falls below a defined tolerance threshold, indicating that the inference policy underperforms its training-side counterpart, the update is rejected and the system rolls back to the previous checkpoint. If the proxy passes the acceptance criterion, the candidate is accepted as the new policy.

Experiment

MIPU is evaluated under FP8-quantized rollout, a high training-inference mismatch setting, on Qwen3 models of two scales using mathematical reasoning benchmarks. It achieves both better final performance and sustained training stability, while baseline methods degrade after early peaks. Ablation studies show that Step 1 improves the candidate update direction and Step 2 conditions synchronization on a post-update inference-gap signal, addressing complementary failure modes; random rollback fails to prevent collapse, demonstrating that the signal-driven acceptance, not mere update rejection, is essential.

Under FP8-quantized rollout, MIPU achieves the highest average accuracy on both Qwen3-4B (66.71%) and Qwen3-1.7B (53.97%), outperforming baselines across five mathematical benchmarks. It is the only method that remains stable throughout continued training, while other approaches exhibit collapse or sharp degradation after reaching competitive intermediate scores. Gains are especially pronounced on AMC23 and Minerva for the 4B model, and on MATH-500 and OlympiadBench for the 1.7B model. MIPU is the only method that avoids training collapse under extended FP8-quantized rollout, whereas all baselines (Baseline, MIS, LR-decay) eventually degrade. MIPU sets the best average accuracy on both model sizes, with standout improvements on AMC23 and Minerva for Qwen3-4B, and on MATH-500 and OlympiadBench for Qwen3-1.7B.

Under FP8-quantized rollout, combining sampler-referenced candidate construction (Step 1) with inference-gap-aware acceptance (Step 2) yields the highest average accuracy and top scores on most benchmarks. Step 1 alone improves some tasks but still accepts all synchronized candidates, while Step 2 alone prevents collapse but cannot improve candidate quality and can reduce performance. The full method demonstrates that corrected candidate updates and selective acceptance are complementary. Adding Step 1 alone raises average accuracy over the baseline but falls short of the full method because every synchronized candidate is accepted without filtering. Step 2 alone lowers average accuracy below the baseline and substantially degrades AMC23, indicating that gap-aware rejection without better candidate construction undermines performance. The full method attains the highest average and the best scores on MATH 500, AIME 24, Minerva, and AMC23, with only a minor drop on Olympiad compared to Step 1 alone. Combining both steps improves AIME 24 and Minerva relative to either single-step variant, showing that corrected candidate updates and inference-gap filtering provide complementary benefits.

The evaluation assesses MIPU under FP8-quantized rollout on mathematical reasoning tasks, showing it consistently achieves the highest accuracy across multiple benchmarks and model sizes, while remaining stable during extended training where other methods collapse. An ablation study confirms that sampler-referenced candidate construction and inference-gap-aware acceptance are complementary: using either step alone fails to match the full method, with construction alone lacking quality filtering and acceptance alone degrading performance. Overall, MIPU's dual strategy produces robust improvements without the degradation observed in baselines.