One-sentence Summary

The authors propose Parallel-RL, a multi-task training paradigm for large language models that decouples tasks by exploiting the insight that reinforcement learning yields sparse, near-orthogonal gradient updates due to variance-limited interference, in contrast to the norm-limited conflicts of supervised fine-tuning, thereby improving efficiency and flexibility.

Key Contributions

• Multi-stage supervised fine-tuning (SFT) causes a severe performance decline across tasks (averaging 23.1% below the base model), whereas multi-stage reinforcement learning (RL) achieves stable, cumulative improvements (24.9% average gain), demonstrating a coexistence property of RL in multitask settings.
• Gradient interference analysis reveals distinct mechanisms: SFT interference is norm-limited and scales with absolute gradient magnitudes, while RL interference is variance-limited, bounded by low gradient variance from advantage normalization and on-policy optimization, producing sparse, approximately orthogonal updates that avoid task conflicts.
• Parallel-RL is proposed as a paradigm that decouples multi-task training by independently training tasks based on the near-orthogonal RL updates. Experiments show it matches single-task performance, improves training efficiency, and ablation studies confirm effective decoupling of task-specific capabilities.

Introduction

Enhancing reasoning in large language models via supervised fine-tuning (SFT) and reinforcement learning (RL) is central to the pursuit of robust multi-task capabilities, yet prior work has mostly studied these methods in single-task settings, leaving their multi-task behavior underexplored. Existing practice often uses mixed-data training for SFT to avoid forgetting, while RL frequently adopts a multi-stage, per-task approach; the reasons for this divergence and its impact were not systematically understood. The authors address this gap with a combined empirical and theoretical analysis, revealing that multi-stage SFT suffers severe performance collapse due to conflicting parameter updates, whereas RL yields nearly orthogonal updates that coexist and enable stable cumulative gains across tasks. Building on the finding that RL’s gradient interference is variance-limited rather than norm-limited, they introduce Parallel-RL, a paradigm that trains tasks independently and merges updates, achieving strong multi-task performance with improved efficiency.

Method

The authors begin by establishing a theoretical foundation for their proposed method, analyzing the distinct parameter update behaviors of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). They demonstrate that RL updates exhibit sparsity, small magnitude, and near-orthogonality across different tasks. This orthogonality arises from two key mechanisms in RL algorithms like GRPO. First, the advantage function acts as a filter that removes the dense mean gradient direction, leaving only intra-group residuals. Second, the on-policy nature of RL ensures these residuals remain minimal. Consequently, gradient interference in SFT is norm-limited and prone to dense conflicts, whereas RL interference is strictly variance-limited by the diversity of intra-group rollouts.

To empirically validate this geometric property, the authors analyze the distribution of score functions sampled during training.

As illustrated in the figure above, RL score functions form distinct, separable clusters for different tasks, indicating that RL updates occupy nearly orthogonal subspaces. In contrast, SFT distributions exhibit heavy overlap, confirming that SFT suffers from dense, conflicting updates while RL naturally decouples task optimization.

Motivated by this insight, the authors propose Parallel-RL, a decoupled post-training paradigm that leverages the orthogonal nature of RL updates. In this framework, a set of diverse reasoning tasks is optimized independently. Formally, given a set of tasks $\mathcal{T} = \{T_1, \dots, T_N\}$T={T1​,…,TN​}, the authors launch $N$N parallel RL training processes. Each process $i$i produces a task-specific parameter update $\Delta W_i$ΔWi​. The final model parameters $W_{final}$Wfinal​ are then obtained by merging these independent updates:

$$W_{final} = W_{base} + \mathcal{M}(\Delta W_1, \dots, \Delta W_N)$$Wfinal​=Wbase​+M(ΔW1​,…,ΔWN​)

where $\mathcal{M}$M represents a merging function, such as linear averaging or Singular Value Decomposition. This approach allows for the harmonious coexistence of different task updates within a single model without the interference typically observed in multi-task SFT.

Experiment

This study compares SFT and RL (GRPO) on a Qwen model across math, science, code, and logic tasks. Multi-stage SFT causes catastrophic forgetting, while RL supports stable growth because its updates are sparse, minimal, and nearly orthogonal across tasks, thus avoiding interference. This natural decoupling enables Parallel-RL, where task-specific RL updates can be merged with negligible conflict and even surpass single-task performance after light adaptation.

When training on multiple tasks, mixing data during supervised fine-tuning yields small improvements, but training tasks one after another causes severe performance collapse, especially in logic. In contrast, reinforcement learning benefits from both mixed-data and sequential strategies, with the sequential multi-stage approach delivering the largest gains, particularly in science and logic. Multi-stage supervised fine-tuning led to catastrophic forgetting, with logic accuracy plummeting far below the base model, while mixed-data fine-tuning avoided this collapse. Reinforcement learning proved robust across strategies: multi-stage training achieved the highest accuracy on every task, with especially large improvements in science and logic. Mixed-data reinforcement learning improved all tasks but was outperformed by its multi-stage counterpart, which saw cumulative growth rather than interference.

After single-task training, supervised fine-tuning (SFT) improves the target task but causes clear degradation on other tasks, whereas reinforcement learning (RL) not only boosts the target task more but also yields small, consistent gains across all other evaluated tasks. SFT on a single task caused nontarget accuracy drops as large as 21 percentage points (logic after science SFT). RL on math improved math and simultaneously lifted science by over 7 points, showing positive cross-task transfer.

Supervised fine-tuning (SFT) yields score function vectors with large magnitude and moderate inter-task cosine similarity, indicating dense and overlapping updates. In contrast, reinforcement learning (RL) produces score functions with much smaller magnitude, a bounded residual term, and near-zero cosine similarity, showing that RL updates are nearly orthogonal and naturally decoupled across tasks. The score function L2 norm for SFT is approximately 7.1, while for RL it drops to roughly 0.1, reflecting substantially smaller update magnitudes. Inter-task cosine similarity for RL is around 0.001 compared to about 0.1 for SFT, confirming that RL gradients occupy almost orthogonal subspaces.

Parallel reinforcement learning (RL) merges task-specific updates with far less conflict than supervised fine-tuning (SFT): naive sum merging retains 95% of single-task RL performance versus only 66% for SFT. Sparsification strategies like TIES and SVD further raise retention to 96–98%, and a light adaptation step after merging pushes performance beyond single-task models (102.8% retention) while achieving a 9.4% average gain over the base model. Naive Parallel-RL (sum) retains 95% of single-task RL performance, whereas Parallel-SFT (sum) retains only 66%, showing RL updates are more compatible. Sparsifying RL updates with TIES or SVD increases retention to 96–98%, reducing task interference. Adapted Parallel-RL with a small post-merge fine-tuning step surpasses single-task models (102.8% retention) and delivers a 9.4% improvement over the base model.

Removing a task-specific update from the Naive Parallel-RL merged model causes a large accuracy drop on that task (averaging -7.1%), while performance on the remaining tasks stays robust, even showing a small average improvement of 0.6%. This demonstrates that Parallel-RL yields strongly decoupled task representations, allowing selective module exclusion without harming unrelated capabilities. Excluding the math update reduces MATH500 accuracy by 3.6 points (from 86.4 to 82.8), while the other three benchmarks collectively improve by 0.9 points on average. Excluding the science update causes a 10.5-point drop on MMLU (48.0 to 37.5), yet the other tasks' scores are nearly unchanged, shifting by only -0.1 points on average.

The experiments assess supervised fine-tuning (SFT) and reinforcement learning (RL) on math, science, and logic tasks under single-task, sequential multi-stage, and parallel merging strategies. RL updates are naturally decoupled and near-orthogonal across tasks, preventing catastrophic forgetting and enabling positive cross-task transfer, while SFT produces dense, interfering updates that cause severe performance drops in untargeted skills. Parallel RL merging preserves up to 95% of single-task performance with simple summation, and sparsification plus light adaptation can yield models that surpass single-task counterparts, with removal of individual task modules demonstrating perfectly isolated task representations.