One-sentence Summary

The authors provide evidence that LLMs exhibit the Superposition Linearity Hypothesis, in which linearly combined inputs produce superposed next-token distributions, show that this linearity is intrinsic to Transformer architectures yet diminishes with pretraining and can be substantially restored by lightweight fine-tuning, and introduce a guided decoding procedure that disentangles superposed outputs to generate two coherent continuations from a single forward pass.

Key Contributions

• The paper demonstrates that decoder-only Transformers exhibit superposition linearity, where averaging embeddings from two text streams yields a next-token distribution close to the average of the individual next-token distributions and retains high probability mass on tokens favored by each independent stream.
• This work shows that superposition linearity is an intrinsic architectural property that degrades during pretraining, and that lightweight fine-tuning with less than 0.025% of the original pretraining data substantially restores it, reducing divergence between mixed and averaged next-token distributions.
• A guided decoding procedure is introduced to disentangle superposed outputs, enabling recovery of two coherent continuations from a single forward pass, theoretically offering a 2× increase in inference throughput and halving the KV-cache footprint per active stream.

Introduction

The authors investigate decoder-only Transformer language models, which are built from highly non-linear attention and MLP components and are typically treated as a single coherent semantic stream. As a result, processing multiple independent inputs requires separate forward passes, sequential processing, or architectural modifications to avoid destructive interference. Although recent work shows approximate affine structure between consecutive layers in the residual stream, it is unclear whether this linearity extends to end-to-end input-output behavior. The authors formalize a Superposition Linearity Hypothesis and show that when token embeddings from two documents are averaged and passed through a pretrained LLM, the model retains substantial probability mass for both streams, with both ground-truth next tokens frequently appearing in the top 10. They find this behavior is strongest at initialization and degrades during pretraining, indicating it is an architectural property rather than a learned capability, and they show it can be substantially restored with lightweight fine-tuning using less than 0.025% of the original pretraining data. Their main contribution includes a decoding procedure that disentangles the mixed hidden state, enabling two distinct continuations to be recovered from a single forward pass.

Method

The authors leverage a targeted optimization strategy to enhance the intrinsic linear superposition capabilities of pre-trained Transformer models. While standard pre-training does not explicitly incentivize this behavior, a lightweight fine-tuning phase is introduced to align the model weights to explicitly support the parallel processing of superposed inputs.

To achieve this, the authors employ a self-distillation framework designed to minimize the discrepancy between the model output on a mixed input and the mixture of its independent outputs. The framework initializes a student model with pre-trained weights and utilizes a frozen copy of the same architecture as the teacher model. For a given pair of distinct text sequences, the target probability distribution is defined as the arithmetic mean of the teacher model's independent predictions:

The student model processes the element-wise average of the input embeddings from both sequences, formulated as $z = \frac{1}{2} (E(x^{(A)}) + E(x^{(B)}))$z=21​(E(x(A))+E(x(B))). The optimization objective is to minimize the Kullback-Leibler divergence between the student output and the target mixture:

This fine-tuning procedure substantially reduces the divergence between the predicted and target distributions. The intervention successfully reverses the interference observed in the base model, preserving both ground-truth streams with high fidelity at the output layer. Notably, the probability of the true next token appearing in the top ranks rises significantly after optimization.

As shown in the figure below, the cumulative rank distribution demonstrates significant improvements across different model architectures following this lightweight fine-tuning phase.

The plot illustrates that the chance of a single-stream predicted token remaining in the top ranks of the mixed distribution increases markedly. Furthermore, the optimization effectively rehabilitates the parallel processing of complex semantic content, recovering signals for hard content tokens where the base model previously collapsed.

Beyond fitting the mixed distribution, the authors address the challenge of decoding the two streams separately from a single mixed forward pass. Direct sampling from the mixed distribution is obstructed by a geometric mean effect, where the resulting probabilities scale with the geometric mean of the independent distributions:

This creates an intrinsic decoding challenge because any token highly probable in one stream but unlikely in the other is heavily penalized. To disentangle the mixed hidden state back into its constituent streams, the authors propose a Joint Contrastive Decoding mechanism. This method utilizes a small auxiliary model to provide per-stream guidance during the process. The disentangled logits are computed by adjusting the large model's mixed logits with the auxiliary model's independent predictions:

The symmetric formulation is applied for the second stream. The scalar coefficients are initialized to one and trained jointly with the backbone on a symmetric per-stream cross-entropy loss, effectively mitigating the geometric mean obstruction and enabling practical parallel inference.

Experiment

The experiments assess whether standard Transformers can process superposed inputs by averaging embeddings from two streams without architectural changes. Rank and distributional analyses show that pretrained models preserve both token-level and distributional signal well above chance, with this linearity strongest early in training and deep layers remaining near-linear. Attention patching indicates that structural attention shape and frequency priors help predictable positions, but embedding mixing retains more semantic content on hard positions, and self-distillation fine-tuning further restores this parallel processing at some cost to single-stream quality. Finally, decoding two streams from a mixed pass is limited by a geometric-mean interference effect, though joint contrastive decoding offers a proof-of-concept recovery.

At context length 32, all evaluated models approximate the target arithmetic-mean mixture better than a random baseline, with normalized divergence ratios below 1 for KL, JS, and Wasserstein metrics. The smallest model shows the strongest relative approximation across all metrics, while the largest model has the highest absolute divergences. KL ratios are consistently the lowest, and Wasserstein ratios are consistently the closest to 1. The smallest Pythia model achieves the best KL, JS, and Wasserstein ratios, indicating the closest relative match to the target mixture. Wasserstein ratios are higher than KL and JS ratios across models, suggesting weaker improvement over the random baseline on the embedding-aware metric.

Predictable positions, roughly two-thirds of tokens, keep the vanilla top-1 token near the top when attention shape is preserved under embedding mixing or donor patching. Content positions are much more fragile, with large rank increases under those base-model perturbations, while permutation patching destroys attention structure and collapses top-1 recovery across both token types. Fine-tuning restores much stronger exact agreement and low median rank on hard content positions. Under base embedding mixing and donor patching, predictable positions maintain low median ranks and moderate exact agreement, while content positions degrade substantially. Permutation patching collapses top-1 recovery across both predictable and content positions, showing that preserving attention shape is critical. Fine-tuning improves content-position recovery dramatically, reducing median rank from hundreds to single digits and raising exact agreement from single digits to over one-fifth.

In single-stream perturbation experiments, a donor patch preserves top-rank target placement far more often than permuted attention, with top-10 recall falling from a majority to about one in ten and the median top-1 rank worsening from single digits to thousands. Permutation also increases KL-based divergence from vanilla outputs, indicating that attention structure is important beyond the frequency prior alone. Permuting attention collapses top-10 target recall to roughly a tenth and pushes the median top-1 rank from single digits to several thousand. KL-based divergence measures are larger under permutation than with a donor patch, showing a stronger departure from vanilla behavior.

Embedding mixing retains more signal for hard content prediction than donor patching, with higher LAMBADA accuracy and a much lower LAMBADA target median rank. Donor patching yields a better A1 top-1 median rank but much weaker target-token recovery. Prior LSTM and N-gram baselines are near zero, placing both methods in a difficult low-accuracy regime. Mixing achieves substantially higher LAMBADA accuracy than donor patching. Mixing yields a much better LAMBADA target median rank, indicating stronger recovery of hard target content. Donor patching achieves a better A1 top-1 median rank than mixing. LSTM and N-gram baselines are near zero, so the task remains difficult even with mixing.

Joint Contrastive decoding improves mean LAMBADA accuracy on superposed forward passes compared with raw pretrained mixing across the Qwen, Llama, and Pythia entries shown, while also reducing Jaccard token-overlap on FineWeb generations, indicating better stream separation. The mixed-stream accuracy still remains below the small model's single-stream baseline in every reported configuration. These results support the method as a proof of concept that the superposed signal is exploitable but not fully decoded. Joint Contrastive decoding raises LAMBADA mixed mean accuracy over pretrained mixing across the reported model pairings, with the largest gain on Llama-3.2-3B guided by Llama-3.2-1B. Jaccard token-overlap on FineWeb generations decreases under Joint Contrastive decoding across the shown configurations, indicating cleaner separation of the two streams. Even with Joint Contrastive decoding, mixed-stream accuracy trails the small model's single-stream baseline, leaving a residual gap consistent with the geometric-mean obstruction.

Experiments assess how well models approximate a target arithmetic-mean mixture, finding that all evaluated models beat a random baseline and that smaller models show the strongest relative fit. Attention perturbation analyses show that preserving attention shape is critical: predictable positions remain relatively stable under embedding mixing or donor patching, while content positions degrade sharply and permutation collapses recovery, though fine-tuning substantially restores content-position performance. Embedding mixing retains more signal for hard content prediction than donor patching, and joint contrastive decoding improves mixed-stream accuracy and stream separation over raw pretrained mixing while still leaving a gap below single-stream performance.