One-sentence Summary

Researchers from Xiamen University and Shopee propose Knowledge–Geometry Decoupling (KGD), a streaming recommendation framework that decouples behavioral knowledge, learned via Behavioral Multi-Token Prediction, from task-specific geometry, captured through read-only cross-attention and Anchored Calibration Residual, thereby enabling continual model refresh without task interference, achieving 4–12% gains on eight public benchmarks and a 1.75% GMV lift on Shopee Homepage Search.

Key Contributions

• Behavioral Multi-Token Prediction (BMTP) retains only collaboratively or semantically related future items as supervision, filtering out spurious cross-session transitions and producing cleaner, more transferable behavioral knowledge.
• Knowledge–Geometry Decoupling (KGD) splits the architecture into a refreshable encoder and a task learner that accesses encoder states through read-only cross-attention, while Anchored Calibration Residual (ACR) writes task-specific geometry orthogonal to the pretrained embedding, enabling continual encoder refresh without task-gradient interference.
• KGD improves over strong pretrain–transfer baselines by 4–12% on eight public benchmarks, sustains its advantage across a 90-day streaming production trace, and yields a 1.75% lift in GMV per user and a 1.53% increase in advertising revenue in a live A/B test on Shopee Homepage Search.

Introduction

Industrial recommenders increasingly adopt the generative pretraining paradigm, training transformers on user behavior sequences with next-item objectives and transferring the representations to downstream ranking tasks. However, continual distribution drift from item and audience turnover corrupts supervision: adjacent items in a sequence often belong to unrelated sessions, so treating adjacency as dependency injects noise. Additionally, refreshing the pretrained encoder forces a conflict between behavioral knowledge and task-specific geometry on shared parameters. Prior denoising methods down-weight irrelevant items or rewrite sequences via large language models, but they either still supervise false transitions or are impractically expensive at scale. Transfer approaches such as full fine-tuning overwrite pretrained knowledge, frozen representations forbid geometric adaptation, and adapter-based tuning provides insufficient flexibility. The authors propose Knowledge–Geometry Decoupling (KGD) to address both what to learn and how to transfer. Behavioral Multi-Token Prediction (BMTP) filters future items by collaborative or semantic relevance to avoid noisy supervision, while the task learner accesses encoder states through read-only cross-attention and writes its own geometry via an Anchored Calibration Residual (ACR) that stays orthogonal to the pretrained embedding. This decoupling allows daily encoder refreshes without invalidating the task-specific geometry, making pretrained knowledge and downstream adaptation coexist under continuous drift.

Method

The authors propose Knowledge-Geometry Decoupling (KGD) to resolve the conflict between pretrained behavioral knowledge and task-specific geometry in sequential recommendation. In recommendation streams, user behavior sequences often concatenate separate interest sessions, meaning adjacency does not imply dependency. Furthermore, the gradients of pretraining and downstream task objectives on shared parameters are often negatively correlated or near-orthogonal, making it impossible for a single parameter set to serve both purposes.

As shown in the figure below, these challenges manifest as noisy representations from unrelated session transitions and severe gradient conflicts during fine-tuning.

To address this, KGD separates pretrained knowledge and task-specific geometry into two distinct layers under separate ownership. Refer to the framework diagram for an overview of the architecture.

The framework consists of two main components: a Sequence Encoder pretrained with Behavioral Multi-Token Prediction (BMTP) to build a clean base geometry, and a Task Learner that adapts this geometry via decoupled read-write interfaces.

Behavioral Multi-Token Prediction (BMTP)

Instead of standard next-token prediction, which encodes noisy session boundaries, BMTP supervises the encoder only on transitions that carry persistent knowledge: collaborative and semantic dependencies. For each position $t$t, the method retains the nearest subsequent items whose similarity to $i_t$it​ exceeds a threshold $\tau_a$τa​ on either the collaborative axis (proximity on a co-occurrence graph) or the semantic axis (cosine similarity of text embeddings). The set of qualifying items is defined as:

The pretraining loss supervises these filtered items:

By focusing on collaborative and semantic relations rather than raw adjacency, BMTP produces a clean base geometry free of transient noise, which remains safe to refresh continually as the data stream shifts.

Decoupled Read-Write Ownership

To adapt the pretrained geometry into discriminative structure without destroying it, the authors assign behavior-transition knowledge and task geometry to two separate owners: the pretrained encoder and the task learner. These are coupled through two one-way interfaces that leave the encoder's weights untouched.

Anchored Calibration Residual (ACR): At the embedding level, the task learner forms a task-adapted embedding by superposing a task-owned residual onto a detached copy of the pretrained embedding. The authors observe that real tasks require a task-conditioned, direction-dependent deformation of the embedding space, but this deformation must not cancel the pretrained direction. This is achieved by ensuring the added component is orthogonal to the pretrained embedding $e_i^{\text{pre}}$eipre​:

where $k$k indexes downstream tasks, $\text{sg}(\cdot)$sg(⋅) denotes stop-gradient, and $Z^{(k)}$Z(k) and $B^{(k)}$B(k) are task-owned low-rank factors kept orthogonal to $e_i^{\text{pre}}$eipre​ via regularization. This allows the task to write discriminative directions in the orthogonal complement while the pretrained embedding remains intact.

Read-only Cross-Attention: At the representation level, the task learner needs to project the user's contextualized interest into a task-discriminative space. Since sharing the transformer backbone reintroduces gradient conflicts, KGD decouples the backbones and bridges them with read-only cross-attention. The task transformer reads the encoder's final-layer hidden states $\tilde{H}$H~ through:

where $\rho$ρ is a task-side reader token and $W_Q, W_K, W_V$WQ​,WK​,WV​ are owned by the task transformer. The stop-gradient on keys and values ensures the interface is strictly one-way, so task optimization never pollutes the pretrained knowledge.

Training and Serving Pipeline

The architecture is instantiated as a standard self-attention transformer for the encoder and a transformer with both self- and read-only cross-attention for the task learner. The instantiation of the reader token $\rho$ρ follows the task's demand: a user-side vector for retrieval tasks to produce a single user embedding, and a candidate-side token for ranking tasks to aggregate candidate features.

Under a daily streaming schedule, the training pipeline operates in two steps. First, the encoder is refreshed on the new daily data for a single pass. Second, the encoder-owned parameters are frozen, and the combined encoder-learner graph is run to update only the learner parameters. This design ensures that refreshing the transition backbone never disturbs the task backbone, and because the task geometry is written as a residual anchored relative to the pretrained embedding, a refresh moves the base beneath a residual that stays valid. At serving time, the per-request latency aligns with single-backbone baselines, as the query token still passes through the shared encoder.

Experiment

Evaluation spans public benchmarks, an industrial streaming setting, and an online A/B test to isolate the contributions of BMTP pretraining, transfer mechanisms, and refresh schedules. Public results demonstrate that BMTP provides richer behavioral knowledge that fine-tuning erases, while decoupled ownership via KGD preserves this knowledge and lets the task learner develop its own discriminative geometry without conflict. Industrial experiments show refresh only pays off under decoupled ownership, with frozen transfer degrading over time, and ablation reveals that ownership rather than capacity drives the gains; online deployment delivers significant GMV and revenue uplifts at manageable cost.

Under full fine-tuning (TE&FT), replacing next-token prediction (NTP) with behavior-masked token prediction (BMTP) improves Recall@50 on five of eight public datasets, led by Toys (+13.4%), Office (+11.7%), and Arts (+8.0%). The gain is not universal: Phones shows a 2.7% Recall@50 drop and an 8.2% NDCG@50 decline, while Beauty and Games see minor changes. BMTP achieves the highest overall Recall@50 on seven datasets, with NTP outperforming it only on Phones. BMTP delivers the largest Recall@50 improvements over NTP on Toys (+13.4%), Office (+11.7%), and Arts (+8.0%) under TE&FT. On Phones, BMTP reduces Recall@50 by 2.7% and NDCG@50 by 8.2% compared to NTP, the only dataset where BMTP hurts both metrics. Across all shown TE&FT rows, BMTP attains the best Recall@50 on seven of eight datasets, with NTP slightly ahead only on Phones (0.0808 vs. 0.0786).

Decoupled ownership of knowledge and geometry yields the best performance on the 28-day industrial stream, with KGD leading all metrics. Continual refresh only helps when parameters are decoupled; under shared ownership, refreshing degrades or does not help, and removing decoupling drops performance below a randomly initialized model. Complementary pretraining filters and a separate task backbone are essential for stable and refreshable long-term gains. KGD achieves the top click and order AUC/GAUC on the 28-day stream. Daily refresh improves results only with decoupled ownership; with shared parameters, TA&FT's click AUC drops from 0.7852 to 0.7837. Removing decoupling interfaces from KGD collapses performance to 0.7785, below the scratch model. BMTP's graph and semantic filters are complementary: removing the graph filter hurts head users, while removing the semantic filter hurts tail users. ACR is essential; reading pretrained embeddings directly without it reduces performance across all metrics. Even with ACR, a shared backbone underperforms a separate task backbone because knowledge and geometry compete on the same parameters. Without pretraining, continual methods offer little benefit: buffer replay falls to 0.7732, below scratch. LoRA underperforms due to insufficient capacity to reshape geometry. Frozen transfer looks competitive in a short window but degrades over 90 days, showing that refreshability must be judged on long trajectories.

Ablating KGD components reveals that semantic and collaborative pretraining filters complement each other for tail and head users, respectively, while removing the adaptive calibration module or forcing a shared backbone consistently degrades performance, even when parameter capacity is matched. The gains stem from the decoupled architecture that keeps knowledge in the encoder and task-specific geometry in a separate learner. On industrial data, dropping the semantic filter costs 1% AUC on tail users (bottom 30% by activity), whereas dropping the collaborative filter costs 0.5% AUC on head users. Removing the adaptive calibration module (ACR) hurts across all benchmarks and industrial metrics, showing that reading pretrained embeddings directly cannot provide the discriminative geometry needed for the task. Placing ACR on a shared backbone—whether frozen or fully fine-tuned—underperforms full KGD, and aligning the shared backbone’s parameter count to match KGD still trails and even regresses on sparse public datasets.

In public dataset experiments, behavior-masked token prediction (BMTP) generally improves recall over next-token prediction under full fine-tuning, though it can underperform on certain datasets like Phones. On an industrial recommendation stream, a decoupled architecture (KGD) that separately owns pretrained knowledge and task-specific geometry achieves the best long-term performance, with continual refresh only beneficial when parameters are decoupled and with complementary semantic and collaborative filters aiding tail and head users respectively. Ablation studies confirm that removing the adaptive calibration module or forcing knowledge and geometry to share a backbone consistently degrades results, underscoring the necessity of decoupled ownership for stable and refreshable gains.