il y a 4 jours

Yifan Zhang Steve Ta Jasper Zhang Jichen Feng Shuzhen Li Yongxin Zhang Yifeng Liu Huizhuo Yuan Mengdi Wang Quanquan Gu

Table des matières

Résumé

Les mémoires récurrentes à poids rapides et les modèles d'espace d'état sélectifs compressent un contexte en expansion en un état récurrent de taille fixe, faisant de la transition d'état une règle d'apprentissage en ligne. Nous étudions cette règle sous une sémantique autorégressive de lecture-après-écriture. Pour l'objectif de prédiction de préfixe considéré ici, l'exemple de mémoire rapide locale révélé à l'étape t est la paire alignée sur le préfixe $( \mathbf { x } _ { t } , \mathbf { y } _ { t } ) \ = \ ( \phi ( \mathbf { k } _ { t - 1 } ) , \mathbf { v } _ { t } )$ . L'association courante à la même étape (ϕ(kt), vt) reste causale, mais optimise un objectif interne différent. Nous dérivons des mises à jour normalisées de premier ordre pour les objectifs de régression par erreur quadratique et de produit scalaire négatif. La famille de régression comprend Falcon-1 (une mise à jour NLMS scalaire), Falcon-2 (son extension par colonne) et Falcon-3 (une mise à jour par mini-lots à fenêtre glissante) ; Falcon-1A/Falcon-2A/Falcon-3A sont les variantes correspondantes de produit scalaire. Nous fournissons des formes récurrentes, parallèles masquées et parallèles par blocs, ainsi qu'une renormalisation à décroissance positive numériquement stable. Les variantes représentatives restent compétitives en modélisation du langage et améliorent l'extrapolation de longueur sur l'addition de chiffres variables. Ce cadre sépare l'alignement temporel, la plasticité, l'oubli et la répétition bornée dans les modèles séquentiels récurrents.

One-sentence Summary

Researchers from Bytedance Seed, Princeton, Tsinghua, UCLA, and Hyperbolic Labs propose a fast-weight attention framework for continual learning that derives normalized first-order updates for squared-error regression and negative inner-product objectives, introducing $\text{Falcon-1/2/3}$ variants (with inner-product counterparts $\text{Falcon-1A/2A/3A}$ ) that remain competitive in language modeling and improve length extrapolation on variable-digit addition.

Key Contributions

Introduces a unified family of normalized fast-weight updates, Falcon-1/2/3 for squared-error regression and Falcon-1A/2A/3A for negative inner-product objectives, derived under read-after-write autoregressive semantics with prefix-aligned causal pairs (φ(kₜ₋₁), vₜ).
Provides recurrent, masked-parallel, and chunk-parallel implementations with numerically stable positive-decay renormalization, separating temporal alignment, plasticity, forgetting, and bounded rehearsal while remaining compatible with SSD-style chunk-parallel training.
Shows that representative scalar regression and scalar/sliding inner-product variants remain competitive in language modeling, and the best inner-product variants improve length extrapolation on variable-digit addition.

Introduction

Transformers dominate modern language modeling because self-attention captures global dependencies, but their quadratic scaling in sequence length creates major bottlenecks in both compute and memory, especially for long contexts. Recurrent alternatives like SSMs and fast-weight models compress memory into a fixed state, yet their state-update rules are often presented at the architectural level, leaving the underlying learning objective implicit. This ambiguity causes a temporal alignment mismatch: many recurrences bind the same-step pair (key, value), whereas autoregressive prediction requires pairing the newly revealed target with the prefix feature available at prediction time, a distinction that affects how fast memory is trained.

The authors address this by recasting state-based sequence modeling as autoregressive next-latent prediction, making the fast-memory training objective explicit. They derive a unified family of normalized first-order updates, Falcon-1, Falcon-2, and Falcon-3, along with inner-product counterparts, Falcon-1A, Falcon-2A, and Falcon-3A, which separate temporal alignment, plasticity, forgetting, and bounded rehearsal. These updates remain compatible with chunk-parallel training, and the authors provide implementations and empirical evaluations showing competitive language modeling performance and improved arithmetic length extrapolation.

Method

2 Background

2.2 Linear Attention

Linear attention circumvents the $O(N^2)$ complexity of standard attention by replacing the softmax with a kernel feature map $\phi(\cdot): \mathbb{R}^d \to \mathbb{R}^m$ such that $\kappa(\mathbf{q}_t, \mathbf{k}_j) = \phi(\mathbf{q}_t)^\top \phi(\mathbf{k}_j)$ . Exploiting the associativity of matrix multiplication, the output $\mathbf{o}_t \in \mathbb{R}^{d_v}$ for the t-th token is:

\mathbf{o}_t = \frac{\sum_{j=1}^t \phi(\mathbf{q}_t)^\top \phi(\mathbf{k}_j) \mathbf{v}_j}{\sum_{j=1}^t \phi(\mathbf{q}_t)^\top \phi(\mathbf{k}_j)} = \frac{\left(\sum_{j=1}^t \phi(\mathbf{k}_j) \mathbf{v}_j^\top\right)^\top \phi(\mathbf{q}_t)}{\phi(\mathbf{q}_t)^\top \sum_{j=1}^t \phi(\mathbf{k}_j)},

This formulation allows the context to be compressed into a recurrent matrix state $\mathbf{S}_t \in \mathbb{R}^{m \times d_v}$ and a normalizer $\mathbf{z}_t \in \mathbb{R}^m$ :

\mathbf{o}_t = \frac{\mathbf{S}_t^\top \phi(\mathbf{q}_t)}{\mathbf{z}_t^\top \phi(\mathbf{q}_t) + \varepsilon_{\mathrm{attn}}}, \quad \mathbf{S}_t = \mathbf{S}_{t-1} + \phi(\mathbf{k}_t) \mathbf{v}_t^\top, \quad \mathbf{z}_t = \mathbf{z}_{t-1} + \phi(\mathbf{k}_t).

For a fresh sequence, Eq. (2.2) is exactly equivalent to Eq. (2.1) when $\mathbf{S}_0 = \mathbf{0}$ , $\mathbf{z}_0 = \mathbf{0}$ , $\varepsilon_{\mathrm{attn}} = 0$ , and the denominator is nonzero. With $\varepsilon_{\mathrm{attn}} > 0$ , it is the usual stabilized positive-feature variant. The normalized form is attention-like only when the read normalizer is nonnegative; signed-feature caveats are deferred to Appendix C.2.

Causality and indexing. Eq. (2.2) follows the standard Transformer convention: at position $t$ the paper read from the updated state $(\mathbf{S}_t, \mathbf{z}_t)$ , and the resulting representation is used to predict token $t+1$ . The authors' next-latent alignment shifts the write stream by one: after observing $\mathbf{v}_t$ , it is written under the previous write feature $\phi(\mathbf{k}_{t-1})$ , or equivalently, $(\phi(\mathbf{k}_i), \mathbf{v}_{i+1})$ under standard indexing with $i = t-1$ . This yields the read-after-write recurrence:

\mathbf{o}_t = \frac{\mathbf{S}_t^\top \phi(\mathbf{q}_t)}{\mathbf{z}_t^\top \phi(\mathbf{q}_t) + \varepsilon_{\mathrm{attn}}}, \quad \mathbf{S}_t = \mathbf{S}_{t-1} + \phi(\mathbf{k}_{t-1}) \mathbf{v}_t^\top, \quad \mathbf{z}_t = \mathbf{z}_{t-1} + \phi(\mathbf{k}_{t-1}),

where $\varepsilon_{\mathrm{attn}} \geq 0$ is a small stabilizer. Defining the shifted write-feature stream:

\widetilde{\mathbf{x}}_1 := \mathbf{0}, \qquad \widetilde{\mathbf{x}}_t := \phi(\mathbf{k}_{t-1}) \quad \text{for } t \geq 2,

Eq. (2.3) is exactly Eq. (2.2) with the standard write-feature stream $\{\phi(\mathbf{k}_t)\}_{t=1}^T$ replaced by $\{\widetilde{\mathbf{x}}_t\}_{t=1}^T$ when $\varepsilon_{\mathrm{attn}} = 0$ ; for $\varepsilon_{\mathrm{attn}} > 0$ , it is the corresponding stabilized shifted variant. The boundary condition is therefore imposed in feature space rather than raw-key space.

Normalized vs. unnormalized linear attention. The denominator in Eq. (2.1) uses the normalizer state $\mathbf{z}_t$ to rescale the readout. Many SSM/SSD-style architectures instead drop the denominator (and $\mathbf{z}_t$ ) and use an unnormalized inner-product read:

\mathbf{o}_t = \mathbf{S}_t^\top \phi(\mathbf{q}_t), \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \phi(\mathbf{k}_{t-1}) \mathbf{v}_t^\top,

with the unshifted convention recovered by replacing $\mathbf{k}_{t-1}$ with $\mathbf{k}_t$ . In this denominator-free form, state magnitude and the effective memory timescale are controlled by explicit decay (e.g., $\lambda_t > 0$ ) and/or gain control. Section 4.3 shows that the numerator-state update in Eq. (2.4) is exactly gradient descent on an inner-product objective. The auxiliary normalizer $\mathbf{z}_t$ in the normalized variants is a separate bookkeeping state for the read denominator; it is not itself obtained from that objective.

Accordingly, Eq. (2.4) is the gradient-descent update for the inner-product objective in Section 4.3. When $\lambda_t = 0$ , the write is purely additive rank-one Hebbian learning; when $\lambda_t > 0$ , it is additive plus scalar shrinkage. As noted in Section 2.1, Mamba-2 proves that the no-normalizer recurrence is functionally equivalent to a specific class of SSMs under the SSD framework.

2.3 Delta Networks

Fast Weight Programmers and Delta Networks formulate sequence modeling as the online learning of a value-retrieval function. Let $\mathbf{S}_{t-1} \in \mathbb{R}^{d \times d_v}$ denote the fast-weight state matrix. Instead of purely additive accumulation, standard Delta Networks employ an error-driven update derived from the gradient of the instantaneous squared error between the state's reconstruction of the current key and value:

\ell_t(\mathbf{S}) := \frac{1}{2} \left\| \mathbf{S}^\top \mathbf{k}_t - \mathbf{v}_t \right\|_2^2.

Here, $\mathbf{S}^\top \mathbf{k}$ represents the model's prediction of value $\mathbf{v}_t$ given key $\mathbf{k}$ . The gradient with respect to the state is $\nabla_{\mathbf{S}} \ell_t(\mathbf{S}) = \mathbf{k}_t \left( \mathbf{S}^\top \mathbf{k}_t - \mathbf{v}_t \right)^\top$ .

Delta rule. the paper denote the gradient step size by $\eta_t$ ; later, $\beta_t$ denotes a dimensionless normalized gain and $\eta_t$ the induced step size. A single online gradient step gives:

\mathbf{S}_t = \mathbf{S}_{t-1} - \eta_t \mathbf{k}_t (\mathbf{S}_{t-1}^\top \mathbf{k}_t - \mathbf{v}_t)^\top = (\mathbf{I} - \eta_t \mathbf{k}_t \mathbf{k}_t^\top) \mathbf{S}_{t-1} + \eta_t \mathbf{k}_t \mathbf{v}_t^\top.

The rank-one factor performs a targeted shrinkage/edit along the current key direction; it becomes an orthogonal projection only when $\eta_t = 1 / \|\mathbf{k}_t\|_2^2$ . Gated Delta Networks add an explicit global decay gate; notation and comparison details are in Appendix C.3.

3 Autoregressive Next-Latent Prediction

In this section, the authors cast the recurrent write as an explicit online optimization problem. Under the read-after-write convention, the causal example revealed at step $t$ pairs the newly observed target with the prefix write feature available when that target was predicted, namely $\mathbf{x}_t = \phi(\mathbf{k}_{t-1})$ and $\mathbf{y}_t = \mathbf{v}_t$ . Standard DeltaNet instead uses the same-step pair $(\phi(\mathbf{k}_t), \mathbf{v}_t)$ ; that pairing remains causal, but it corresponds to a different local fast-memory objective. The authors therefore model $\mathbf{S}$ as an online linear predictor from $\mathbf{x}_t$ to $\mathbf{y}_t$ and optimize an instantaneous ridge-regression loss:

\ell_t(\mathbf{S}) \triangleq \frac{1}{2} \left\| \mathbf{S}^\top \mathbf{x}_t - \mathbf{y}_t \right\|_2^2 + \frac{\lambda_t}{2} \|\mathbf{S}\|_F^2,

where $\lambda_t \geq 0$ is a regularization coefficient. While full-batch minimization of the cumulative loss corresponds to the offline solution found in methods like MesaNet, efficient autoregressive modeling requires an online approximation. The authors therefore employ Online Gradient Descent (OGD).

Notation. In kernelized linear attention, the key that writes to memory is typically a feature vector $\phi(\mathbf{k}) \in \mathbb{R}^m$ rather than the raw key $\mathbf{k} \in \mathbb{R}^d$ . All derivations in this section are interpreted with:

\mathbf{x}_t \equiv \phi(\mathbf{k}_{t-1}) \in \mathbb{R}^m, \qquad \mathbf{S}_t \in \mathbb{R}^{m \times d_v},

so $\mathbf{x}_t$ is treated as the generic write feature. Queries used for retrieval, e.g. $\phi(\mathbf{q}_t)$ in linear attention, live in the same feature space but need not equal $\mathbf{x}_t$ . Whenever a summary sentence informally refers to a pairing in raw-key space, the mathematically exact kernelized object is the corresponding write feature.

Implicit fast-memory objective. Eq. (3.1) is the instantaneous objective whose gradient step defines the fast-memory write rule; it is not an additional supervised loss beyond the outer autoregressive likelihood. During training, the authors differentiate through the update so that the slow weights (which produce $(\mathbf{q}, \mathbf{k}, \mathbf{v})$ as well as $\beta_t, \lambda_t$ ) learn representations and gates that make these local updates useful.

Indexing and causality conventions. The read-after-write (RAW) convention is used throughout: after token $t$ is observed and written, the updated state $\mathbf{S}_t$ is read to predict token $t+1$ . Under next-latent alignment, the causal write pair is therefore $(\phi(\mathbf{k}_{t-1}), \mathbf{v}_t)$ , or equivalently $(\phi(\mathbf{k}_i), \mathbf{v}_{i+1})$ under standard indexing. The authors set $\mathbf{S}_0 = \mathbf{0}$ and impose the feature-space boundary $\mathbf{x}_1 := \mathbf{0}$ . For update rules with explicit shrinkage, the boundary sentinel is also assigned $\eta_1 := 0$ (equivalently, $\alpha_1 = 0, \gamma_1 = 1$ in the log-space notation); otherwise $\lambda_1 > 0$ would decay a carried state even though no data pair is written. Detailed RAW/RBW and boundary conventions are deferred to Appendix C.5.

With this convention, the internal fast-memory prediction is $\widehat{\mathbf{y}}_t := \mathbf{S}_{t-1}^\top \mathbf{x}_t$ , with residual $\mathbf{r}_t = \mathbf{y}_t - \widehat{\mathbf{y}}_t$ . This is distinct from the model readout at position $t$ , which under RAW uses the updated state $\mathbf{S}_t$ .

Fast memory as continual learning. The recurrent state is the fast memory updated within the forward pass; each token provides a local training pair $(\mathbf{x}_t, \mathbf{y}_t)$ . Across the family, $\beta_t$ controls plasticity and $\lambda_t$ controls shrinkage/forgetting; the realized $\eta_t$ depends on the local normalization statistic, which is smoothness-matched for regression and energy-based for the inner-product implementations.

3.1 Online Gradient Descent Update

The gradient of the instantaneous loss in Eq. (3.1) with respect to the state $\mathbf{S}$ is:

\nabla_{\mathbf{S}} \ell_t(\mathbf{S}) = \mathbf{x}_t (\mathbf{S}^\top \mathbf{x}_t - \mathbf{y}_t)^\top + \lambda_t \mathbf{S}.

Applying a single gradient descent step with learning rate $\eta_t$ yields the update rule:

\begin{aligned} \mathbf{S}_t &\leftarrow \mathbf{S}_{t-1} - \eta_t \nabla_{\mathbf{S}} \ell_t(\mathbf{S}_{t-1}) \\ &= \mathbf{S}_{t-1} - \eta_t \left[ \mathbf{x}_t (\mathbf{S}_{t-1}^\top \mathbf{x}_t - \mathbf{y}_t)^\top + \lambda_t \mathbf{S}_{t-1} \right] \\ &= (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{r}_t^\top, \end{aligned}

where $\mathbf{r}_t \triangleq \mathbf{y}_t - \mathbf{S}_{t-1}^\top \mathbf{x}_t$ is the residual (prediction error). Note that, unlike standard Delta Networks, $\mathbf{r}_t$ measures the discrepancy between the prediction from the previous write feature $\mathbf{x}_t = \phi(\mathbf{k}_{t-1})$ and the current value $\mathbf{v}_t$ .

This is gradient descent on the instantaneous ridge objective. With the normalized step size below, it becomes a normalized update. In the special case $\lambda_t = 0$ and $\varepsilon = 0$ , it reduces exactly to the classical NLMS recursion; for $\varepsilon > 0$ , it is the usual stabilized NLMS variant. Specifically, the loss $\ell_t$ is $L_t$ -smooth with respect to the Frobenius norm, with smoothness constant $L_t = \|\mathbf{x}_t\|_2^2 + \lambda_t$ . To ensure numerical stability and scale robustness, the authors adopt the normalized step size:

\eta_t = \frac{\beta_t}{\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon}, \quad \beta_t \in (0, 2), \varepsilon \geq 0.

The authors adopt the convention $\eta_t := 0$ when $\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon = 0$ , and also at the boundary sentinel $t = 1$ when $\mathbf{x}_1 = \mathbf{0}$ is used to denote "no causal pair." In analysis, $\varepsilon = 0$ may be taken and $L_t > 0$ assumed. In implementations, $\varepsilon > 0$ is used for numerical robustness; any $\varepsilon > 0$ only decreases $\eta_t$ and therefore preserves the descent guarantees below.

Appendix C.6 records secondary implementation details, including the $\beta_t > 1$ sign-flip regime and the positive-decay interpretation of ridge shrinkage under log-space unrolling.

3.2 Analysis

The authors show that the normalized step size yields per-step descent in the instantaneous regularized objective $\ell_t$ , a basic local stability property. This statement is pointwise in $t$ : it does not imply monotone decrease of the cumulative online loss $\sum_s \ell_s$ or of the outer autoregressive training objective.

Lemma 3.1 (Per-step Descent for Smooth Losses). Let $f: \mathbb{R}^{d_x \times d_v} \to \mathbb{R}$ be $L$ -smooth with respect to the Frobenius norm. For any step size $\eta \in (0, 2/L)$ , the gradient step $\mathbf{S}^+ = \mathbf{S} - \eta \nabla f(\mathbf{S})$ satisfies:

f(\mathbf{S}^+) \leq f(\mathbf{S}) - \frac{\eta (2 - \eta L)}{2} \|\nabla f(\mathbf{S})\|_F^2.

Proof. By $L$ -smoothness, for any $\mathbf{S}$ and $\mathbf{S}'$ :

f(\mathbf{S}') \leq f(\mathbf{S}) + \langle \nabla f(\mathbf{S}), \mathbf{S}' - \mathbf{S} \rangle + \frac{L}{2} \|\mathbf{S}' - \mathbf{S}\|_F^2.

Set $\mathbf{S}' = \mathbf{S}^+ = \mathbf{S} - \eta \nabla f(\mathbf{S})$ and simplify.

Step-size parametrization. Choosing $\eta = \beta / L$ with $\beta \in (0, 2)$ (hence requiring $L > 0$ ) yields a decrease coefficient $\beta (2 - \beta) / (2L)$ . For $L > 0$ , the stabilized choice $\eta = \beta / (L + \varepsilon)$ with $\varepsilon \geq 0$ also lies in $(0, 2/L)$ . When $L = 0$ , this interval is undefined; in the degenerate cases arising here ( $\mathbf{x}_t = \mathbf{0}$ and $\lambda_t = 0$ , or the corresponding windowed analogue), the gradient is zero, so the update is defined to be a no-op by setting $\eta := 0$ .

3.3 Delta Networks as Regression

In this section, the authors interpret many previous fast weight models under this optimization framework. They observe that Delta Networks and Linear Attention can be viewed as gradient updates induced by specific online objective functions.

By substituting the regression assignments $\mathbf{x}_t \gets \phi(\mathbf{k}_{t-1})$ and $\mathbf{y}_t \gets \mathbf{v}_t$ , Eq. (3.3) recovers the functional form of the Delta Network update rule, but with the critical index shift:

\mathbf{S}_t = \underbrace{\left((1 - \eta_t \lambda_t) \mathbf{I}_{d_x} - \eta_t \mathbf{x}_t \mathbf{x}_t^\top\right)}_{\text{Decay \& Targeted Forget}} \mathbf{S}_{t-1} + \underbrace{\eta_t \mathbf{x}_t \mathbf{y}_t^\top}_{\text{Write}}, \quad \mathbf{x}_t = \phi(\mathbf{k}_{t-1}), \mathbf{y}_t = \mathbf{v}_t.

Here, the rank-one term $\mathbf{x}_t \mathbf{x}_t^\top \mathbf{S}_{t-1}$ is the left Hessian action of the squared-error loss along the current write-feature direction. Intuitively, it reduces the component of the current predictor that acts on $\mathbf{x}_t = \phi(\mathbf{k}_{t-1})$ before adding the new target $\mathbf{y}_t = \mathbf{v}_t$ .

In contrast, replacing the regression (MSE) loss with the inner-product objective of Section 4.3 removes the residual term and yields an additive write. With the standard unshifted assignment $(\mathbf{x}_t, \mathbf{y}_t) = (\phi(\mathbf{k}_t), \mathbf{v}_t)$ (or $(\mathbf{k}_t, \mathbf{v}_t)$ in the unkernelized case), this is the familiar Linear Attention / Mamba-2 accumulation. With the next-latent assignment $(\mathbf{x}_t, \mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}), \mathbf{v}_t)$ , it becomes the one-step-shifted variant used by the authors' methods (e.g., Falcon-3A in Section 4.5).

This regression perspective motivates a key algorithmic improvement that the authors analyze in Section 4: with objective-matched normalization, the rank-one regression step uses $L_t = \|\mathbf{x}_t\|_2^2 + \lambda_t$ , while the sliding rule uses $L_t^{(B)} = \lambda_{\max}(\bar{\mathbf{C}}_t^{(B)}) + \lambda_t$ . This suggests that fixed learning rates are scale-mismatched for regression-style fast-weight updates; for inner-product writes, the same normalization is better viewed as a magnitude stabilizer than as a curvature requirement.

Appendix A.2 gives reference pseudocode for the sequential first-order online-ridge update.

4 Falcon: Fast Weight Attention

Fast-weight memories and linear-attention architectures can be interpreted as online models that update a recurrent memory during the forward pass, tracing back to classical fast-weight mechanisms and their modern instantiations in linear Transformers and Delta-style rules. In this section, the authors derive Falcon from two local objectives under the shifted $\phi(\mathbf{k}_{t-1}) \mathbf{v}_t$ alignment: squared-error regression and a negative inner-product objective. This yields normalized step sizes, explicit forgetting controls, and sliding-window variants.

Naming and formula summary. All variants use the same causal pair:

\mathbf{x}_t := \phi(\mathbf{k}_{t-1}), \qquad \mathbf{y}_t := \mathbf{v}_t, \qquad \mathbf{x}_1 := \mathbf{0}, \qquad \eta_1 := 0,

and read after writing, $\mathbf{o}_t = \mathbf{S}_t^\top \phi(\mathbf{q}_t)$ . Let $\mathbf{r}_t := \mathbf{y}_t - \mathbf{S}_{t-1}^\top \mathbf{x}_t$ and $\mathrm{Diag}(\boldsymbol{\eta}_t)$ denote the diagonal matrix of per-column step sizes. The regression family is:

\text{Falcon-1:} \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{r}_t^\top, \quad \eta_t = \frac{\beta_t}{\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon},

\text{Falcon-2:} \quad \mathbf{S}_t = \mathbf{S}_{t-1} (\mathbf{I}_{d_v} - \lambda_t \mathrm{Diag}(\boldsymbol{\eta}_t)) + \mathbf{x}_t (\boldsymbol{\eta}_t \odot \mathbf{r}_t)^\top, \quad \eta_{j,t} = \frac{\beta_{j,t}}{\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon},

\text{Falcon-3:} \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \frac{\eta_t}{B_t} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \left(\mathbf{y}_j - \mathbf{S}_{t-1}^\top \mathbf{x}_j\right)^\top, \quad \eta_t = \frac{\beta_t}{\mu_t^{(B)} + \lambda_t + \varepsilon}.

Here $\mu_t^{(B)} = \lambda_{\max}\left( B_t^{-1} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \mathbf{x}_j^\top \right)$ . The inner-product family replaces the residual regression write by direct target writes:

\text{Falcon-1A:} \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{y}_t^\top, \quad \eta_t = \frac{\beta_t}{E_t + \lambda_t + \varepsilon},

\text{Falcon-2A:} \quad \mathbf{S}_t = \mathbf{S}_{t-1} (\mathbf{I}_{d_v} - \lambda_t \mathrm{Diag}(\boldsymbol{\eta}_t)) + \mathbf{x}_t (\boldsymbol{\eta}_t \odot \mathbf{y}_t)^\top, \qquad \eta_{j,t} = \frac{\beta_{j,t}}{E_t + \lambda_t + \varepsilon},

\text{Falcon-3A:} \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \bar{\mathbf{N}}_t^{(B)}, \quad \eta_t = \frac{\beta_t}{\bar{E}_t^{(B)} + \lambda_t + \varepsilon},

where $\mathcal{E}_t = \|\mathbf{x}_t\|_2^2$ , $\bar{\mathbf{N}}_t^{(B)} = B_t^{-1} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \mathbf{y}_j^\top$ , and $\bar{E}_t^{(B)} = B_t^{-1} \sum_{j \in \mathcal{I}_t} \|\mathbf{x}_j\|_2^2$ . Thus, the index $1/2/3$ denotes scalar, per-column, and sliding-window dynamics, respectively. The suffix "A" denotes the inner-product objective.

4.1 Scaled Linear Attention and Scaled DeltaNet

Unlike softmax attention, which uses the scaled dot product $\langle \mathbf{q}, \mathbf{k} \rangle / \sqrt{d}$ , fast-weight recurrences are directly sensitive to the norms of queries and keys: (i) dot-product reads grow with $\|\mathbf{q}_t\|_2 \|\mathbf{k}\|_2$ , and (ii) additive (inner-product) writes grow with the write-feature norm. To stabilize both the read and the write streams, especially under long decoding horizons and mixed precision, the authors use explicit feature scaling/normalization.

Scaled features. The authors define a generic RMS normalization operator for a vector $\mathbf{u} \in \mathbb{R}^{d_u}$ :

\mathrm{RMSNorm}(\mathbf{u}) := \frac{\mathbf{u}}{\sqrt{\|\mathbf{u}\|_2^2 / d_u + \varepsilon_{\mathrm{rms}}}},

where $\varepsilon_{\mathrm{rms}} > 0$ is a small stabilizer. Unless stated otherwise, RMSNorm is applied to the $(\mathbf{q}_t, \mathbf{k}_t)$ projections used by fast-weight reads/writes, before forming dot products or outer products. This default differs from common $\ell_2$ -normalized DeltaNet variants (e.g., in Gated DeltaNet implementations) and is substantially more stable in mixed precision because standard RMSNorm keeps coordinate magnitudes $\Theta(1)$ . Moreover:

\left\| \mathrm{RMSNorm}(\mathbf{u}) \right\|_2^2 = \frac{d_u \|\mathbf{u}\|_2^2}{\|\mathbf{u}\|_2^2 + d_u \varepsilon_{\mathrm{rms}}} \leq d_u,

so in the usual regime $\|\mathbf{u}\|_2^2 \gg d_u \varepsilon_{\mathrm{rms}}$ the paper indeed have $\|\mathrm{RMSNorm}(\mathbf{u})\|_2^2 \approx d_u$ . By default, values $\mathbf{v}_t$ are not normalized; value normalization (VNorm) is optional and disabled unless explicitly enabled.

Scaled Linear Attention. Under next-latent alignment, the RMS-normalized projections are used in the feature space. The scalar denominator-free inner-product recurrence, denoted Falcon-1A below, is:

\mathbf{o}_t = \mathbf{S}_t^\top \phi(\mathbf{q}_t), \quad \mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{v}_t^\top, \quad \mathbf{x}_t := \phi(\mathbf{k}_{t-1}).

Its per-column counterpart, Falcon-2A, replaces the scalar write gain by a vector $\boldsymbol{\eta}_t \in \mathbb{R}^{d_v}$ :

\mathbf{S}_t = \mathbf{S}_{t-1} (\mathbf{I}_{d_v} - \lambda_t \mathrm{Diag}(\boldsymbol{\eta}_t)) + \mathbf{x}_t (\boldsymbol{\eta}_t \odot \mathbf{v}_t)^\top.

The corresponding normalized numerator/denominator recurrence is recorded in Appendix C.1; in signed-feature settings, the denominator caveat in Appendix C.2 applies.

Scaled DeltaNet and the common ridge parameterization. For regression-style fast weights, the same scaled write-feature $\mathbf{x}_t$ is used inside the NLMS update:

\mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \left(\mathbf{v}_t - \mathbf{S}_{t-1}^\top \mathbf{x}_t\right)^\top, \quad \eta_t = \frac{\beta_t}{\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon}.

In scaled implementations, the network may emit a dimensionless base ridge $\bar{\lambda}_t$ that is converted to the actual coefficient used by the recurrence, $\lambda_t = \bar{\lambda}_t E_t$ , where $E_t$ is the appropriate normalization statistic: $\|\mathbf{x}_t\|_2^2$ for Falcon-2 and $\mu_t^{(B)} := \lambda_{\max}(\bar{\mathbf{C}}_t^{(B)})$ for Falcon-3. The authors then set:

\eta_t = \frac{\beta_t}{E_t + \lambda_t + \varepsilon}, \quad \alpha_t := \eta_t \lambda_t, \quad \gamma_t := 1 - \alpha_t.

Here $E_t = \|\mathbf{x}_t\|_2^2$ for the non-sliding scalar/per-column rules (Falcon-1/Falcon-2 and Falcon-1A/Falcon-2A), while $E_t = \mu_t^{(B)}$ for Falcon-3 and $E_t = \hat{E}_t^{(B)}$ for Falcon-3A. For regression, $E_t$ is the local smoothness scale of the data term; for inner-product writes, the corresponding energy statistics in Sections 4.3 and 4.5 are practical write-magnitude controls. Exact scale-robustness identities, detached-statistics details, the normalized linear-attention recurrence, and log-space positive-decay handling are deferred to Appendix C.1 and Appendix C.6.

4.2 Regression Loss (Delta Network)

The authors formulate the state update as an autoregressive linear regression problem. At time step $t$ , let the state be $\mathbf{S}_{t-1} \in \mathbb{R}^{d_x \times d_v}$ , the write feature be $\mathbf{x}_t \triangleq \phi(\mathbf{k}_{t-1}) \in \mathbb{R}^{d_x}$ , and the target be $\mathbf{y}_t \triangleq \mathbf{v}_t \in \mathbb{R}^{d_v}$ . (In the unkernelized case, $\phi$ is the identity and $d_x = d$ .) The state $\mathbf{S}_{t-1}$ acts as a linear predictor mapping the prefix feature $\mathbf{x}_t$ to the newly observed target $\mathbf{y}_t$ .

The instantaneous squared-error loss is:

f_t(\mathbf{S}) := \frac{1}{2} \left\| \mathbf{S}^\top \mathbf{x}_t - \mathbf{y}_t \right\|_2^2.

Evaluated at the pre-update state, the residual is $\mathbf{r}_t \triangleq \mathbf{y}_t - \mathbf{S}_{t-1}^\top \mathbf{x}_t$ . The gradient with respect to the state is:

\nabla_{\mathbf{S}} f_t(\mathbf{S}_{t-1}) = \mathbf{x}_t (\mathbf{S}_{t-1}^\top \mathbf{x}_t - \mathbf{y}_t)^\top = -\mathbf{x}_t \mathbf{r}_t^\top.

Delta update (standard vs. next-latent). A single step of Online Gradient Descent (OGD) with learning rate $\eta_t$ yields the Delta update:

\begin{aligned} \mathbf{S}_t &= \mathbf{S}_{t-1} - \eta_t \nabla_{\mathbf{S}} f_t(\mathbf{S}_{t-1}) \\ &= \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{r}_t^\top \\ &= (\mathbf{I}_{d_x} - \eta_t \mathbf{x}_t \mathbf{x}_t^\top) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{y}_t^\top. \end{aligned}

Under next-latent alignment the paper have $\mathbf{x}_t = \phi(\mathbf{k}_{t-1})$ and $\mathbf{y}_t = \mathbf{v}_t$ . Replacing $\mathbf{k}_{t-1}$ with $\mathbf{k}_t$ (equivalently, $\mathbf{x}_t \gets \phi(\mathbf{k}_t)$ ) recovers the unshifted DeltaNet update of Schlag et al.

$L_2$ Regularization. Adding an $L_2$ penalty $\frac{\lambda_t}{2} \|\mathbf{S}\|_F^2$ to the instantaneous loss yields the shrinkage term $-\eta_t \lambda_t \mathbf{S}_{t-1}$ in the online update (cf. Eq. (3.3)). Under normalized step sizes, this results in the multiplicative factor $(1 - \eta_t \lambda_t)$ , which the authors treat as a simple and controllable forgetting mechanism.

Falcon-2: Adaptive Learning Rates. The authors propose Falcon-2, which combines NLMS-style normalization with per-channel adaptive learning rates. In Appendix F, they derive a vectorized dual form that makes column-wise adaptivity computationally tractable on GPUs. This allows the step size to be a vector $\boldsymbol{\eta}_t \in \mathbb{R}^{d_v}$ , tailoring the update magnitude for each value channel (column of $\mathbf{S}$ ) independently. In the models studied here, the standard multi-head factorization is used: each head carries its own fast state, and the same per-column rule is applied independently within that head.

Per-column NLMS step sizes. Concretely, an NLMS-style normalizer shared across columns is used with learned per-column gains (equivalently, per output-feature / column gains):

\eta_{j,t} = \frac{\beta_{j,t}}{\|\mathbf{x}_t\|_2^2 + \lambda_t + \varepsilon}, \quad \beta_{j,t} \in (0, 2), \varepsilon > 0.

When $\lambda_t = 0$ , $\varepsilon = 0$ , and the gains are tied across channels ( $\beta_{j,t} \equiv \beta_t$ ), this reduces to the classical scalar NLMS step size; otherwise, it is a column-wise NLMS generalization with a shared normalizer. Since the squared-error (ridge) loss decomposes across value coordinates (columns of $\mathbf{S}$ ), the per-step descent argument from Lemma 3.1 applies column-wise provided $0 < \beta_{j,t} < 2$ for all $j$ . Equivalently, Falcon-2 is a collection of $d_v$ independent scalar-step updates on a separable objective, rather than a single Frobenius-gradient step with a full matrix-valued learning rate.

Update rule. Let $\boldsymbol{\eta}_t = (\eta_{1,t}, \dots, \eta_{d_v,t})^\top$ . The update can be written compactly as:

\mathbf{S}_t = \mathbf{S}_{t-1} \left( \mathbf{I}_{d_v} - \lambda_t \mathrm{Diag}(\boldsymbol{\eta}_t) \right) + \mathbf{x}_t \left( \boldsymbol{\eta}_t \odot \mathbf{r}_t \right)^\top,

An expanded edit decomposition, separating the per-column shrinkage path from the feature-direction edit, is deferred to Appendix F.

Relation to RWKV-7 and Kimi Linear Attention. A brief comparison to RWKV-7 and Kimi Linear Attention is moved to Appendix C.6.

The equivalent column-wise recursion and the log-space positive-decay renormalization used by the chunk-parallel kernels are deferred to Appendix F.

Chunk-parallel implementation. Falcon-2 can be trained sequence-parallel by chunking the length axis and using a Gram/WY representation within each chunk. In the multi-head setting used here, this computation is applied independently within each head. Within a chunk, the WY/Gram form builds a shared key Gram matrix and a channel-dependent unit-lower-triangular system. The injected-value and projected-history paths share this triangular factor, so the implementation solves one merged residual system rather than two. This removes one batched TriSolve per chunk in the forward pass without changing the recurrence or asymptotic complexity, and mirrors the single-inversion form emphasized in Comba. Algorithm 1 gives the single-head Falcon-2 chunk-wise forward pass with the same positive-decay convention used by the implementation. The no-ridge rank-one case is recovered by setting $\lambda_t = 0$ , in which case $\gamma_{t,j} = 1$ and the chunk-local rescaling becomes the identity. Appendix F gives the exact WY/Gram algebra and complexity analysis.

4.3 Inner Product Loss (Linear Attention and Mamba-2)

The authors now consider an Inner Product objective that encourages alignment between the state prediction and the target. It is written in minimization form with an optional $L_2$ penalty:

\ell_t^{\mathrm{ip}}(\mathbf{S}) \triangleq -\langle \mathbf{S}^\top \mathbf{x}_t, \mathbf{y}_t \rangle + \frac{\lambda_t}{2} \|\mathbf{S}\|_F^2, \quad \lambda_t \geq 0.

When $\lambda_t = 0$ , the objective is linear in $\mathbf{S}$ (no finite minimizer) and gradient descent reduces to purely additive Hebbian writes.

Standard vs. next-latent alignment. If the unshifted features $(\mathbf{x}_t, \mathbf{y}_t) = (\phi(\mathbf{k}_t), \mathbf{v}_t)$ are chosen (or $(\mathbf{k}_t, \mathbf{v}_t)$ in the unkernelized case), the additive update below matches the usual Linear Attention write $\phi(\mathbf{k}_t) \mathbf{v}_t^\top$ (Eq. (2.2)). The next-latent framework instead uses $(\mathbf{x}_t, \mathbf{y}_t) = (\phi(\mathbf{k}_{t-1}), \mathbf{v}_t)$ , yielding a one-step shifted write stream.

Falcon-A variants and notation. The suffix "A" is used for the inner-product objective and the numerical index is kept aligned with the regression family. Thus Falcon-1A is the scalar non-sliding inner-product rule, Falcon-2A is the per-column non-sliding inner-product rule, and Falcon-3A is the sliding-window inner-product rule in Section 4.5. As in Falcon-1/Falcon-2/Falcon-3, $\beta_t$ denotes the dimensionless gain, $\lambda_t$ the actual shrinkage coefficient used by the recurrence (obtained directly or via the same scale-coupled construction described above), and $\eta_t$ the resulting step size. For the inner-product family, this step size should be read as an energy-normalized write gain rather than as a curvature-matched denominator. Any decay fraction $\alpha_t := \eta_t \lambda_t$ is derived rather than independently parameterized.

Gradient and update. The gradient of Eq. (4.5) is:

\nabla_{\mathbf{S}} \ell_t^{\mathrm{ip}}(\mathbf{S}) = -\mathbf{x}_t \mathbf{y}_t^\top + \lambda_t \mathbf{S}.

A scalar gradient step gives the Linear-Attention/Mamba-2 style update, which the authors denote Falcon-1A:

\mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{y}_t^\top,

where $\mathbf{x}_t = \phi(\mathbf{k}_{t-1})$ and $\mathbf{y}_t = \mathbf{v}_t$ under next-latent alignment. Setting $\lambda_t = 0$ recovers the usual additive write $\mathbf{S}_t = \mathbf{S}_{t-1} + \eta_t \mathbf{x}_t \mathbf{y}_t^\top$ .

Falcon-2A: per-column inner-product write. Because Eq. (4.5) decomposes over value coordinates, each column can use its own energy-normalized learning rate. Define:

\boldsymbol{\eta}_t := \left( \eta_{1,t}, \dots, \eta_{d_v,t} \right)^\top.

The per-column inner-product update is:

\mathbf{S}_t = \mathbf{S}_{t-1} (\mathbf{I}_{d_v} - \lambda_t \mathrm{Diag}(\boldsymbol{\eta}_t)) + \mathbf{x}_t (\boldsymbol{\eta}_t \odot \mathbf{y}_t)^\top.

Equivalently, the $j$ -th column evolves as:

\mathbf{s}_{t,j} = (1 - \eta_{j,t} \lambda_t) \mathbf{s}_{t-1,j} + \eta_{j,t} y_{t,j} \mathbf{x}_t.

Thus Falcon-2A is not the scalar inner-product rule; it is the per-column inner-product analogue of Falcon-2.

Falcon-1A/Falcon-2A step sizes. Unlike regression, the inner-product objective is $\lambda_t$ -smooth independently of the write-feature energy. Accordingly, the objective-matched denominator would depend only on $\lambda_t$ ; in the inner-product implementations studied here, an energy-normalized write gain is instead retained to control the magnitude of the additive write. Let $E_t := \|\mathbf{x}_t\|_2^2$ , and if the scale-coupled parameterization is active set $\lambda_t := \bar{\lambda}_t E_t$ before applying the update. The scalar Falcon-1A step size is:

\eta_t = \frac{\beta_t}{E_t + \lambda_t + \varepsilon}, \quad \beta_t \in (0, 2), \varepsilon \geq 0.

The per-column Falcon-2A step sizes are:

\eta_{j,t} = \frac{\beta_{j,t}}{E_t + \lambda_t + \varepsilon}, \quad \beta_{j,t} \in (0, 2), \varepsilon \geq 0.

As in Eq. (3.4), $\eta_t := 0$ (or $\eta_{j,t} := 0$ for all $j$ ) is set when the denominator vanishes, and also at the boundary sentinel $t = 1$ when $\mathbf{x}_1 = \mathbf{0}$ is imposed. When $\lambda_t > 0$ , these choices satisfy $\eta_t < 2/\lambda_t$ and $\eta_{j,t} < 2/\lambda_t$ for any admissible $\beta_t, \beta_{j,t}$ . Since $\ell_t^{\mathrm{ip}}$ is $\lambda_t$ -smooth (its Hessian is $\lambda_t \mathbf{I}$ ), Lemma 3.1 yields per-step descent for the unclamped scalar update, and the same argument applies column-wise to Eq. (4.7). If the later positive-decay clamp is activated for log-space unrolling, the implemented shrinkage should be interpreted as using the effective ridge coefficient $\lambda_t := \alpha_t / \eta_t$ in the scalar case, or $\bar{\lambda}_{j,t} := \alpha_{j,t} / \eta_{j,t}$ in the per-column case, whenever the corresponding step size is positive. The $E_t$ term is not required by curvature, but stabilizes the write magnitude and yields a sensible $\lambda_t \to 0$ limit.

Decay positivity. Some parallel/unrolled forms (Section 4.6) use $\gamma_t := 1 - \eta_t \lambda_t$ in log space and therefore require $\gamma_t > 0$ . In implementations, compute $\alpha_t := \eta_t \lambda_t$ and, if necessary, clamp $\alpha_t \leftarrow \min(\alpha_t, 1 - \varepsilon_\gamma)$ before computing $\log \gamma_t = \log 1\mathrm{p}(-\alpha_t)$ (fp32). The clamp is inactive whenever $\eta_t \lambda_t < 1 - \varepsilon_\gamma$ ; in that regime, the dynamics match $\gamma_t = 1 - \eta_t \lambda_t$ exactly. As above, the descent statement pertains to the unclamped recurrence. When the clamp activates, the implemented recurrence should be viewed as a numerically safe surrogate whose shrinkage path uses the effective ridge coefficient:

\widetilde{\lambda}_t := \alpha_t / \eta_t

whenever $\eta_t > 0$ (and $\widetilde{\lambda}_t := 0$ when $\eta_t = 0$ ), while the additive write gain remains $\eta_t$ .

4.4 Mini-batch Update Rule for Regression (Falcon-3)

To better capture local dependencies and reduce noise accumulation, the authors introduce a sliding-state mechanism: instead of updating the state from only the instantaneous residual, a single mini-batch gradient step is taken on a finite history window of nominal size $B$ .

Falcon-3 can be viewed as a sliding-window specialization of the internal-objective view exemplified by ATLAS, but here instantiated with a linear matrix memory, a squared-error objective, and strict next-latent alignment. Exact continuation across segment boundaries additionally requires a fixed-width tail of the last $B-1$ causal pairs; Appendix C.7 records the details.

Sequence-parallel training. After zero-padding each active window to width $B$ , Falcon-3 becomes a fixed-rank- $B$ low-rank recurrence. Algorithm 2 gives the reference sequential update, and Algorithm 3 gives the chunk-parallel ParallelFlow implementation based on the positive-decay reduction in Eq. (4.16). Appendix H records the driver construction and mask conventions. The inner-product counterpart (Falcon-3A) admits an explicit masked linear-attention form (Section 4.6), enabling fully vectorized training over the sequence dimension.

For $t \geq 2$ , let the active window indices be $\mathcal{I}_t = \{ j \mid \max(2, t - B + 1) \leq j \leq t \}$ and denote the realized window size by $B_t := |\mathcal{I}_t| \leq B$ (so $B_t \geq 1$ ). Define the write feature $\mathbf{x}_j := \phi(\mathbf{k}_{j-1})$ for $j \geq 2$ (so $\mathbf{x}_j = \mathbf{k}_{j-1}$ when $\phi$ is the identity), and impose the boundary convention $\mathbf{x}_1 := \mathbf{0}$ . The authors set $\eta_1 := 0$ , so the $t = 1$ write is a no-op; all windowed objectives/updates below are defined for $t \geq 2$ . To make the update magnitude (and hence the effective decay) invariant to the nominal window size $B$ , the window-average squared loss is optimized:

\ell_t^{\mathrm{reg},(B)}(\mathbf{S}) := \frac{1}{2 B_t} \sum_{j \in \mathcal{I}_t} \| \mathbf{S}^\top \mathbf{x}_j - \mathbf{v}_j \|_2^2 + \frac{\lambda_t}{2} \|\mathbf{S}\|_F^2, \qquad (t \geq 2).

Sufficient Statistics. The authors define the sliding covariance $\mathbf{C}_t^{(B)}$ and cross-covariance $\mathbf{N}_t^{(B)}$ matrices:

\mathbf{C}_t^{(B)} \triangleq \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \mathbf{x}_j^\top, \quad \mathbf{N}_t^{(B)} \triangleq \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \mathbf{v}_j^\top.

Define the window-averaged statistics:

\bar{\mathbf{C}}_t^{(B)} := \frac{1}{B_t} \mathbf{C}_t^{(B)}, \quad \bar{\mathbf{N}}_t^{(B)} := \frac{1}{B_t} \mathbf{N}_t^{(B)}.

Then the gradient evaluated at the pre-update state is:

\nabla_{\mathbf{S}} \ell_t^{\mathrm{reg},(B)}(\mathbf{S}_{t-1}) = \bar{\mathbf{C}}_t^{(B)} \mathbf{S}_{t-1} - \bar{\mathbf{N}}_t^{(B)} + \lambda_t \mathbf{S}_{t-1}.

Update Rule. A block-normalized gradient step is applied:

\mathbf{S}_t = \mathbf{S}_{t-1} - \eta_t \nabla_{\mathbf{S}} \ell_t^{\mathrm{reg},(B)}(\mathbf{S}_{t-1}).

Substituting the gradient yields the affine update:

\mathbf{S}_t = \left( \mathbf{I}_{d_x} - \eta_t (\bar{\mathbf{C}}_t^{(B)} + \lambda_t \mathbf{I}_{d_x}) \right) \mathbf{S}_{t-1} + \eta_t \bar{\mathbf{N}}_t^{(B)}.

Equivalently, collecting residuals at the pre-update state yields:

\mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \frac{\eta_t}{B_t} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \left( \mathbf{v}_j - \mathbf{S}_{t-1}^\top \mathbf{x}_j \right)^\top.

This is the direct mini-batch analogue of Eq. (3.3): all window residuals are evaluated at the pre-update state $\mathbf{S}_{t-1}$ , and the update averages their rank-one gradients.

For $t \geq 2$ , let $\mathbf{X}_t \in \mathbb{R}^{d_x \times B_t}$ stack the active-window write-features, so that $\bar{\mathbf{C}}_t^{(B)} = \mathbf{X}_t \mathbf{X}_t^\top / B_t$ . The authors normalize by the exact local smoothness scale of the windowed ridge objective rather than by its trace upper bound:

\mu_t^{(B)} := \lambda_{\max}(\bar{\mathbf{C}}_t^{(B)}) = \frac{\|\mathbf{X}_t\|_2^2}{B_t} = \frac{\lambda_{\max}(\mathbf{X}_t^\top \mathbf{X}_t)}{B_t}, \quad \eta_t = \frac{\beta_t}{\mu_t^{(B)} + \lambda_t + \varepsilon}, \quad \beta_t \in (0, 2), \varepsilon > 0.

Then $L_t^{(B)} = \mu_t^{(B)} + \lambda_t$ , so whenever $L_t^{(B)} > 0$ this normalization ensures $\eta_t \in (0, 2/L_t^{(B)})$ for any $\beta_t \in (0, 2)$ and Lemma 3.1 yields per-step descent for Eq. (4.14) before any positive-decay clamp. If $L_t^{(B)} = 0$ (equivalently, $\bar{\mathbf{C}}_t^{(B)} = \mathbf{0}$ and $\lambda_t = 0$ ), then $\bar{\mathbf{N}}_t^{(B)} = \mathbf{0}$ as well and the update is a no-op. Crucially, because the window average is optimized, $\bar{\mathbf{N}}_t^{(B)}$ is an average and $\mu_t^{(B)}$ is the spectral norm of an average covariance, so neither quantity grows linearly with the nominal window size $B$ . If the write feature itself is RMS-normalized, then:

\mu_t^{(B)} \leq \bar{E}_t^{(B)} := \mathrm{tr}(\bar{\mathbf{C}}_t^{(B)}) \approx d_x,

so the denominator remains $O(d_x)$ rather than $O(B d_x)$ ; for a generic kernel map $\phi$ , the correct statement is simply that $\mu_t^{(B)}$ tracks the realized windowed smoothness scale. Consequently, neither the injection $\eta_t \bar{\mathbf{N}}_t^{(B)}$ nor the decay fraction $\alpha_t := \eta_t \lambda_t$ is systematically amplified by increasing $B$ . If the scale-coupled ridge parameterization of Section 4.1 is enabled, replace $\lambda_t$ throughout this subsection by $\lambda_t^{\mathrm{eff}} := \bar{\lambda}_t \mu_t^{(B)}$ . In the current implementation, this smoothness statistic can be treated as a statistics-only multiplier when constructing $\lambda_t^{\mathrm{eff}}$ (detached / stop-gradient through the multiplier), while the step-size denominator still uses the live $\mu_t^{(B)}$ . Importantly, one need not materialize the $d_x \times d_x$ matrix $\mathbf{C}_t^{(B)}$ to evaluate either the gradient or the step size: if the window write-features are stacked into $\mathbf{X}_t \in \mathbb{R}^{d_x \times B_t}$ , then:

\bar{\mathbf{C}}_t^{(B)} \mathbf{S}_{t-1} = \frac{1}{B_t} \mathbf{X}_t \left( \mathbf{X}_t^\top \mathbf{S}_{t-1} \right), \quad \mu_t^{(B)} = \frac{\lambda_{\max}(\mathbf{X}_t^\top \mathbf{X}_t)}{B_t}.

Since $B_t \leq B$ is small, $\mu_t^{(B)}$ can be computed exactly from the $B_t \times B_t$ Gram matrix or approximated with a few power iterations. For the implemented positive-decay recurrence, define:

\alpha_t^{\mathrm{raw}} := \eta_t \lambda_t, \quad \alpha_t := \min(\alpha_t^{\mathrm{raw}}, 1 - \varepsilon_\gamma), \quad \gamma_t := 1 - \alpha_t,

c_0 := 1, \qquad c_t := \prod_{r=1}^t \gamma_r, \quad \widetilde{\mathbf{S}}_t := \mathbf{S}_t / c_t.

When $\alpha_t = \alpha_t^{\mathrm{raw}}$ , this is exactly the ridge-gradient recurrence in Eq. (4.14). If the clamp activates, the implemented shrinkage path should instead be interpreted as using the effective coefficient:

\widetilde{\lambda}_t := \begin{cases} \alpha_t / \eta_t, & \eta_t > 0, \\ 0, & \eta_t = 0, \end{cases}

while keeping the same residual injection gain $\eta_t$ . Thus the descent claim above applies to the unclamped update; the clamped update is a positive-decay surrogate. With this convention, the implemented recurrence is equivalent to:

\widetilde{\mathbf{S}}_t = \widetilde{\mathbf{S}}_{t-1} + \frac{\widehat{\eta}_t}{B_t} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \left( \frac{\mathbf{v}_j}{c_{t-1}} - \widetilde{\mathbf{S}}_{t-1}^\top \mathbf{x}_j \right)^\top, \quad \widehat{\eta}_t := \eta_t / \gamma_t.

Algorithm 3 realizes Eq. (4.16) chunk-locally via log-prefix decays; Appendix C.7 records the corresponding continuation and chunk-boundary details.

Boundary/scan details. The offline overlap interpretation, exact continuation requirement, and explicit associative chunk map are given in Appendix C.7 and Appendix H.1.

4.5 Mini-batch Update Rule for Inner Product Loss (Falcon-3A)

The authors apply the same sliding-window principle to the inner-product objective, while keeping the same parameter semantics as in the rest of the family: $\beta_t$ is the dimensionless gain, $\lambda_t$ is the actual shrinkage coefficient used by the recurrence (after any optional scale coupling), and $\eta_t$ is the resulting normalized step size.

For $t \geq 2$ , let $B_t := |\mathcal{I}_t| \leq B$ (and skip the boundary update at $t = 1$ ). The windowed loss is:

\ell_t^{\mathrm{ip},(B)}(\mathbf{S}) := -\frac{1}{B_t} \sum_{j \in \mathcal{I}_t} \left\langle \mathbf{S}^\top \mathbf{x}_j, \mathbf{v}_j \right\rangle + \frac{\lambda_t}{2} \|\mathbf{S}\|_F^2.

Define the window-averaged cross-covariance and write energy:

\bar{\mathbf{N}}_t^{(B)} := \frac{1}{B_t} \sum_{j \in \mathcal{I}_t} \mathbf{x}_j \mathbf{v}_j^\top, \quad \bar{E}_t^{(B)} := \frac{1}{B_t} \sum_{j \in \mathcal{I}_t} \|\mathbf{x}_j\|_2^2.

Then the gradient evaluated at the pre-update state is:

\nabla_{\mathbf{S}} \ell_t^{\mathrm{ip},(B)}(\mathbf{S}_{t-1}) = -\bar{\mathbf{N}}_t^{(B)} + \lambda_t \mathbf{S}_{t-1}.

Unlike Falcon-3, whose regression step size uses the local smoothness $\mu_t^{(B)}$ , the inner-product windowed rule uses the window-energy statistic as a practical write-gain normalizer. Let $E_t^{(B)} := \bar{E}_t^{(B)}$ , and if the scale-coupled parameterization is active set $\lambda_t := \bar{\lambda}_t E_t^{(B)}$ before computing the step size:

\eta_t = \frac{\beta_t}{E_t^{(B)} + \lambda_t + \varepsilon}, \qquad \beta_t \in (0, 2), \varepsilon > 0,

with the boundary convention $\eta_1 := 0$ . Applying one gradient step gives:

\mathbf{S}_t = (1 - \eta_t \lambda_t) \mathbf{S}_{t-1} + \eta_t \bar{\mathbf{N}}_t^{(B)}.

When $\lambda_t > 0$ , the objective is $\lambda_t$ -smooth, so Eq. (4.18) implies $\eta_t < 2/\lambda_t$ for any $\beta_t \in (0, 2)$ , and Lemma 3.1 yields per-step descent in $\ell_t^{\mathrm{ip},(B)}$ for the unclamped update. If the positive-decay clamp below activates, the implemented shrinkage should again be interpreted as using the effective ridge coefficient $\widetilde{\lambda}_t := \alpha_t / \eta_t$ when $\eta_t > 0$ (and 0 when $\eta_t = 0$ ), rather than as an exact gradient step for the original $\lambda_t$ . When $\lambda_t = 0$ , the objective is linear and unbounded below, so the same normalization should be interpreted as a magnitude stabilizer for the additive write rather than as a bounded-objective guarantee.

For log-space unrolling, the authors introduce only the derived quantities:

\alpha_t^{\mathrm{raw}} := \eta_t \lambda_t, \quad \alpha_t := \min(\alpha_t^{\mathrm{raw}}, 1 - \varepsilon_\gamma), \quad \gamma_t := 1 - \alpha_t \in [\varepsilon_\gamma, 1],

with $\alpha_1 := 0$ and $\gamma_1 := 1$ . Thus, $\alpha_t$ and $\gamma_t$ are implementation variables derived from the same $(\beta_t, \lambda_t, \eta_t)$ parameterization used elsewhere; they are not separate learned controls.

Because the window average is optimized, neither the typical scale of $\bar{\mathbf{N}}_t^{(B)}$ nor the decay fraction $\alpha_t = \eta_t \lambda_t$ grows systematically with the nominal window size $B$ . The authors again set $\eta_t := 0$ when the denominator in Eq. (4.18) vanishes (in practice, $\varepsilon > 0$ is taken). Appendix C.7 records the stationary calculation and the exact boundary-state requirements. For $B = 1$ and $t \geq 2$ , Eq. (4.19) reduces to the scalar non-sliding inner-product update, i.e. Falcon-1A; in the additive ablation $\lambda_t \equiv 0$ , the write is purely additive. The per-column non-sliding analogue is Falcon-2A in Eq. (4.7).

4.6 Parallel (Attention) Form of Mini-batch Inner Product Update

The authors show that the recurrent sliding-window update in Section 4.5 can be written as the sum of (i) a decayed-history term from the incoming boundary state and (ii) an (unnormalized) dot-product attention matrix with a structured causal mask. This view enables GPU-parallel training and matches the usual unrolling of gated linear recurrences into masked attention.

For the unclamped exposition, let $\lambda_s$ denote the actual shrinkage coefficient after any optional scale coupling, and define:

\eta_s := \frac{\beta_s}{\bar{E}_s^{(B)} + \lambda_s + \varepsilon}, \quad \gamma_s := 1 - \eta_s \lambda_s, \quad \delta_t := \prod_{r=1}^t \gamma_r.

Then the recurrence $\mathbf{S}_s = \gamma_s \mathbf{S}_{s-1} + \eta_s \bar{\mathbf{N}}_s^{(B)}$ unrolls to:

\mathbf{S}_t = \delta_t \mathbf{S}_0 + \sum_{j=2}^t M_{t,j} \mathbf{x}_j \mathbf{v}_j^\top,

where, under the boundary convention $\mathbf{x}_1 := \mathbf{0}$ and $\eta_1 = 0$ :

M_{t,j} := \sum_{s=j}^{\min(t, j+B-1)} \frac{\eta_s}{B_s} \prod_{r=s+1}^t \gamma_r, \qquad 2 \leq j \leq t,

and $M_{t,j} := 0$ otherwise. Hence, the read-after-write output is:

\mathbf{o}_t = \delta_t \mathbf{S}_0^\top \phi(\mathbf{q}_t) + \sum_{j=2}^t M_{t,j} \langle \phi(\mathbf{q}_t), \mathbf{x}_j \rangle \mathbf{v}_j.

Equivalently, stacking query features $\mathbf{Q} \in \mathbb{R}^{L \times d_x}$ , write features $\mathbf{X} \in \mathbb{R}^{L \times d_x}$ , values $\mathbf{V} \in \mathbb{R}^{L \times d_v}$ , and mask $\mathbf{M} \in \mathbb{R}^{L \times L}$ with entries $\mathbf{M}_{t,j} = M_{t,j}$ , the authors obtain:

\mathbf{O} = \mathrm{Diag}(\boldsymbol{\delta}) \mathbf{Q} \mathbf{S}_0 + (\mathbf{Q} \mathbf{X}^\top \odot \mathbf{M}) \mathbf{V}, \quad \boldsymbol{\delta} := (\delta_1, \dots, \delta_L)^\top.

For the fresh-sequence default $\mathbf{S}_0 = \mathbf{0}$ , only the masked-attention term remains. Appendix D.1 gives the full derivation, the stationary special case, and the chunk-local log-space evaluation used by Algorithm 4.

Backward pass. The numerically stable backward pass is given in Appendix D.2; it differentiates through the structured mask, chunk-local log-decay renormalization, and normalized step-size computation.

Relation to prior internal-objective views. A brief comparison to test-time training and Titans- and ATLAS-style internal memory objectives is moved to Appendix C.4.

Experiment

The evaluation trains 124M-130M parameter language models on a 50B-token budget, comparing Falcon variants against Transformer and recurrent baselines via perplexity, downstream accuracy, and a variable-length addition diagnostic. Falcon-1.3 leads in perplexity and one-shot accuracy, while Falcon-1A.2 tops zero-shot averages, with normalized updates and context-conditioned parameters yielding targeted gains. On addition, Falcon-3A.3 and Falcon-1A.3 achieve the best extrapolation, outperforming baselines, supporting the claim that shifted, normalized fast-weight updates improve causal storage and length extrapolation without sacrificing language-model quality.

At 124M-130M parameters and a 50B-token budget, the proposed Falcon-1.3 model achieves the best FineWeb-Edu perplexity among all listed models, outperforming both the Transformer and recurrent baselines. Among recurrent baselines, Gated DeltaNet is the strongest, while Falcon-1A.3 leads the inner-product variants. The results show that the proposed aligned, normalized updates maintain competitive language modeling quality. Falcon-1.3 achieves the lowest FineWeb-Edu perplexity, surpassing all baselines including the Transformer and Gated DeltaNet. Gated DeltaNet is the best recurrent baseline on FineWeb-Edu, but still trails Falcon-1.3. Falcon-1A.3 is the best inner-product variant, with perplexity close to the top recurrent baseline. QK-RMSNorm improves FineWeb-Edu perplexity over QK-l2 normalization in the small scalar ablation.

On the 124M-130M downstream evaluation, the proposed Falcon variants achieve competitive zero-shot and one-shot averages, with Falcon-1A.2 leading zero-shot and Falcon-1.3 leading one-shot among recurrent models. The results show no uniform win over baselines, but the aligned, normalized updates preserve language-model quality while enabling controlled arithmetic gains. Falcon-1A.2 achieves the best zero-shot average among listed models, while Falcon-1.3 has the best recurrent one-shot average. Among recurrent baselines, DeltaNet and Mamba-2 outperform RetNet/LightningAttn on the zero-shot average. The proposed variants do not uniformly beat baselines, but maintain competitive downstream accuracy across tasks.

On variable-length multi-digit addition, the proposed Falcon variants generally match or exceed baseline models in validation accuracy, while showing stronger extrapolation to longer digit lengths. Among the listed models, Falcon-1A.3 achieves the highest mean accuracy and the best accuracy on the longest tested lengths, surpassing both the Transformer and recurrent baselines. Falcon-1A.3 attains the highest mean accuracy among all listed models, outperforming the Transformer and RetNet/LightningAttn. All Falcon variants reach near-perfect validation accuracy, with Falcon-1A.3 slightly lower at 99.8%. On the longest digit lengths, Falcon-1A.3 achieves the best accuracy, while the Transformer and Mamba-2 lag notably.

The evaluation covers language modeling, downstream tasks, and arithmetic reasoning. At 124M-130M parameters, Falcon-1.3 achieves the best FineWeb-Edu perplexity among all baselines, with Gated DeltaNet as the strongest recurrent competitor and Falcon-1A.3 leading inner-product variants. On downstream zero-shot and one-shot averages, Falcon variants remain competitive but do not uniformly outperform baselines, with Falcon-1A.2 leading zero-shot and Falcon-1.3 leading one-shot among recurrent models. In variable-length multi-digit addition, Falcon variants match or exceed baselines in validation accuracy and show stronger extrapolation to longer digit lengths, with Falcon-1A.3 achieving the highest mean accuracy and best performance on the longest tested lengths. Overall, the aligned, normalized updates preserve language-model quality while enabling controlled arithmetic gains.

PDF source Voir le code

Table des matières

Créer de l'IA avec l'IA

De l'idée au lancement — accélérez votre développement IA avec le co-codage IA gratuit, un environnement prêt à l'emploi et le meilleur prix pour les GPU.

Codage assisté par IA

GPU prêts à l’emploi

Tarifs les plus avantageux

Commencer Voir les tarifs

HyperAI Newsletters

Abonnez-vous à nos dernières mises à jour

Nous vous enverrons les dernières mises à jour de la semaine dans votre boîte de réception à neuf heures chaque lundi matin

Propulsé par MailChimp

Command Palette

Attention à poids rapides pour l'apprentissage continu

Yifan Zhang Steve Ta Jasper Zhang Jichen Feng Shuzhen Li Yongxin Zhang Yifeng Liu Huizhuo Yuan Mengdi Wang Quanquan Gu1 more