One-sentence Summary

Researchers at Nagoya University's Institute of Materials and Systems for Sustainability propose Cumulant Attention, an attention mechanism that applies cumulant expansion from statistical mechanics to approximate softmax attention at linear computational complexity while preserving its nonlinearity, outperforming linear attention and matching softmax accuracy on ViT-based image and video classification tasks.

Key Contributions

• The paper introduces Cumulant Attention, an attention mechanism that incorporates a statistical-mechanics perspective and cumulant expansion into the transformer attention layer to balance efficiency and accuracy.

• A low-order variant of Cumulant Attention reduces attention computation to linear order while preserving nonlinear behavior associated with softmax attention.

• Experiments on image classification with ViT on ImageNet-100 and video classification with ViViT on UCF-101 show that Cumulant Attention outperforms linear attention and achieves accuracy comparable to softmax attention.

Introduction

Transformer attention powers modern models across language, vision, speech, graphs, and multimodal tasks, but its standard softmax attention scales quadratically with sequence length, creating a serious efficiency bottleneck for large-scale data. Prior efficient attention methods, such as linear attention, reduce cost through kernelized approximations, yet they can struggle with spatial selectivity and local feature extraction in computer vision. The authors leverage ideas from statistical mechanics to design an efficient attention mechanism that interprets attention through physical concepts and cumulants, aiming to retain expressive relevance modeling while reducing computational overhead. Their method achieves strong performance on CV tasks, outperforming linear attention and approaching softmax attention accuracy.

Method

The authors recast standard scaled dot product attention as a gradient of a statistical mechanical free energy. They consider the tokens $x^l$xl as possible states of a random variable $X$X, uniformly drawn from the set of $L$L keys. The query side is treated as an external field, with the field direction determined by the query-key interaction through the usual key and query projections. The partition function is defined as

and the log partition function

plays the role of a negative free-energy potential. Differentiating this quantity with respect to the external field gives the normalized expectation under a Boltzmann-type distribution,

After applying the value projection and the usual query-key scaling, this derivative reproduces the attention output,

Thus, attention is interpreted not merely as a learned mixture of value vectors, but as the response of a physical system to an external field. The softmax denominator corresponds to the partition function, and attention corresponds to the derivative of the free-energy potential.

The second central idea is to approximate the free-energy potential using a cumulant expansion. Cumulants are coefficients in the power series expansion of $\log Z(p)$logZ(p) and represent correlations among random variables. The first cumulant is the mean, the second cumulant is the covariance, and higher cumulants describe genuine higher-order statistical dependence. The expansion takes the form

where the third central cumulant is

The method truncates this expansion to obtain a controlled approximation to attention. Keeping only the first and second cumulants yields the second-order cumulant attention

where $\bar{k}$kˉ and $\bar{v}$vˉ denote the average key and value vectors, respectively. This expression replaces the softmax normalization with an additive correction built from centered key-value statistics. The leading correction to this approximation comes from the diagonal part of the third cumulant, producing

In this formulation, the approximation operates through moments of the key and value embeddings. Second-order attention corresponds to retaining mean and covariance information, which is equivalent to assuming that the underlying statistical variables follow a multivariate normal distribution. The third-order term introduces additional dependence on higher-order embedding correlations and serves as a systematic correction to that Gaussian approximation. A notable property of the cumulant formulation is that the linked-cluster theorem removes the denominator, yielding an additive approximation whose structure is determined by centered key-value moments.

Experiment

Experiments evaluated Cumulant Attention on image and video classification using ViT on ImageNet-100 and ViViT on UCF-101. The results show that cumulant attention consistently outperforms linear attention and approaches softmax attention, reflecting its selective capability and more localized attention patterns. A similar trend is observed for video classification, although the model tends to overfit because of the small dataset size.