One-sentence Summary

HSE University and ITMO University researchers propose a physiologically constrained MEG speech decoder that replaces 2D Fourier attention with a spherical harmonics layer, adds trainable temporal filters, and reduces subject-specific branches to $K = 25$K=25, mapping learned weights to canonical speech-perception cortical sources while achieving $39.75 \pm 0.34\%$39.75±0.34% Top-1 accuracy with approximately $20\times$20× fewer parameters, and occlusion reveals that silence, sound intensity, vowels, and acoustic onsets drive retrieval most while narrative structure strongly impacts recoverable information.

Key Contributions

• A decoder with a spherical-harmonics front end and trainable temporal filters reaches 39.75% top-1 accuracy on MEG-MASC, using roughly 20× fewer parameters than previous work.
• The architecture’s interpretability allows learned weights to be mapped to cortical sources, recovering the canonical speech-perception network and showing that left-hemisphere branches carry higher-frequency rhythmic dynamics.
• Paired MEG occlusion and narrative-substitution experiments reveal that the model relies on acoustic features such as silence, intensity, vowels, and onsets, that coherent speech context improves retrievable information, and that feature-use patterns remain stable across six training seeds.

Introduction

Decoding perceived speech from non-invasive magnetoencephalography (MEG) has reached high retrieval accuracy, but the resulting models remain opaque: their learned weights do not correspond to recognizable neural sources, rhythms, or time courses, so the high scores cannot be linked to specific cortical computations. While compact factorized architectures that separate spatial from temporal filtering have been used in brain decoding, earlier attempts to interpret them overlooked the mutual dependence of jointly trained filters and were not applied to whole-head MEG during complex natural speech tasks. The authors address this gap by extending a physiologically grounded front-end that factorizes spatial and temporal processing, equipping it with spherical-harmonic attention, trainable depthwise temporal filters, a small bottleneck of 25 branches, and explicit removal of ocular and cardiac artifacts. This architecture achieves competitive retrieval accuracy while allowing the trained weights to be directly mapped onto cortical source locations and their second-order dynamics. Through paired MEG substitution experiments, the authors further reveal which stimulus properties — spanning acoustics, phonetics, and contextual surprisal — the decoder actually uses, transforming the network from a black-box benchmark into an instrument for neurophysiological discovery.

Dataset

The authors use the MEG-MASC dataset, a collection of simultaneous audio and magnetoencephalography (MEG) recordings from 27 English-speaking participants listening to narrated stories from the MASC corpus. The dataset comprises 49 session recordings (22 participants contributed two sessions, five contributed one), each roughly one hour long.

Key dataset characteristics and processing steps:

• Audio preprocessing: The speech is resampled to 16 kHz and cut into 3-second windows with a 1-second stride. Windows whose peak absolute amplitude falls below $10^{-4}$10−4 are discarded. A window is kept only if at least 50 % of the duration of one or more annotated words falls inside it.
• MEG preprocessing: Ocular and cardiac ICA components are removed. The signals are downsampled from 1000 Hz to 100 Hz. For each participant–session–story recording, the per-channel mean over the 0.5 s before the first stimulus onset is subtracted, channels are robust-scaled with the median and interquartile range, standardized to zero mean and unit variance, and clipped to ±20 standard deviations.
• Target representation: For every retained audio window, the target is obtained by passing the audio through the wav2vec 2.0 Base model and averaging the outputs of the last four hidden layers at each model time step.
• Train/validation/test split:
  • The development set (2698 segments) consists of the full stories LW1, Cable Spool Fort, and Easy Money, plus the first five pieces of Black Willow. Within this set, the fifth piece of Black Willow is held out as the validation set.
  • The test set (1005 segments) comprises the last seven pieces of Black Willow.
  • For Black Willow, scaling parameters are fitted only on samples before the first test piece to avoid leakage.
• Audio–MEG pairing: Each 3-second audio segment is paired with the 3-second MEG segment starting 150 ms later to account for auditory response latency.
• Test segment alignment: Unlike common practice, test segments are not aligned to word onsets, making the retrieval setting more challenging.

Method

The authors address the retrieval task by training a network to construct embeddings for MEG data that align with audio embeddings produced by wav2vec 2.0, utilizing a CLIP-style objective. The proposed architecture replaces standard spatial-attention layers with a physically motivated 3D spatial attention layer, augments the model with a temporal-filtering layer, and modifies the convolutional decoder.

As shown in the figure below:

The interpretable front-end processes the input MEG data through a factorized spatial-temporal structure. The spatial filtering stage begins with a 3D spatial attention layer. Because MEG sensors occupy a three-dimensional, approximately spherical arrangement, the authors parameterize each of the $J = 270$J=270 virtual channels using real spherical harmonics. The unnormalized coefficient for virtual channel $j$j and sensor $m$m is computed as:

$$c_{jm} = \sum_{\ell=0}^{L-1} \sum_{q=-\ell}^{\ell} \gamma_j^{q,\ell} Y_\ell^q(\theta_m, \varphi_m)$$cjm​=ℓ=0∑L−1​q=−ℓ∑ℓ​γjq,ℓ​Yℓq​(θm​,φm​)

where $(\theta_m, \varphi_m)$(θm​,φm​) are the polar and azimuthal angles of sensor $m$m, $Y_\ell^q$Yℓq​ is a real spherical-harmonic basis function, and $\gamma_j^{q,\ell}$γjq,ℓ​ is a learned parameter. The coefficients are normalized across the $M$M sensors using a softmax function and applied to the input signal.

Following the spatial attention, a shared $1 \times 1$1×1 unmixing convolution applies a learned affine transformation in the channel space. A subject-specific layer then projects this representation to $K$K interpretable branches. The effective participant-specific spatial filtering matrix is defined as $\mathbf{W}^{(s)} = \mathbf{W}_s \mathbf{W}_u \mathbf{C}$W(s)=Ws​Wu​C, and the corresponding branch-wise bias is $\mathbf{b}^{(s)} = \mathbf{W}_s \mathbf{b}_u$b(s)=Ws​bu​. The branch signals before temporal filtering are computed as $\mathbf{a}_s(t) = \mathbf{W}^{(s)} \mathbf{x}_s(t) + \mathbf{b}^{(s)}$as​(t)=W(s)xs​(t)+b(s).

Refer to the framework diagram:

The front-end is designed as a collection of branches where each branch adapts to a particular neural source with specific spatial and dynamical properties. To target specific frequency ranges, the authors apply one trainable 1-D depthwise temporal filter to each of the $K$K branch signals. Each filter has 15 samples, corresponding to 150 ms at the MEG sampling rate of 100 Hz. The temporal filters are shared across participants, whereas the preceding spatial projection is participant-specific. The output of branch $k$k is obtained by applying its temporal filter to the spatially filtered signal:

$$r_{s,k}(t) = (a_{s,k} * h_k)(t)$$rs,k​(t)=(as,k​∗hk​)(t)

The $K$K branch-wise signals produced by the interpretable front-end are then passed to a non-linear temporal decoder. This decoder comprises $B$B temporal convolutional blocks followed by a convolutional head. Each temporal block contains three one-dimensional convolutions with specific dilation factors, batch normalization, and GELU activation. The convolutional head projects the decoder channels to the 768-dimensional wav2vec feature channels.

The training objective is a one-directional MEG-to-audio contrastive cross-entropy loss. MEG-derived embeddings are compared with the unique audio targets represented in the current minibatch. Similarities are computed after $L_2$L2​ normalization over the feature-time dimensions and divided by a learned temperature parameter. The models are trained using the AdamW optimizer with early stopping based on validation loss.

To understand the contribution of the spatial and temporal factorization, the authors perform architectural ablations of the front-end components.

As shown in the figure below:

The ablation results indicate that the full spatial-temporal factorization performs best. The largest degradation in retrieval accuracy occurs when subject-conditioned spatial mappings are removed, highlighting the necessity of adapting the spatial projection to individual subjects due to variations in anatomy and sensor geometry. Removing the attention layer or replacing the 3D attention with a 2D version also reduces performance, supporting the use of a sensor-geometry-aware parameterization.

The authors also investigate the effect of temporal-filter support on retrieval accuracy.

As shown in the figure below:

Retrieval depends mainly on whether the filter has sufficient temporal support. The one-sample condition, which contains no temporal context, performs worse than the 150 ms default. Performance generally improves as temporal support increases up to approximately 150 ms, after which gains become less systematic, indicating that the benefit of filter length begins to saturate around this scale.

Furthermore, the authors analyze the capacity of the model by varying the number of interpretable branches $K$K and the number of convolutional blocks in the decoder.

As shown in the figure below:

Accuracy increases sharply from very small $K$K to approximately $K = 10-25$K=10−25, then enters a broad plateau. Larger values of $K$K do not produce systematic gains and can mildly degrade performance. Across decoder depths, the 0-block model is consistently weaker, while models with 2 to 5 convolutional blocks form a similar high-performing regime. The main configuration with 2 convolutional blocks and $K = 25$K=25 branches lies on this compact high-accuracy plateau.

Experiment

A series of experiments validated an interpretable MEG-to-speech retrieval decoder trained contrastively on narrative listening data, using paired occlusion, spatial clustering, and architectural ablations. The decoder relied on a compact set of stimulus features, including silence, loudness, vowels, and acoustic onsets, with spatially organized filters concentrated over bilateral auditory, frontal, and superior temporal cortices. Performance improved monotonically with longer MEG-audio segments, and the audio target representation could be drastically compressed along the feature axis through a learned low-dimensional subspace, while temporal resolution remained critical. Subject-specific spatial mapping, 3D geometry-aware attention, and temporal filtering all contributed to retrieval, with performance resting on a broad plateau across many architectural configurations.