One-sentence Summary

Researchers from Columbia University et al. propose BrainLoRA, a parameter-efficient fine-tuning method that adapts multimodal backbones up to 104B parameters for brain encoding via low-rank adapters, lightweight readouts, and single-pass causal multi-timepoint prediction, achieving state-of-the-art Algonauts 2025 performance with about 5% trainable parameters and ranking first with a 200+ model ensemble.

Key Contributions

• The paper presents BrainLoRA, a parameter-efficient fine-tuning method that adapts large pretrained multimodal backbones for brain encoding using low-rank adapters and a lightweight readout. It improves brain encoding across nine multimodal backbones and achieves the best published Algonauts 2025 Challenge performance on the same Qwen3-Omni backbone used by prior work while using roughly 5% of the trainable parameters.
• The work makes large-scale adaptation computationally practical by exploiting the backbone's causal structure for single-pass multi-timepoint feature extraction, reducing feature-extraction cost by 350x compared with prior state of the art, and by introducing a multi-LoRA kernel for batched adapter training. This supports adapting backbones up to 104B parameters and training an ensemble of more than 200 models that ranks first on the public Algonauts post-challenge leaderboard.
• The study shows that adapted multimodal backbones remain interpretable: modality attribution recovers canonical sensory topography normally obtained by combining separate unimodal encoders, and adaptation increases audio input contributions and cross-modal mixing compared with frozen backbones.

Introduction

Brain encoding models predict neural responses to stimuli and are usually built by freezing a large pretrained model as a feature extractor and training only a readout on neural data. This approach benefits from strong pretrained priors but cannot specialize the backbone to brain data, and adapting backbones is difficult because neural recordings are costly and scarce. Prior adaptation results have been mixed: training smaller models from scratch does not reach state-of-the-art whole-brain performance, full fine-tuning can underperform freezing, and low-rank adaptation had only been tested on smaller vision models. The authors address this gap by treating parameter-efficient fine-tuning as a scalable framework for brain encoding. They introduce BrainLoRA, which pairs LoRA with a lightweight readout on large multimodal backbones, and make adaptation practical through single-pass feature extraction that reduces cost by roughly 350 times. BrainLoRA improves over frozen backbones across nine multimodal models up to 104B parameters and sets a new state of the art on the Algonauts 2025 whole-brain fMRI benchmark while using about 5 percent of the trainable parameters of the prior best method.

Method

The authors formalize brain encoding as a mapping from a multimodal stimulus $X$X to the evoked neural response $Y$Y. The model consists of a pretrained backbone $f_\theta$fθ​ followed by a readout $g_\phi$gϕ​, trained by minimizing

where $\mathcal{L}$L is a task-appropriate loss such as squared error or negative correlation, and $g_\phi$gϕ​ also handles temporal alignment between the stimulus representation and the neural sampling rate.

Instead of full fine-tuning, the method constrains backbone adaptation to a low-dimensional parameterization. Given pretrained weights $\theta_0$θ0​, the optimization set satisfies

with $k \ll d$k≪d and $\Delta(0) = 0$Δ(0)=0. This formulation is architecture-agnostic, and LoRA, adapter layers, and prompt tuning are all instances of $\Delta$Δ. For the main architecture based on Qwen3-Omni, the authors insert rank-16 LoRA into attention modules through layer 32 and truncate the decoder at that layer, introducing 8.91 million trainable parameters. The adapters and readout are jointly optimized for one epoch.

The feature extraction pipeline replaces the overlapping sliding-window protocol used by earlier methods. Because the backbone is causally masked, a 149-second window, corresponding to 100 fMRI TRs, is encoded with a single forward pass, and representations for all timesteps are extracted simultaneously. A 60-second context lead is prepended to each window to recover the temporal context lost at window boundaries. This single-pass design reduces feature extraction cost by roughly 350 times and makes fine-tuning through the backbone computationally affordable. The authors also verify that the locally bidirectional modality encoders, which operate over short audio and visual blocks before the decoder, introduce only small numerical differences rather than meaningful future stimulus leakage.

For the readout, backbone features are first pooled into one vector per modality at each 2 Hz timestep and aggregated into eight log-spaced temporal-lag blocks aligned with the fMRI sampling rate. The linear readout pools roughly 19.3k tokens to 894 vectors per input window and then applies a final linear projection to 1,000 brain parcels, with 52.3 million trainable parameters. As an alternative, per-TR cross-attention over an expanded set of about 3.2k tokens replaces the fixed temporal-lag aggregation, adding only 0.2 million parameters.

After joint optimization with SGD, the linear readout $W$W is re-estimated in closed form. Forward passes accumulate subject-specific Gram matrices, after which $W$W is solved using parcel-wise ridge regression. This closed-form re-fit provides an additional improvement beyond the SGD-trained readout.

Experiment

Using mean Pearson correlation across four subjects on an audiovisual fMRI benchmark, the experiments evaluate parameter-efficient adaptation across nine multimodal backbones, compare small adapters against large readouts, ablate model components and input design, interpret modality contributions, and study large adapter ensembles. The findings show that PEFT consistently improves brain encoding over frozen backbones, with lightweight LoRA or cross-attention adapters outperforming much larger frozen readouts and setting a new state of the art, while efficient single-pass inputs recover lost performance after adaptation. Feature attribution recovers canonical cortical organization but adaptation shifts modality weighting toward audio and increases cross-modal mixing. Large ensembles of independently adapted models further improve accuracy, with useful diversity arising mainly from architectural changes rather than seed or training variations.

BrainLoRA sets a new state of the art on the Algonauts 2025 benchmark by adapting a frozen backbone with lightweight LoRA modules instead of training a large readout. A LoRA plus linear adapter with a small fraction of MIRAGE's trainable parameters already outperforms the previous single-model state of the art on held-out ID and OOD test sets, and a cross-attention variant improves further. Full fine-tuning of the feature extraction stack does not beat LoRA, while ensembling many independently adapted models yields the highest public leaderboard score. A LoRA plus linear adapter trains roughly 5 percent of the prior MIRAGE model's parameters and already surpasses it on both ID and OOD held-out scores. A cross-attention readout further improves single-model performance, exceeding the published 15-model MIRAGE ensemble on ID and matching it on OOD. Full fine-tuning with hundreds of times more trainable parameters performs on par with or slightly below LoRA, and a large ensemble of adapters achieves the top public leaderboard result.

The ablation shows that adapting the frozen backbone with LoRA yields the largest performance gain, and the closed-form re-fit adds a smaller but consistent improvement. Alternative readout and rank choices produce modest differences, with cross-attention showing the highest score among the readout variants. Removing audio causes a substantial drop, indicating its importance to the encoding model. LoRA adaptation provides the largest single improvement over the frozen-backbone baseline. Removing the closed-form re-fit lowers the adapted model score, confirming its consistent contribution. Rank-32 LoRA and gated-MLP readout perform close to the main configuration, while cross-attention readout scores slightly higher. No audio leads to a substantial performance drop relative to the main configuration.

These experiments evaluate BrainLoRA on the Algonauts 2025 benchmark, using a frozen backbone adapted with lightweight LoRA modules and different readouts. LoRA adaptation provides the largest gain over the frozen baseline, while the closed-form re-fit adds a smaller but consistent improvement and removing audio causes a substantial performance drop. A LoRA plus linear adapter already surpasses the prior MIRAGE model with far fewer parameters, and a cross-attention readout improves single-model performance further. Full fine-tuning does not outperform LoRA, and ensembling adapted models yields the best leaderboard result.