One-sentence Summary

MiniMax Sparse Attention (MSA) is a Grouped Query Attention variant that employs a lightweight Index Branch to independently select Top-k key-value blocks per group, enabling exact block-sparse computation that matches standard performance while reducing per-token compute by 28.4x at one million tokens and delivering 14.2x prefill and 7.6x decoding speedups on H800 GPUs through a co-designed kernel leveraging exp-free Top-k selection and KV-outer sparse attention.

Key Contributions

• MiniMax Sparse Attention (MSA) is a blockwise sparse attention mechanism built on Grouped Query Attention that employs a lightweight index branch to independently score and select a top-k subset of key-value blocks for each GQA group prior to exact block-sparse attention.
• A co-designed GPU execution path implements exp-free top-k selection and KV-outer sparse attention to maximize tensor-core utilization under block-granular memory access patterns.
• Evaluations on a 109B-parameter multimodal model demonstrate that the approach maintains performance parity with standard Grouped Query Attention while reducing per-token attention compute by 28.4x at a one-million-token context, yielding 14.2x prefill and 7.6x decoding wall-clock speedups on H800 hardware.

Introduction

Long-context modeling in transformer-based language models requires efficient attention mechanisms to mitigate the quadratic computational and memory overhead of dense softmax attention. Prior approaches typically substitute attention with linear or recurrent alternatives, apply fixed content-agnostic sparse patterns, or implement adaptive sparsification that either inherits full-attention training costs or suffers from fragmented memory access and unoptimized inference kernels. The authors leverage a per-GQA-group Top-k sharing strategy combined with block-level selection to preserve contiguous KV cache reads while maintaining adaptive context awareness. They further accelerate the framework by adapting the FlashAttention algorithmic skeleton with loop ordering specifically tuned to this access pattern, effectively translating theoretical FLOP reductions into measurable wall-clock speedups.

Experiment

Two 109B-scale experiments validate replacing dense attention with a sparse mechanism by either training from scratch or continuing pretraining from a full-attention checkpoint. The native sparse approach demonstrates that the model can stably adapt its representations to learn essential attention structures without hard-coded constraints, while the continued pretraining route validates a practical and stable conversion pathway from dense checkpoints. Both methods maintain competitive performance across language, multimodal, and long-context benchmarks despite a strict key-value token budget. Ultimately, the sparse architecture delivers substantial computational efficiency and sustained long-context capabilities, establishing it as a scalable alternative to dense attention.