Transformer architectures have become the backbone of generative AI, powering models like GPT, DeepSeek, and Llama by enabling them to process entire input sequences in parallel, capturing long-range dependencies. At the heart of this capability is the self-attention mechanism, which, while powerful, suffers from quadratic computational and memory complexity. This limitation creates significant bottlenecks when handling long context windows in modern large language models. FlashAttention is an algorithmic innovation designed to overcome these challenges. It computes the same result as standard attention but does so more efficiently by being input/output-aware. FlashAttention reduces both memory usage and computation time, enabling faster training and inference, and allowing models to work with much longer sequences—critical for applications like processing high-resolution images or maintaining extended conversational context. FlashAttention-4 (FA4) is the latest evolution, specifically co-designed for NVIDIA’s Blackwell architecture, including the HGX B200. FA4 achieves a peak performance of 1,605 TFLOPS/s, utilizing 71% of the hardware’s theoretical maximum. It addresses key bottlenecks introduced by Blackwell’s asymmetric scaling, where compute power has doubled while memory bandwidth has not kept pace. FA4 leverages several hardware-specific features to deliver substantial gains. It uses Tensor Memory (TMEM), a 256 KB on-chip memory per SM, to store intermediate backward pass values like attention scores and gradients directly, drastically reducing reliance on shared memory (SMEM) and minimizing traffic. This allows for larger computation tiles—up to 128×128—and deeper pipelines, improving efficiency. The algorithm also restructures the backward pass to reduce pressure on the MUFU (Mixed-precision Unified Functional Unit), which handles computationally expensive operations like softmax exponentials. By using FMA-based polynomial approximations and software-emulated exponentials, FA4 reduces the runtime of these operations by 25–60% compared to matmul time. To maximize compute utilization, FA4 introduces fully asynchronous pipelines that overlap matrix multiplication, softmax, and memory operations. This prevents idle time in tensor cores despite sequential dependencies. It also minimizes non-matmul work through algorithmic optimizations, such as conditional softmax rescaling that only triggers when necessary. Development was accelerated using CUDA 13 and CUDA-X tooling, with the CuTe DSL in Python cutting compile times by 20–30x compared to previous versions, while maintaining kernel expressivity. Performance results show dramatic improvements. On a Blackwell GPU with a head dimension of 128, FA4 delivers a 3.6x speedup in the forward pass and a 3.15x speedup in the backward pass over FA2 at a sequence length of 32,768. These gains are especially pronounced in multi-GPU, multi-node setups. FA4 is already integrated into inference frameworks like SGLang and vLLM for prefill operations. NVIDIA has also incorporated FA4 techniques into cuDNN 9.14, enhancing deep learning performance across the board. The algorithm exemplifies the power of hardware-software co-design in unlocking the full potential of next-generation AI accelerators.