PyTorch continues its Profiling in PyTorch series with Part 3, focusing on the performance analysis of attention mechanisms within Transformer architectures. Utilizing an NVIDIA A100-SXM4-80GB GPU, this analysis benchmarks naive implementations against PyTorch's scaled dot product attention backends, demonstrating how algorithmic design and backend selection manifest in profiler traces. A hand-crafted naive attention module reveals the primitive operations underlying self-attention: matrix multiplications for score computation and value reweighting, scaling, causal masking, and softmax normalization. Profiling this implementation yields five GPU kernels but exposes a hidden memory copy kernel caused by the out-of-place masked fill operation. PyTorch defaults to out-of-place operations to preserve tensor values required for the backward pass. Switching to the in-place masked fill variant eliminates the memory copy kernel and reduces CPU overhead. This optimization, safe within a no-grad inference context, illustrates how minor implementation choices can introduce significant hidden latency in deep learning workloads. PyTorch's scaled dot product attention function abstracts these primitives into a unified API, dispatching to specialized backends: Math, Efficient, Flash, and cuDNN. The Math backend serves as the reference implementation, prioritizing numerical stability over performance. Traces show this backend launching 20 GPU kernels, significantly more than the naive version. Performance degrades notably because the Math backend upcasts inputs to FP32, bypassing A100 Tensor Cores in favor of general CUDA cores. Furthermore, it materializes the causal mask on every forward pass and employs a NaN-safe softmax variant, introducing overhead kernels for row-wise normalization and safety checks. This comparison highlights how API convenience can obscure computational inefficiencies. The Efficient backend, upstreamed from Meta's xformers library, collapses the attention pipeline into a single fused kernel operating in bfloat16. This approach leverages Tensor Cores and minimizes memory bandwidth pressure. Similarly, the Flash backend utilizes FlashAttention-2 to fuse operations into one kernel. Profiler traces indicate low occupancy, approximately 13 percent, for Flash. This metric is a deliberate design feature; the kernel consumes high register and shared memory resources per block to keep attention tiles on-chip. By avoiding the materialization of the full score matrix in high bandwidth memory, Flash drastically reduces data movement, which typically dominates attention latency. The cuDNN backend generates kernels dynamically tuned to specific problem dimensions. Traces reveal zero transpose operations due to native tensor layout support and the use of the driver-level cuLaunchKernelEx API. Reported occupancy reads zero percent due to profiling measurement gaps, while the footprint confirms high resource utilization per block. While cuDNN matches Flash in GPU efficiency for many shapes, it incurs higher CPU overhead as the runtime engine selects optimization plans dynamically. This processing shift results in a distinct CPU bottleneck, demonstrating that reduced kernel count does not always equate to lower overall system cost. Comparative analysis confirms that fused kernels reduce launch overhead and high bandwidth memory traffic. The Math backend's relative slowdown underscores the cost of upcasting and intermediate matrix materialization. Engineers are advised to form hypotheses before examining traces; mismatches between expectations and profiler data, such as hidden memory copies or opaque fused kernels, often reveal critical optimization opportunities. Profiling remains an essential discipline for diagnosing performance bottlenecks and validating architectural decisions in large model development.