This post explores a technique for optimizing token generation in PyTorch-based autoregressive decoder models using CUDA stream interleaving. While such models are central to modern generative AI, their sequential nature often leads to underutilized GPU compute due to host-device synchronization during early stopping checks. The method described here addresses this bottleneck by overlapping the GPU computation of the next token with the CPU-side EOS (end-of-sequence) check for the current token. The experiments use a GPT-2 model from Hugging Face’s transformers library, running on an NVIDIA L40S GPU with PyTorch 2.10.0. A baseline implementation generates tokens one at a time without optimization, resulting in O(N²) runtime complexity and high memory fragmentation. This inefficiency is mitigated through KV caching, which reduces the complexity to O(N) by reusing key and value tensors across steps. Further improvements come from expandable memory allocations and static KV caching. The latter pre-allocates cache memory, reducing fragmentation and enabling better memory management. However, it incurs computational overhead due to attention operations on full cache sizes, even for irrelevant tokens. Model compilation via torch.compile significantly boosts performance, especially when combined with static caching, as it enables JIT optimizations on fixed-size tensors. A key bottleneck is the early stopping condition. The check torch.all(finished) requires synchronizing the GPU with the CPU via .item(), which blocks the CPU and causes GPU idle time. Profiling with NVIDIA Nsight Systems reveals this idle period, which can be substantial in high-throughput scenarios. To eliminate this delay, the post introduces a CUDA stream interleaving technique. Two streams alternate in processing tokens: one stream computes the next token while the other checks the EOS condition from the previous step. By using non-blocking memory copies and explicit stream synchronization, the CPU can launch the next GPU kernel before completing the stop check. This creates a pipelined, ping-pong pattern that keeps the GPU continuously busy. The results show a measurable performance gain—up to 11.6% improvement at low batch sizes—where kernel loading time is a larger fraction of total runtime. The benefit diminishes with larger batch sizes, where computation dominates and idle time is less impactful. However, the technique introduces risks. Mismanaged stream synchronization can lead to CUDA errors or silent data corruption. Additionally, per-stream memory allocation can increase overall memory reservation and fragmentation, potentially offsetting gains in memory-constrained environments. The best performance—nearly five times faster than the baseline—is achieved by combining static caching, model compilation, and CUDA stream interleaving. Yet, the effectiveness of stream interleaving is highly dependent on workload characteristics, particularly the ratio of kernel launch overhead to compute time. In summary, CUDA stream interleaving is a powerful but delicate optimization for PyTorch-native inference. It effectively hides synchronization latency and improves GPU utilization, but must be applied with care. For production systems, dedicated inference engines like vLLM or TensorRT-LLM typically offer superior performance and stability. Still, for development, testing, or custom workflows, this technique provides a valuable tool for squeezing more throughput from PyTorch-based decoder models.