One-sentence Summary

The MiMo Team at Xiaomi introduces a full-pipeline inference optimization for the MiMo-V2.5 model family, achieving strict $^{??}(^{??})$??(??) SWA storage and high cache hit rates through KVCache optimizations (layerwise prefetch, SWA-aware prefix cache trees), a GCache distributed cache with RDMA-optimized networking, and a KVCache-affinity router, making it the first production serving system for the Hybrid SWA + MoE + multimodal composite architecture.

Key Contributions

• A KVCache optimization scheme for Hybrid Sliding Window Attention combines layerwise prefetch, SWA-aware prefix cache trees, and specialized placement strategies, achieving near-optimal storage and high cache hit rates in production.
• GCache, a distributed cache infrastructure with RDMA-optimized networking, and a KVCache-affinity router reduce redundant computation while preserving load balancing across the serving cluster.
• The system incorporates multimodal optimizations such as GPU image preprocessing, parallel video decoding, and multimodal cache sharing, and constitutes the first large-scale production serving system to fully cover the Hybrid SWA, sparse MoE, and multimodal architecture. Cost savings are passed to users through API price reductions, and a subset of the optimizations is contributed to the SGLang open-source community.

Introduction

The MiMo-V2.5 model family combines Hybrid Sliding Window Attention (Hybrid SWA), sparse mixture-of-experts (MoE), and multimodal encoders to achieve strong efficiency and long-context reasoning. While Hybrid SWA theoretically reduces attention compute and KVCache storage to roughly one-seventh that of full attention, production systems struggle to realize these gains because of complex KVCache hit-rate management, prefix matching, dual-semantic consistency between full-attention and SWA layers, data movement across storage tiers, and distributed cache synchronization. Additional bottlenecks arise from MoE load balancing and multimodal encoder throughput. The authors address these gaps with an end-to-end inference engineering practice that systematically optimizes KVCache management, tiered caching, SWA-aware prefix cache trees, scheduling, Prefill/Decode pipelines, and multimodal serving, translating the architecture’s theoretical efficiency into real-world, long-context production performance.

Method

The authors begin by analyzing KVCache memory usage, observing that Sliding Window Attention (SWA) layers only need to retain key-value pairs within the sliding window rather than the full sequence. This structural sparsity drops KVCache memory usage to nearly one-seventh of traditional implementations. Since the decode phase is predominantly memory-bound, this reduction translates almost directly into lower inference costs for long sequences.

As shown in the figure below, KVCache storage varies substantially across different model architectures. Within their respective parameter-scale groups, the MiMo-V2.5 models demonstrate the second-lowest estimated KV cache memory requirements, highlighting the efficiency of their hybrid architecture.

To fully exploit this structural advantage, the authors refactor the caching system with a KVCache Dual-Pool Design. Hybrid SWA introduces a fundamental storage conflict: Full Attention layers require storing the full sequence KV, while SWA layers only need the sliding window. Under a traditional single-pool design, the system must allocate memory for the full sequence across all layers, negating the benefits of SWA. The authors resolve this by splitting the KVCache into two independent physical pools. The SWA pool is sized strictly for the window and supports independent eviction, while the logical layer exposes a single sequence view to upper layers. The system validates capacity constraints for both pools during request admission and performs cross-tier data movement based solely on the SWA mask to avoid redundant bandwidth consumption.

To support this optimized caching strategy at scale, the team developed GCache, a high-performance distributed cache infrastructure.

Refer to the framework diagram. GCache employs decentralized metadata management where consistent hashing on keys determines storage locations, enabling unlimited cluster scaling. A Raft-based Master handles service discovery and heartbeats, but IO paths bypass it entirely. The architecture supports both memory and disk caching on the server side; cold data is evicted to disk while hot data is promoted to memory, which is highly favorable for inference scenarios with varying session activity. Furthermore, the multi-language SDK utilizes dedicated threads for request slicing and dispatch, ensuring IO sizes remain within RDMA-friendly ranges and improving concurrency without consuming user thread resources.

Beyond caching, the authors address scheduling and prefill bottlenecks. While the hybrid architecture improves compute efficiency, throughput still degrades noticeably as sequence length increases.

As illustrated in the figure below, relative prefill throughput drops significantly as the cache sequence length grows under a fixed 16K compute chunk. In agentic scenarios, ultra-long requests often coexist with shorter ones. When scheduled to the same model instance, short requests become bottlenecked by long ones during DP-Attention synchronization and Chunked Prefill interference.

To mitigate these load imbalance issues, the authors implement a three-tier length bucketing strategy, categorizing requests into buckets of 0–64K, 64K–256K, and 256K–1M tokens. By aggregating requests with similar load characteristics into the same bucket for computation, they prevent short requests from being dragged down by long-prefix computations, significantly improving average production prefill throughput. Additionally, they optimize parallelism configuration by reducing Expert Parallelism size, which was previously constrained by the need to store full KVCache, further boosting end-to-end performance.

Experiment

These experiments evaluate inference system improvements for agentic scenarios. The SWA KVCache design extends cache TTL and achieves high hit rates by reducing storage pressure. Length bucketing resolves load imbalance among requests of varying lengths, substantially improving prefill throughput. GPU memory optimizations expand effective KVCache capacity, enabling higher decode concurrency. Activating MTP during prefill delivers a notable speedup for early token generation, especially in short-output contexts.

Encoder throughput doubled from 15 to 30 queries per second after optimization. Average latency remained nearly unchanged, while P90 tail latency dropped significantly, indicating improved consistency under load. Throughput improved by 100% with no regression in average latency. P90 latency decreased by approximately 18%, showing better tail performance.

The experiment evaluated encoder performance under load by measuring throughput and latency. Optimization doubled throughput with no regression in average latency, while tail latency decreased significantly, demonstrating improved consistency and efficiency.