One-sentence Summary

The authors propose FreeToken, an edge-native Mixture-of-Experts serving system that co-designs model loading, expert residency, CPU-GPU execution, and memory management to continuously map computation and model state onto available heterogeneous resources rather than fixed offloading, supporting over 20 MoE models and scaling from a 35B model on a laptop to the 753B GLM-5.2 on a single workstation GPU.

Key Contributions

• FreeToken is an edge-native MoE serving system that treats a personal machine as a unified elastic inference platform across GPU, CPU, host memory, and interconnect, supporting more than 20 MoE models and real coding and tool-using agents on hardware from an 8 GB laptop GPU to a single workstation GPU.
• The system introduces bandwidth-adaptive CPU co-execution and an elastic full-expert cache with unified prefill/decode residency, derives the CPU/GPU division from measured bandwidths as a closed-form ratio that remains device-resident inside captured CUDA graphs, and applies hierarchical-memory management to the expert pool rather than fixed offloading or host-side heuristics.
• FreeToken enables models from 35B to 753B parameters on personal hardware, including a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU, while consistently outperforming existing edge serving systems and providing cross-request prefix reuse for multi-turn agentic sessions.

Introduction

Open-weight models such as DeepSeek-V4-Flash and GLM-5.2 are closing the capability gap with proprietary systems, but running them remains expensive because frontier models usually require datacenter GPUs or costly hosted APIs, and agentic applications increase inference demand. Mixture-of-experts models make local serving more feasible by activating only a few experts per token, but the full expert pool still exceeds consumer GPU memory, creating hard prefill, decode, and resource-management problems. Existing edge serving systems address only fragments of this problem and suffer from dense expert working sets during long prefills, cache misses without principled CPU/GPU partitioning, and fixed policies under variable consumer hardware. The authors propose FreeToken, a serving system that treats heterogeneous consumer machines as a unified inference platform and combines bandwidth-adaptive execution, semantic-aware caching, and elastic resource management to serve frontier-scale MoE models on personal hardware.

Method

The authors propose FreeToken, a system designed to optimize edge Mixture-of-Experts (MoE) serving by organizing computation around a two-level expert-memory hierarchy. In this architecture, the CPU-resident expert pool holds the complete routed-expert weights and serves as the source of truth, while non-expert weights remain resident on the GPU. The remaining GPU memory is transformed into a single elastic expert cache shared across all MoE layers. Each slot in this cache stores every tensor required to evaluate one layer-expert pair, allowing residency, lookup, and execution to operate on logical identifiers rather than tensor shards.

Refer to the framework diagram:

During the prefill stage, FreeToken addresses the bottlenecks of massive expert transfer and redundant recomputation through two dedicated mechanisms. First, the system employs full-layer double buffering to hide transfer latency behind computation. Because prefill activates nearly the entire expert set of every layer, FreeToken allocates two full-layer buffers from the global slot pool. While the GPU computes the routed experts of layer $l$l from one buffer, a dedicated transfer stream simultaneously loads the complete expert set of layer $l+1$l+1 into the other. This ensures continuous background weight movement. Second, to handle agentic context edits that invalidate recurrent states, the authors introduce a semantic-aware state cache. This mechanism maintains a small pool of recurrent-state checkpoints attached to nodes of a radix prefix tree. Checkpoints are strategically placed at semantic anchors, such as special-token boundaries marking thinking segments or tool calls. When a context edit occurs, the system restores from the deepest surviving checkpoint, allowing full-attention layers to reuse their KV cache and recurrent layers to resume from the anchor, thereby re-prefilling only the new suffix.

In the decode stage, FreeToken leverages the strong temporal expert locality observed across consecutive steps. Instead of static expert placement, the system maintains a shared LRU residency space that continuously tracks the experts selected by the router. A cache hit refreshes an expert's recency, while eviction removes the least recently demanded expert, ensuring scarce GPU memory tracks the current working set.

To handle cache misses that inevitably occur due to cold starts or capacity limits, FreeToken implements a bandwidth-adaptive execution strategy governed by the $q^\star$q⋆ policy. This mechanism dynamically divides the $m$m missing experts between PCIe transfer and CPU execution based on the measured pinned expert-transfer bandwidth ($B_P$BP​) and host-side expert-processing bandwidth ($B_H$BH​). The misses are split into a cache-fill set $\mathcal{F}$F and a CPU-execution set $\mathcal{C}$C. Experts in $\mathcal{F}$F are transferred to the GPU cache and executed there, while experts in $\mathcal{C}$C execute directly from the CPU-resident pool. The optimal split ratio $q^\star$q⋆ is derived to balance the concurrent branches:

This formulation ensures that cache fills proceed at the full PCIe rate while the CPU utilizes the residual host bandwidth, converting it into progress without suspending cache updates. The CPU and GPU compute their respective partial sums and merge them to preserve the exact MoE output.

Beneath these phases, an elastic expert-memory lifecycle treats GPU cache capacity as a runtime-adjustable resource. FreeToken allows runtime cache reconfiguration, enabling the system to rebuild the GPU expert cache for a revised VRAM budget at any scheduler safe point without restarting the engine. Furthermore, fast engine bootstrap is achieved by reading expert weights directly into their final host layout and pinning memory only after filling, eliminating the need for a warm-up phase.

From an implementation perspective, the authors ensure that all routing-dependent control remains strictly on the GPU to avoid costly device synchronization. For each MoE layer, a single-pass kernel deduplicates routed experts, classifies them against the residency table, derives the fetch count $q$q, and selects eviction victims. This dynamic control is represented as data inside a statically captured CUDA graph. The CPU branch of the bandwidth-adaptive execution is also captured into the same graph, utilizing a persistent C++ worker pool pinned to physical cores. To facilitate rapid loading, the system normalizes model-specific checkpoint layouts into the FreeToken Weight (FTW) format, which stores expert weights merged into a runtime bank layout, allowing parallel direct I/O straight into exact-size host banks.

Experiment

The evaluation compares FreeToken with llama.cpp, Ollama, KTransformers, and MoE-Infinity on six edge and workstation GPUs serving large MoE models that exceed VRAM across four agentic workloads, including math reasoning and coding and email agents. The main results show that FreeToken delivers higher and more stable decode throughput and much lower tail time-to-first-token than baselines under agentic serving. Breakdown experiments attribute these gains to pipelined prefill overlap and global expert-cache locality, while cross-hardware tests show the advantage persists from an 8 GB laptop GPU to a frontier-scale model on a workstation GPU.

The measured systems span consumer and workstation GPUs, covering a wide range of PCIe transfer bandwidth and CPU-side expert kernel bandwidth. Across all listed systems, the effective CPU-side expert kernel bandwidth is higher than the host-to-device PCIe transfer bandwidth. Rented server configurations are capped and pinned to emulate edge-class hosts, and their CPU-side bandwidth lands in the same range as the real desktop and laptop systems. PCIe transfer bandwidth scales from the laptop's PCIe 4.0 x8 link through the PCIe 4.0 x16 systems to the PCIe 5.0 x16 systems. The gap between CPU-side expert kernel bandwidth and PCIe transfer bandwidth is largest on the laptop, where CPU-side bandwidth is several times higher than PCIe bandwidth. The workstation system provides the largest memory capacity and the highest CPU-side expert kernel bandwidth, well above the consumer platforms.

The evaluation covers consumer laptops, desktops, workstations, and rented server configurations that are capped to emulate edge-class hosts, spanning PCIe 4.0 x8 through PCIe 5.0 x16 links. Across all systems, the effective CPU-side expert kernel bandwidth is consistently higher than the host-to-device PCIe transfer bandwidth. The gap is largest on the laptop, where CPU-side bandwidth is several times higher than PCIe bandwidth, while the workstation provides the largest memory capacity and the highest CPU-side expert kernel bandwidth.