One-sentence Summary

Researchers from the National University of Singapore and the University of Oxford propose Flide, an exact decision-inference engine that decouples logical request trees from flattened-token execution, where ForkAttention uses GPU-resident indices across standard attention and recurrent Gated DeltaNet layers to ensure branches match isolated forward passes, achieving up to $9.5$9.5 times faster than vLLM and SGLang while staying closest to a float32 reference for agent decision intelligence.

Key Contributions

• Flide is an efficient and exact inference engine for decision models that decouples the logical tree structure of a request from its physical execution within a single flattened token stream, eliminating redundant prefix recomputation and delimiter-induced output distortion.
• ForkAttention uses GPU-resident indices to navigate request dependencies across standard attention and recurrent Gated DeltaNet layers, guaranteeing in exact arithmetic that every query branch evaluates identically to an isolated forward pass.
• By expressing branch topologies in GPU-resident metadata with a compact set of pre-recorded execution graphs, Flide supports mixed request workloads and lets expanding agent histories resume from checkpoints saved in earlier passes. Evaluations across hybrid and full-attention architectures, decision benchmarks, and real-world agent trajectories report up to 9.5 times faster query answering than vLLM and SGLang while staying closest to a float32 reference in total variation distance.

Introduction

Large language models increasingly act as decision engines for routing, evaluating responses, and selecting agent actions, where they output probability distributions over discrete choices rather than long text. Prior serving systems are optimized for single-sequence generation: they either recompute shared context for every question or append delimiter tokens that alter hidden states and shift probabilities; hybrid recurrent states are also restored only at coarse boundaries, and batch-dependent reduction order can change decisions. To close this gap, the authors propose Flide, an inference engine that keeps each request as a logical tree in GPU metadata but executes all branches together as one flat token stream. Its ForkAttention mechanism propagates attention and Gated DeltaNet states so each question computes exactly what it would compute alone, avoiding delimiter corruption and repeated prefix work while supporting optional batch-invariant floating-point behavior.

Method

The authors introduce Flide, a system designed to efficiently process decision requests where each request consists of a shared state and multiple associated questions. Rather than processing each question as an independent sequence, Flide maintains the tree structure of each request as its logical foundation while executing every round of requests as a single flat token stream on the GPU. This approach minimizes redundant computation of the shared state and significantly reduces GPU launch overhead.

As shown in the figure below:

To ensure that every question retains its exact standalone computation without prompt modifications, the authors design a module called ForkAttention. This module structures each decision request as a computational tree, where the shared state forms a common trunk and the questions fork as independent branches. By flattening an entire round of requests into a continuous token stream without padding, Flide maps this logical tree onto the physical GPU layout. Each token carries metadata indicating its request index, branch index, and sequential position, ensuring that positional embeddings for questions match those of a standalone sequence.

For full-attention layers, Flide enforces a strict visibility rule. A query token attends to a key token only if they belong to the same request, the key resides in the shared state or the query's specific question, and the key does not appear later in the sequence. This guarantees that each question attends exclusively to its trunk and itself.

Hybrid models often replace attention with Gated DeltaNet layers, which compress all prior tokens into a single matrix-valued recurrent state $\mathbf{S}_{t}$St​ of fixed size by the gated delta rule:

where $\mathbf{q}_{t}$qt​, $\mathbf{k}_{t}$kt​, and $\mathbf{v}_{t}$vt​ are the query, key, and value vectors of the token, and the decay $\alpha_{t}$αt​ and write strength $\eta_{t}$ηt​ are computed from the token. Since this recurrent state inherently contains every earlier token, a simple mask cannot isolate sibling branches. To resolve this, Flide executes each Gated DeltaNet layer in two distinct calls over the flattened stream. The first call processes the state segments of all requests to output the recurrent state and convolution window for each request. The second call processes all question segments simultaneously, with each question continuing the recurrence from the recurrent state of its own request via an index, avoiding any state copying.

To address the challenge of minimal GPU launch overhead when handling rounds with highly variable numbers and lengths of requests, the authors implement Flat-Graph Execution. While CUDA graphs eliminate sequential kernel launches, they typically fix the shape of every launch. Flide reconciles this by keeping the tree structure of a round purely as GPU-resident metadata, including segment lengths, positional indices, and key ranges. The system captures a family of flat graphs indexed by stream capacity at initialization. During execution, Flide selects the smallest graph capable of accommodating the flattened stream and its metadata, allowing different request mixes to share the same execution graph without altering the launch configuration. Furthermore, a selective readout mechanism ensures that the output head only computes the final layer operations for the last token of every question, skipping unnecessary work.

Finally, to handle the incremental growth of an agent's state over successive steps, Flide employs Incremental State Forking. Successive steps often share most of their state trunk. Flide exploits this overlap by checkpointing the state of a request at an internal position. Each checkpoint stores the recurrent state and convolution window of every Gated DeltaNet layer, along with the keys and values of the full-attention layers. When generating the checkpoint during a forward pass, Flide divides the state segments into two levels. The Gated DeltaNet layer processes these levels sequentially, saving the final states of the first level as checkpoints. This allows a subsequent request to resume from the checkpoint, computing only the new tokens and a short uncached tail once for all questions, thereby maximizing computational efficiency.

Experiment

Across workloads built from RACE, QuALITY, MultiRC, and the Decision Index, the experiments compare Flide with vLLM, SGLang, multi-item scoring, and specialized decision-model baselines on NVIDIA H200 hardware, measuring latency, throughput, answer fidelity, accuracy, and stability. The results show that Flide is faster even for single-question states and gains substantially when many questions share one state, including full-attention models, while preserving accuracy and remaining closest to float32 reference answers. Ablations attribute these gains to state forks, CUDA graphs, flat graphs, and selective readout, and further analyses confirm advantages on dedicated decision models, avoid the fidelity loss caused by delimiter-based sharing, and scale efficiently as questions per state and state length grow.

Current decision-serving engines force tradeoffs among one-pass execution, same-input semantics, hybrid forking, graph replay, and bitwise reproducibility. Flide is the only documented engine that satisfies all five properties at once, while SGLang MIS supports one-pass hybrid forks but loses same-input semantics and graph replay. Other engines retain same-input semantics and graph replay but cannot fork hybrid state or complete decision serving in a single pass. Flide is the only listed engine with all five capabilities: one pass, same input, hybrid fork, graph replay, and bitwise outputs. SGLang MIS achieves one-pass hybrid forking and bitwise outputs but lacks same-input semantics and graph replay. vLLM, SGLang, and vllm-jev preserve same-input semantics and graph replay but do not support one-pass or hybrid fork.

The workloads range from single-question requests on short passages to multi-question requests on long documents, with some settings asking dozens of questions per state. In the stream prompt, the state is included once and each question is appended once, while the one-sequence-per-question prompt repeats the state for every question. As a result, stream token counts remain modest relative to sequence token counts, and the gap widens as questions per request increase. Per-request question counts span from a single question to dozens, with MultiRC the most question-dense workload. In single-question workloads, stream and one-sequence-per-question prompt lengths are identical, but in multi-question workloads sequence prompt lengths become several times larger than stream prompt lengths.

Flide outperforms vLLM and SGLang across evaluated latency and throughput settings, including single-question states and full-attention models. Its advantage is greatest when many questions share one state, especially on question-dense workloads such as QuALITY and MultiRC, where it avoids repeated computation or restoration of shared context. Flide reduces latency even when a state has only one question, with the largest lone-request speedup on the smallest Qwen3.5 model. For multiple questions per state, Flide achieves lower latency and higher throughput than the best baseline at all evaluated model sizes and concurrency levels, with the biggest gains on QuALITY and MultiRC. Flide remains faster than vLLM's prefix cache on full-attention Gemma 4, indicating an advantage beyond caching from processing the whole request in one forward pass.

Across Qwen3.5 model sizes and workloads, Flide's top answer agrees with the float32 reference more often than vLLM or SGLang, and its accuracy stays within 1.2 points of vLLM in every cell. Batching changes answer stability for all engines, but Flide shifts the least, and in its fixed computation mode it returns identical bits at any load. Flide has the highest top-answer agreement with the float32 reference across every evaluated workload. Flide's accuracy remains within 1.2 points of vLLM in every cell while maintaining its speed advantage. Flide's answer stability changes by at most 0.2 percent between one and 64 clients, compared with larger shifts for SGLang and vLLM.

An ablation on Qwen3.5-4B isolates the performance contribution of each Flide component. Removing state forks causes the largest latency penalty on multi-question QuALITY workloads, while removing CUDA graphs severely hurts short-input latency. Disabling flat graphs cuts short RACE throughput by more than a third, and selective readout adds further throughput gains across workloads. Without state forks, multi-question QuALITY latency increases by more than an order of magnitude. Without CUDA graphs, a single request launches thousands of sequential kernels and incurs a severe short-input latency penalty. Without flat graphs, short RACE throughput drops by over one third because every request requires a distinct replay. Selective readout delivers additional throughput gains by avoiding redundant token projections across all workloads.

The evaluation first establishes that Flide uniquely combines one-pass execution, same-input semantics, hybrid forking, graph replay, and bitwise reproducibility, while SGLang MIS and other engines each miss some capabilities. Across workloads from single-question short passages to many-question long documents, stream prompts keep token counts much lower than repeated-state sequence prompts. Flide outperforms vLLM and SGLang in latency and throughput, especially on question-dense multi-question workloads, and also shows higher top-answer agreement with the float32 reference and more stable answers under batching. An ablation on Qwen3.5-4B attributes the largest multi-question gains to state forks, short-input latency gains to CUDA graphs, throughput gains to flat graphs, and further throughput gains to selective readout.