HyperAIHyperAI

Command Palette

Search for a command to run...

MMFlash: 비전-언어 모델을 위한 더 빠른 추측 디코딩

Sicheng Feng Zigeng Chen Song Wang Gongfan Fang Xinchao Wang

초록

추측 디코딩은 경량 드래프터가 제안한 토큰을 목표 모델이 병렬로 검증하도록 하여 생성을 가속화한다. DFlash와 같은 확산 드래프터는 단일 순전파로 전체 토큰 블록을 제안하지만 주로 텍스트 전용 작업에 초점을 맞추어 멀티모달 드래프팅은 충분히 탐구되지 않았다. 이 설정을 연구하기 위해, 우리는 먼저 시각 입력이 있을 때와 없을 때의 다음 토큰 분포 간 KL 발산에 기반한 토큰 수준 시각 의존성 지표를 도입한다. 우리는 수락률이 시각 의존성이 가장 낮은 사분위수에서 46%에서 가장 높은 사분위수에서 22%로 감소하며, 이러한 격차는 멀티모달 학습 후에도 지속됨을 발견한다. 이 관찰에 동기를 얻어, 우리는 드래프터가 잘 예측하지 못하는 시각 의존 토큰을 적응적으로 상향 가중하는 간단하고 효과적인 프레임워크인 MMFlash를 제안한다. 긴 비디오 입력의 경우, MMFlash는 희소 어텐션을 추가로 사용하여 중복 어텐션 계산을 줄이고 드래프팅 품질을 향상시킨다. 다양한 멀티모달 벤치마크에서 MMFlash는 최대 3.58배의 무손실 디코딩 속도 향상을 달성하여 EAGLE-3와 DFlash를 각각 최대 195.4%와 17.1% 능가한다.

One-sentence Summary

Researchers from the National University of Singapore propose MMFlash, a speculative decoding framework for vision-language models that adaptively upweights visually dependent tokens the drafter predicts poorly and applies sparse attention for long video inputs, achieving lossless speedups of up to 3.58×3.58\times3.58× and outperforming EAGLE-3 and DFlash by up to 195.4%195.4\%195.4% and 17.1%17.1\%17.1%, respectively.

Key Contributions

  • The paper introduces a token-level visual dependence metric based on the KL divergence between next-token distributions with and without visual input, showing that draft acceptance rates drop from 46% in the lowest visual-dependence quartile to 22% in the highest.
  • The paper proposes MMFlash, a diffusion drafting framework that adaptively upweights visually dependent tokens the drafter predicts poorly and employs sparse attention for long videos to reduce redundant attention computation.
  • Across seven image benchmarks and three long video benchmarks, MMFlash achieves lossless decoding speedups up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, and generalizes to other model families and scales.

Introduction

Vision-language models (VLMs) perform well on multimodal tasks such as visual question answering and video understanding, but their autoregressive decoding limits inference efficiency, especially as reasoning models generate longer responses. Speculative decoding accelerates this process by having a small drafter propose tokens for parallel verification, yet autoregressive drafters still require sequential forward passes and often sacrifice draft quality for lightweight design. Diffusion-based drafters such as DFlash can emit an entire block in one forward pass, but extending them to VLMs exposes a key limitation: acceptance rates decline sharply on visually dependent tokens even after multimodal training. The authors propose MMFlash, a diffusion drafting framework that measures visual dependence via KL divergence and reweights training to emphasize poorly predicted, visually dependent tokens, while adding sparse attention for long videos and achieving substantial decoding speedups.

Method

The authors identify a key limitation when extending block diffusion drafting to vision-language models: the acceptance rate drops significantly for tokens that heavily depend on visual inputs. To quantify this, they introduce a token-level measure of visual dependence, defined as the KL divergence between the target model's next-token distributions with and without the visual input, conditioned on the same textual prefix:

d=KL(p(⋅∣prefix,I)∥p(⋅∣prefix))d = \mathrm{KL} \big (p (\cdot \mid \text{prefix}, I) \parallel p (\cdot \mid \text{prefix}) \big)d=KL(p(⋅∣prefix,I)∥p(⋅∣prefix))

As shown in the figure below, empirical analysis reveals that as visual dependence increases, the acceptance rate of drafted tokens falls monotonically, while truncation points concentrate heavily in the most visually dependent quartiles. This observation motivates MMFlash, which explicitly directs the drafter's training capacity toward visually dependent tokens rather than treating all positions alike.

The visual dependence score dtd_tdt​ is precomputed offline for every token in the training set. To ensure the drafter focuses on tokens where the target model has a definite prediction, the score is scaled by the target's confidence, measured by the entropy HtH_tHt​ of its with-image distribution:

gt=g~t⋅exp⁡(−Ht)g_t = \tilde{g}_t \cdot \exp(-H_t)gt​=g~​t​⋅exp(−Ht​)

where g~t\tilde{g}_tg~​t​ is the quantile of dtd_tdt​ in the training dataset.

The training objective replaces the purely positional weight with a product of three factors. For the B−1B-1B−1 masked positions in a block, the weight wkw_kwk​ is defined as:

wk=bk(1+αgk(1−sg(qk)))w_k = b_k \left(1 + \alpha g_k \left(1 - \mathrm{sg}(q_k)\right)\right)wk​=bk​(1+αgk​(1−sg(qk​))) LMMFlash=∑k=1B−1wk(−log⁡qk)\mathcal{L}_{\mathrm{MMFlash}} = \sum_{k=1}^{B-1} w_k \left(-\log q_k\right)LMMFlash​=k=1∑B−1​wk​(−logqk​)

Here, bkb_kbk​ is a verification-aware positional weight estimating the chain survival probability from the drafter's own confidence:

bk=∏i=1kqˉi,qˉi=(1−λ)sg(qi)+λb_k = \prod_{i=1}^k \bar{q}_i, \quad \bar{q}_i = (1 - \lambda) \mathrm{sg}(q_i) + \lambdabk​=i=1∏k​qˉ​i​,qˉ​i​=(1−λ)sg(qi​)+λ

The term (1−sg(qk))(1 - \mathrm{sg}(q_k))(1−sg(qk​)) acts as a dynamic component representing the drafter's current probability of missing the token. This ensures the extra training effort is largest while a visually grounded token is unlearned and fades as the drafter fits it, preventing overfitting to visual tokens at the expense of textual ones.

When applying the drafter to long video contexts, the authors observe that full-context attention is inefficient. Analysis of the trained drafter's attention patterns shows that attention concentrates heavily on the question, the generated answer, and recent positions, while distant visual frames receive diffuse attention.

As shown in the figure below, applying a simple attention window reduces acceptance length on image benchmarks but improves it on video benchmarks. Based on these observations, the authors design a sparse attention pattern specifically for video tasks. This pattern retains the first 4 sink tokens, visual tokens sampled with a stride of 32, all text tokens, and a local window of 256 tokens. The same pattern is applied consistently during both training and inference to ensure stable context access and improve acceptance length on video tasks.

Experiment

To motivate multimodal drafting, the authors measure per-token visual dependence through the KL divergence between target model predictions with and without the image, finding that higher visual dependence leads to lower draft acceptance rates and more truncation points. The main experiments train MMFlash on roughly 900K multimodal samples from LLaVA-OV-1.5 for Qwen3.5-4B and Qwen3.5-9B and evaluate on seven multimodal benchmarks, where MMFlash achieves average speedups of 3.00 to 3.05 times under greedy decoding and 2.54 to 2.70 times at temperature 1, outperforming MTP, EAGLE-3, and DFlash baselines. Additional results show improved speedups on video benchmarks after lightweight fine-tuning and generalization to Gemma4 and Qwen3.8-27B, while diagnostic studies confirm robustness under high serving concurrency and validate the chosen block size, drafter depth, and number of target hidden states.

MMFlash consistently achieves the highest acceptance length and decoding speedup across text/document, general VQA, and STEM benchmarks on Qwen3.5 models. Under greedy decoding its average speedup is around 3.0 to 3.1 times, and at temperature 1 it is around 2.5 to 2.7 times, outperforming DFlash by 6.7 to 10.9 percent. The advantage holds across target models and benchmark categories. MMFlash surpasses MTP, EAGLE-3, and DFlash in acceptance length and decoding speedup across diverse benchmark groups. Relative to DFlash, MMFlash adds roughly 7 to 11 percent higher decoding speedup at greedy and temperature 1 settings on Qwen3.5 models.

On three long-context video benchmarks, the sparse-attention MMFlash variant achieves the highest decoding speedups and improves acceptance length over the base MMFlash. The gains are consistent across benchmarks, with the largest speedup on LongVideoBench and more modest improvements on Video-MME. Compared with DFlash, MMFlash variants deliver higher speedups on every video benchmark. The sparse-attention MMFlash configuration reaches the best speedup on all three video benchmarks, with the largest gain on LongVideoBench. Adding sparse attention improves both acceptance length and speedup over the base MMFlash, and fine-tuning further strengthens the advantage over DFlash.

MMFlash improves decode throughput over the baseline in SGLang serving at every concurrency level from 1 to 32, with the largest speedups at lower concurrency and a narrowing advantage as decoding becomes more compute-bound. At concurrency 16, MMFlash still exceeds the throughput the baseline achieves at concurrency 32 across all tasks. Average acceptance length remains high on the reported Qwen3.5-4B tasks, around 5.7 to 6.2 tokens. Speedups over the baseline are highest at concurrency 1 and decline steadily as concurrency increases from 4 to 32. MMFlash at concurrency 16 delivers higher throughput than the baseline at concurrency 32 across all six tasks. On Qwen3.5-4B CharXiv and MMBench, average acceptance length stays near six tokens even as throughput scales.

Across both Qwen3.5-4B and Qwen3.5-9B, MMFlash with a test-time block size of 16, which matches its training configuration, achieves higher acceptance lengths and speedups than block size 8 on all evaluated tasks. Reducing the test-time block size to 8 preserves most of the speedup, while acceptance length declines more substantially. This indicates the drafter generalizes to smaller block sizes but under-utilizes its capacity when the block cap is lower. A test-time block size of 16 consistently yields higher acceptance length and speedup than block size 8 across all evaluated tasks and both model sizes. Using block size 8 leads to only a modest average speedup reduction but a much larger average acceptance length drop, reflecting capped draft block capacity. On ChartQA and CharXiv, a notable share of draft blocks saturate the maximum acceptance length of 8, showing the smaller block size limits accepted tokens per draft.

At greedy decoding, MMFlash with a block size of 16 attains the highest acceptance length and decoding speedup across ChartQA, MMStar, and DynaMath on Gemma4-E4B-it compared with MTP and EAGLE-3 drafters. The advantage holds across reported model families, with speedups on Qwen3.8-27B exceeding those on Gemma4-E4B-it. MMFlash shows the largest relative gains on DynaMath among the Gemma benchmarks. MMFlash achieves the best acceptance length and speedup on every listed benchmark for Gemma4-E4B-it. EAGLE-3 outperforms MTP variants but remains below MMFlash across the evaluated tasks. Reported decoding speedups are higher on Qwen3.8-27B than on Gemma4-E4B-it, while MMFlash maintains the top rank.

The experiments evaluate MMFlash as a speculative decoding drafter across text/document, general VQA, STEM, long-context video, and serving workloads on Qwen and Gemma models. MMFlash consistently achieves the highest acceptance length and decoding speedup compared with DFlash, MTP, and EAGLE-3 under both greedy and temperature 1 settings. A sparse-attention variant further improves long-context video performance, throughput gains remain positive across serving concurrency levels despite narrowing at higher load, and using the training-matched block size of 16 performs best while a smaller block size preserves most speedups.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp