One-sentence Summary

The authors propose Omni-Woodpecker (OWP), a training-free framework that corrects crossmodal hallucinations in Omni LLMs by identifying conflicts and using orthogonal decomposition to rebalance fused representations, suppressing misleading priors while restoring ignored evidence; with Qwen2.5-Omni-7B as the backbone, it improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, and generalizes to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks.

Key Contributions

• Omni-Woodpecker is a training-free framework that reframes hallucination mitigation in omni LLMs as conflict-conditioned representation rebalancing rather than output-level correction.
• The method detects cross-modal conflicts from complementary modality-specific views, applies orthogonal decomposition to separate grounding evidence from competing priors in hidden states, and adaptively suppresses misleading priors while restoring ignored evidence before decoding, allocating correction across attention paths and residual states according to the diagnosed conflict.
• Experiments with Qwen2.5-Omni-7B improve CMM and AVHBench by 12.9 and 8.6 points, respectively, with gains of 0.67 to 2.25 points across four general omni-modal reasoning benchmarks; the approach also transfers to VideoLLaMA2-AV, improving CMM and AVHBench by 4.6 and 3.9 points.

Introduction

Omni LLMs extend large language models to jointly process text, vision, and audio, enabling cross-modal reasoning but also creating a reliability challenge: when modalities provide incomplete or conflicting cues, the model may correctly encode evidence yet let a strong learned prior override it during fusion. Existing mitigation methods such as contrastive decoding, modality-adaptive generation, and preference optimization treat hallucination as an output-level anomaly by calibrating decoding distributions or response preferences, but in early-fusion models grounding evidence and priors are already entangled in hidden representations before decoding begins, so these approaches leave the underlying conflict unresolved. The authors propose Omni-Woodpecker, a training-free framework that shifts mitigation to representation-level rebalancing by detecting query-conditioned cross-modal conflicts, separating grounding evidence from competing priors, and restoring suppressed evidence while attenuating misleading priors before decoding, without modifying model parameters or the language head. This reframes omni-modal hallucination as an evidence-prior representation imbalance rather than a perception failure.

Method

The authors propose Omni-Woodpecker (OWP), a training-free framework designed to correct cross-modal hallucinations in Omni Large Language Models (OLLMs) by rebalancing grounding evidence and learned priors within the fused representation. Rather than manipulating final decoding distributions, OWP operates directly on the internal hidden states of a frozen model. The pipeline consists of three sequential stages: query-conditioned conflict diagnosis, evidence-preserving conflict decomposition, and conflict-adaptive representation rebalancing.

In the first stage, Query-Conditioned Conflict Diagnosis, the framework identifies the target modality required by the query and the competing modality that causes the hallucination. The authors run the frozen model on four distinct input views: the full multimodal input $x^F = (q, v, a)$xF=(q,v,a), audio-only $x^A = (q, a)$xA=(q,a), video-only $x^V = (q, v)$xV=(q,v), and text-only $x^T = q$xT=q, where $q$q, $v$v, and $a$a represent the query, video, and audio, respectively. To determine the target view, a diagnostic query is appended to prompt the model for the necessary modality, and the target is selected via softmax probabilities over the next-token logits. The competing view is then identified by evaluating which single-modality branch agrees with the full-view answer while exhibiting a higher source-normalized attention mass than the target branch.

The second stage, Evidence-Preserving Conflict Decomposition, translates the differences between these views into precise representation edits. The authors first compute a calibrated evidence direction $u_\ell$uℓ​ that anchors on the target evidence while retaining only the cross-view components that support the target answer. This is achieved by calculating preference scores and trust terms to weight the target evidence, non-target information, and fusion shift. Next, they derive an orthogonal conflict direction $p_\ell$pℓ​ and an evidence deficit $g_\ell$gℓ​ by projecting the fusion shift onto the orthogonal complement of the evidence direction. This ensures that suppressing the competing prior leaves the faithful evidence component untouched. Concurrently, head-level conflict localization identifies specific attention heads to edit. By analyzing source attention and answer-aligned value contributions, the framework partitions heads into a competing set $\mathcal{H}_\ell^-$Hℓ−​ and an evidence set $\mathcal{H}_\ell^+$Hℓ+​ for each edited layer.

In the final stage, Conflict-Adaptive Representation Rebalancing, the framework applies the computed edits by splitting a single correction budget between attention-path modifications and residual-state updates. The allocation is governed by a path share $\lambda(x)$λ(x), which is determined by the full-view answer preference. A weak preference shifts more control to the path level, while a strong preference directs the correction to the residual state. The path-level edit reweights the token-head routes before multi-head attention aggregation by subtracting source-aligned values from the competing heads and adding evidence-aligned values from the evidence heads. The residual-level edit then suppresses the orthogonal prior component and restores the measured evidence deficit. The updated state is written back to the same layer, ensuring that the context is preserved while the hallucination-inducing cues are suppressed before decoding.

Experiment

The experiments evaluate Omni-Woodpecker on cross-modal conflict benchmarks, audio-visual matching, and general omni-modal reasoning tasks using Qwen2.5-Omni-7B and VideoLLaMA2-AV, with controlled comparisons against decoding and representation-editing baselines. The results validate that the framework corrects conflicts in both directions, preserves joint audio-visual evidence, transfers across backbones, and improves general reasoning while remaining more efficient. Ablations and mechanism analyses show that prior suppression and evidence enhancement work complementarily, with residual edits as the main correction site and layer-wise separation between evidence recovery and prior removal.

Among the listed external omni backbones, Qwen3-Omni achieves the strongest overall CMM and AVHBench performance, while VITA-1.5 consistently lags. OmniVinci is competitive on CMM measures but falls behind Qwen3-Omni on AVHBench. The proposed Omni-Woodpecker framework substantially improves the Qwen2.5-Omni-7B backbone, lifting hallucination resistance and perception accuracy and outperforming Qwen3-Omni on both benchmarks. Qwen3-Omni posts the best overall results among the listed external backbones, whereas VITA-1.5 is the weakest across most metrics. Omni-Woodpecker with Qwen2.5-Omni-7B raises CMM hallucination resistance from 57.0% to 85.5% and exceeds Qwen3-Omni on both CMM and AVHBench by 4.3 and 6.2 points.

The experiments evaluate hallucination resistance and perception accuracy on the CMM and AVHBench benchmarks using several external omni backbones and the proposed Omni-Woodpecker framework. Among the external backbones, Qwen3-Omni performs best overall, while VITA-1.5 consistently lags and OmniVinci is competitive on CMM but weaker on AVHBench. The proposed Omni-Woodpecker with a Qwen2.5-Omni-7B backbone substantially improves both measures and surpasses Qwen3-Omni on both benchmarks.