One-sentence Summary

Researchers from Tel Aviv University and Simon Fraser University analyze the “attention triangle” in audio-video diffusion models, revealing that bidirectional audio-video cross-attention edges, shaped by parameter biases, cause semantic leakage when prompts conflict with learned priors, and they introduce attention-derived diagnostic signals to probe and deliberately incur leakage, guiding inference-time interventions that improve cross-modal alignment while preserving generation quality.

Key Contributions

• Analyzes the “attention triangle” in audio-video diffusion models, showing that the audio-video cross-attention edge is bidirectional and bias-driven, making it a major source of semantic leakage when prompts conflict with learned priors.
• Introduces attention-derived diagnostic signals that expose how semantics are distributed across modalities, enabling controlled leakage induction and isolation of individual cross-modal interactions in a pretrained generator.
• Applies these signals to training-free inference-time interventions that improve cross-modal alignment, with experiments on LTX-2 demonstrating better semantic grounding while preserving generation quality.

Introduction

Diffusion models, particularly Diffusion Transformers (DiT), have made attention the core mechanism for conditional generation, using cross-attention to bind text, audio, and video modalities. However, attention mixes information globally and softly without explicit constraints, leading to attribute leakage and unstable semantic bindings. In trimodal audio-video generation, this problem compounds: the model must reconcile text intent, audio temporal structure, and video spatiotemporal realization, but competing signals and modality dominance often cause semantic associations to spread to unintended entities. Prior work on leakage in text-to-image models treats it as an attention-routing failure and applies inference-time interventions, yet audio-video models inherit and amplify this issue, especially along the audio-video pathway, where a 1D audio embedding attending to 3D video patches strips spatial distinctness and leaves sound-source grounding underconstrained.

The authors analyze this leakage through the structure of an "attention triangle" among text, audio, and video, focusing on source attribution failures. They identify the audio-video edge as a weak link, where learned cross-modal biases can override prompt-specified bindings and reroute sound semantics to visually canonical but incorrect sources. Building on this, they extract attention-derived signals as a diagnostic tool to probe and deliberately induce leakage, then use these signals to guide inference-time interventions. Their experiments show that steering individual triangle edges only partially resolves issues, such as fixing appearance without attribution or vice versa, while joint steering of all edges restores consistent sound-to-source grounding across diverse prompts.

Method

The authors introduce a training-free, inference-time steering algorithm designed to mitigate semantic leakage in joint text-to-video and text-to-audio generation models. They formalize the interaction between modalities as an "Attention Triangle," where text, audio, and video tokens are connected via pairwise cross-attention. This coupled system allows semantic information to route directly and indirectly, but often leads to attributes intended for one entity leaking to another, such as a parrot's speech being incorrectly attributed to a pirate. To resolve this, the method applies pre-softmax additive logit biases to all three cross-attention surfaces simultaneously.

Refer to the framework diagram:

The steering biases are constructed from three families of anchors derived from a single unsteered baseline pass:

1. Text Anchors: The user annotates the intended sound source, the sound or action, and an optional competing source in the prompt. These phrases are mapped to specific token indices.
2. Visual Anchors: Baseline frames are decoded and processed using a segmentation model prompted with the intended-source text to generate hard binary masks for the source and competing regions.
3. Audio Anchors: Soft sound masks are derived by aggregating audio-query and text-key attention weights that target the sound phrase tokens across denoising steps, as no external audio segmenter is utilized.

The algorithm applies specific biases to the edges of the triangle to reground the streams:

Audio-Video Edge: The authors compute a soft agreement matrix $\mathbf{G}_{VA}$GVA​ that assigns high values to intended source-sound pairs and non-source/non-sound pairs, while assigning low values to mismatched pairs. A bias is applied to suppress mismatched audio-video logits:

where $\lambda > 0$λ>0.

Text-Conditioned Edges: For the text-to-video and text-to-audio surfaces, the method defines "intended" and "conflicting" cell masks. For the video-query and text-key surface, the intended mask $\mathbf{M}_{VT}^{\text{int}}$MVTint​ pairs source video patches with source text tokens, while the conflicting mask $\mathbf{M}_{VT}^{\text{conf}}$MVTconf​ pairs source patches with competing text and vice versa. The bias is formulated as:

A symmetric formulation is applied to the text-to-audio surface using audio masks. The authors use fixed hyperparameters across experiments, placing greater weight on suppressing conflicting associations than reinforcing intended ones.

As shown in the figure below:

This visualization demonstrates the necessity of the full joint steering. The baseline generation exhibits strong leakage where the pirate inherits the speaking role. Text steering restores the pirate's appearance but fails to localize speech correctly. Audio-video steering correctly localizes speech to the parrot but leaves appearance leakage. Only the full triangle steering jointly restores correct appearance and localizes the sound to the intended source, confirming that addressing all three edges is required to close the triangle and eliminate residual leakage.

Experiment

The evaluation validates the proposed steering framework through attention visualizations, qualitative comparisons, and human preference studies. First- and second-order attention analyses confirm that the method redirects audio-mediated attention from incorrect to intended sources, mitigating four recurring leakage modes in open-domain scenarios. Comparisons against the native baseline, a bounded-attention adaptation, and partial steering variants show that only the full intervention simultaneously preserves appearance and grounds sound correctly; ablations isolate the complementary roles of text and audio-video edge steering. Quantitative metrics and pairwise user studies consistently favor the full method on attribution, leakage, and overall quality, with all preference margins statistically significant.

The proposed full intervention achieves the highest sound-source attribution score and the best visual consistency and aesthetic quality among all methods, while remaining competitive on audio-text alignment. Partial variants show that steering the audio-video edge alone improves attribution but not as much as the full intervention, and the video-only baseline is not scored on attribution. The full intervention outperforms the native baseline and all other variants on sound-source attribution, with the largest gain over the bounded attention baseline. Steering only the audio-video edge yields a higher attribution score than steering only the text edges, but the full intervention combines both for the best result. The full intervention maintains or improves visual consistency and aesthetic quality relative to the native baseline, with negligible change in audio-text alignment.

The evaluation compares the full intervention against partial variants and baselines, showing that it achieves the highest sound-source attribution while preserving visual consistency and aesthetic quality, with only negligible impact on audio-text alignment. Steering the audio-video edge alone improves attribution more than steering text edges, but the full intervention, which combines both, yields the best result. The largest attribution gain is observed over the bounded attention baseline, and the video-only baseline is excluded from attribution scoring.