One-sentence Summary

Researchers at The University of Hong Kong propose reframing full-duplex omni models as stateful computational middleboxes in a human-centered feedback loop, introducing cross-stage coordination of network-aware inference scheduling, execution-aware transport prioritization, and playback control in CONFLUX to jointly optimize communication, computation, and playback for improved response latency and playback deadline adherence under network degradation.

Key Contributions

• The paper proposes a conceptual shift: treating full-duplex omni models as stateful computational middleboxes within a human-centered feedback loop, which breaks the traditional boundaries between network transport, inference serving, and playback stages.
• It introduces CONFLUX, an AI-native real-time communication framework that jointly coordinates these stages through network-aware inference scheduling, execution-aware transport prioritization, and playback control that adapts to both network and model variability.
• Preliminary experiments with CONFLUX demonstrate substantial improvements in response latency and playback deadline adherence under network degradation, providing evidence for the gains of cross-stage optimization.

Introduction

The authors address the shift from turn-based language models to real-time, full-duplex multimodal interaction, where users can speak, interrupt, and share visual context while the model continuously listens and responds. This closed-loop paradigm breaks the traditional separation between network transport, model serving, and playback, revealing that locally optimized components undermine end-to-end quality when network delay, inference timing, and user experience are tightly coupled. Prior systems lack cross-stage coordination, and existing attempts to use semantic cues for joint optimization are costly and impractical. The authors introduce CONFLUX, an AI-native communication framework that treats chunks as a universal coordination substrate. Chunks carry execution-level signals (such as remaining latency budget) across transport, inference, and playback, enabling stages to jointly decide transmission, scheduling, and presentation timing without interpreting the content. The framework is built on three insights: network- and session-aware inference scheduling, model-defined transport requirements based on modality urgency and reliability, and adaptive per-response playback start that balances time to first response against deadline misses. The result is a practical, semantics-free coordination layer that improves interaction quality under constrained networks.

Method

The authors propose CONFLUX, a system designed to coordinate transport, inference, and playback in full-duplex AI media. As illustrated in the framework diagram, CONFLUX operates as an end-to-end control loop that gathers signals from three domains: model signals (modality requirements, inference latency, throughput), network signals (RTT, bandwidth, jitter), and playback signals (Time to First Response, Deadline Miss).

These signals are aggregated by a central gatherer to drive three coordinated runtime actions: sender-side cross-layer transport, network-aware model inference, and receiver-side playback scheduling. This architecture ensures that the system adapts dynamically to the evolving interaction state.

Full-duplex omni-models treat chunks as the fundamental transport objects. As shown in the figure below, the process begins with the uplink, where each chunk represents a bounded interaction interval containing aligned modalities like audio, video frames, and text.

These chunks are transmitted via channels such as WebRTC. On the server side, the input chunks enter a model pipeline where they are queued, batched, and processed by an inference engine. The autoregressive LLM samples an action, such as silence, delegate, or response. If a response is generated, it is rendered by modality-specific generators and packaged into output chunks for downlink delivery. The timeline illustrates the critical path from the last input capture to playback start, defining the Time to First Response. Late arrivals on the downlink introduce gaps and user-perceived glitches.

The sender acts as the first control point in this loop. It utilizes a model-centric specification that defines available media channels, tunable parameters like bitrate and resolution, and adaptation policies. By observing runtime signals like network RTT and server-side backpressure, the sender applies a policy to make joint transmission decisions. It determines which chunks to degrade, defer, or discard. For instance, speech and control events take precedence over less urgent visual content. The adaptation policy allows the sender to reduce less critical dimensions first, such as lowering video resolution before audio bitrate, to maintain interaction quality.

To handle network-conditioned model behavior, the inference scheduler maintains lightweight per-session context, including duplex phase, turn states, recent Quality of Experience, and network quality. Instead of relying solely on request arrival order, the scheduler assigns a priority score to each pending chunk. This score reflects whether the chunk changes the model state, such as a barge-in event, and accounts for sessions with poor interaction quality or degraded network conditions. The scheduler aggregates priorities within candidate batches. If the aggregate priority exceeds a threshold, the batch is admitted immediately to shorten batching delay for urgent events; otherwise, it waits to improve serving efficiency.

The receiver controls the final stage by setting the playback buffer duration after the first output chunk arrives. This balances responsiveness against smoothness. The receiver derives a congestion signal from recent playback dynamics and adapts the buffer time $b_t$bt​ using a congestion-control-inspired policy:

Here, $L_t$Lt​ is the observed end-to-end latency, $\gamma > 1$γ>1 is the multiplicative increase factor, and $\delta > 0$δ>0 is the additive decrease step. The thresholds $\phi_{\text{high}}$ϕhigh​ and $\phi_{\text{low}}$ϕlow​ define a hysteresis region. If latency exceeds $\phi_{\text{high}}$ϕhigh​, the buffer increases rapidly to absorb gaps. If it falls below $\phi_{\text{low}}$ϕlow​, the buffer decreases gradually to improve responsiveness. This prevents oscillation from small latency fluctuations.

Experiment

A preliminary evaluation of CON-FLUX was conducted using MiniCPM-o-4.5 deployed on cloud instances with emulated network conditions ranging from high-bandwidth, low-latency to severely constrained links. The results show that CON-FLUX significantly reduces time-to-first-response and per-chunk playback lag compared to the baseline, with the greatest gains observed under degraded network profiles. These findings demonstrate that the design enhances both initial responsiveness and strict adherence to playback deadlines, even under challenging network conditions.

A barge-in example illustrates that the system can successfully interrupt an ongoing response to answer a new question, but it may also fail to do so and continue the previous answer, revealing inconsistent barge-in behavior. In the successful case, the model immediately stops the initial description of a large language model and answers the interrupting question about the object on the hand with "It is a white cup." In the failure case, the model ignores the same barge-in request and continues generating the previous response about artificial intelligence.

As network conditions degrade from good to bad, the latency for a user barge-in to affect the model's output increases substantially. Under good conditions, the barge-in latency is around half a second, but it balloons to over five seconds under severely constrained links, causing the model to continue producing stale responses. Barge-in latency grows more than 9x from good to bad network profiles, reaching over 5 seconds. The jump from limited to bad conditions is the most pronounced, with latency nearly quadrupling.

Modalities exhibit distinct transport requirements: control events and audio are urgent, while video and text are not. Reliability and replaceability vary sharply, with control and text demanding high reliability and no replacement, video being replaceable and low-reliability, and audio offering partial adaptability. These differences motivate exposing modality-specific constraints to the transport layer for efficient, accuracy-bounded delivery. Control events and audio are classified as urgent, while video and text are only of medium urgency. Control events and text require high reliability and cannot be replaced, whereas video is replaceable and has low reliability requirements. Audio and video are adaptable, but control events and text are not, limiting how they can be adjusted in transit.

A conversational AI system using CON-FLUX was tested across four network profiles ranging from high-bandwidth, low-latency (L1) to severely constrained (L4). It maintained fast time-to-first-response (TTFR) and very low playback-lag, with relative gains over a baseline that grew substantially as network quality deteriorated. CON-FLUX kept TTFR between 1.39 and 1.88 seconds across all profiles, while the baseline ranged from 1.77 to 5.41 seconds. The TTFR improvement over the baseline increased from 21.3% under the best network (L1) to 74.4% under the most constrained (L4). CON-FLUX held per-chunk playback lag below 10 ms in every profile, whereas the baseline lag rose from 46 ms to 94 ms as network conditions worsened. TTFR under L4 was lower than under L3, suggesting the scheduling policy reacts differently across regimes and leaving room for further tuning.

An evaluation of barge-in behavior and transport scheduling in a conversational AI system reveals that responsiveness degrades sharply under poor network conditions, with barge-in latency growing dramatically and modality-specific constraints (urgent and reliable control and audio, replaceable video) motivating tailored transport policies. The CON-FLUX system maintains consistently fast time-to-first-response and very low playback lag across all network profiles, and its relative performance advantage over a baseline increases substantially as network quality deteriorates.