One-sentence Summary

Researchers from Harbin Institute of Technology, the Shenzhen Loop Area Institute, and The Chinese University of Hong Kong propose Lychee-FD, a full-duplex SLM framework that traces modality interference to gradient conflicts between acoustic and semantic modeling and resolves them through hierarchical parameter separation with a dedicated semantic alignment channel, achieving state-of-the-art results on full-duplex benchmarks.

Key Contributions

• Analysis of optimization dynamics in full-duplex Spoken Language Models identifies modality interference as arising from gradient conflicts between acoustic and semantic modeling in shared deep parameter spaces, manifesting as optimization divergence and semantic dilution.

• Lychee-FD is introduced as a native end-to-end full-duplex framework that mitigates this interference through hierarchical parameter separation, decoupling conflicting modalities in deep layers while preserving cross-modality coherence through a dedicated semantic alignment channel.

• Experiments on multiple full-duplex benchmarks show state-of-the-art improvements, including +7.4% on Spoken QA and +28.5% on FullDuplexBench 1.5, demonstrating gains in speech intelligence and full-duplex interaction fluidity.

Introduction

The authors situate the work in spoken language models, where moving from text chat to native voice interaction can make assistants more natural, but current systems are usually limited to rigid turn taking instead of the fluid full duplex conversation typical of humans. Prior approaches struggle with modality interference when a model must simultaneously process speech and preserve semantic knowledge; this leads to degraded reasoning, and alternative designs often add training complexity or latency. The authors analyze gradient conflicts between acoustic and semantic objectives, then introduce Lychee-FD, a native end-to-end full duplex framework that separates acoustic and semantic deep layers while using a semantic alignment channel to retain robust reasoning.

Dataset

• The authors construct a synthetic full-duplex dialogue dataset because open-source full-duplex data are scarce.

• The dataset contains approximately 140K full-duplex dialogue instances covering three interaction behaviors: Interruptions, User Backchannels, and AI Backchannels.

• Dialogue data are generated with multiple agents that simulate realistic User-Assistant conversations.

• Rule-based constraints are injected to trigger diverse interruption types, such as topic switching and follow-up queries, as well as natural backchannels.

• Transcripts are synthesized into speech using CosyVoice 2 with 80K predefined voice prompts for zero-shot cloning, providing acoustic variety and robustness.

• Samples are filtered to remove logical inconsistencies and low audio quality, yielding a curated and reliable training set.

• The final dataset is used as the primary training and experiment data for modeling full-duplex conversational behavior.

Method

Method and Architecture

The authors leverage optimization diagnostics to identify the root cause of modality interference in native full-duplex models. Their analysis starts from a native continuous dialogue model initialized with weights from StepAudio-2-mini. For a subset of training samples, they perform forward passes and accumulate gradients for the text token generation loss $\mathcal{L}_{\mathrm{text}}$Ltext​ and the speech token generation loss $\mathcal{L}_{\mathrm{speech}}$Lspeech​, but they do not update the model parameters. For layer $l$l, the gradient vectors are obtained by flattening the gradients with respect to the layer parameters $\theta^{(l)}$θ(l):

$$\mathbf{g}_{\mathrm{text}}^{(l)} = \nabla_{\boldsymbol{\theta}^{(l)}} \mathcal{L}_{\mathrm{text}},$$gtext(l)​=∇θ(l)​Ltext​, $$\mathbf{g}_{\mathrm{speech}}^{(l)} = \nabla_{\theta^{(l)}} \mathcal{L}_{\mathrm{speech}}.$$gspeech(l)​=∇θ(l)​Lspeech​.

Using these vectors, the authors quantify the alignment between the two learning objectives with cosine similarity:

$$\mathcal{S}^{(l)} = \cos \left( \mathbf{g}_{\mathrm{text}}^{(l)}, \mathbf{g}_{\mathrm{speech}}^{(l)} \right),$$S(l)=cos(gtext(l)​,gspeech(l)​),

and they examine the relative strength of semantic supervision through the gradient magnitude ratio:

$$\mathcal{R}^{(l)} = \frac{\left\| \mathbf{g}_{\mathrm{text}}^{(l)} \right\|}{\left\| \mathbf{g}_{\mathrm{speech}}^{(l)} \right\|}.$$R(l)=​gspeech(l)​​​gtext(l)​​​.

As shown in the optimization dynamics visualization below:

The analysis reveals a depth-dependent interaction between modalities. In shallow layers, the cosine similarity is positive, meaning text and speech gradients point in broadly compatible directions. This supports the idea that early layers can benefit from shared low-level feature processing. In deeper layers, however, the similarity becomes negative and unstable, indicating that acoustic modeling and semantic modeling begin to pull the shared parameters in opposing directions. The gradient ratio analysis further shows that enforcing temporal alignment with padding tokens reduces the relative magnitude of text gradients. Sparse text supervision therefore weakens the language modeling signal, allowing acoustic reconstruction to dominate optimization and contributing to knowledge degradation.

The proposed Lychee-FD framework directly addresses these two issues with hierarchical parameter separation and a semantic alignment channel. It is designed as a native end-to-end system, while avoiding the latency and redundancy of Thinker-Talker pipelines and the optimization conflicts of fully shared architectures. The comparative system design is shown below:

The backbone begins with Step Audio-2, selected for its public availability and strong half-duplex foundation. Incoming speech is processed with a Whisper-v3-large encoder, and acoustic outputs are represented using the CosyVoice 2 tokenizer at a 25 Hz frame rate. This fixed 25 Hz acoustic stream is important for full-duplex timing, especially when the model must listen, respond, and manage turn transitions continuously.

The core architectural innovation is a hierarchical Transformer that preserves shared computation where modalities are compatible but separates parameters where gradients conflict. Let $\mathbf{E} \in \mathbb{R}^{N \times d}$E∈RN×d be the input embedding sequence. The model first passes through shared Transformer layers:

$$\mathbf{H}_{\mathrm{shared}} = \mathcal{F}_{\mathrm{shared}}(\mathbf{E}; \theta_{\mathrm{shared}}).$$Hshared​=Fshared​(E;θshared​).

In deeper layers, the network branches into three specialized heads: a Semantic Head for text generation, an Acoustic Head for speech synthesis, and a Control Head for interaction management such as start and stop behavior. For each head $m \in \{T, A, C\}$m∈{T,A,C}, the output logits are computed as:

$$\mathbf{O}^{m} = \mathcal{F}_{\mathrm{head}}^{m}(\mathbf{H}_{\mathrm{shared}}; \theta_m),$$Om=Fheadm​(Hshared​;θm​),

where $T$T, $A$A, and $C$C denote text, acoustic, and control outputs. The model is optimized with a summed next-token prediction loss:

$$\mathcal{L} = - \sum_{m \in \{T, A, C\}} \sum_t \log P(y_t^m \mid y_{<t}, \mathbf{E}; \theta),$$L=−m∈{T,A,C}∑​t∑​logP(ytm​∣y<t​,E;θ),

where $y_t^m$ytm​ is the ground-truth token for modality $m$m at step $t$t. This structure isolates conflicting objectives in deep layers while still allowing shared low-level representation learning.

The second key module is the semantic alignment channel. Instead of relying only on sparse text tokens inserted to align with audio frames, the model learns to generate an internal monologue that provides dense semantic supervision. The parallel generation streams are organized as:

$$Y^{\mathrm{T}} = [t_1, t_2, \ldots, t_n, \langle \mathrm{EOT} \rangle, \langle \mathrm{pad} \rangle, \ldots, \langle \mathrm{pad} \rangle],$$YT=[t1​,t2​,…,tn​,⟨EOT⟩,⟨pad⟩,…,⟨pad⟩], $$Y^{\mathrm{A}} = [a_1, a_2, \ldots, a_n, a_{n+1}, a_{n+2}, \ldots, \langle \mathrm{EOS} \rangle],$$YA=[a1​,a2​,…,an​,an+1​,an+2​,…,⟨EOS⟩], $$Y^{\mathrm{C}} = [\langle \mathrm{Start} \rangle, c_1, \ldots, c_k, \langle \mathrm{Stop} \rangle].$$YC=[⟨Start⟩,c1​,…,ck​,⟨Stop⟩].

Here, the text stream carries the monologue, the acoustic stream carries speech tokens, and the control stream manages speaking onset and offset. By making the text channel explicit, the system maintains stronger gradient flow for language modeling and reduces the semantic dilution caused by padding-heavy alignment schemes.

To assess whether the observed gradient conflict is specific to one architecture, the authors repeat the same geometric analysis on a different native end-to-end full-duplex model. The per-layer trend is shown below:

In that comparison, shallow and middle layers again show positive gradient similarity, while deep layers transition into negative similarity. This pattern suggests that deep-layer divergence between semantic and acoustic objectives is a general bottleneck in fully shared end-to-end spoken language models, rather than an artifact of one particular design. The evidence strengthens the rationale for separating acoustic and semantic parameters in the higher layers.

The training workflow supports this architecture with a synthesized full-duplex dialogue pipeline. Because native full-duplex supervision is scarce, the authors simulate realistic user-assistant conversations using multiple agents. Rule-based constraints are injected to trigger behaviors such as interruptions, follow-up queries, topic switches, and backchannels. The generated transcripts are then converted into speech using CosyVoice 2 with zero-shot voice cloning, producing audio that covers diverse acoustic styles. After synthesis, the corpus undergoes filtering to remove logical inconsistencies, low-quality audio, and unusable alignment cases. The resulting dialogue data is used to train the model in a single-epoch regime.

For optimization, the system is trained with AdamW and a cosine learning-rate schedule, with a warmup ratio of 0.1. The implementation uses a global batch size of 32 and a learning rate of $3 \times 10^{-6}$3×10−6. The architecture configuration includes 24 shared Transformer layers, 4 layers for the text head, 4 layers for the speech head, and 2 layers for the control head. During inference, both text and speech tokens are generated with greedy sampling, and evaluation performance is averaged over multiple random seeds.

Experiment

The experiments evaluate the proposed full-duplex speech-language model against system-level and native baselines, using synthesized full-duplex training data and benchmarks for spoken question answering, interactive conversation, and speech generation. The results indicate that explicit semantic modeling and hierarchical parameter separation improve knowledge retention, turn-taking, interruption handling, and natural speech quality compared with shared or pipeline-based approaches. Ablations and gradient analysis further support the conclusion that separating text and speech optimization reduces modality interference and addresses a common bottleneck in fully shared full-duplex architectures.

The evaluated model leads both speech-to-text and speech-to-speech accuracy averages, improving over native and system-level comparisons. Its advantage is strongest when answers are produced as speech, while preserving stable full takeover behavior. Ablations indicate that replacing semantic supervision with sparse time-aligned text and sharing parameters degrade speech answer accuracy. Full-duplex accuracy advantage is maintained without turn-taking sacrifice Semantic alignment channel sustains knowledge more effectively than sparse time-aligned text Hierarchical parameter separation reduces modality interference in speech-to-speech answering

The proposed model leads most full-duplex interaction and efficiency measures, with especially strong results in interruption response, backchanneling, turn-taking, and pause handling. It also shows low first-response and interruption-response latency, supporting efficient real-time interaction. The improvement on challenging interruption handling suggests better native full-duplex decision-making than baselines that rely on external VAD. It achieves the best results on most interaction metrics, while baselines often trade accuracy for overly frequent or delayed responses. Low first-response delay and short stop latency suggest the architecture avoids visible real-time computational overhead. Higher interruption-response rate and lower backchannel takeover rate indicate more selective turn control than aggressive takeover behavior. The substantial gain on the challenging interruption benchmark shows a clear advantage over systems that depend on external VAD.

The experiments evaluate the model on speech-to-text and speech-to-speech answering accuracy as well as full-duplex interaction and efficiency. The results show consistent accuracy gains, especially for speech answers, while maintaining stable takeover behavior and improving selective control over interruptions, backchannels, turns, and pauses. Ablation studies validate the value of semantic alignment supervision and hierarchical parameter separation, whereas sparse time-aligned supervision and shared parameters harm speech answering performance.