One-sentence Summary

Researchers at XPeng Inc. propose X-AuT, a progressive audio-encoder compression framework for speech large language models that selects layer combinations via behavioral probes, restores pruned models through cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning, reducing macroaverage error from 5.61% to 5.27% on Qwen3-ASR-0.6B with $18 \rightarrow 16$18→16 layers and achieving 5.75% error with 20.7% fewer audio-tower parameters while outperforming direct pruning.

Key Contributions

• X-AuT is a progressive pruning framework that selects audio-encoder layer combinations through short behavioral probes and restores the pruned model via representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning, while keeping the language-model backbone frozen.
• The recovery recipe uses a transcript-consistency pipeline that assigns agreement tiers from external ASR hypotheses and applies source reweighting, together with a scheduled distillation strategy that blends teacher-forced and on-policy supervision to stabilize training.
• On ten Chinese–English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 layers reduces macro-average error from 5.61% to 5.27%, the 14-layer model reaches 5.75% error with 20.7% fewer audio-tower parameters, and progressive pruning outperforms direct pruning (5.75% vs. 6.73%) and self-distillation (5.55% vs. 8.45% with a 1.7B teacher), establishing two practical operating points.

Introduction

Speech large language models combine a deep audio encoder with an autoregressive text decoder, but the encoder must process every input frame, making its depth a key contributor to first-token latency in streaming, mobile, and in-vehicle systems. Post-training depth reduction is attractive because it removes entire Transformer blocks and yields a regular, deployment-friendly model, yet simply dropping layers alters the audio embeddings fed to the decoder and can trigger premature end-of-sequence predictions and large deletion errors. Prior ASR compression work has explored distillation, low-rank factorization, sparsity, and supernet training, but it does not fully address which combinations of layers can be removed and recovered under a fixed budget when the pretrained decoder remains frozen, nor how to handle both hidden-state mismatch and errors induced by the student’s own decoding history. The authors introduce X-AuT, a progressive pruning and recovery framework that uses short behavioral probes to evaluate candidate layer sets, then realigns the pruned encoder through cross-scale distillation from a larger teacher, scheduled student-policy training, and low-rank adaptation of decoder attention, all while preserving the pretrained language model backbone. This approach yields a 14-layer model that matches the baseline’s accuracy while reducing audio-tower parameters by 20.7% and measured encoder latency by 21.4% on an in-vehicle accelerator.

Dataset

The authors construct a large multilingual ASR dataset by combining public and proprietary corpora, then filter and label it for distillation training. Key details:

• Source pool: Public and proprietary ASR corpora, including AISHELL-1/4/5, CommonVoice, Emilia, GigaSpeech, KeSpeech, LibriSpeech, WenetSpeech, and cockpit-domain speech. The pool exceeds 280k hours before quality filtering. All audio is capped at 40 seconds.
• Manifest construction: Each corpus is converted to a unified JSONL manifest. Each record contains an utterance identifier, audio reference, source transcript, language and split tags, duration, sampling rate, and channel count. Text normalization applies Unicode NFKC normalization, width conversion, Traditional-to-Simplified conversion for Mandarin, removal of invisible characters and numeric separators, dash canonicalization, and whitespace normalization. Both original and normalized transcripts are retained.
• Filtering: Reference-bearing utterances are decoded offline by Qwen3-ASR-1.7B and Qwen3.5-Omni. Records missing an inference result, a valid audio reference, or nonempty supervision are excluded. The remaining records receive consistency labels (as described in Section 3.2).
• Training data usage: The distillation runs use only the class-1 subset of the manifest. The first-hop log yields approximately 299k weighted target records per epoch; the second-hop log yields approximately 292k. In stage 2, class-1 data is kept but corpus weights are adjusted: AISHELL-4/5 and cockpit-query contributions are increased, while several weakly matched web-speech sources are dropped. These numbers represent the realized loader indices, not the full 280k-hour source pool.

Method

The authors leverage an input-embedding conditioning architecture where an input waveform $\mathbf{x}$x is encoded by an $N$N-layer audio encoder $E_{\theta}$Eθ​ and a bridge $B_{\phi}$Bϕ​ into audio embeddings $\mathbf{e} = B_{\phi}(E_{\theta}(\mathbf{x}))$e=Bϕ​(Eθ​(x)). These embeddings are placed at audio-placeholder positions in the token embedding sequence, and a causal language-model decoder $D_{\psi}$Dψ​ predicts transcription tokens through a tied output projection $H_{\omega}$Hω​. A pruning operation retains an ordered subset $\mathcal{I} \subset \{1, \ldots, N\}$I⊂{1,…,N} and forms $E_{\theta, \mathcal{I}}$Eθ,I​ from those blocks. The goal is to find a recoverable subset and parameters that minimize the aggregate text error rate (TER) under a target depth:

The pretrained weights of $D_{\psi}$Dψ​ remain frozen. LoRA parameters attached to its q/k/v/o attention projections are trainable, and $H_{\omega}$Hω​ is trainable during the initial stages and frozen later.

To handle heterogeneous supervision in the source pool, the authors employ a transcript-consistency filtering mechanism. For each reference-bearing utterance, two strong ASR systems produce offline hypotheses. After language-aware normalization, they compute the three pairwise edit rates among the source transcript and the two hypotheses, using CER for Chinese and WER for English. The maximum edit rate $e_{\mathrm{max}}$emax​ measures the largest disagreement. Exact agreement, Mandarin homophone agreement, and a consistency vote assign one of nine tiers, with lower tier numbers indicating stronger transcript agreement. The reported configuration uses class 1 for both distillation stages.

As shown in the figure below, the complete data pipeline encompasses corpus unification, text normalization, dual-model quality estimation, and quality-ranked label selection to construct the final training pool.

The pruning process occurs in two hops, reducing the encoder from 18 to 16, and then to 14 layers. The first hop removes original layers {1, 18}. For the second hop, every candidate starts from the same recovered 16-layer checkpoint and is trained with a short LoRA warm-up. The authors evaluate each remaining layer as a single removal, then evaluate a fixed set of adjacent and non-adjacent layer pairs. Candidate selection uses the lowest aggregate TER on a fixed development suite. This procedure directly measures post-removal behavior under the available recovery budget, acknowledging that recovery after removing several layers cannot be predicted reliably from corresponding single-layer scores.

Refer to the framework diagram for a comprehensive overview of the X-AuT pipeline, which integrates transcript-consistency filtering, behavioral probes for layer selection, progressive pruning, and the three-stage recovery process.

Each pruning hop utilizes a three-stage recovery recipe. The student is the pruned model, and the teacher is a larger model with a deeper audio encoder. Teacher parameters are frozen and discarded at inference.

Stage 0 occupies the first 5% of the distillation epoch and combines intermediate-layer, bridge, logit, and transcript losses:

Both representation losses sum mean-squared error and cosine distance. Teacher layers are divided uniformly into $M$M ordered groups, and student layer $m$m aligns to the last teacher layer in group $m$m. A learned two-layer MLP with a 256-dimensional bottleneck projects teacher hidden and bridge features into the student space. Logit KD uses temperature-scaled KL divergence under gold prefixes. The pruned audio encoder and bridge are fully trainable, while decoder base weights remain frozen.

Stage 1 disables intermediate-layer loss and uses bridge alignment, teacher-forced logit KD, and gold-transcript CE:

After 20% of Stage 1 has elapsed, every fifth optimizer step is scheduled for student-policy supervision. The student greedily generates a prefix, and student and teacher are evaluated on the same generated context. Their distributions are compared over the union of each model's top-k support. Rollout safeguards prevent degenerate prefixes from entering the KD loss by enforcing minimum token generation, duration-aware maximums, and rejecting budget-exhausted or repetitive rollouts.

Stage 2 initializes from the best Stage 1 checkpoint and optimizes gold-transcript CE for one epoch. The audio encoder, bridge, and decoder LoRA adapters remain trainable, while the tied output embedding is frozen. The data index is reweighted toward target-domain sources. This stage contains no teacher loss:

Experiment

The evaluation compares depth-reduced ASR models distilled from a larger cross-scale teacher against the original 0.6B baseline, using a multi-stage recovery process and a macro-average of five development subsets for checkpoint selection. Results show that the 16-layer model surpasses the baseline on average, while the 14-layer model trades a small error increase for larger parameter savings; a stronger teacher is critical, as self-distillation degrades performance. Layer removal experiments reveal that pair interactions and progressive pruning matter more than individual layer importance, and representation recovery can partially but not fully compensate for lost encoder capacity.

A nine-tier hierarchy ranks transcript consistency by measuring the maximum pairwise edit rate between a source transcript and two ASR hypotheses. Lower tiers indicate stronger agreement, from full exact match down to a 20–30% edit rate band, with a separate Mandarin homophone tier. The reported setup retains only Tier-1 transcripts for both distillation stages, decoupling strict consistency filtering from phase-specific source reweighting. Tier 1 (full agreement) is used for both distillation stages, keeping only transcripts where both ASR hypotheses exactly match the source. The hierarchy spans from exact match to a maximum edit rate of 30%, with a dedicated homophone tier for Mandarin where pinyin character error rate is zero.

Removing two layers from the 18-layer audio encoder produces a 16-layer model with roughly 10% fewer parameters and a 6.1% lower macro-average error. Stage 2 finetuning improves every benchmark over the Stage 1 checkpoint, with the largest gains on CommonVoice zh, CommonVoice en, and WenetSpeech-meeting, while Tedlium shows a small regression. The 16-layer model achieves a lower macro-average error than the 18-layer baseline, with a relative error reduction of 6.1%. Stage 2 finetuning consistently outperforms Stage 1 across all evaluated benchmarks, including AISHELL-1, Fleurs, LibriSpeech, THCHS-30, and Tedlium.

The 14-layer audio encoder reduces parameters by 20.7% while macro error increases only 0.14 percentage points over the 18-layer baseline. Stage 2 recovery improves every benchmark, with LibriSpeech test-clean nearly returning to baseline performance. The largest degradation occurs on Fleurs-en (0.93 pp), and progressive pruning from 18 to 16 to 14 layers is essential for achieving this small error increase. The 14-layer model uses 147.8M audio-tower parameters, 20.7% fewer than the 186.4M baseline, while macro error rises to 5.75%, just 0.14 pp above baseline. Stage 2 recovery improves all benchmarks; LibriSpeech test-clean WER drops from 3.09% to 2.45%, nearly matching the baseline 2.48%. Fleurs-en WER degrades by 0.93 pp (5.10% vs. 4.17%), the largest single-benchmark loss. Progressive pruning (18→16→14) achieves 5.75% mean error, substantially better than direct 18→14 pruning (6.73%), highlighting the importance of the pruning schedule. The small macro-error increase demonstrates a favorable tradeoff: substantial parameter reduction with minimal average accuracy cost.

A nine-tier transcript consistency hierarchy is used to filter training data, retaining only Tier-1 transcripts where both ASR hypotheses exactly match the source. Pruning the audio encoder from 18 to 16 layers reduces parameters by about 10% and lowers macro-average error, while further pruning to 14 layers cuts parameters by 20.7% with only a marginal error increase, and progressive pruning proves essential to avoid larger degradation. Stage 2 finetuning consistently improves all benchmarks over Stage 1, recovering near-baseline performance on most sets and demonstrating that strict consistency filtering combined with gradual encoder compression yields substantial parameter savings at minimal accuracy cost.