One-sentence Summary

Researchers from Fudan University, Tencent Inc., and South China University of Technology propose ScriptMoE, a script-aware Mixture-of-Experts recognizer that uses an image-level router to dispatch images to top-2 script-aligned experts and a shared expert; trained on the TextMuSS-10M dataset spanning 10 scripts and 229 languages, it achieves 82.06% on TextMuSS-Bench and lifts CC-OCR $F_1$F1​ to 80.89%.

Key Contributions

• Introduces TextMuSS-10M, a large-scale synthetic scene text dataset covering 10 scripts and 229 languages, and TextMuSS-Bench, a real-image benchmark spanning all ten scripts with 10,899 images.
• Proposes ScriptMoE, a script-aware Mixture-of-Experts architecture with a shared visual encoder and a sparse MoE decoder where an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert.
• Experiments report 82.06% average accuracy on TextMuSS-Bench, 1.31% above the strongest STR baseline, and replacing the PP-OCRv5 recognizer with ScriptMoE on the CC-OCR task raises F1 from 65.71% to 80.89%, surpassing the best VLM score of 80.73% at a fraction of the parameter count.

Introduction

Scene text recognition in the wild is a core OCR task, but most accurate models focus on English and Chinese, while deployed systems must handle many scripts such as Arabic, Cyrillic, Hindi, Japanese, Korean, and Thai. Prior multilingual approaches either use per-language expert recognizers with a separate language-identification step, which raises cost and error accumulation, or rely on large vision-language models that are too heavy for edge deployment; moreover, real training data for non-English and non-Chinese scripts is scarce. The authors address these gaps by building TextMuSS-10M, a balanced synthetic dataset covering 10 scripts and 229 languages, and TextMuSS-Bench for real evaluation, and by proposing ScriptMoE, a script-aware Mixture-of-Experts recognizer that shares a visual encoder while routing each image to top-2 script experts plus an always-on shared expert. This gives a single lightweight all-in-one model that improves average accuracy on their multilingual benchmark and reaches end-to-end OCR performance on par with much larger vision-language models.

Dataset

The authors build a large synthetic multilingual scene-text dataset called TextMuSS-10M after concluding that SynthMLT is too small and covers too few languages for high-accuracy training.

• Dataset composition and sources

  • TextMuSS-10M contains 1M synthetic samples per script across ten target scripts, for a total of 10M samples.
  • Backgrounds come from 8,000 text-free scene images or pure-color images.
  • Text corpora are collected per script, with 100K to 1M words per script.
  • For Latin, which covers many languages, the authors collect more words and balance across languages.
  • Longer text is simulated by concatenating word-level text with spaces.
  • Random permutations of the character table add meaningless text to balance character and length distributions and cover rare characters.
  • News Crawl newspaper corpora provide real sentence-level text, from which phrases and sentences of various lengths are extracted.

• Key processing details

  • The authors adapt the UnionST synthesis engine.
  • A character vocabulary is built for each target script using PP-OCRv5 MLT and standard script definitions.
  • This character set is used for filtering and sampling expansion.
  • Layout templates include horizontal, vertical, multidirectional rotation, and curved text.
  • Text layers receive effects such as shadow, distortion, and perspective before being overlaid on backgrounds.
  • Script-specific adjustments include:
    • Chinese, Japanese, and Korean use 20% vertical text synthesis.
    • Other scripts use 5% vertical text synthesis.
    • Arabic text is rendered from right to left and saved in logical order.

• How the dataset is used

  • TextMuSS-10M serves as the foundation for subsequent script-balanced, high-accuracy scene text recognition models.
  • The section does not specify a training split, mixture ratios, cropping strategy, or metadata construction details.

Method

The authors identify data scarcity for long-tail scripts as a primary obstacle in building a high-accuracy multilingual scene text recognizer. To address this, they construct TextMuSS-10M, a balanced multilingual synthetic dataset. They adapt the UnionST synthesis engine by collecting specific character vocabularies and diverse corpora for each target script. The corpus collection includes word-level text, concatenated phrases to simulate longer text, random permutations for rare characters, and extracted sentences from newspaper corpora to simulate real semantics. Language-specific adjustments are applied, such as a higher proportion of vertical text for CJK scripts and right-to-left rendering for Arabic. The specific synthesis flow involves selecting text-free scene images as backgrounds, rendering text using preset layout templates including horizontal, vertical, rotated, and curved formats, applying effects like shadow and distortion, and overlaying the text layer onto the background. The resulting synthesis examples are displayed below.

To overcome the capacity limitations of a single dense decoder, the authors propose ScriptMoE, which leverages the single-image-few-script prior. Refer to the framework diagram for the full pipeline overview.

The architecture begins with a hierarchical visual encoder (SVTRv2) that maps an input image $\mathbf{x} \in \mathbb{R}^{H \times W \times 3}$x∈RH×W×3 to a sequence of visual tokens $\mathbf{F} \in \mathbb{R}^{L \times d}$F∈RL×d. These tokens are fed into a script-aware Transformer decoder where the standard feed-forward network (FFN) is replaced by a Mixture-of-Experts (MoE) block. The remaining components operate identically to a vanilla Transformer decoder, producing output autoregressively. Let $\mathbf{h}_t \in \mathbb{R}^d$ht​∈Rd be the decoder hidden state at step $t$t. The MoE layer computes the output as:

$$\mathrm{MoE}(\mathbf{h}_t) = \alpha_t \mathrm{FFN}_{\text{share}}(\mathbf{h}_t) + (1 - \alpha_t) \sum_{i=1}^n g_i \mathrm{FFN}_i(\mathbf{h}_t)$$MoE(ht​)=αt​FFNshare​(ht​)+(1−αt​)i=1∑n​gi​FFNi​(ht​)

where the gating weights $g_i$gi​ are determined by:

$$g_i = \frac{p_i \cdot \mathbb{I}\{i \in \mathrm{TopK}(\mathbf{p})\}}{\sum_j p_j \cdot \mathbb{I}\{j \in \mathrm{TopK}(\mathbf{p})\}}$$gi​=∑j​pj​⋅I{j∈TopK(p)}pi​⋅I{i∈TopK(p)}​

and the routing probabilities $\mathbf{p}$p are calculated via:

$$\mathbf{p} = \text{softmax}\big(W_g(\bar{\mathbf{F}} \odot (1 + \sigma \cdot \boldsymbol{\epsilon}))\big)$$p=softmax(Wg​(Fˉ⊙(1+σ⋅ϵ)))

Here, $\bar{\mathbf{F}} \in \mathbb{R}^d$Fˉ∈Rd is the image-level router input obtained by mean-pooling the visual tokens $\mathbf{F}$F. $\boldsymbol{\epsilon} \sim \mathcal{N}(0, I)$ϵ∼N(0,I) is a multiplicative router jitter applied during training, $W_g$Wg​ is the router projection, and TopK selects the highest-scoring experts (Top-2). A learnable per-token gate $\alpha_t = \text{sigmoid}(\mathbf{w}_s^\top \mathbf{h}_t)$αt​=sigmoid(ws⊤​ht​) balances the shared and routed branches.

Crucially, the router input is formed once per image rather than per token, meaning all output tokens of an image are processed by the same routed experts. This design reduces routing cost, avoids token-level instability, and allows each expert to act as a script specialist. The authors group the ten target scripts into four categories based on character morphology: Alphabet (Latin, Cyrillic), CJK (Chinese, Japanese, Korean), Arabic family, and Others (Hindi, Bangla, Tibetan, Thai).

The shared expert $\mathrm{FFN}_{\text{share}}$FFNshare​ is always activated regardless of the routing decision. It absorbs script-invariant features such as digits, punctuation, and geometric distortions, preserving cross-script transfer and preventing the fragmentation of common knowledge.

To ensure the router aligns with actual scripts rather than discovering arbitrary groupings, the authors introduce a script-classification signal. An auxiliary four-way classification head $h_{\text{scls}}: \mathbb{R}^d \to \mathbb{R}^4$hscls​:Rd→R4 is attached to the same pooled router input $\bar{\mathbf{F}}$Fˉ. This head is trained using cross-entropy loss against the script-group label $y_x$yx​, which is derived automatically by mapping the Unicode ranges of the ground-truth transcription characters to one of the four script groups:

$$\mathcal{L}_{\text{scls}} = -\frac{1}{|\mathcal{B}|} \sum_{x \in \mathcal{B}} \log \text{softmax}\big(h_{\text{scls}}(\bar{\mathbf{F}}_x)\big)_{y_x}$$Lscls​=−∣B∣1​x∈B∑​logsoftmax(hscls​(Fˉx​))yx​​

where $\mathcal{B}$B is the mini-batch. This classifier shares the router input but not the router weights, providing a script-aware learning signal while allowing the router to form useful within-script sub-populations.

The entire ScriptMoE model is trained end-to-end by minimizing the combined objective:

$$\mathcal{L} = \mathcal{L}_{\text{ar}} + \lambda_{\text{scls}} \mathcal{L}_{\text{scls}}$$L=Lar​+λscls​Lscls​

where $\mathcal{L}_{\text{ar}}$Lar​ is the standard autoregressive cross-entropy loss. The auxiliary term $\mathcal{L}_{\text{scls}}$Lscls​ is weighted with $\lambda_{\text{scls}} = 0.1$λscls​=0.1 to orient the experts towards scripts without overriding the router flexibility.

Experiment

ScriptMoE is evaluated on the TextMuSS-Bench STR benchmark covering ten scripts and on end-to-end multilingual OCR with CC-OCR, using a fixed detector and training baselines on identical data for fair comparison. The main results show that ScriptMoE improves recognition especially on low-resource scripts and that its router learns script-specific expert specialization, while broad generalist OCR systems do not guarantee multilingual accuracy. Ablations confirm that synthetic data is essential for unseen low-resource scripts, that a moderately sparse mixture-of-experts decoder is more effective than dense or overly specialized variants, and that script-aware components provide the largest benefits for low-resource scripts.

The benchmark comprises 10,899 images across ten major scripts. Latin is the dominant script group, contributing more than half of all images, while the remaining scripts have much smaller and uneven representation. Among non-Latin scripts, Russian and Thai have the largest evaluation sets, and Chinese and Tibetan are among the smallest. Latin contributes the majority of images in the benchmark. Non-Latin script representation is uneven, with Russian and Thai larger than Chinese and Tibetan. Several scripts, including Arabic, Bangla, Hindi, and Tibetan, have relatively small evaluation sets.

ScriptMoE attains the highest average text recognition accuracy among STR-class models on TextMuSS-Bench and outperforms the strongest baseline, with the largest gains on low-resource scripts such as Arabic, Thai, and Tibetan. General and OCR-specialized vision-language systems trail substantially, and several fail almost completely on Arabic, Bangla, and Tibetan. ScriptMoE achieves this while activating far fewer parameters per image than VLM-based alternatives. ScriptMoE reaches 82.06% average word accuracy, improving over the strongest STR baseline by 1.31 percentage points. Gains are concentrated on low-resource scripts, with notable improvements for Arabic, Thai, and Tibetan. Generalist vision-language OCR systems trail by 18 to 70 average points and some collapse on Arabic, Bangla, and Tibetan. ScriptMoE activates far fewer parameters per image than VLM-based systems.

On the multilingual end-to-end OCR benchmark, general vision-language models vary widely, with Qwen3.5-9B leading and InternVL3.5-8B clearly trailing. Replacing the PP-OCRv5 MLT recognizer with ScriptMoE lifts overall F1 from about 66 to about 81, narrowly surpassing the strongest generalist and specialized baselines. Because detection is fixed, the gain is attributed to stronger script-aware recognition. Among general VLMs, Qwen3.5-9B posts the highest total F1, followed closely by Gemini-1.5-Pro and Qwen2.5-VL-72B; InternVL3.5-8B lags substantially, especially on Arabic, Russian, and German. ScriptMoE-based OCR edges past the strongest zero-shot general VLM, the best OCR-specialized VLM, and expert OCR baselines while using a fixed detector, indicating recognition quality drives the improvement. Latin-script languages such as Spanish, Portuguese, French, and Italian tend to score higher across systems, while performance on Korean, Japanese, and Arabic is more variable.

Real-only training performs well on the seven scripts it sees but fails on Russian, Thai, and Tibetan. Synthetic-only training covers all ten scripts and already surpasses real-only on the seven-script average, while the combined setup achieves the highest overall accuracy. The main trade-offs are small drops in Latin and Russian, attributed to visual confusion between Latin and Cyrillic. Real-only training collapses on Russian, Thai, and Tibetan, while synthetic-only training lifts those scripts from zero to substantial accuracy and already beats real-only on the seven-script average. Combining real and synthetic data produces the best overall accuracy, though adding synthetic data slightly lowers Latin and adding real data slightly lowers Russian, indicating Latin-Cyrillic cross-script confusion.

The MoE decoder is essential for multi-script recognition: the pure AR baseline without experts trails MoE configurations except on Latin. Four experts with top-2 routing achieve the best average accuracy, while scaling to one expert per script or activating more experts per item does not improve overall performance. Script-aware components provide further gains, with the shared expert being especially important for low-resource scripts. The no-MoE baseline underperforms MoE variants on average and only remains competitive on Latin, the most data-rich script. Four experts with top-2 routing yield the best average accuracy, with notable gains on Thai and Chinese compared with the baseline. Using ten experts, one per script, reduces accuracy relative to four experts because each expert sees too few samples to specialize. Top-1 routing beats the no-MoE baseline but is weaker than top-2 on Thai and Chinese; top-4 routing adds cost without meaningful average benefit and hurts Japanese and Korean. Token-level routing performs similarly to image-level routing overall, with small per-script trade-offs but higher router cost. Removing the shared expert causes a larger average drop than removing the script-classification signal, concentrated on low-resource scripts such as Arabic and Tibetan.

The experiments evaluate multilingual scene text recognition on a ten-script benchmark where Latin dominates and non-Latin scripts are unevenly represented, and they also test end-to-end OCR with a fixed detector. ScriptMoE outperforms strong STR baselines and generalist or OCR-specialized vision-language models, with the largest gains on low-resource scripts, and it lifts end-to-end OCR F1 from about 66 to about 81, indicating that recognition quality drives the improvement. Data ablations show that real-only training fails on unseen scripts, synthetic data is essential for coverage, and combined training works best except for small Latin/Cyrillic confusion. MoE ablations confirm that the expert decoder is necessary, four experts with top-2 routing are most effective, and the shared expert especially benefits low-resource scripts.