One-sentence Summary

The authors propose AnyText, a diffusion-based framework for multilingual visual text generation and editing that addresses illegible character synthesis by integrating an auxiliary latent module processing text glyph, position, and masked image inputs with an OCR-driven text embedding module that fuses stroke data with image caption embeddings, while text-control diffusion and text perceptual losses ensure accurate and coherent text rendering.

Key Contributions

• AnyText is a diffusion-based framework that renders accurate multilingual text within synthesized images through an auxiliary latent module and a text embedding module. The auxiliary latent module encodes text glyphs, spatial positions, and masked regions, while the text embedding module fuses OCR-derived stroke embeddings with standard caption tokens to ensure seamless background integration.
• The training pipeline incorporates a text-control diffusion loss and a text perceptual loss in the image space to compensate for the limited character sensitivity of standard tokenizers. These objectives directly optimize character formation and structural coherence during both generation and editing tasks.
• The AnyWord-3M dataset and the AnyText-benchmark are introduced to systematically evaluate visual text generation accuracy and quality. The framework supports multi-line layouts, deformed regions, and cross-lingual prompts while maintaining consistent font styles during both text generation and editing.

Introduction

Diffusion-based text-to-image models have transformed digital content creation but consistently fail to render legible, accurate text, which severely limits their deployment in design, advertising, and media workflows. Prior methods struggle with this bottleneck due to a lack of large-scale datasets containing precise OCR annotations, text encoders that process vocabulary rather than individual characters, and loss functions optimized for global image quality instead of textual fidelity. The authors leverage a redesigned diffusion pipeline that incorporates an auxiliary latent module for spatial and glyph control, paired with an OCR-based embedding system that fuses stroke information directly with caption tokens. By training with dedicated text-control and perceptual losses, they deliver a plug-and-play solution for accurate multilingual text generation and editing, accompanied by a massive new dataset to standardize future evaluation.

Dataset

• Dataset Composition and Sources: The authors compile AnyWord-3M from publicly available images spanning diverse text-rich scenes like street views, book covers, advertisements, and movie frames. The primary sources include the Noah-Wukong corpus for Chinese content, LAION-400M for English content, and several established OCR datasets including ArT, COCO-Text, RCTW, LSVT, MLT, MTWI, and ReCTS.
• Subset Details and Filtering Rules: The final collection contains 3,034,486 images with over 9 million text lines. The authors divide the data into a training set and an evaluation set, randomly extracting 1,000 images from the Wukong and LAION subsets to form the Chinese and English AnyText-benchmark respectively. Image-level filtering enforces minimum dimensions of 256 pixels, an aspect ratio between 0.67 and 1.5, a text region covering at least 10% of the image, and a maximum of eight text lines per image. Text-line filtering requires a minimum height of 30 pixels, an OCR confidence score of at least 0.7, and non-empty content, with stricter thresholds applied to the English subset.
• Processing and Metadata Construction: The authors process non-OCR images using the PP-OCRv3 pipeline for detection and recognition. Captions are regenerated via BLIP-2, with specific placeholders removed and standardized phrases concatenated based on text line counts. These placeholders are later replaced with glyph embeddings during training. In the v1.1 update, Chinese annotations were refreshed using PP-OCRv4 to correct initial error rates, while English annotations were replaced with MARIO-LAION outputs for better handling of deformed text. The dataset also includes a wm_score metadata field that estimates watermark probability, enabling the removal of watermarked images during training.
• Training Strategy and Usage: The training split comprises approximately 1.6 million Chinese images, 1.39 million English images, and 10,000 images in other languages. The authors later adjusted the mixture to maintain an approximate 1:1 English to Chinese ratio. During fine-tuning, they implement an inv_mask technique that identifies low-confidence or undersized text lines and masks them out, setting the loss to zero in those regions to reduce pseudo-text artifacts. The final training phase filters out roughly 25% of Chinese and 8% of English data using the watermark score, applies a perceptual loss weight of 0.003, and switches the base generator to the Realistic_Vision community model for improved aesthetics.

Method

The authors leverage a diffusion-based framework for multi-lingual text generation and editing, termed AnyText, which integrates a text-control diffusion pipeline with two core auxiliary modules: the auxiliary latent module and the text embedding module. The overall framework is designed to generate text in diverse configurations, including arbitrary shapes and multiple languages, by combining semantic guidance with explicit visual control. As shown in the figure below, the process begins with an input image $x_0$x0​, which is encoded into a latent representation $z_0$z0​ using a Variational Autoencoder (VAE). This latent space serves as the foundation for the diffusion process, where noise is progressively added to $z_0$z0​ to produce a noisy latent image $z_t$zt​ at time step $t$t. The denoising process is guided by a network $\epsilon_\theta$ϵθ​ that predicts the noise component, conditioned on the time step $t$t, an auxiliary feature $z_a$za​ from the auxiliary latent module, and a text embedding $c_{te}$cte​ generated by the text embedding module. The training objective combines a text-control diffusion loss $\mathcal{L}_{td}$Ltd​ and a text perceptual loss $\mathcal{L}_{tp}$Ltp​, weighted by a hyperparameter $\lambda$λ, to ensure both faithful text generation and high accuracy in rendered text.

The auxiliary latent module generates the auxiliary feature $z_a$za​ by integrating three distinct conditions: glyph, position, and masked image. The glyph condition $l_g$lg​ is produced by rendering the target text using a uniform font style (Arial Unicode) onto an image based on the specified text locations. To handle text in curved or irregular regions, the position condition $l_p$lp​ is introduced, which marks the text positions on the image. During training, these positions are derived from OCR detection or manual annotation, while in inference, they are provided by the user. Additionally, a masked image $l_m$lm​ is used to indicate which regions of the image should be preserved during generation, enabling both text-to-image and text editing modes. These conditions are processed through separate blocks: a glyph block and a position block for $l_g$lg​ and $l_p$lp​, and a VAE encoder for $l_m$lm​, each downsampling the input to match the spatial resolution of $z_t$zt​. The resulting feature maps are then fused into a single latent feature $z_a$za​ using a convolutional fusion layer $f$f, which combines the outputs of the individual blocks.

The text embedding module addresses the challenge of multilingual text generation by transforming the text encoder to incorporate both semantic and glyph information. Instead of relying solely on character-level text encoders, which often struggle with non-Latin scripts, the authors render each text line into an image and use a pre-trained OCR recognition model (PP-OCRv3) to extract glyph features. These features are then integrated into the text embedding process. Specifically, the input caption $y$y is processed with special placeholders for each text line to be generated. After tokenization and embedding, the placeholder embeddings are replaced with the OCR-extracted glyph features through a linear transformation. The resulting composite embeddings are passed through a CLIP text encoder to generate a fused intermediate representation $c_{te}$cte​, which is then mapped to the UNet via cross-attention. This approach enables the model to generate text in multiple languages by leveraging visual glyph information rather than language-specific character encoders.

The text perceptual loss is introduced to improve the accuracy of text generation by supervising the process at the pixel level. After the denoising network predicts the noise $\epsilon_t$ϵt​, the model reconstructs the original image $x'_0$x0′​ using the VAE decoder. By leveraging the position condition $l_p$lp​, the framework can precisely locate the generated text region. The OCR model is then applied to both the original image $x_0$x0​ and the reconstructed image $x'_0$x0′​ to extract feature maps representing the text content. The text perceptual loss computes the Mean Squared Error (MSE) between these feature maps across all text regions, with a weight adjustment function $\varphi(t)$φ(t) that accounts for the quality of text at different diffusion steps. This loss ensures that the generated text is not only visually consistent but also accurate in terms of character recognition and spelling.

Experiment

The evaluation compares AnyText against leading text-to-image models and specialized baselines using standardized sampling parameters and combined quantitative and qualitative assessments for English and Chinese text generation. Results demonstrate that AnyText achieves superior text-background integration, visual realism, and generation accuracy, particularly excelling in complex Chinese character rendering where competing methods struggle. Ablation studies further validate the critical role of its specialized text embedding, positional encoding, and perceptual loss components in ensuring stable convergence and high-fidelity output. Collectively, these findings confirm that the framework delivers state-of-the-art text generation capabilities without introducing significant computational overhead.

The authors conducted an ablation study to evaluate the impact of different components in their model, using a small-scale dataset for training and validation. The results show that certain modules, particularly those related to text embedding and perceptual loss, significantly improve performance, while the overall parameter size and computational overhead remain comparable to the baseline model. The inclusion of a trainable text embedding module significantly improves text generation accuracy. The use of a pre-trained OCR model leads to a substantial increase in text generation performance. The addition of text perceptual loss further enhances accuracy, despite increasing training time.

The authors evaluate their model against existing methods using a benchmark that includes both English and Chinese text generation tasks. Results show that their approach achieves superior performance in both OCR accuracy and realism, particularly in Chinese text generation, outperforming all competing methods. The model demonstrates strong integration of generated text with backgrounds, as observed in qualitative comparisons. Ablation studies confirm the effectiveness of key components such as text embedding and perceptual loss, while maintaining computational efficiency comparable to baseline models. AnyText outperforms competing methods in both English and Chinese text generation, achieving higher OCR accuracy and better realism. The model shows significant improvements in Chinese text generation, where other methods struggle with character recognition and integration. Ablation studies validate the importance of text embedding and perceptual loss in enhancing generation quality while maintaining similar computational overhead.

The authors compare the parameter sizes of ControlNet and AnyText, showing that AnyText has a slightly higher total parameter count but maintains a similar scale to ControlNet. The additional modules in AnyText, such as the OCR model and text embedding components, contribute to the increase in parameters while still keeping the overall size manageable. AnyText has a total parameter count that is slightly larger than ControlNet but remains in a similar scale. The OCR model and text embedding components are the main contributors to the increased parameter count in AnyText. Despite the added modules, the overall parameter size of AnyText does not significantly exceed that of ControlNet.

The authors analyze the composition of the AnyText-benchmark dataset, highlighting the distribution of text across different languages and character counts. The dataset includes a substantial number of lines and characters, with English and Chinese texts being the most prominent, and a high proportion of short lines containing 20 or fewer characters. The dataset contains a large number of text lines and characters, with English and Chinese being the primary languages. A significant portion of the lines in the dataset have 20 or fewer characters. The dataset includes a diverse range of character types, with a high count of unique characters and words.

The authors compare different versions of their model, focusing on performance improvements in text generation for both English and Chinese. Results show a significant reduction in error rates for both watermark and pseudo-text generation, with the v1.1 version demonstrating substantial improvements over v1.0 in both languages. The v1.1 version shows a substantial improvement over v1.0 in reducing watermark and pseudo-text generation errors for both English and Chinese. The improvement in Chinese text generation is more pronounced, with a notable decrease in error rates for both watermark and pseudo-text types. The v1.1 version achieves a significant reduction in error rates, indicating enhanced accuracy and reliability in text generation.

The authors evaluate their model through comprehensive benchmark testing on English and Chinese text generation tasks, supplemented by ablation studies and iterative version comparisons. These experiments validate the contribution of core architectural components, demonstrating that integrating trainable text embeddings, perceptual loss, and pre-trained OCR modules substantially enhances generation accuracy and visual fidelity. Qualitative assessments highlight particularly strong performance in Chinese character rendering and seamless text-background integration, while maintaining computational efficiency comparable to baseline architectures. Overall, the findings confirm that strategic module additions and iterative refinements consistently improve generation reliability without imposing significant parameter overhead.