One-sentence Summary

PhotoMaker is an efficient personalized text-to-image method that encodes an arbitrary number of input ID images into a unified stacked ID embedding, and when trained via an ID-oriented data construction pipeline, it surpasses test-time fine-tuning approaches in identity fidelity and generation speed while preserving strong text controllability and robust generalization.

Key Contributions

• This work introduces PhotoMaker, an efficient personalized text-to-image framework that encodes an arbitrary number of reference images into a stacked ID embedding to comprehensively capture identity characteristics. This unified representation enables flexible identity merging and customization during inference without additional fine-tuning.
• An automated data construction pipeline is proposed to assemble a dedicated ID-oriented training dataset comprising multiple images per subject across diverse viewpoints, attributes, and scenarios. This pipeline automatically generates captions with corresponding class words to align with the framework training requirements.
• Extensive evaluations demonstrate that PhotoMaker achieves superior identity preservation compared to test-time fine-tuning baselines while delivering significant inference speedups. The framework produces high-quality generation results with strong generalization capabilities across diverse customization applications.

Introduction

Text-to-image diffusion models have transformed personalized portrait generation, yet practitioners struggle to simultaneously achieve high efficiency, strict identity fidelity, and flexible text controllability. Existing methods force a compromise between slow, computationally heavy test-time fine-tuning and faster embedding-based approaches that often memorize irrelevant traits or fail to preserve accurate identity features. To resolve these limitations, the authors introduce PhotoMaker, a feed-forward framework that encodes an arbitrary number of reference images into a unified stacked ID embedding. This design enables the base diffusion model to adaptively integrate comprehensive identity information without additional parameters, delivering generation speeds over one hundred times faster than traditional fine-tuning. Furthermore, the researchers develop an automated pipeline to construct a diverse, ID-oriented training dataset, which significantly improves editability, viewpoint generalization, and the ability to merge distinct identities while maintaining real-time performance.

Dataset

1. Dataset composition and sources

• The authors construct an identity-oriented human text-image dataset sourced from celebrity rosters in VoxCeleb1 and VGGFace2.
• The collection is divided into a training corpus and a strictly non-overlapping evaluation set.
• The training corpus contains approximately 112,000 images organized under roughly 13,000 distinct identity names, while the evaluation set combines manually selected identities with a portion of the MyStyle dataset.

2. Key details for each subset

• Training subset: The pipeline initially downloads roughly 100 images per celebrity, filtering out any image with a shortest dimension below 512 pixels.
• Face and identity validation: RetinaNet detects faces, discarding images that lack bounding boxes of at least 256 by 256 pixels. Remaining faces undergo ArcFace verification, where similarity scores are aggregated to isolate the correct identity. The authors filter out inconsistent identities using a threshold set at eight times the standard deviation of the group scores.
• Evaluation subset: Each test identity provides four reference images for metric computation, including DINO, CLIP-I, and Face Similarity. Single-embedding baselines receive one randomly selected image per identity. The test prompts span 40 variations covering clothing, accessories, actions, expressions, views, and backgrounds.

3. How the paper uses the data

• The training data drives the PhotoMaker model by enabling the sampling of multiple images per identity to construct stacked identity embeddings.
• The authors utilize the dataset as a dedicated identity-consistent training corpus without specified mixture ratios, prioritizing cross-attribute and cross-scene diversity within each identity group.
• The evaluation split isolates unseen identities to rigorously test zero-shot identity transfer, ensuring no overlap with the training set to measure genuine generalization.

4. Cropping strategy, metadata construction, and other processing details

• Cropping: Each image is framed into a square crop centered on the verified face, ensuring the facial region occupies at least 10 percent of the final image.
• Mask generation: Mask2Former performs panoptic segmentation on the person class. The mask overlapping most with the verified face bounding box is retained, while images lacking a valid mask or overlap are discarded.
• Captioning and annotation: BLIP2 generates initial captions, which are regenerated in random mode until a singular class word appears. The authors then annotate the precise position of this class word. For captions containing multiple class words, they use frequency analysis per identity group, dependence parsing, and combined CLIP and SentenceFormer similarity scores to accurately mark the word corresponding to the target identity.
• Final metadata: Every processed image is paired with its corresponding identity mask and the precisely annotated caption to facilitate text-image embedding fusion during training.

Method

The authors leverage a novel framework designed for efficient and controllable personalized human image generation, enabling the creation of photo-realistic images that retain identity characteristics while adhering to new textual prompts. The core of the method lies in a simple yet effective stacked ID embedding mechanism, which allows the system to process multiple input identity images without requiring model fine-tuning, thereby achieving the efficiency of tuning-free approaches. This framework operates within a diffusion model architecture, where identity information is encoded and fused with text prompts to guide the generation process. The overall system consists of several key modules: an image encoder, a text encoder, a fusion module, and a merging mechanism that integrates identity and text information into the diffusion model.

Refer to the framework diagram. The process begins with user inputs consisting of one or more identity images. These images are first processed by an image encoder, specifically the CLIP image encoder $\mathcal{E}_{img}$Eimg​, which extracts feature embeddings for each input. To ensure the model focuses on the subject of interest and ignores background clutter, the non-body regions of each input image are masked with random noise prior to encoding. To enhance the encoder's ability to extract identity-specific features from these masked images, the authors fine-tune a portion of the transformer layers within the image encoder during training. The extracted embeddings, denoted as $\{e^{i} \in \mathbb{R}^{D} \mid i=1 \ldots N\}${ei∈RD∣i=1…N}, where $D$D is the projected dimension, are then fed into a fusion module.

For a given text prompt $T$T, the CLIP text encoder $\mathcal{E}_{text}$Etext​ generates a text embedding $t \in \mathbb{R}^{L \times D}$t∈RL×D, where $L$L is the sequence length. To integrate identity information with the text prompt, the method marks the corresponding class word (e.g., "man" or "woman") in the input caption. The feature vector at the position of this class word in the text embedding is extracted and fused with each image embedding $e^{i}$ei using two multi-layer perceptrons (MLPs). This fusion creates a comprehensive representation of each input ID, which is crucial for both identity fidelity and semantic controllability during inference, as demonstrated by the ability to modify attributes like age and gender by altering the class word.

As shown in the figure below, the fused embeddings are concatenated along the length dimension to form a single, variable-length stacked ID embedding $s^* \in \mathbb{R}^{N \times D}$s∗∈RN×D. This representation serves as a unified vector that captures the identity information of all input images, regardless of the number of inputs. This stacking process is a key differentiator from tuning-based methods like DreamBooth, which require multiple forward passes and model updates. The stacked ID embedding is then used to modify the text embedding. Specifically, the original text embedding $t$t is updated by replacing the feature vector at the class word's position with the stacked ID embedding $s^*$s∗, resulting in an updated text embedding $t^* \in \mathbb{R}^{(L+N-1) \times D}$t∗∈R(L+N−1)×D. This updated embedding is then fed into the diffusion model.

The final step involves merging the ID information into the diffusion process. The authors utilize the inherent cross-attention mechanism of the diffusion model. The updated text embedding $t^*$t∗ is used as the key and value ($\mathbf{K}, \mathbf{V}$K,V) in the cross-attention layers, while the query ($\mathbf{Q}$Q) is derived from the latent features of the denoising process. The cross-attention operation is formulated as:

$$\left\{ \begin{array} { l l } { \mathbf { Q } = \mathbf { W } _ { Q } \cdot \phi ( z _ { t } ) ; \; \mathbf { K } = \mathbf { W } _ { K } \cdot t ^ { * } ; \; \mathbf { V } = \mathbf { W } _ { V } \cdot t ^ { * } } \\\\ { \mathrm { A t t e n t i o n } ( \mathbf { Q } , \mathbf { K } , \mathbf { V } ) = \operatorname { s o f t m a x } ( \frac { \mathbf { Q } \mathbf { K } ^ { T } } { \sqrt { d } } ) \cdot \mathbf { V } , } \end{array} \right.$$⎩⎨⎧​Q=WQ​⋅ϕ(zt​);K=WK​⋅t∗;V=WV​⋅t∗Attention(Q,K,V)=softmax(d​QKT​)⋅V,​

where $\phi(\cdot)$ϕ(⋅) is the embedding from the UNet denoiser, and $\mathbf{W}_{Q}$WQ​, $\mathbf{W}_{K}$WK​, and $\mathbf{W}_{V}$WV​ are projection matrices. To further enhance the model's sensitivity to the ID information, the authors train LoRA residuals for the attention layer matrices. This allows for fine-grained control over the influence of each input ID, as the degree of participation can be adjusted through prompt weighting. This modular design enables the system to generate diverse, high-fidelity human images that are both identity-consistent and text-controllable.

Experiment

The proposed method was evaluated on unseen identities using diverse prompts and benchmarked against established baselines through quantitative metrics and a comprehensive user study to assess identity fidelity, text alignment, generation quality, and facial diversity. Experiments validated its capabilities across several key applications, including universal recontextualization, age and gender transformation, multi-identity blending, and artistic stylization. Qualitative assessments and user feedback consistently demonstrated that the approach successfully preserves target identities while producing high-quality, diverse outputs with strong prompt adherence. Ultimately, the method outperforms existing techniques in balancing identity retention with creative editability, although it currently prioritizes single-subject portraits and inherits underlying dataset biases.

The authors evaluate their method using a set of metrics that measure identity fidelity, text consistency, face similarity, and face diversity. The results show that the proposed stacked ID embedding approach achieves a balance between high identity fidelity and diverse facial generation, outperforming the baseline methods in most metrics. The stacked approach also demonstrates improved performance in maintaining identity information compared to linear combination methods. The stacked ID embedding method achieves higher identity fidelity and face diversity compared to the linear combination approach. The stacked method outperforms the linear method in face similarity and face diversity metrics. The stacked approach shows improved performance in maintaining identity information while preserving facial diversity.

The authors compare different methods for generating personalized images using varying numbers of input ID images. Results show that using multiple ID images improves identity fidelity, particularly in terms of DINO and Face Similarity metrics, while also enhancing face diversity. The proposed method achieves a balance between high-quality generation and identity preservation. Using multiple ID images improves identity fidelity and face diversity compared to single-image methods. The proposed method achieves higher face diversity while maintaining strong identity fidelity. The proposed method balances high-quality generation with effective identity preservation across multiple metrics.

The authors compare their PhotoMaker method with existing approaches in a recontextualization setting, evaluating ID fidelity, text consistency, face diversity, and generation quality. Results show that PhotoMaker achieves the highest scores in ID-related metrics and face diversity, while maintaining competitive performance in text fidelity and generation quality. The method also demonstrates superior inference speed compared to most baselines. PhotoMaker achieves the highest ID fidelity and face diversity among all methods. PhotoMaker maintains strong text consistency and generation quality while outperforming other methods in inference speed. PhotoMaker demonstrates superior performance in ID-related metrics compared to DreamBooth, Textual Inversion, FastComposer, and IPAdapter.

The authors present a comparative evaluation of their PhotoMaker method against several baselines, including IPAdapter, FastComposer, and DreamBooth, using a set of metrics such as ID fidelity, quality, diversity, and text fidelity. The results show that PhotoMaker achieves the highest scores across all evaluated metrics, particularly in text fidelity and diversity, while also maintaining strong performance in ID fidelity and quality. The comparison highlights the effectiveness of the stacked ID embedding approach in enhancing identity preservation and generation quality. PhotoMaker outperforms all compared methods in text fidelity and diversity metrics. PhotoMaker achieves the highest score in ID fidelity and quality, demonstrating superior identity preservation and generation quality. The method shows consistent advantages across all evaluation dimensions, indicating a balanced improvement in performance.

The evaluation compares a stacked identity embedding approach against established baselines by testing single versus multiple reference images and validating performance across identity preservation, text consistency, facial diversity, and generation quality. The experiments demonstrate that stacking embeddings and utilizing multiple inputs significantly strengthens both identity fidelity and facial diversity without compromising text alignment or visual quality. Ultimately, the proposed method consistently outperforms existing techniques across all dimensions while offering faster inference speeds, confirming its ability to balance high-quality generation with robust identity preservation.