One-sentence Summary

ByteDance Seed researchers show that in visual generation, converged diffusion loss scales with structured language in prompts, decreasing approximately linearly with the white-box GPG metric and following a power law with the black-box ED metric, and they leverage these scaling properties to improve diffusability through structured prompts and promptability through supervised fine-tuning, cold-start, and verifier-gated on-policy distillation, outperforming all evaluated open-weight models on nearly every compositional, reasoning, and world-knowledge benchmark.

Key Contributions

• Introduces empirical scaling properties for text conditioning in visual generation by adapting a white-box grounding metric (GPG) and a black-box attribute metric (ED), and shows converged diffusion loss decreases approximately linearly with GPG and follows a power law with ED.
• Improves diffusability by constructing structured prompts with semantic and geometric annotations derived from images, and improves promptability by training a prompter through supervised fine-tuning, cold-start, and verifier-gated on-policy distillation; matched natural-language retraining shows the structured-representation gains are not due to extra training alone.
• The resulting system outperforms all evaluated open-weight models on nearly every compositional, reasoning, and world-knowledge benchmark, and matches or surpasses the strongest closed-weight models on most evaluations.

Introduction

Text-to-image generation has scaled model size and data, but unlike language models it learns from image-caption pairs rather than from the visual signal itself. Captions often bind visual content ambiguously or omit it, so longer natural-language prompts do not reliably improve generation and existing systems still struggle with dense prompts involving objects, layouts, relations, and visual coherence. The authors reframe the problem around caption information rather than caption length. They introduce structured prompts, a JSON-based semantic schema with explicit fields for scene context, per-element properties and geometry, and cross-element relations, and they adapt image-grounded metrics to quantify caption informativeness. These metrics predict converged diffusion loss under fixed training conditions, and an LLM prompter is trained to infer the structured fields at test time, yielding an end-to-end system with improved compositional generation without changing the diffusion architecture.

Dataset

Dataset Description

• Corpus and annotation objective The authors use a fixed set of training images and source annotations, but the excerpt does not report raw corpus size, original data sources, or filtering rules. Each training image is annotated with a faithful, full-schema L10 scene prompt, and the resulting SP levels are used as structured supervision for diffusion training.

• Annotation pipeline The authors construct image-to-SP supervision through a five-stage pipeline:

  • Stage 1: Reads the full image to establish global semantics: intent, scene, atmosphere, style, lighting, camera setup, plus an ordered element inventory with identifiers and bounding boxes.
  • Stage 2: Revisits each element crop so the VLM can resolve local descriptions, attributes, actions, and photography. Human elements also receive Sapiens 133-keypoint pose overlays to help determine body-side orientation and joint geometry.
  • Stage 3: Adds geometric evidence from DepthAnything V2 for relative depth and SAM 2.1 for masks and occlusion cues. Bounding boxes, masks, and depth support overlap, containment, relative position, and depth order.
  • Stage 4: Reconciles global semantics, crop-level semantics, and geometric evidence to serialize a well-formed L10 SP. Expert outputs constrain the annotation rather than being copied raw; inferred pose may be stored textually, while keypoints and masks remain intermediate evidence.
  • Stage 5: Projects the L10 annotation into the controlled field ladder L5-L9.

• Subsets and controlled field ladder L5 to L9 are derived from each full L10 SP by deterministic masking of predefined field groups. Moving from L5 to L10 successively restores:

  • bounding boxes
  • scene context
  • dynamic attributes
  • depth and relations
  • element-level photography

  Each level is a projection of the same L10 annotation, so images and source annotations stay fixed; only the caption content and organization exposed to the learner vary. NL controls are verbalized independently from the same underlying annotation evidence rather than converted from SP JSON.

• Use in the model The SP levels serve as structured supervision in diffusion training. For evaluation, a frozen Gemini 3 Pro generates one L10 SP zero-shot per user prompt. L5-L9 are derived from that same output and rendered by the BAGEL diffuser trained at the corresponding level. The paper reports monotonic improvement in GenEval2 GM and order-swapped GSB as schema richness increases. A field-wise ablation identifies global scene context as the largest individual contributor, followed by bounding-box conditioning.

• Processing and metadata notes Key processing details include bounding-box-defined element crops, pose keypoint overlays, depth maps, masks, and occlusion cues. The final metadata is a serialized full-schema SP with field groups; lower SP levels are deterministic masks, not new annotations.

Method

The authors propose a text-to-image generation framework centered around a Structured Prompt (SP), a typed JSON caption that organizes image-grounded variables into named fields. This SP serves as the shared interface between a large language model (LLM) prompter and a diffusion model, aiming to maximize both diffusability and promptability.

The SP organizes image-grounded information at three distinct scopes. Global fields describe the overall intent, scene, atmosphere, and lighting. Each foreground object receives a per-element entry detailing its identity, attributes, actions, bounding-box position, and optional depth. Cross-element relationships bind these entries together. The authors demonstrate through a fixed-backbone reconstruction probe that progressively restoring these SP fields improves reconstruction metrics, whereas simply lengthening natural-language captions yields flat performance.

To quantify the information exposed by different caption formats, the authors introduce two complementary metrics: Grounded Perplexity Gain (GPG) and Effective Detailness (ED). GPG measures the log-likelihood gain when a paired image is revealed to a frozen vision-language model, calculated as:

ED offers a semantic measure by matching caption attributes against an image-grounded reference set, reported as $F_{0.5}(P_{A}, R_{A})$F0.5​(PA​,RA​).

The authors establish scaling properties showing that converged diffusion training loss is well fit by a linear function of GPG and follows a power-law trend in ED. This calibration allows caption informativeness to predict matched-budget converged diffusion loss.

To raise diffusability at corpus scale, the authors develop a five-stage image-to-SP annotation pipeline. This pipeline combines general-purpose vision-language models with frozen domain experts to extract pose and geometry evidence.

The pipeline begins by establishing a shared semantic frame, recovering global intent and creating an element inventory. It then revisits each element crop to resolve local descriptions, utilizing Sapiens for human pose keypoints. Complementary geometric evidence is supplied via DepthAnything V2 for relative depth and SAM 2.1 for masks and occlusion cues. A final assembly stage reconciles this evidence with global and crop-level semantics to serialize a well-formed SP. To separate schema richness from annotation quality, the pipeline deterministically masks predefined field groups to derive lower-level SPs from the full annotation.

At inference, no paired image is available, so an LLM prompter must infer plausible visual details from a user request to instantiate the SP. The authors raise promptability through a staged training pipeline comprising Supervised Fine-Tuning (SFT), cold-start distillation, and Reinforcement Fine-Tuning (RFT). SFT teaches the SP content distribution expected by the diffuser. Cold-start distillation teaches the prompter to reason from a user prompt to a detailed SP using traces generated by an image-conditioned teacher. RFT moves beyond offline traces to the prompter's own rollouts. A verifier filters these trajectories using conservative thresholds, and On-Policy Self-Distillation (OPSD) trains the prompter on accepted trajectories by minimizing the divergence between the image-free prompter and an image-conditioned teacher:

This process ensures the prompter learns to generate detailed, structurally coherent SPs that effectively condition the diffusion model.

Experiment

The experiments evaluate the complete prompter-diffuser pipeline against representative text-to-image models, then isolate promptability by varying the LLM backend and training stages while fixing the schema and diffuser, and finally test iterative inference-time refinement. The structured-prompt interface yields broad gains over baselines and matched natural-language controls, while trained prompters transfer LLM progress and improve structural composition, layout, and text fidelity, with task-specific training contributing more than zero-shot schema filling. Iterative refine-render-judge refinement further improves structure and alignment, but gains saturate after a few rounds, indicating that prompter training absorbs most of the benefit and long-horizon prompt-side reasoning is not strongly required.

Restoring progressively richer structured-prompter fields from the base level to the full schema yields monotonic gains in both GenEval2 GM and GSB measured relative to the base level. Gains are especially notable when scene context, dynamic attributes, and depth and relationships are introduced. Because each level uses a BAGEL diffuser trained at that level, these results are not directly comparable with Qwen-Image systems. Both evaluation metrics improve with each richer field group, while average tokens per structured prompter also rise from 447 to 1374. The largest single-step gain in GSB over the base level occurs when scene context is added, and the largest GenEval2 improvement occurs when dynamic attributes are added.

The structured-prompt text-to-image system achieves leading or tied results among open-weight models across object-centric, dense prompt-following, world knowledge, and reasoning-focused benchmarks, and it matches or surpasses representative closed systems on most metrics. Matched comparisons on the same Qwen-Image backbone show that official prompt enhancement and matched natural-language retraining improve scores but remain well below the structured-prompt system, indicating the shared structured caption interface is responsible for the gains. These benefits appear across diverse prompt regimes and evaluators rather than being limited to one setting. Among evaluated open-weight systems, the structured-prompt model leads or ties nearly every reported metric and matches or surpasses closed systems on most, with gains spanning object alignment, dense prompt following, world knowledge, and reasoning. Controlled comparisons on the same architecture show that official prompt enhancement and matched natural-language retraining lag well behind the structured-prompt system, pointing to the structured caption interface as the key factor rather than model scale or training alone.

Holding the schema and image generator fixed, zero-shot structured prompting improves alignment and preference over natural-language rewriting but lowers DPG-Bench and structure scores. Training the prompt producer closes that gap, yielding the strongest single-turn backend across all four metrics. Native multi-turn coding agents provide additional gains, and an iterative refine-render-judge loop with the trained prompter surpasses the strongest coding agent while shortening the refinement trajectory. Zero-shot schema filling improves alignment and GSB over the same LLM's natural-language rewrite but consistently reduces DPG-Bench and structure. Training the prompt producer makes it the strongest single-turn backend on DPG-Bench, structure, alignment, and GSB. Coding-agent rows change both the model and inference procedure, so they do not isolate the backend model alone. With iterative refinement, the trained prompter reaches higher GSB than the strongest coding agent and needs fewer rounds on average than the base prompter.

Cumulative prompter training improves structure and preference, with supervised fine-tuning providing the largest single-stage structural gain and cold-start further improving structure and preference while slightly reducing alignment. Among RFT variants, verifier-gated OPSD yields the strongest endpoint across DPG-Bench, structure, alignment, and preference, outperforming verifier-reward GRPO and ungated OPSD. This indicates that combining high-confidence rollout selection with dense image-conditioned targets is more effective than reward-only or ungated optimization. SFT produces the largest single-stage structural gain and establishes a clear preference over the base prompter. Verifier-gated OPSD achieves the strongest overall metrics, while cold-start improves structure and preference at a slight alignment cost.

With the schema, image backbone, judge, and prompter weights held fixed, both the zero-shot and trained prompters improve as the refinement budget increases. Gains are larger for structural and layout quality than for basic alignment. The trained prompter starts from a much stronger position, reaches passing outputs in fewer rounds, and shows quickly diminishing returns from additional refinement. Increasing the number of refine-render-judge rounds improves structure more than alignment for both prompters, suggesting feedback mainly repairs object decomposition, relations, and layout. The trained prompter at a single round outperforms the zero-shot prompter's best eight-round result on structure and net preference, and it reaches a passing decision in fewer average rounds. For the trained prompter, moving from four to eight rounds adds only a small amount of structure and preference gain, so iterative correction saturates quickly.

The experiments evaluate a structured-prompt text-to-image pipeline through schema ablations, controlled model comparisons, backend variants, prompter training stages, and refinement budgets. Richer structured fields consistently improve generation quality, with scene context and dynamic attributes yielding the largest gains, while same-backbone comparisons attribute the improvements mainly to the structured caption interface rather than model scale or training alone. Training the prompt producer, especially with verifier-gated OPSD, produces the strongest single-turn results, and iterative refine-render-judge refinement further improves structure and preference but saturates after a few rounds.