One-sentence Summary

Researchers from Nanyang Technological University and SenseTime Research propose Unified Autoencoding (UAE), introducing the Prism Hypothesis that semantic encoders capture low-frequency global meaning while pixel encoders retain high-frequency details. UAE innovates with a frequency-band modulator harmonizing these representations within one latent space, effectively unifying semantic abstraction and pixel fidelity for state-of-the-art performance on ImageNet and MSCOCO benchmarks.

Key Contributions

• The paper identifies a critical fragmentation in foundation models where semantic encoders (focused on high-level meaning) and pixel encoders (preserving fine details) operate separately, causing representational conflicts and reduced training efficiency due to heterogeneous feature spaces.
• It introduces the Prism Hypothesis, revealing that semantic encoders predominantly capture low-frequency spectral components for abstract meaning while pixel encoders retain high-frequency details, and proposes Unified Autoencoding (UAE) with a frequency-band modulator to harmonize these into a single latent space.
• Extensive validation on ImageNet and MSCOCO shows UAE achieves state-of-the-art performance across reconstruction (rFID, PSNR, gFID) and perception (Accuracy) tasks, outperforming contemporaries like RAE and SVG by effectively unifying semantic abstraction and pixel fidelity.

Introduction

The authors address a critical fragmentation in vision foundation models where semantic encoders (capturing high-level meaning) and pixel encoders (preserving visual detail) operate in separate, incompatible latent spaces. This separation forces downstream systems to reconcile heterogeneous representations, reducing training efficiency and causing representational conflicts that degrade performance in joint perception-generation tasks. Prior approaches either transfer semantic knowledge into generation pipelines—sacrificing fine detail recovery—or inject pixel-level awareness into semantic models through distillation and hierarchical features, achieving only partial integration via compromises rather than true unification.

The authors introduce the Prism Hypothesis, which posits that natural inputs project onto a continuous frequency spectrum: low-frequency bands encode global semantics (e.g., categories and relations), while high-frequency bands capture local texture and geometry. Leveraging this insight, they propose Unified Autoencoding (UAE), a tokenizer that harmonizes semantic and pixel representations within a single latent space using a frequency-band modulator. This modulator explicitly factorizes content into a fundamental semantic band and residual pixel bands, enabling controllable detail reconstruction. UAE outperforms contemporary unified tokenizers like RAE, SVG, and UniFlow on ImageNet and MS-COCO across reconstruction (rFID, PSNR) and perception (gFID, accuracy) metrics, demonstrating that spectral factorization resolves the abstraction-fidelity tension without trade-offs.

Dataset

The authors use two primary datasets for training and evaluation:

• Training data: ImageNet-1K training set (standard 1.28M images), processed at 256x256 resolution. No additional filtering or cropping strategies are described beyond this resolution.
• Evaluation data:
  • ImageNet-1K validation set (standard 50k images) for classification benchmarks.
  • MS-COCO 2017 validation set for cross-dataset generalization tests.

Key implementation details:

• The model (UAE) trains exclusively on the ImageNet-1K training split, using DINOv2-B or DINOv2-L as semantic encoders.
• Training occurs in three stages:
  1. Decoder training with reconstruction loss (frozen encoder).
  2. Full-model fine-tuning with semantic-focused loss and reconstruction loss.
  3. End-to-end fine-tuning with noise injection and GAN loss (following RAE methodology).
• No dataset mixture ratios, metadata construction, or explicit preprocessing steps beyond resolution scaling are specified.

Method

The authors leverage the Prism Hypothesis to design Unified Autoencoding (UAE), a framework that unifies semantic abstraction and pixel-level fidelity within a single latent space by explicitly modeling the spectral decomposition of visual representations. The core insight is that natural inputs project onto a shared frequency spectrum, where low-frequency components encode semantic structure and high-frequency bands capture fine-grained visual detail. UAE operationalizes this by decomposing the latent representation into frequency bands and modulating them to preserve both global semantics and local fidelity.

Refer to the framework diagram, which illustrates how natural inputs—both image and text—are mapped through a frequency-band modulator into a unified latent spectrum. Semantic encoders such as DINOv2 or CLIP primarily occupy the low-frequency region, encoding visual abstraction, while pixel encoders like SD-VAE extend into higher frequencies to retain visual fidelity. UAE bridges these modalities by initializing its encoder from a pretrained semantic model and then expanding its spectral coverage through joint optimization.

The architecture begins with a unified encoder initialized from a pretrained semantic encoder (e.g., DINOv2), which processes the input image into a latent grid $\mathbf{z} \in \mathbb{R}^{B \times C \times H \times W}$z∈RB×C×H×W. This latent is then decomposed into $K$K frequency bands $\mathbf{z}_{\mathrm{f}} \in \mathbb{R}^{B \times K \times C \times H \times W}$zf​∈RB×K×C×H×W via an FFT Band Projector and an Iterative Split procedure. The projector applies a 2D discrete Fourier transform, partitions the spectrum using smooth radial masks $\mathbf{M}_k$Mk​, and transforms each band back to the spatial domain. The iterative split ensures disentanglement: starting with $\mathbf{r}^{(0)} = \mathbf{z}$r(0)=z, each band $\mathbf{z}_{\mathrm{f}}^{(k)} = \mathcal{P}_k(\mathbf{r}^{(k)})$zf(k)​=Pk​(r(k)) is extracted, and the residual is updated as $\mathbf{r}^{(k+1)} = \mathbf{r}^{(k)} - \mathcal{P}_k(\mathbf{r}^{(k)})$r(k+1)=r(k)−Pk​(r(k)). This yields a multi-band latent representation where low bands encode global semantics and high bands capture edges and textures.

As shown in the figure below, the decomposed bands are processed by a frequency band modulator. During training, a noise injection module perturbs only the high-frequency bands using a binary mask $\mathbf{m}$m and Gaussian noise $\mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I})$N(0,σ2I), enhancing decoder robustness. The processed bands are then concatenated along channels and passed through a spectral transform module—a two-layer convolutional block with SiLU activations—that predicts a residual $\Delta$Δ. The final fused latent $\mathbf{q} = \Delta + \sum_{k=0}^{K-1} \widehat{\mathbf{b}}^{(k)}$q=Δ+∑k=0K−1​b(k) maintains the original spatial dimensions and is fed into a ViT-based pixel decoder to reconstruct the RGB image.

Training is guided by two complementary objectives. The semantic-wise loss enforces alignment between the unified encoder and the frozen semantic encoder only on the lowest $K_{\text{base}}$Kbase​ bands (typically $K_{\text{base}} = 1$Kbase​=1), computed as: