One-sentence Summary

ReChannel, proposed by UCSD and HKUST researchers, repurposes a pretrained DiT by replacing its VAE decoder with a lightweight token-to-patch linear head ($\sim 33\text{K}$∼33K parameters) to directly output task-native pixel-space fields, achieving state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation while being $2.48\times$2.48× faster than an edit-plus-latent-decode counterpart in a matched $4\text{B}$4B setting.

Key Contributions

• ReChannel reinterprets the output tokens of a frozen pretrained Diffusion Transformer (DiT) as spatial carriers, mapping each token to a fixed pixel patch whose channels carry task-native quantities (e.g., depth, matting alpha) instead of RGB, thereby eliminating the target-side VAE decoder.
• The method uses a minimal, unified interface: a frozen text-to-image DiT backbone, lightweight task-specific LoRA adaptation, and a shared token-local linear head (about 33K parameters) that directly reads out pixel-space patches without any spatial decoder or upsampling, handling geometry, masks, segmentation, and heatmaps in one form.
• Across six dense prediction tasks and more than a dozen benchmarks, ReChannel achieves new state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation, stays competitive on surface normals, saliency, and pose, and in a matched 4B-parameter setting is more accurate and 2.48× faster than an edit-plus-latent-decode counterpart.

Introduction

RGB-based dense prediction estimates pixel-aligned fields such as depth, masks, mattes, and heatmaps from a single image, a capability central to many vision tasks. Current approaches either rely on discriminative foundation encoders with task-specific decoders or repurpose text-to-image generative models as priors. The generative route has proven effective, but existing methods largely treat dense targets as images to be synthesized: they encode task fields into an RGB-trained VAE latent space and then decode them back to pixels. This target-side rendering interface is indirect, imposing an RGB reconstruction burden on quantities that are not appearance, and it adds computational overhead. The authors argue that the generative prior’s value lies in the rich spatial field it already organizes, not in its ability to generate the target. They introduce ReChannel, a method that drops the target-side VAE. Instead, a frozen diffusion transformer backbone is lightly adapted with task-specific LoRA, and a token-local linear projection directly reads out task-native patch fields from the adapted token grid. This reframes dense prediction as rechanneling the spatial carriers rather than generating an image, yielding a unified, minimal interface that achieves state-of-the-art results across six tasks while being up to 2.48 times faster than an edit-plus-latent-decode counterpart.

Method

The authors propose ReChannel, a framework that repurposes a pretrained text-to-image diffusion transformer for dense prediction tasks without using the generative decoder. The core idea is to treat the spatial token grid of the transformer as a readout substrate, directly projecting each token to a local pixel-space patch via a lightweight linear mapping. This avoids the conventional image-generation interface and keeps the dense targets in their native pixel-aligned format.

The input pipeline retains the original RGB pathway of the pretrained T2I model. An input image $x$x is encoded by the frozen VAE encoder, producing a latent that lies in the pretrained model’s distribution. The latent is then fed into the diffusion transformer (DiT) backbone, which is run in a deterministic zero-noise mode. The backbone weights stay frozen, and task-specific adaptation is injected through low-rank (LoRA) parameters $\Delta_t$Δt​:

$Z_t = F_{\theta + \Delta_t} \left(\mathrm{VAE}_{\mathrm{enc}}(x); \sigma = 0, c_t\right),$Zt​=Fθ+Δt​​(VAEenc​(x);σ=0,ct​),

where $F_\theta$Fθ​ is the pretrained transformer, $c_t$ct​ is the text condition (e.g., a referring expression), and $Z_t = \{z_{ij}^t\}$Zt​={zijt​} is the resulting spatial token field for task $t$t. This design preserves the pretrained RGB field structure while steering it toward target semantics through LoRA.

The key innovation is the readout stage. Instead of a generative decoder or a multi-stage spatial head, each adapted token $z_{ij}^t$zijt​ is mapped independently to a local dense patch using a shared token-local linear projection:

$\hat{Y}_{ij}^t = \text{reshape}\left(W_t z_{ij}^t + b_t\right), \quad \hat{Y}_{ij}^t \in \mathbb{R}^{p \times p \times K_t}.$Y^ijt​=reshape(Wt​zijt​+bt​),Y^ijt​∈Rp×p×Kt​.

Here $p$p is the patch size and $K_t$Kt​ the number of output channels for the task. The projection is applied identically at every token, with no inter-token spatial mixing. The reshape operation merely unpacks the projection into pixel space; the spatial structure of the prediction is entirely inherited from the adapted token field. This is the ReChannel readout: it reinterprets each token’s channels from RGB appearance to task-native fields while leaving the spatial carrier intact.

The same readout mechanism is instantiated across diverse tasks. For each task, a separate LoRA adapter and readout projection are trained, with $K_t$Kt​ and the supervision loss chosen according to the target representation. Scalar fields like depth, alpha mattes, and saliency use $K_t = 1$Kt​=1; surface normals use $K_t = 3$Kt​=3; pose estimation uses multi-channel heatmaps. Continuous regression tasks are trained with standard pixel-space losses, while binary mask tasks use pixel-space mask supervision. The loss functions are kept conventional, as the focus is on the output interface rather than task-specific loss engineering.

Experiment

ReChannel is evaluated on six dense prediction tasks by adapting a frozen FLUX-Klein text-to-image backbone with a LoRA adapter and a token-local linear head, following each benchmark's standard protocol. It achieves state-of-the-art or competitive results across depth, matting, referring segmentation, and pose, while ablations confirm that LoRA adaptation and a strong pretrained prior are necessary, and that the adapted token field already encodes the output, making extra output machinery or target-side VAE decoding redundant. The token-local readout is the most accurate configuration and also the fastest, as it eliminates the decoder entirely.

ReChannel-9B achieves the best depth and surface normal estimates on most benchmarks, surpassing both specialized and generative baselines. For monocular depth, it leads on KITTI and ScanNet while narrowly trailing one method on NYU. On surface normals, ReChannel-9B attains the lowest mean angular error on all three datasets, and its smaller ReChannel-4B variant is already competitive with the strongest generative approaches. ReChannel-9B sets the best absolute relative depth error on KITTI and ScanNet, improving over Edit2Perc on those two datasets. On surface normals, ReChannel-9B obtains the lowest mean angular error on NYU, ScanNet, and iBims, while ReChannel-4B matches the top generative baselines. Generative methods like GeoWizard are consistently outperformed by ReChannel on both depth and normals, with larger gaps on KITTI depth and iBims normals. Specialized discriminative methods (e.g., DAv2, DSINE) are surpassed by ReChannel-9B on all geometric tasks except NYU depth, where DAv2 still holds an edge.

ReChannel-9B sets a new state of the art on trimap-free matting, outperforming the strongest specialized and generative-prior baselines by clear margins. The advantage is even more pronounced in the zero-shot AIM-500 setting, where the error is less than half that of the previous best generative method, indicating improved cross-domain generalization. The token-local readout removes the target-side VAE, which is particularly beneficial for this boundary-sensitive task. ReChannel-9B achieves new state-of-the-art SAD on both P3M-500-P (5.69) and P3M-500-NP (6.67), surpassing the specialized ViTAE-S by 0.55 and 0.92 points respectively, and the generative-prior GenPercept by 4.06 and 6.10 points. On zero-shot AIM-500, ReChannel-9B reaches 34.90 SAD, less than half of GenPercept's 75.5, showing that avoiding a target-side VAE round-trip enhances both in-domain boundary accuracy and out-of-distribution transfer.

ReChannel models achieve state-of-the-art referring segmentation on all RefCOCO splits without any mask-proposal head, LLM decoder, or task-specific architecture. The 4B variant already outperforms all prior 7B–8B LLM-based methods and the previous best PSALM-1.3B, while the 9B model further raises the average cIoU to 82.0. This indicates that a token-local linear readout from an adapted T2I backbone alone provides both the spatial grounding and segmentation precision that earlier work sought through dedicated output machinery. ReChannel-4B surpasses every prior method, including the 1.3B PSALM (77.1 avg cIoU) and 7B–8B LLM-based models, reaching 80.3 avg cIoU. ReChannel-9B is the best on all eight splits and sets a new state of the art at 82.0 avg cIoU, without any dedicated mask head or LLM decoder.

ReChannel-9B reaches 79.2 AP on COCO pose estimation, surpassing the specialized ViTPose-L by 0.9 points without any pose-specific architecture. For saliency detection, it achieves the best Fmax and MAE on DUTS-TE and ECSSD among all reported methods. These results show the same token-local readout adapts to structured heatmaps and binary masks. ReChannel-9B improves COCO AP by 0.9 over ViTPose-L using a generic readout and no pose-specific design. On saliency, ReChannel-9B sets the highest Fmax and lowest MAE on both DUTS-TE and ECSSD.

When the backbone is frozen and only a linear head is trained, surface normals and matting collapse to near-trivial predictions, revealing that the pretrained token field is not natively aligned with task outputs. Using a random initialization instead of the pretrained prior leaves a large accuracy gap, and adding extra output capacity or generative decoding paths does not improve upon a simple token-local linear readout. The token-local readout is the most accurate and fastest configuration, as the target structure is already present in the adapted token field. Head-only training on a frozen body causes surface normal error to exceed 44° and matting SAD to surpass 170, close to a constant-prediction regime. Random initialization of the same architecture yields roughly double the matting error and significantly worse normals, confirming reliance on the strong pretrained prior. A 13× larger convolutional head and full fine-tuning of the 4B backbone do not outperform the token-local linear readout, and generative interfaces (latent, VAE-decoded, edit) are consistently worse. The token-local linear readout adds no measurable latency, while generative alternatives are 1.56× to 2.48× slower due to additional decoding passes.

ReChannel models are evaluated across a wide range of dense prediction tasks, including monocular depth, surface normals, matting, referring segmentation, pose estimation, and saliency, using standard benchmarks and comparisons against specialized and generative baselines. The experiments validate that a simple token-local linear readout from a pretrained diffusion backbone can achieve state-of-the-art results on all these tasks without any task-specific architectures, outperforming both specialized discriminative methods and generative approaches. Ablation studies confirm that the strong pretrained prior is essential and that the token-local readout is both the most accurate and efficient design, as the target spatial structure is already captured in the adapted token field.