HyperAIHyperAI

Command Palette

Search for a command to run...

Jeveration: 시스템 원 모델의 생성 능력

Yuhe Liu Qi Li Shihua Zhang Xinchao Wang

초록

Jev와 같은 시스템 원 모델은 빠르고 병렬적인 예측을 통해 결정을 내린다. 본 논문에서는 동일한 결정 패러다임이 단일 순방향 패스만으로도 고품질 이미지를 생성할 수 있음을 보여주는 Jeveration을 소개한다. Jeveration은 동결된 비전-언어 모델을 기반으로 하며, 그 시각-의미 지식을 전체적 이미지 생성에 재활용한다. 핵심 과제는 전역적 일관성을 유지하는 것이다. 하나의 프롬프트는 여러 유효한 이미지를 허용하지만, 병렬 예측은 일관된 전체로 수렴해야 한다. 따라서 모든 위치를 공유 무작위 상태로 조건화하여 병렬 예측을 조정함으로써, 각 추출마다 서로 다르지만 일관된 이미지를 얻는다. 직접적인 이미지 지도 하에서는 모델이 이 상태를 무시하는 경향이 있으므로, 먼저 이산 이미지 코드에 대한 분포를 학습하여 단일 패스 생성을 안내한다. 텍스트-이미지 생성 외에도 동일한 아키텍처 내에서 이미지-이미지 생성과 밀집 예측을 통합하여, 하나의 체크포인트가 세 가지 작업을 각각 한 번의 순방향 패스로 지원하도록 한다. 실험 결과, Jeveration은 512×512 이미지를 78ms에 생성하여 기존 통합 모델 대비 최대 122배 빠르면서도 경쟁력 있는 이미지 품질과 밀집 예측 성능을 유지하는 것으로 나타났다.

One-sentence Summary

Researchers at the National University of Singapore propose Jeveration, a System One model that repurposes a frozen vision-language model with a shared random state and a learned distribution over discrete image codes to generate coherent images in a single forward pass, unifying text-to-image generation, image-to-image generation, and dense prediction while producing 512×512512 \times 512512×512 images in 78 ms, up to 122×122 \times122× faster than existing unified models.

Key Contributions

  • Jeveration is introduced as a parallel decision model that generates 512×512 images in a single forward pass by repurposing a frozen vision-language model's visual-semantic knowledge.
  • To maintain global consistency, the method conditions all image code positions on a shared random state and learns a distribution over discrete image codes through token-level conditional matching with a masked model, avoiding the tendency to ignore the state under direct image supervision.
  • A single checkpoint unifies text-to-image generation, image-to-image generation, and dense prediction through shared adapters and task-specific output heads; experiments report 78 ms generation, up to 122× faster than existing unified models, with competitive image quality and dense prediction performance.

Introduction

Image generation is currently dominated by autoregressive and diffusion models, but both rely on many sequential model evaluations, which increases inference latency. Prior System One models can return typed visual decisions in parallel, but they have been applied mainly to understanding and scoring tasks rather than synthesis, and naive parallel image prediction tends to produce spatially incoherent regions. The authors introduce Jeveration, which adapts a frozen vision-language model with LoRA adapters and output heads to return an entire image as one joint decision in a single forward pass. To achieve coherence, all code positions are conditioned on a shared random state, and a masked model trained to learn the image distribution provides target conditionals for guiding the parallel decisions. The resulting single checkpoint covers text-to-image generation, image-to-image generation, and dense prediction tasks.

Method

Jeveration enables a vision-language model (VLM) to generate images through parallel typed decisions in a single forward pass, while simultaneously supporting dense prediction tasks within the same framework. For image synthesis, the model selects a codebook entry at each code position, whereas for dense prediction, it produces task-specific classes or scores. A shared random state provides common stochastic conditioning to all code positions, ensuring that each draw yields a different but globally coherent image.

Given a conditioning input yyy, such as a text prompt or a source image, Jeveration formulates the output as typed decisions. An image is represented as a joint choice of discrete codes c=(c1,…,cN)\boldsymbol{c} = (c_1, \dots, c_N)c=(c1​,…,cN​) from an image tokenizer. A direct approach that predicts a categorical distribution at each code position and samples codes independently in parallel fails to capture dependencies between codes. This independent factorization matches only the marginals, often resulting in codes that do not fit together to form a coherent image.

To overcome this, Jeveration draws one random state sequence ξ\xiξ per image and conditions every code position on this shared state:

qθ(c∣y)=Eξ∼p0[∏i=1Nqθ,i(ci∣y,ξ)]q_{\theta}(\boldsymbol{c} \mid y) = \mathbb{E}_{\xi \sim p_0} \left[ \prod_{i=1}^{N} q_{\theta, i}(c_i \mid y, \xi) \right]qθ​(c∣y)=Eξ∼p0​​[i=1∏N​qθ,i​(ci​∣y,ξ)]

Although the codes are sampled independently given (y,ξ)(y, \xi)(y,ξ), their shared dependence on ξ\xiξ allows the resulting image distribution to capture dependencies between positions. This enables the model to coordinate its code choices while deciding all positions in a single forward pass.

The authors implement these visual decisions by appending decision tokens to the input of a VLM. Text tokens encode the prompt and task instruction, and a frozen vision tower encodes any input image into visual tokens. The decision tokens are initialized from a sampled random state for image generation and serve as spatial queries for dense prediction. A frozen backbone with shared LoRA adapters processes the input and decision tokens in a single forward pass. In its full-attention layers, decision tokens attend to each other bidirectionally while the input prefix remains causal. Finally, task-specific heads map the decision tokens' final hidden states to image codes or dense predictions.

For image generation, the image tokenizer is a frozen TA-TiTok with N=128N = 128N=128 codes per image. The initial embeddings of the decision tokens encode the random state ξ∼p0\xi \sim p_0ξ∼p0​, which masks each position with probability ρ\rhoρ, fills the remaining positions with uniformly random codes, and adds Gaussian noise of scale σ\sigmaσ to all embeddings. A transformer head reads the final hidden states of these tokens together with ξ\xiξ and outputs a feature uiu_iui​ for each position. Each code is predicted from uiu_iui​ using two branches. The codebook branch fcbf_{\mathrm{cb}}fcb​ projects uiu_iui​ to the code space and applies a classifier initialized from the tokenizer's codebook. The wide branch fwidef_{\mathrm{wide}}fwide​, initialized at zero, maps uiu_iui​ directly to the code logits. The code distribution sums the two:

qθ,i(⋅∣y,ξ)=softmax(fcb(ui)+fwide(ui))q_{\theta, i}(\cdot \mid y, \xi) = \text{softmax} \left(f_{\mathrm{cb}}(u_i) + f_{\mathrm{wide}}(u_i)\right)qθ,i​(⋅∣y,ξ)=softmax(fcb​(ui​)+fwide​(ui​))

At inference, all codes are sampled in parallel as one joint decision and rendered by the tokenizer's decoder. For dense prediction, the decision tokens act as spatial queries aligned with the visual tokens of the input image, and task-specific heads decode each query's final hidden state into segmentation, depth, or surface-normal predictions.

The training procedure teaches the model to use ξ\xiξ to make coherent image decisions. Direct supervision on target codes does not encourage the model to use ξ\xiξ, since the random state is sampled independently of the target image. Therefore, the authors first learn a model of the image distribution through masked prediction, then use it to guide image decisions from random states.

In the first phase, the architecture is trained as a masked model with the VLM frozen. For target codes c\boldsymbol{c}c, a random set M\mathcal{M}M is masked, and the hidden codes are predicted from the visible codes cMˉ\boldsymbol{c}_{\bar{\mathcal{M}}}cMˉ​ and condition yyy:

Lmask(ϕ)=E[1∣M∣∑i∈M−log⁡pϕ,i(ci∣cMˉ,y)]\mathcal{L}_{\mathrm{mask}}(\phi) = \mathbb{E} \left[ \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} -\log p_{\phi, i}(c_i \mid \boldsymbol{c}_{\bar{\mathcal{M}}}, y) \right]Lmask​(ϕ)=E[∣M∣1​i∈M∑​−logpϕ,i​(ci​∣cMˉ​,y)]

The visible codes supply image-specific context, enabling the model to learn dependencies between positions.

In the second phase, the decision model θ\thetaθ starts from the first-phase model and learns to turn random states into whole images. From (y,ξ)(y, \xi)(y,ξ), θ\thetaθ predicts logits and samples a code sequence c^\hat{\boldsymbol{c}}c^ in a single pass. A random subset M\mathcal{M}M of c^\hat{\boldsymbol{c}}c^ is re-masked, and two masked models predict the hidden codes from the same visible codes. The frozen first-phase model ϕ\phiϕ represents the image distribution, while an auxiliary model ψ\psiψ tracks the current distribution of θ\thetaθ. The auxiliary model begins as a copy of ϕ\phiϕ and continues training on θ\thetaθ's samples. The two predictions agree when θ\thetaθ matches the image distribution. The authors use their difference to update θ\thetaθ by minimizing:

LDM(θ)=E[1∣M∣∑i∈Msg(pψ,i−pϕ,i)⊤ℓθ,i]\mathcal{L}_{\mathrm{DM}}(\theta) = \mathbb{E} \left[ \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \mathrm{sg}(p_{\psi, i} - p_{\phi, i})^\top \ell_{\theta, i} \right]LDM​(θ)=E[∣M∣1​i∈M∑​sg(pψ,i​−pϕ,i​)⊤ℓθ,i​]

where sg\mathrm{sg}sg stops gradients. This pushes θ\thetaθ to coordinate its codes through the shared state ξ\xiξ.

In the final part of the second phase, confidence-guided correction adds direct supervision on generated samples. For some samples in each batch, the half of the positions with the lowest confidence are re-masked, and the frozen model completes them over several rounds into cref\boldsymbol{c}^{\mathrm{ref}}cref. The model θ\thetaθ is then trained to predict these corrected codes from the same (y,ξ)(y, \xi)(y,ξ):

Lgen=LDM+λLcorr,Lcorr=Eselected[1∣R∣∑i∈R−log⁡qθ,i(ciref∣y,ξ)].\begin{array}{l} \mathcal{L}_{\mathrm{gen}} = \mathcal{L}_{\mathrm{DM}} + \lambda \mathcal{L}_{\mathrm{corr}}, \\ \mathcal{L}_{\mathrm{corr}} = \mathbb{E}_{\mathrm{selected}} \left[ \frac{1}{|\mathcal{R}|} \sum_{i \in \mathcal{R}} -\log q_{\theta, i}(c_i^{\mathrm{ref}} \mid y, \xi) \right]. \end{array}Lgen​=LDM​+λLcorr​,Lcorr​=Eselected​[∣R∣1​∑i∈R​−logqθ,i​(ciref​∣y,ξ)].​

Dense tasks are supervised with ground truth, using cross-entropy for segmentation, an ℓ1\ell_1ℓ1​ depth error, and 1−cos⁡1 - \cos1−cos for surface normals. The frozen model, the auxiliary model, and the correction are used only in training, ensuring that inference remains a single forward pass of θ\thetaθ.

Experiment

The experiments evaluate a single Jeveration checkpoint trained with rank-256 LoRA adapters on a frozen VLM for text-to-image generation, image-to-image tasks, and dense prediction, using generation benchmarks such as GenEval, DPG-Bench, and HPSv2.1, along with structure fidelity, NYUv2, and latency metrics. Text-to-image results show that the model generates coherent images in one forward pass and is faster than both unified and generation-only baselines while maintaining competitive quality. Image-to-image and dense prediction experiments demonstrate that the same checkpoint handles controllable generation, segmentation, and depth estimation with strong structural fidelity and little added latency. Ablations confirm that the shared random state, correction target, dual image-head branches, source conditioning, and joint training each contribute to image quality, controllability, or task flexibility.

Jeveration generates 512x512 images in a single forward pass with the lowest end-to-end latency in the comparison and stronger GenEval than the listed unified and same-resolution generation-only baselines. It is roughly two orders of magnitude faster than autoregressive Janus-Pro baselines and outperforms several one-step generation-only models on GenEval despite their higher output resolution. The results indicate a frozen vision-language backbone can provide competitive text-to-image quality without iterative sampling. At 512-px resolution, it achieves higher GenEval than multi-step generation-only models while being several times faster end to end. It is roughly two orders of magnitude faster than the autoregressive Janus-Pro baselines at batch 1 on the same hardware.

On held-out image-to-image examples, Jeveration achieves strong structure fidelity across edge, depth, and colorization tasks. It improves over a decoder without source conditioning, particularly in colorization and edge fidelity, and achieves higher style PSNR. Jeveration attains the best edge and depth fidelity among the compared conditions. Source-image conditioning sharply improves colorization fidelity over source-free decoding and lifts edge fidelity above direct input copying. Jeveration achieves better style PSNR than the decoder without source, indicating closer alignment with paired style targets.

On NYUv2 dense prediction, the unified model matches dense-only segmentation accuracy from the same backbone and far exceeds DeepLabV3+. Depth accuracy is close to the dense-only baselines, while surface normal error is somewhat higher. Each prediction is produced in a single forward pass, supporting one checkpoint for both image generation and dense prediction. The unified model matches dense-only segmentation scores from the same backbone while outperforming DeepLabV3+ by a wide margin. Depth error is close to the dense-only baselines and much lower than DeepLabV3+, but surface normal accuracy trails the dense-only models.

Ablations indicate that a shared random state improves both composition and image quality over a fixed state. A correction target adds a further small quality gain while leaving composition similar, and combining wide and codebook image-head branches outperforms either branch alone. The results support the selected configuration for coherent, high-quality generation. A shared random state yields higher GenEval and HPS than a fixed state, indicating better coherence and image quality. Using a correction target improves HPS while GenEval remains similar, suggesting it refines details rather than composition. Removing the correction target slightly lowers image quality with no meaningful change in composition. The wide plus codebook image-head combination outperforms the wide-only and codebook-only variants.

Joint training across text-to-image, image-to-image, and dense prediction tasks preserves text-to-image quality almost entirely. The all-task model performs on par with a text-to-image only baseline on GenEval, HPS, and DPG metrics. The added tasks introduce little interference while enabling dense prediction and image-to-image generation. All-task training matches text-to-image-only quality, with only marginal differences across GenEval, HPS, and DPG. Adding image-to-image and dense tasks comes at almost no cost to text-to-image performance. For dense prediction, joint training matches dense-only segmentation and shows only a modest decline in depth and normal accuracy.

Jeveration is evaluated as a unified vision-language model across text-to-image, image-to-image, and dense prediction tasks. It produces 512x512 images in a single forward pass with lower latency and stronger GenEval than several generation-only baselines, while image-to-image experiments show high edge, depth, and colorization fidelity that improves with source conditioning. For dense prediction, the unified model matches segmentation quality from the same backbone and outperforms DeepLabV3+, with depth close to dense-only baselines; ablations confirm that shared random states, the correction target, and combined image-head branches improve quality, and joint training preserves text-to-image performance with little interference.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp