Command Palette
Search for a command to run...
Open Vision Banana: 생성형 시각 인식을 위한 완전 공개 레시피
Open Vision Banana: 생성형 시각 인식을 위한 완전 공개 레시피
Arpit Jadon Valentinos Pariza Jona Ruthardt Ryousuke Yamada A. Sophia Koepke Andrei Bursuc Yuki M. Asano
초록
최근 연구들은 사전 학습된 이미지 생성기가 이미지 합성 이상의 실제 세계 이해를 내재하고 있음을 시사한다. Vision Banana는 시각 과제를 조건부 이미지 편집으로 재구성함으로써 이러한 특성을 활용한다. 그러나 해당 연구의 데이터, 가중치, 코드 및 구현 세부 사항은 공개되어 있지 않다. 본 논문에서는 사전 학습된 생성기를 통합 멀티태스크 인식 시스템으로 변환하기 위한 완전히 공개되고 재현 가능한 레시피인 Open Vision Banana(OVB)를 제시한다. 우리의 분석은 색상 인코딩, 프롬프트 설계, 파라미터 적응, 태스크 균형화 등 일반화에 가장 중요한 설계 선택들을 규명한다. 이러한 선택들은 서로 상호작용하므로, 개별적으로는 유익한 선택도 다른 선택과 결합될 때 성능을 저해할 수 있다. 4B 및 9B 파라미터의 공개 가중치 기반 모델에서 시작하여 공개 데이터셋의 210만~380만 개 학습 쌍을 사용한 OVB는 Vision Banana가 보고한 Cityscapes 제로샷 성능(73.9 vs. 69.9 mIoU)을 능가하며, 최신 아핀 불변 깊이 추정기와 견줄 만한 성능을 달성한다. 비제로샷 변형은 지시 및 추론 분할에서 Vision Banana와 5포인트 이내의 차이를 보이지만, 인스턴스 분할에서는 여전히 더 큰 격차를 보인다. 모든 모델은 기본 모델의 이미지 편집 능력을 대체로 유지한다. 범례 앙상블을 통해 우리는 생성기 기반 인식을 위한 테스트 시간 확장 기법을 소개한다. 이 기법은 여러 클래스-색상 범례 하에서 이미지를 분할하고 다수결 투표로 결과를 융합하여 불안정한 색상-클래스 결합을 교정하며, 재학습 없이 Cityscapes에서 최대 8.7포인트, COCO-Object에서 최대 15.0포인트의 mIoU 향상을 달성한다. 우리는 데이터셋 구축 및 처리 파이프라인, 학습 및 평가 코드, 그리고 최종, 중간, 및 제거 실험 체크포인트를 공개한다.
One-sentence Summary
Researchers from Fundamental AI Lab, University of Technology Nuremberg, DLR, AIST, and other institutions present Open Vision Banana, a fully open and reproducible recipe for converting pretrained generators into unified multitask perception systems that surpasses Vision Banana on Cityscapes zero-shot and introduces legend ensembling, a test-time scaling method that improves mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.
Key Contributions
-
Open Vision Banana (OVB) provides a fully open and reproducible recipe for converting pretrained image generators into unified multitask perception models for four segmentation tasks, depth estimation, and surface normal estimation, with released data pipeline, training code, and model checkpoints. Starting from 4B and 9B open-weight base models and 2.1M to 3.8M public training pairs, OVB surpasses Vision Banana's reported Cityscapes zero-shot performance (73.9 vs. 69.9 mIoU), rivals state-of-the-art affine-invariant depth estimators, and largely retains image-editing ability.
-
Controlled experiments identify design choices that most affect generalization, including color encoding, prompt design, parameter adaptation, and task balancing, and show that these choices interact. The resulting recipe favors randomized, well-separated color palettes, relative inverse depth, and full transformer adaptation, while revealing a trade-off between perception accuracy and retained generation quality.
-
Legend ensembling is introduced as a training-free test-time scaling method for generator-based segmentation. It segments each image under multiple class-color legends and fuses the results by majority vote, correcting unstable color-class bindings and improving mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.
Introduction
Image generation models now produce realistic visual content, and recent work suggests this capability can be repurposed for general visual perception. Vision Banana showed that a proprietary image generator can be adapted to output predictions as RGB images, matching domain specialists on segmentation and geometry while retaining editing ability, but its closed base model, data, and code made the recipe hard to reproduce and the important design choices unclear. To address this, the authors introduce Open Vision Banana (OVB), a fully open recipe and model family that turns public image generators into perception models. They systematically study training data, target representations, prompt design, adaptation parameters, and inference readout, and introduce legend ensembling, a training-free test-time method that improves segmentation accuracy by fusing multiple randomized palette predictions.
Dataset
The authors describe a public data mixture used to fine-tune Vision Banana.
-
Dataset composition and sources:
- The full Vision Banana recipe fine-tunes a proprietary base model on proprietary vision data mixed with the base model’s original training data.
- For the public recipe, only public datasets are used.
- The mixture combines Perception data for the target tasks with replay data from public text-to-image, image-editing, and identity-reconstruction datasets.
-
Dataset scale:
- Zero-shot models use 2.1M training pairs.
- Non-zero-shot models use 3.8M training pairs.
-
How the data is used:
- The resulting pools serve as fine-tuning data for Vision Banana.
- Sampling weights for each task and dataset shares within each task are reported in Appendix Tables 20 and 21.
-
Processing details:
- The excerpt does not describe cropping, metadata construction, or other preprocessing steps beyond dataset mixing and sampling weights.
Method
The authors frame Open Vision Banana as a recipe for converting a pretrained image generator into a general perception model without task-specific heads. Given an input image x and an instruction p, the model generates an RGB image y that a deterministic, task-specific readout converts into a prediction. This design unifies segmentation, geometry estimation, and related tasks under a single conditional image generation formulation.
At the core are fine-tuned FLUX.2-Klein generators at 4B and 9B scales. The open-weight models condition on the input image, and all transformer parameters are trained, from the input and context embedders to the output layers. The VAE and text encoder remain frozen.
Each task is represented through a deterministic RGB encoder and decoder. The encoder renders an annotation as an RGB target, while the decoder reads a generated image back into a prediction. Both directions use per-sample metadata such as color legends, depth scaling, and resizing. For segmentation, prompts specify colors as RGB triplets, hexadecimal codes drawn from a palette of 64 most-separated colors, or one of 37 common color names, with black reserved for background.
In semantic segmentation, the prompt lists the requested classes and their colors. Colors are assigned randomly per sample using the same procedure in training and inference. The target paints each prompted class in its assigned color and leaves all other pixels black. At inference, each generated pixel is assigned to the nearest prompted color or to the background. The training prompt distribution includes random subsets of present classes, all present classes, and subsets augmented with absent negative classes. Negative classes teach the model not to hallucinate absent masks. For instance segmentation, the prompt names one category and lists 32 colors. The 32 largest instances are sorted by their leftmost pixel and painted in fixed color order, making training targets unique for each image and prompt. During evaluation, predictions are matched to ground-truth instances by overlap without penalizing alternative color assignments. The readout adds color snapping, gap filling, and GrabCut refinement. For referring segmentation, the expression is paired with a color sampled independently from the named palette, and the model renders the referred region in that color before binary mask readout.
Geometry tasks use a similar color-channel representation. Depth is encoded as affine-invariant inverse depth, normalized per image to the RGB range and replicated across channels, so nearer points are brighter. At inference, the prediction is mapped back to inverse depth and evaluated after per-image affine alignment. For surface normals, normals are converted to a common camera-space convention and each vector component is mapped linearly to one normalized RGB channel. The resulting RGB prediction is mapped back to camera-space vectors and normalized to unit length. Prompts explicitly state the expected output representation.
Because all targets are RGB images, training uses the base generator's rectified-flow objective. The target is encoded as zy, Gaussian noise ϵ is sampled, and an interpolation coefficient t∈(0,1] is drawn uniformly. The noisy latent is formed as
zt=(1−t)zy+tϵ.Conditioned on the clean image x and text prompt p, the transformer predicts v^=vθ(zt,t,p,x) with flow target ϵ−zy. Letting mj indicate whether latent position j is supervised and C denote the number of latent channels, the flow-matching loss is
LFM=E(x,p,y),t,ϵ[C∑jmj∑jmj∥v^j−(ϵ−zy)j∥22].A validity mask is the only task-dependent training component. Geometry annotations contain undefined regions encoded by placeholder colors that a model might otherwise learn to reproduce. These regions account for about 8% of target pixels in the non-zero-shot pool and are excluded from the loss. Zero-shot models and non-geometry tasks use no mask and standard flow matching. During masked training, the validity mask follows image transforms and is area-resized to the latent grid. Positions with at least 50% valid coverage contribute to the loss. The final models use no erosion, exclude samples with no valid latent positions, and normalize each eligible sample by its valid area.
Full-scale models train for 100K steps at a global batch size of 192 on aspect-ratio buckets of about 5122 pixels, including 512×512, 576×448, 608×416, and their transposes. The optimizer is AdamW with weight decay 10−4, using FSDP, bf16 autocast, and fp32 parameters. A constant learning rate follows linear warmup: 5×10−6 with 500 warmup steps for the 4B model, and 2×10−6 with 1,000 warmup steps for the 9B model. Each epoch draws 1M samples with replacement and splits them across tasks. Final models use weight-space averaging of a few late transformer checkpoints.
Experiment
The experiments evaluate Open Vision Banana models finetuned from FLUX.2 Klein 4B and 9B under zero-shot and non-zero-shot regimes on semantic, referring, instance segmentation, geometry, and retained image generation, using standard dense-prediction metrics and legend ensembling for test-time scaling. Zero-shot 9B models often match or surpass specialized baselines on semantic and instance tasks, while ensembling mainly corrects class-color binding errors and yields stronger consensus predictions. Ablations show that random color legends and subset or full-set prompt mixtures prevent fixed class-color memorization, balanced generation and editing data preserve generation ability, and updating most transformer parameters improves segmentation and normals, with fine-tuning also sharpening frozen feature representations.
OVB consistently outperforms the listed frozen zero-shot vision learners across supported perception tasks, with test-time ensembling and non-zero-shot adaptation providing additional gains. The 9B zero-shot model is especially strong on Cityscapes semantic segmentation and depth, while non-zero-shot training improves referring segmentation, instance segmentation, and surface normal error. Across OVB variants, larger scale generally favors depth and instance accuracy, but the 4B non-zero-shot model achieves the best semantic and reasoning segmentation. OVB 9B zero-shot reaches higher Cityscapes mIoU than the cited specialist and generalist baselines, and ensembling adds substantial gains over its single-pass mean. Non-zero-shot OVB improves over its zero-shot counterpart on most tasks, with the largest relative gains in referring segmentation and surface normal error. The frozen zero-shot vision learners score much lower on semantic segmentation and do not support several tasks, while OVB provides coverage across all six benchmarks.
On COCO-Object, the reported specialist methods range from under 38 mIoU to just under 50 mIoU, with CorrCLIP leading and diffusion-based DiffSegm and iSeg scoring lowest among the compared methods. The zero-shot 9B model with test-time scaling surpasses all listed specialists, while its mean single-pass performance is only on par with DiffSegm and iSeg. Test-time scaling is especially beneficial on this 80-class benchmark because many errors come from colors assigned to the wrong class. CorrCLIP is the strongest listed specialist on COCO-Object, ahead of OV-Stitcher and FSeg-LLaVA, while DiffSegm and iSeg are the weakest in the comparison. With test-time scaling, the zero-shot 9B model outperforms all listed COCO-Object specialists, but its mean single-pass performance is comparable to the diffusion-based DiffSegm and iSeg.
On ReasonSeg val, the zero-shot 4B model with query rewriting reaches the highest gIoU among methods that do not use Gemini 2.5 Pro, ahead of zero-shot SegZero and RSVP-GPT-4o as well as target-trained LISA and SenseNova-Vision. SAM 3 and Vision Banana still score higher, but both rely on Gemini 2.5 Pro as a reasoner. Query rewriting adds more gIoU after fine-tuning than in the pretrained base model. The zero-shot 4B model with rewritten queries outperforms both zero-shot and target-trained baselines on ReasonSeg val. SAM 3 and Vision Banana remain stronger but use Gemini 2.5 Pro as their reasoner.
On the RefCOCOg referring segmentation benchmark, X-SAM, SenseNova-Vision, and HyperSeg are the strongest target-trained entries shown. The evaluated zero-shot 9B model outperforms HybridGL but trails LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana. With target training, the 9B model improves substantially but remains behind the top target-trained specialists. Among target-trained methods, X-SAM, SenseNova-Vision, and HyperSeg achieve the highest reported overlap scores. The zero-shot 9B model is ahead of HybridGL but behind LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana.
This comparison isolates the roles of the pretrained base model, OVB fine-tuning, and query rewriting on ReasonSeg. The pretrained base model gains little from explicit rewrites, whereas OVB 4B improves substantially with rewritten queries. Fine-tuning also lifts raw-query performance, suggesting the model learns to resolve implicit expressions itself. The pretrained base model stays low across raw and rewritten queries, so query rewriting alone is not enough without fine-tuning. OVB 4B in the zero-shot setting improves by about 27 gIoU from raw queries to explicit rewrites, with all three rewriters providing large gains. Moving from zero-shot to non-zero-shot improves raw-query performance more than Gemma4-rewritten performance, indicating the model partly learns to interpret implicit queries itself.
The experiments evaluate zero-shot and non-zero-shot OVB variants across semantic, referring, instance, and reasoning segmentation benchmarks against frozen zero-shot learners and specialist models. Zero-shot OVB consistently outperforms the listed frozen zero-shot baselines, with test-time ensembling or scaling providing additional gains, especially on Cityscapes and COCO-Object. Fine-tuning and query rewriting yield substantial improvements on reasoning and referring segmentation, and fine-tuning helps the model interpret implicit queries rather than relying only on explicit rewrites, though top specialist models remain stronger on some referring benchmarks.