Command Palette
Search for a command to run...
Open Vision Banana:生成的な視覚認識のための完全公開レシピ
Open Vision Banana:生成的な視覚認識のための完全公開レシピ
Arpit Jadon Valentinos Pariza Jona Ruthardt Ryousuke Yamada A. Sophia Koepke Andrei Bursuc Yuki M. Asano
概要
近年の研究は、事前学習済み画像生成器が画像合成を超えた実世界の理解を符号化していることを示唆している。Vision Bananaは、視覚タスクを条件付き画像編集として捉えることでこれを活用する。しかし、そのデータ、重み、コード、実装の詳細は公開されていない。我々は、事前学習済み生成器を統合されたマルチタスク認識システムへ変換するための、完全に公開され再現可能なレシピであるOpen Vision Banana(OVB)を提案する。我々の分析により、色符号化、プロンプト設計、パラメータ適応、タスクバランシングなど、汎化に最も重要な設計上の選択が明らかになった。これらの選択は相互に作用するため、個別には有益な選択も他と組み合わせると有害になり得る。4Bおよび9Bパラメータの公開重みベースモデルを出発点とし、公開データセットから210万~380万の学習ペアを用いて、OVBはVision Bananaが報告したCityscapesのゼロショット性能を上回り(73.9対69.9 mIoU)、最先端のアフィン不変深度推定器に匹敵する。非ゼロショット版は参照セグメンテーションと推論セグメンテーションにおいてVision Bananaと5ポイント差以内に迫るが、インスタンスセグメンテーションでは依然として大きく遅れをとる。すべてのモデルはベースモデルの画像編集能力をほぼ保持している。凡例アンサンブルにより、生成器ベースの認識のためのテスト時スケーリング手法を導入する。これは複数のクラス・色凡例の下で画像をセグメンテーションし、多数決で結果を融合することで、不安定な色とクラスの対応付けを修正し、再学習なしでCityscapesでは最大8.7ポイント、COCO-Objectでは最大15.0ポイントのmIoU向上を達成する。我々はデータセット構築・処理パイプライン、学習・評価コード、ならびに最終・中間・アブレーション用チェックポイントを公開する。
One-sentence Summary
Researchers from Fundamental AI Lab, University of Technology Nuremberg, DLR, AIST, and other institutions present Open Vision Banana, a fully open and reproducible recipe for converting pretrained generators into unified multitask perception systems that surpasses Vision Banana on Cityscapes zero-shot and introduces legend ensembling, a test-time scaling method that improves mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.
Key Contributions
-
Open Vision Banana (OVB) provides a fully open and reproducible recipe for converting pretrained image generators into unified multitask perception models for four segmentation tasks, depth estimation, and surface normal estimation, with released data pipeline, training code, and model checkpoints. Starting from 4B and 9B open-weight base models and 2.1M to 3.8M public training pairs, OVB surpasses Vision Banana's reported Cityscapes zero-shot performance (73.9 vs. 69.9 mIoU), rivals state-of-the-art affine-invariant depth estimators, and largely retains image-editing ability.
-
Controlled experiments identify design choices that most affect generalization, including color encoding, prompt design, parameter adaptation, and task balancing, and show that these choices interact. The resulting recipe favors randomized, well-separated color palettes, relative inverse depth, and full transformer adaptation, while revealing a trade-off between perception accuracy and retained generation quality.
-
Legend ensembling is introduced as a training-free test-time scaling method for generator-based segmentation. It segments each image under multiple class-color legends and fuses the results by majority vote, correcting unstable color-class bindings and improving mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.
Introduction
Image generation models now produce realistic visual content, and recent work suggests this capability can be repurposed for general visual perception. Vision Banana showed that a proprietary image generator can be adapted to output predictions as RGB images, matching domain specialists on segmentation and geometry while retaining editing ability, but its closed base model, data, and code made the recipe hard to reproduce and the important design choices unclear. To address this, the authors introduce Open Vision Banana (OVB), a fully open recipe and model family that turns public image generators into perception models. They systematically study training data, target representations, prompt design, adaptation parameters, and inference readout, and introduce legend ensembling, a training-free test-time method that improves segmentation accuracy by fusing multiple randomized palette predictions.
Dataset
The authors describe a public data mixture used to fine-tune Vision Banana.
-
Dataset composition and sources:
- The full Vision Banana recipe fine-tunes a proprietary base model on proprietary vision data mixed with the base model’s original training data.
- For the public recipe, only public datasets are used.
- The mixture combines Perception data for the target tasks with replay data from public text-to-image, image-editing, and identity-reconstruction datasets.
-
Dataset scale:
- Zero-shot models use 2.1M training pairs.
- Non-zero-shot models use 3.8M training pairs.
-
How the data is used:
- The resulting pools serve as fine-tuning data for Vision Banana.
- Sampling weights for each task and dataset shares within each task are reported in Appendix Tables 20 and 21.
-
Processing details:
- The excerpt does not describe cropping, metadata construction, or other preprocessing steps beyond dataset mixing and sampling weights.
Method
The authors frame Open Vision Banana as a recipe for converting a pretrained image generator into a general perception model without task-specific heads. Given an input image x and an instruction p, the model generates an RGB image y that a deterministic, task-specific readout converts into a prediction. This design unifies segmentation, geometry estimation, and related tasks under a single conditional image generation formulation.
At the core are fine-tuned FLUX.2-Klein generators at 4B and 9B scales. The open-weight models condition on the input image, and all transformer parameters are trained, from the input and context embedders to the output layers. The VAE and text encoder remain frozen.
Each task is represented through a deterministic RGB encoder and decoder. The encoder renders an annotation as an RGB target, while the decoder reads a generated image back into a prediction. Both directions use per-sample metadata such as color legends, depth scaling, and resizing. For segmentation, prompts specify colors as RGB triplets, hexadecimal codes drawn from a palette of 64 most-separated colors, or one of 37 common color names, with black reserved for background.
In semantic segmentation, the prompt lists the requested classes and their colors. Colors are assigned randomly per sample using the same procedure in training and inference. The target paints each prompted class in its assigned color and leaves all other pixels black. At inference, each generated pixel is assigned to the nearest prompted color or to the background. The training prompt distribution includes random subsets of present classes, all present classes, and subsets augmented with absent negative classes. Negative classes teach the model not to hallucinate absent masks. For instance segmentation, the prompt names one category and lists 32 colors. The 32 largest instances are sorted by their leftmost pixel and painted in fixed color order, making training targets unique for each image and prompt. During evaluation, predictions are matched to ground-truth instances by overlap without penalizing alternative color assignments. The readout adds color snapping, gap filling, and GrabCut refinement. For referring segmentation, the expression is paired with a color sampled independently from the named palette, and the model renders the referred region in that color before binary mask readout.
Geometry tasks use a similar color-channel representation. Depth is encoded as affine-invariant inverse depth, normalized per image to the RGB range and replicated across channels, so nearer points are brighter. At inference, the prediction is mapped back to inverse depth and evaluated after per-image affine alignment. For surface normals, normals are converted to a common camera-space convention and each vector component is mapped linearly to one normalized RGB channel. The resulting RGB prediction is mapped back to camera-space vectors and normalized to unit length. Prompts explicitly state the expected output representation.
Because all targets are RGB images, training uses the base generator's rectified-flow objective. The target is encoded as zy, Gaussian noise ϵ is sampled, and an interpolation coefficient t∈(0,1] is drawn uniformly. The noisy latent is formed as
zt=(1−t)zy+tϵ.Conditioned on the clean image x and text prompt p, the transformer predicts v^=vθ(zt,t,p,x) with flow target ϵ−zy. Letting mj indicate whether latent position j is supervised and C denote the number of latent channels, the flow-matching loss is
LFM=E(x,p,y),t,ϵ[C∑jmj∑jmj∥v^j−(ϵ−zy)j∥22].A validity mask is the only task-dependent training component. Geometry annotations contain undefined regions encoded by placeholder colors that a model might otherwise learn to reproduce. These regions account for about 8% of target pixels in the non-zero-shot pool and are excluded from the loss. Zero-shot models and non-geometry tasks use no mask and standard flow matching. During masked training, the validity mask follows image transforms and is area-resized to the latent grid. Positions with at least 50% valid coverage contribute to the loss. The final models use no erosion, exclude samples with no valid latent positions, and normalize each eligible sample by its valid area.
Full-scale models train for 100K steps at a global batch size of 192 on aspect-ratio buckets of about 5122 pixels, including 512×512, 576×448, 608×416, and their transposes. The optimizer is AdamW with weight decay 10−4, using FSDP, bf16 autocast, and fp32 parameters. A constant learning rate follows linear warmup: 5×10−6 with 500 warmup steps for the 4B model, and 2×10−6 with 1,000 warmup steps for the 9B model. Each epoch draws 1M samples with replacement and splits them across tasks. Final models use weight-space averaging of a few late transformer checkpoints.
Experiment
The experiments evaluate Open Vision Banana models finetuned from FLUX.2 Klein 4B and 9B under zero-shot and non-zero-shot regimes on semantic, referring, instance segmentation, geometry, and retained image generation, using standard dense-prediction metrics and legend ensembling for test-time scaling. Zero-shot 9B models often match or surpass specialized baselines on semantic and instance tasks, while ensembling mainly corrects class-color binding errors and yields stronger consensus predictions. Ablations show that random color legends and subset or full-set prompt mixtures prevent fixed class-color memorization, balanced generation and editing data preserve generation ability, and updating most transformer parameters improves segmentation and normals, with fine-tuning also sharpening frozen feature representations.
OVB consistently outperforms the listed frozen zero-shot vision learners across supported perception tasks, with test-time ensembling and non-zero-shot adaptation providing additional gains. The 9B zero-shot model is especially strong on Cityscapes semantic segmentation and depth, while non-zero-shot training improves referring segmentation, instance segmentation, and surface normal error. Across OVB variants, larger scale generally favors depth and instance accuracy, but the 4B non-zero-shot model achieves the best semantic and reasoning segmentation. OVB 9B zero-shot reaches higher Cityscapes mIoU than the cited specialist and generalist baselines, and ensembling adds substantial gains over its single-pass mean. Non-zero-shot OVB improves over its zero-shot counterpart on most tasks, with the largest relative gains in referring segmentation and surface normal error. The frozen zero-shot vision learners score much lower on semantic segmentation and do not support several tasks, while OVB provides coverage across all six benchmarks.
On COCO-Object, the reported specialist methods range from under 38 mIoU to just under 50 mIoU, with CorrCLIP leading and diffusion-based DiffSegm and iSeg scoring lowest among the compared methods. The zero-shot 9B model with test-time scaling surpasses all listed specialists, while its mean single-pass performance is only on par with DiffSegm and iSeg. Test-time scaling is especially beneficial on this 80-class benchmark because many errors come from colors assigned to the wrong class. CorrCLIP is the strongest listed specialist on COCO-Object, ahead of OV-Stitcher and FSeg-LLaVA, while DiffSegm and iSeg are the weakest in the comparison. With test-time scaling, the zero-shot 9B model outperforms all listed COCO-Object specialists, but its mean single-pass performance is comparable to the diffusion-based DiffSegm and iSeg.
On ReasonSeg val, the zero-shot 4B model with query rewriting reaches the highest gIoU among methods that do not use Gemini 2.5 Pro, ahead of zero-shot SegZero and RSVP-GPT-4o as well as target-trained LISA and SenseNova-Vision. SAM 3 and Vision Banana still score higher, but both rely on Gemini 2.5 Pro as a reasoner. Query rewriting adds more gIoU after fine-tuning than in the pretrained base model. The zero-shot 4B model with rewritten queries outperforms both zero-shot and target-trained baselines on ReasonSeg val. SAM 3 and Vision Banana remain stronger but use Gemini 2.5 Pro as their reasoner.
On the RefCOCOg referring segmentation benchmark, X-SAM, SenseNova-Vision, and HyperSeg are the strongest target-trained entries shown. The evaluated zero-shot 9B model outperforms HybridGL but trails LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana. With target training, the 9B model improves substantially but remains behind the top target-trained specialists. Among target-trained methods, X-SAM, SenseNova-Vision, and HyperSeg achieve the highest reported overlap scores. The zero-shot 9B model is ahead of HybridGL but behind LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana.
This comparison isolates the roles of the pretrained base model, OVB fine-tuning, and query rewriting on ReasonSeg. The pretrained base model gains little from explicit rewrites, whereas OVB 4B improves substantially with rewritten queries. Fine-tuning also lifts raw-query performance, suggesting the model learns to resolve implicit expressions itself. The pretrained base model stays low across raw and rewritten queries, so query rewriting alone is not enough without fine-tuning. OVB 4B in the zero-shot setting improves by about 27 gIoU from raw queries to explicit rewrites, with all three rewriters providing large gains. Moving from zero-shot to non-zero-shot improves raw-query performance more than Gemma4-rewritten performance, indicating the model partly learns to interpret implicit queries itself.
The experiments evaluate zero-shot and non-zero-shot OVB variants across semantic, referring, instance, and reasoning segmentation benchmarks against frozen zero-shot learners and specialist models. Zero-shot OVB consistently outperforms the listed frozen zero-shot baselines, with test-time ensembling or scaling providing additional gains, especially on Cityscapes and COCO-Object. Fine-tuning and query rewriting yield substantial improvements on reasoning and referring segmentation, and fine-tuning helps the model interpret implicit queries rather than relying only on explicit rewrites, though top specialist models remain stronger on some referring benchmarks.