HyperAIHyperAI

Command Palette

Search for a command to run...

رؤية الموز المفتوحة: وصفة مفتوحة بالكامل للإدراك البصري التوليدي

Arpit Jadon Valentinos Pariza Jona Ruthardt Ryousuke Yamada A. Sophia Koepke Andrei Bursuc Yuki M. Asano

الملخص

تشير أعمال حديثة إلى أن مولدات الصور المدربة مسبقًا ترمّز فهمًا للعالم الحقيقي يتجاوز مجرد تركيب الصور. وتستغل "رؤية الموز" ذلك عبر صياغة مهام الرؤية بوصفها تحريرًا شرطيًا للصور. غير أن بياناتها وأوزانها وكودها وتفاصيل تنفيذها ليست متاحة للعموم. نقدم في هذا العمل "رؤية الموز المفتوحة" (OVB)، وهي وصفة مفتوحة بالكامل وقابلة لإعادة الإنتاج لتحويل المولدات المدربة مسبقًا إلى أنظمة إدراك موحدة متعددة المهام. وتحدد تحليلاتنا خيارات التصميم الأكثر تأثيرًا في التعميم، ومنها ترميز الألوان، وتصميم المطالبات، وتكييف المعاملات، وموازنة المهام. وتتفاعل هذه الخيارات فيما بينها، بحيث قد تضر الخيارات المفيدة فرادى عند دمجها مع خيارات أخرى. وبالانطلاق من نماذج أساسية مفتوحة الأوزان بمعلمين يبلغان 4 مليارات و9 مليارات معامل، واستخدام ما بين 2.1 مليون و3.8 مليون زوج تدريبي من مجموعات بيانات عامة، يتفوق OVB على الأداء المُعلن لـ"رؤية الموز" في اختبار Cityscapes بدون أمثلة (73.9 مقابل 69.9 من حيث mIoU) وينافس أحدث مقدرات العمق الثابتة بالنسبة للتحويلات التقاربية. وتقترب نسخه غير المعتمدة على التعلم بدون أمثلة بفارق خمس نقاط من "رؤية الموز" في مهام التجزئة بالإحالة والتجزئة الاستدلالية، لكنها تظل أبعد في تجزئة النسق. وتحتفظ جميع النماذج إلى حد كبير بقدرات النموذج الأساسي في تحرير الصور. وباستخدام تجميع وسائل الإيضاح، نقدم طريقة لتوسيع نطاق الاستدلال زمنيًا في الإدراك القائم على المولدات؛ إذ تقوم بتجزئة الصورة وفق عدة وسائل إيضاح للفئات والألوان ثم تدمج النتائج بتصويت الأغلبية، مما يصحح الارتباطات غير المستقرة بين الألوان والفئات ويحسن mIoU بما يصل إلى 8.7 نقطة على Cityscapes و15.0 نقطة على COCO-Object دون إعادة تدريب. ونطلق خط أنابيب بناء مجموعة البيانات ومعالجتها، وكود التدريب والتقييم، إضافة إلى نقاط التحقق النهائية والوسيطة ونقاط تحقق دراسات الاستبعاد.

One-sentence Summary

Researchers from Fundamental AI Lab, University of Technology Nuremberg, DLR, AIST, and other institutions present Open Vision Banana, a fully open and reproducible recipe for converting pretrained generators into unified multitask perception systems that surpasses Vision Banana on Cityscapes zero-shot and introduces legend ensembling, a test-time scaling method that improves mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.

Key Contributions

  • Open Vision Banana (OVB) provides a fully open and reproducible recipe for converting pretrained image generators into unified multitask perception models for four segmentation tasks, depth estimation, and surface normal estimation, with released data pipeline, training code, and model checkpoints. Starting from 4B and 9B open-weight base models and 2.1M to 3.8M public training pairs, OVB surpasses Vision Banana's reported Cityscapes zero-shot performance (73.9 vs. 69.9 mIoU), rivals state-of-the-art affine-invariant depth estimators, and largely retains image-editing ability.

  • Controlled experiments identify design choices that most affect generalization, including color encoding, prompt design, parameter adaptation, and task balancing, and show that these choices interact. The resulting recipe favors randomized, well-separated color palettes, relative inverse depth, and full transformer adaptation, while revealing a trade-off between perception accuracy and retained generation quality.

  • Legend ensembling is introduced as a training-free test-time scaling method for generator-based segmentation. It segments each image under multiple class-color legends and fuses the results by majority vote, correcting unstable color-class bindings and improving mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining.

Introduction

Image generation models now produce realistic visual content, and recent work suggests this capability can be repurposed for general visual perception. Vision Banana showed that a proprietary image generator can be adapted to output predictions as RGB images, matching domain specialists on segmentation and geometry while retaining editing ability, but its closed base model, data, and code made the recipe hard to reproduce and the important design choices unclear. To address this, the authors introduce Open Vision Banana (OVB), a fully open recipe and model family that turns public image generators into perception models. They systematically study training data, target representations, prompt design, adaptation parameters, and inference readout, and introduce legend ensembling, a training-free test-time method that improves segmentation accuracy by fusing multiple randomized palette predictions.

Dataset

The authors describe a public data mixture used to fine-tune Vision Banana.

  • Dataset composition and sources:

    • The full Vision Banana recipe fine-tunes a proprietary base model on proprietary vision data mixed with the base model’s original training data.
    • For the public recipe, only public datasets are used.
    • The mixture combines Perception data for the target tasks with replay data from public text-to-image, image-editing, and identity-reconstruction datasets.
  • Dataset scale:

    • Zero-shot models use 2.1M training pairs.
    • Non-zero-shot models use 3.8M training pairs.
  • How the data is used:

    • The resulting pools serve as fine-tuning data for Vision Banana.
    • Sampling weights for each task and dataset shares within each task are reported in Appendix Tables 20 and 21.
  • Processing details:

    • The excerpt does not describe cropping, metadata construction, or other preprocessing steps beyond dataset mixing and sampling weights.

Method

The authors frame Open Vision Banana as a recipe for converting a pretrained image generator into a general perception model without task-specific heads. Given an input image xxx and an instruction ppp, the model generates an RGB image yyy that a deterministic, task-specific readout converts into a prediction. This design unifies segmentation, geometry estimation, and related tasks under a single conditional image generation formulation.

At the core are fine-tuned FLUX.2-Klein generators at 4B and 9B scales. The open-weight models condition on the input image, and all transformer parameters are trained, from the input and context embedders to the output layers. The VAE and text encoder remain frozen.

Each task is represented through a deterministic RGB encoder and decoder. The encoder renders an annotation as an RGB target, while the decoder reads a generated image back into a prediction. Both directions use per-sample metadata such as color legends, depth scaling, and resizing. For segmentation, prompts specify colors as RGB triplets, hexadecimal codes drawn from a palette of 64 most-separated colors, or one of 37 common color names, with black reserved for background.

In semantic segmentation, the prompt lists the requested classes and their colors. Colors are assigned randomly per sample using the same procedure in training and inference. The target paints each prompted class in its assigned color and leaves all other pixels black. At inference, each generated pixel is assigned to the nearest prompted color or to the background. The training prompt distribution includes random subsets of present classes, all present classes, and subsets augmented with absent negative classes. Negative classes teach the model not to hallucinate absent masks. For instance segmentation, the prompt names one category and lists 32 colors. The 32 largest instances are sorted by their leftmost pixel and painted in fixed color order, making training targets unique for each image and prompt. During evaluation, predictions are matched to ground-truth instances by overlap without penalizing alternative color assignments. The readout adds color snapping, gap filling, and GrabCut refinement. For referring segmentation, the expression is paired with a color sampled independently from the named palette, and the model renders the referred region in that color before binary mask readout.

Geometry tasks use a similar color-channel representation. Depth is encoded as affine-invariant inverse depth, normalized per image to the RGB range and replicated across channels, so nearer points are brighter. At inference, the prediction is mapped back to inverse depth and evaluated after per-image affine alignment. For surface normals, normals are converted to a common camera-space convention and each vector component is mapped linearly to one normalized RGB channel. The resulting RGB prediction is mapped back to camera-space vectors and normalized to unit length. Prompts explicitly state the expected output representation.

Because all targets are RGB images, training uses the base generator's rectified-flow objective. The target is encoded as zyz_yzy​, Gaussian noise ϵ\epsilonϵ is sampled, and an interpolation coefficient t∈(0,1]t \in (0,1]t∈(0,1] is drawn uniformly. The noisy latent is formed as

zt=(1−t)zy+tϵ.z_t = (1 - t) z_y + t \epsilon.zt​=(1−t)zy​+tϵ.

Conditioned on the clean image xxx and text prompt ppp, the transformer predicts v^=vθ(zt,t,p,x)\hat{v} = v_\theta(z_t, t, p, x)v^=vθ​(zt​,t,p,x) with flow target ϵ−zy\epsilon - z_yϵ−zy​. Letting mjm_jmj​ indicate whether latent position jjj is supervised and CCC denote the number of latent channels, the flow-matching loss is

LFM=E(x,p,y),t,ϵ[∑jmj∥v^j−(ϵ−zy)j∥22C∑jmj].\mathcal{L}_{FM} = \mathbb{E}_{(x, p, y), t, \epsilon} \left[ \frac{\sum_j m_j \| \hat{v}_j - (\epsilon - z_y)_j \|_2^2}{C \sum_j m_j} \right].LFM​=E(x,p,y),t,ϵ​[C∑j​mj​∑j​mj​∥v^j​−(ϵ−zy​)j​∥22​​].

A validity mask is the only task-dependent training component. Geometry annotations contain undefined regions encoded by placeholder colors that a model might otherwise learn to reproduce. These regions account for about 8% of target pixels in the non-zero-shot pool and are excluded from the loss. Zero-shot models and non-geometry tasks use no mask and standard flow matching. During masked training, the validity mask follows image transforms and is area-resized to the latent grid. Positions with at least 50% valid coverage contribute to the loss. The final models use no erosion, exclude samples with no valid latent positions, and normalize each eligible sample by its valid area.

Full-scale models train for 100K steps at a global batch size of 192 on aspect-ratio buckets of about 5122512^25122 pixels, including 512×512512 \times 512512×512, 576×448576 \times 448576×448, 608×416608 \times 416608×416, and their transposes. The optimizer is AdamW with weight decay 10−410^{-4}10−4, using FSDP, bf16 autocast, and fp32 parameters. A constant learning rate follows linear warmup: 5×10−65 \times 10^{-6}5×10−6 with 500 warmup steps for the 4B model, and 2×10−62 \times 10^{-6}2×10−6 with 1,000 warmup steps for the 9B model. Each epoch draws 1M samples with replacement and splits them across tasks. Final models use weight-space averaging of a few late transformer checkpoints.

Experiment

The experiments evaluate Open Vision Banana models finetuned from FLUX.2 Klein 4B and 9B under zero-shot and non-zero-shot regimes on semantic, referring, instance segmentation, geometry, and retained image generation, using standard dense-prediction metrics and legend ensembling for test-time scaling. Zero-shot 9B models often match or surpass specialized baselines on semantic and instance tasks, while ensembling mainly corrects class-color binding errors and yields stronger consensus predictions. Ablations show that random color legends and subset or full-set prompt mixtures prevent fixed class-color memorization, balanced generation and editing data preserve generation ability, and updating most transformer parameters improves segmentation and normals, with fine-tuning also sharpening frozen feature representations.

OVB consistently outperforms the listed frozen zero-shot vision learners across supported perception tasks, with test-time ensembling and non-zero-shot adaptation providing additional gains. The 9B zero-shot model is especially strong on Cityscapes semantic segmentation and depth, while non-zero-shot training improves referring segmentation, instance segmentation, and surface normal error. Across OVB variants, larger scale generally favors depth and instance accuracy, but the 4B non-zero-shot model achieves the best semantic and reasoning segmentation. OVB 9B zero-shot reaches higher Cityscapes mIoU than the cited specialist and generalist baselines, and ensembling adds substantial gains over its single-pass mean. Non-zero-shot OVB improves over its zero-shot counterpart on most tasks, with the largest relative gains in referring segmentation and surface normal error. The frozen zero-shot vision learners score much lower on semantic segmentation and do not support several tasks, while OVB provides coverage across all six benchmarks.

On COCO-Object, the reported specialist methods range from under 38 mIoU to just under 50 mIoU, with CorrCLIP leading and diffusion-based DiffSegm and iSeg scoring lowest among the compared methods. The zero-shot 9B model with test-time scaling surpasses all listed specialists, while its mean single-pass performance is only on par with DiffSegm and iSeg. Test-time scaling is especially beneficial on this 80-class benchmark because many errors come from colors assigned to the wrong class. CorrCLIP is the strongest listed specialist on COCO-Object, ahead of OV-Stitcher and FSeg-LLaVA, while DiffSegm and iSeg are the weakest in the comparison. With test-time scaling, the zero-shot 9B model outperforms all listed COCO-Object specialists, but its mean single-pass performance is comparable to the diffusion-based DiffSegm and iSeg.

On ReasonSeg val, the zero-shot 4B model with query rewriting reaches the highest gIoU among methods that do not use Gemini 2.5 Pro, ahead of zero-shot SegZero and RSVP-GPT-4o as well as target-trained LISA and SenseNova-Vision. SAM 3 and Vision Banana still score higher, but both rely on Gemini 2.5 Pro as a reasoner. Query rewriting adds more gIoU after fine-tuning than in the pretrained base model. The zero-shot 4B model with rewritten queries outperforms both zero-shot and target-trained baselines on ReasonSeg val. SAM 3 and Vision Banana remain stronger but use Gemini 2.5 Pro as their reasoner.

On the RefCOCOg referring segmentation benchmark, X-SAM, SenseNova-Vision, and HyperSeg are the strongest target-trained entries shown. The evaluated zero-shot 9B model outperforms HybridGL but trails LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana. With target training, the 9B model improves substantially but remains behind the top target-trained specialists. Among target-trained methods, X-SAM, SenseNova-Vision, and HyperSeg achieve the highest reported overlap scores. The zero-shot 9B model is ahead of HybridGL but behind LocalizationHeads-LLaVA, SAM 3 with Gemini, and Vision Banana.

This comparison isolates the roles of the pretrained base model, OVB fine-tuning, and query rewriting on ReasonSeg. The pretrained base model gains little from explicit rewrites, whereas OVB 4B improves substantially with rewritten queries. Fine-tuning also lifts raw-query performance, suggesting the model learns to resolve implicit expressions itself. The pretrained base model stays low across raw and rewritten queries, so query rewriting alone is not enough without fine-tuning. OVB 4B in the zero-shot setting improves by about 27 gIoU from raw queries to explicit rewrites, with all three rewriters providing large gains. Moving from zero-shot to non-zero-shot improves raw-query performance more than Gemma4-rewritten performance, indicating the model partly learns to interpret implicit queries itself.

The experiments evaluate zero-shot and non-zero-shot OVB variants across semantic, referring, instance, and reasoning segmentation benchmarks against frozen zero-shot learners and specialist models. Zero-shot OVB consistently outperforms the listed frozen zero-shot baselines, with test-time ensembling or scaling providing additional gains, especially on Cityscapes and COCO-Object. Fine-tuning and query rewriting yield substantial improvements on reasoning and referring segmentation, and fine-tuning helps the model interpret implicit queries rather than relying only on explicit rewrites, though top specialist models remain stronger on some referring benchmarks.


بناء الذكاء الاصطناعي بالذكاء الاصطناعي

من الفكرة إلى الإطلاق — سرّع تطوير الذكاء الاصطناعي الخاص بك مع المساعدة البرمجية المجانية بالذكاء الاصطناعي، وبيئة جاهزة للاستخدام، وأفضل أسعار لوحدات معالجة الرسومات.

البرمجة التعاونية باستخدام الذكاء الاصطناعي
وحدات GPU جاهزة للعمل
أفضل الأسعار

HyperAI Newsletters

اشترك في آخر تحديثاتنا
سنرسل لك أحدث التحديثات الأسبوعية إلى بريدك الإلكتروني في الساعة التاسعة من صباح كل يوم اثنين
مدعوم بواسطة MailChimp