AI Framework PyVision Lets Models Write Their Own Tools as They Reason Through Visual Tasks

Scale AI, a leading data-labeling startup, has confirmed a major investment from Meta, boosting the company’s valuation to $29 billion. As part of the deal, Scale’s co-founder and CEO Alexandr Wang will step down and join Meta to support its efforts in building superintelligent AI systems. The investment, reported as approximately $14.3 billion for a 49% stake, underscores Meta’s strategic push to strengthen its AI capabilities amid competition from rivals like OpenAI, Google, and Anthropic. Scale AI’s role in providing training data for large language models (LLMs) has been critical, particularly for generative AI development. The company emphasized it will remain independent, with Wang retaining a board director position. Jason Droege, Scale’s current chief strategy officer, will serve as interim CEO. The funds will be allocated to return capital to shareholders and fuel growth, reflecting the high demand for high-quality data in the AI industry. The collaboration highlights a growing trend: AI labs are increasingly relying on specialized data providers to enhance model performance. Recent reports show Scale AI and competitors have intensified hiring for top talent, including PhDs and senior engineers, to meet this demand. This investment also addresses Meta’s need to close gaps in its AI model releases, which have lagged behind industry leaders. The paper introduces PyVision, a framework enabling large multimodal language models (MLLMs) to autonomously generate and execute Python-based tools for visual reasoning tasks. Traditional models struggle with dynamic adaptation, relying on fixed toolkits and linear processing. PyVision breaks this constraint by allowing models to create custom tools during tasks, using Python as a core language. It operates in a multi-turn loop, where the model iteratively refines its approach based on feedback from executed code. PyVision’s process begins with a user query and visual input, prompting the MLLM (e.g., GPT-4.1 or Claude-4.0-Sonnet) to generate Python code. This code runs in an isolated environment, producing results (text, visuals, or numbers) that the model then uses to adjust its strategy. It supports cross-turn persistence, maintaining variable states between interactions for sequential reasoning. Safety measures like process isolation and structured I/O ensure reliability. Libraries such as OpenCV, NumPy, and Pillow enable tasks like segmentation, OCR, and statistical analysis. Benchmark tests show PyVision’s effectiveness: it improved GPT-4.1’s performance on the visual search task V* by 7.8% (to 75.9%) and boosted Claude-4.0-Sonnet’s accuracy on symbolic reasoning benchmarks by over 30%. Other tasks saw gains of 2.4–8.3%, depending on the model’s strengths. The framework enhances existing models rather than replacing them, leveraging their inherent capabilities for complex visual reasoning. PyVision represents a breakthrough in enabling AI systems to dynamically adapt and solve problems iteratively, moving beyond static toolsets. By integrating Python’s flexibility with multimodal reasoning, it addresses a key bottleneck in AI’s ability to handle abstract and context-dependent tasks. Evaluation & Context: PyVision’s approach marks a shift toward agentic AI systems capable of self-directed problem-solving. Experts highlight its potential to revolutionize domains like medical diagnostics and visual math, where adaptability is crucial. The collaboration between Shanghai AI Lab, Rice University, CUHK, NUS, and SII reflects a growing trend of academic-industry partnerships in AI innovation. While the framework enhances existing models, its reliance on Python’s ecosystem could lower barriers for developers. However, challenges remain in scaling safety and ensuring ethical use of self-modifying tools. The paper and GitHub repository offer open access to the research, inviting further exploration.