The MIT team has developed an AI system called SketchAgent that can create sketches and interact with humans during the creative process. The researchers tested SketchAgent’s ability to collaborate with humans or work independently, with the system generating recognizable sketches based on provided text descriptions. They found that certain prompts from human users, such as drawing a bridge or a vase, were essential for achieving high-quality final results. In this collaborative web-based environment, users can share and build upon sketches in real-time with SketchAgent. Each user input adds a stroke, and the process continues until both parties are satisfied with the final image. Green strokes indicate those made by the user, while purple strokes represent SketchAgent’s contributions. Users can also modify the sketches through chat conversations, providing feedback that helps refine the image. The team demonstrated the system's capabilities by generating a variety of conceptual sketches, including robots, DNA double helices, flowcharts, and even abstract art pieces like opera theaters. Looking ahead, SketchAgent holds potential applications in interactive art games and can aid teachers and researchers in visualizing complex concepts, as well as provide a faster way for users to learn sketching. In another experiment, the researchers evaluated SketchAgent’s performance across different language models. They discovered that Claude 3.5 Sonnet, which can generate pixel images (convertable into high-resolution text files), produced the most human-like drawings compared to other models like GPT-4 and Claude 3 Opus. “This result suggests that the model processes visual information in a unique manner,” noted co-author Tamar Rott Shaham. She further added that as the model’s ability to interpret and draw from diverse inputs improves, users will gain a more intuitive and human-like expression method, enhancing the collaborative experience and making AI more accessible and adaptive. Despite its promising potential, SketchAgent currently struggles with detailed depictions, such as fine labels, sentences, and intricate geometrical shapes, which limits its effectiveness in specialized drawings. During the collaboration process, the model sometimes misinterprets user inputs—drawing, for example, a two-headed monster. Vinker explained that this issue may stem from the model's "thought chain" mechanism: when breaking down a drawing task into multiple steps, the model might incorrectly discern which part of the sketch the human user intended to contribute. To improve these outcomes, the team plans to optimize the system for better collaboration by simplifying interactions with multiple language models. The current tool already shows promise in enabling AI to adopt more human-like thinking in the creation of diverse concepts, ultimately producing more harmonious and well-designed illustrations through human-AI interaction.