Computer Use refers to agents using computers like humans—looking at the screen, moving the mouse, typing on the keyboard, and switching between software—without relying on dedicated interfaces. It addresses the "last mile" problem: many software applications lack APIs, or their APIs have limited capabilities, so agents must use software in the same way humans do.

The public starting point for Computer Use was a model capability released by Anthropic in October 2024 (official announcement, where the model operated the desktop directly through screenshots and coordinates. During the same period, Xie et al. constructed the first scalable benchmark for real computer environments in the paper OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.

The evaluation method for Computer Use is execution-based assessment. OSWorld sets up environments in real systems on Ubuntu, Windows, and macOS, providing 369 tasks spanning web and desktop applications, file read/write operations, and multi-application workflows, with each task derived from real-world usage scenarios. Unlike evaluations that only look at the final answer, execution-based assessment checks the system state after task completion, closer to "getting the job done for real."

The technical loop of Computer Use is "Look—Think—Click": multimodal models convert screen screenshots into coordinates and actions. This transforms "knowing how to use a computer" into a measurable and reproducible capability, defining the capability coordinates for general-purpose digital assistants; the challenges are also clear: grounding (locating interface elements), error accumulation in long-horizon tasks, recovery after operational failures, and security risks from misclicks or accidental deletions.