SWE-Serve Exposes Gap Between Local Tests and Live Serving

Researchers have introduced SWE-Serve, a new benchmark designed to evaluate AI coding agents on repository-scale changes to large language model inference and serving software. Developed with direct input from the SGLang team, the framework addresses a critical disconnect in current evaluation methodologies: patches that successfully pass local unit tests frequently fail during live server deployment. SWE-Serve closes this gap by requiring agents to verify changes through a complete inference-serving path, including model loading, request routing, and API response generation. The benchmark derives its 53 executable tasks from 83 merged pull requests to the open-source SGLang repository. These tasks are categorized into six engineering families, spanning speculative decoding, model and backend enablement, kernel optimization, serving APIs, caching, and distributed scheduling. While the majority of tasks run on a single NVIDIA H100 or CPU, nineteen specifically mandate live-server verification. Each task provides the agent with a containerized repository revision preceding the target change and a hidden verifier containing multiple regression and behavioral tests. Success is measured solely by the agent’s ability to satisfy the verifier without reference to the original implementation. Initial evaluations reveal a substantial performance drop when live-serving checks are enforced. Patches that cleared 69.4 percent of tasks without live verification fell to 45.9 percent when the full serving path was tested. The discrepancy highlights the complexity of end-to-end system integration, as failures in expert routing, request batching, and state management often remain undetected until runtime. Performance also degrades significantly when tasks span multiple runtime domains, dropping twenty-one percentage points compared to single-domain evaluations. Testing across eleven leading AI models demonstrated that cost and speed do not strictly dictate capability. GPT-5.6 Solmax and Claude Opus 5max achieved the highest pass rates at 75.5 percent and 75 percent respectively, utilizing maximum reasoning effort configurations. Mid-tier performers clustered around a 64 percent pass rate but exhibited vast differences in processing time and API expenditure, ranging from under a dollar to over seven dollars per task. No single model dominated all engineering families, indicating that specialized optimizations remain necessary for comprehensive inference-stack mastery. The benchmark underwent rigorous validation, screening 786 potential sources and admitting only 53 tasks that met strict correctness and regression criteria. All evaluations were conducted under closed-book conditions to prevent models from retrieving upstream code, with automated safeguards blocking unauthorized external queries. SWE-Serve provides both a public leaderboard and a complete GitHub repository, enabling developers to stress-test their coding agents against realistic, production-grade serving challenges. The initiative establishes a more accurate standard for measuring the readiness of AI-assisted software engineering in complex inference architectures.