SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

Code performance optimization is paramount in real-world software engineeringand critical for production-level systems. While Large Language Models (LLMs)have demonstrated impressive capabilities in code generation and bug fixing,their proficiency in enhancing code performance at the repository level remainslargely unexplored. To address this gap, we introduce SWE-Perf, the firstbenchmark specifically designed to systematically evaluate LLMs on codeperformance optimization tasks within authentic repository contexts. SWE-Perfcomprises 140 carefully curated instances, each derived fromperformance-improving pull requests from popular GitHub repositories. Eachbenchmark instance includes the relevant codebase, target functions,performance-related tests, expert-authored patches, and executableenvironments. Through a comprehensive evaluation of representative methods thatspan file-level and repo-level approaches (e.g., Agentless and OpenHands), wereveal a substantial capability gap between existing LLMs and expert-leveloptimization performance, highlighting critical research opportunities in thisemerging field.