Yanxu Chen Zijun Yao Yantao Liu Jin Ye Jianing Yu Lei Hou Juanzi Li

Abstract

Large language models (LLMs) have recently demonstrated strong capabilitiesas autonomous agents, showing promise in reasoning, tool use, and sequentialdecision-making. While prior benchmarks have evaluated LLM agents in domainssuch as software engineering and scientific discovery, the finance domainremains underexplored, despite its direct relevance to economic value andhigh-stakes decision-making. Existing financial benchmarks primarily teststatic knowledge through question answering, but they fall short of capturingthe dynamic and iterative nature of trading. To address this gap, we introduceStockBench, a contamination-free benchmark designed to evaluate LLM agents inrealistic, multi-month stock trading environments. Agents receive daily marketsignals -- including prices, fundamentals, and news -- and must make sequentialbuy, sell, or hold decisions. Performance is assessed using financial metricssuch as cumulative return, maximum drawdown, and the Sortino ratio. Ourevaluation of state-of-the-art proprietary (e.g., GPT-5, Claude-4) andopen-weight (e.g., Qwen3, Kimi-K2, GLM-4.5) models shows that while most LLMagents struggle to outperform the simple buy-and-hold baseline, several modelsdemonstrate the potential to deliver higher returns and manage risk moreeffectively. These findings highlight both the challenges and opportunities indeveloping LLM-powered financial agents, showing that excelling at staticfinancial knowledge tasks does not necessarily translate into successfultrading strategies. We release StockBench as an open-source resource to supportreproducibility and advance future research in this domain.