One-sentence Summary

Google Cloud AI Research and the University of California, Los Angeles introduce FinanceHarness, an autonomous financial deep research framework that integrates finance-oriented tools and practitioner-guided workflows with FinanceGym, a verifiable point-in-time benchmark combining pre-cutoff and post-cutoff criteria, which reveals that even leading LLMs and agents score below $40\%$40% while FinanceHarness improves the rubric score from $25.3\%$25.3% to $32.4\%$32.4%.

Key Contributions

• FinanceGym is introduced as a point-in-time financial deep research benchmark comprising 400 expert-annotated questions and 2,464 rubric items, with professional expert validation yielding an 82% pass rate.
• FinanceHarness is presented as an end-to-end harness that automates financial deep research using finance-oriented tools, practitioner-guided workflows, and a reward model within the same temporally controlled environment as FinanceGym.
• Leading LLMs and agents score below 40% on FinanceGym, while FinanceHarness improves the rubric score from 25.3% to 32.4% with the same open-weight backbone, and subsequent GRPO training adds a further 0.4 point.

Introduction

Deep research agents powered by large language models have become widely adopted for answering complex, multi-hop questions and generating long-form reports. However, these general-purpose systems struggle in specialized domains like finance, where analysts must gather evidence across disparate sources, validate claims, understand relationships among financial entities and events, and reason about future outcomes. Existing benchmarks either test isolated NLP skills or lack the point-in-time evaluation needed to prevent future information from leaking into reports, making them unsuitable for measuring true financial research capability.

The authors address these gaps by constructing a point-in-time financial search sandbox with cutoff-date access control over a large-scale web corpus. They leverage this sandbox to build FinanceGym, an expert-validated benchmark of 400 research questions with paired pre-cutoff and post-cutoff rubrics that separately assess evidence retrieval and future-outcome reasoning. They also introduce FinanceHarness, an expert-knowledge-guided agent harness that automates environment construction, agent execution, and reward modeling under a strict temporal contract, enabling evaluation and in-environment optimization to share the same framework.

Dataset

The authors construct FinanceGym on top of a point-in-time financial search sandbox. The sandbox is built from a large-scale web corpus of over 100 million articles, collected from thousands of public web domains. Each article carries a reliably extracted publication date, and clean text is obtained with normalized metadata. The corpus deliberately preserves the noise and breadth of web-scale financial search. For retrieval, articles are embedded with Qwen3-Embedding-4B and stored in a FAISS IVF-SQ8 index served through a lightweight API. This API enforces a temporal access contract: agents can search and read historical documents but cannot access articles published after an assigned cutoff date.

FinanceGym itself is generated through a multi-stage pipeline:

• A finance entity graph is extracted from the corpus by filtering to finance-relevant sources and using Gemini-3.5-Flash to produce entity-relation triples. After filtering generic or malformed nodes, the working graph contains 4.37 million edges spanning 1.11 million unique entities from 1.20 million source articles.
• The graph is mined for fine-grained situations using three complementary modes: cross-category linkages, temporal narrative arcs around high-degree entities, and polar divergences (e.g., upgrades vs. downgrades). An entity budget prevents mega-cap firms from dominating the benchmark.
• Each candidate situation is paired with a cutoff date selected by an objective that balances event volume, entity diversity, and relation entropy. An LLM then generates a benchmark record containing an analyst-style question, a reference investment thesis, and a two-tier rubric. Pre-cutoff criteria test facts findable before the cutoff date, while post-cutoff criteria test outcomes verifiable only afterward, separating evidence retrieval from forward-looking synthesis.
• A bottom-up taxonomy is applied after generation: LLMs classify questions into natural categories, which are consolidated into three axes: topic, sector, and reasoning type.
• Generated questions pass through LLM quality gates testing feasibility, institutional relevance, coherence, groundedness, and balance. This yields a data-determined pool of 2,078 questions from 29,669 unconstrained generations.
• An integer linear program selects a 500-question subset that maximizes quality subject to balance constraints across taxonomy axes and monthly cutoff-date buckets.
• The balanced subset undergoes external expert annotation and LLM-assisted curation. Annotators score question feasibility and clarity and mark each rubric item as feasible or infeasible. Curation retains records with sufficiently clear questions and enough feasible rubrics: 411 of the 500 annotated questions meet this bar (82% pass rate). Each sample requires approximately 1.2 hours of expert work.
• A final release gate applies human expert review for quality, privacy, and provenance. Questions that fail any check are removed. The underlying corpus is not publicly released; only the research questions and report submission code are released, with grading running through a private-rubric leaderboard.

The released FinanceGym benchmark contains 400 expert-annotated questions with 2,464 annotated rubric items (mean 6.16 items per question). The dataset is jointly balanced across 9 topics, 11 sectors (merged into 9 leaves), 6 reasoning types, and 12 monthly cutoff-date buckets. Sector coverage spans healthcare/biotech, technology/semiconductors, consumer discretionary, financial services, real estate, energy/natural resources, industrials/transportation, macro/policy, commodities, crypto/digital assets, and fixed income. The expert-annotation curation pass closely preserves the ILP sector balance.

Method

The authors construct a comprehensive benchmark infrastructure comprising a point-in-time (PIT) financial search sandbox, the FinanceGym benchmark, and the FinanceHarness execution stack. The end-to-end pipeline integrates corpus processing, graph-based situation mining, constrained question generation, and expert curation to support both evaluation and model optimization.

The foundation of this system is the PIT financial search sandbox. The authors build a large-scale web corpus of over 100 million articles, extracting publication dates with htmldate and clean text with trafilatura to enforce strict temporal access. Articles are embedded using Qwen3-Embedding-4B to produce normalized dense vectors, which are indexed in a FAISS IVF-SQ8 server. This setup ensures that agents can only retrieve documents published on or before a specific cutoff date.

To generate the benchmark, the authors first construct a finance entity graph by extracting entity-relation triples from the corpus using Gemini-3.5-Flash, resulting in a working graph of 4.37 million edges. Rather than clustering broadly, they mine specific situations mirroring financial analyst workflows through three modes: cross-category linkages, temporal narrative arcs around high-degree entities, and polar divergences like earnings beats versus misses. Each candidate situation is paired with a cutoff date selected via an objective that balances event volume, entity diversity, and relation entropy:

Given a situation and cutoff date, an LLM generates an analyst-style question, a reference investment thesis, and a two-tier rubric without category priming. Pre-cutoff criteria test retrievable facts, while post-cutoff criteria test forward-looking synthesis.

Following generation, the authors assign a bottom-up taxonomy by classifying questions into natural categories, consolidating them into topic, sector, and reasoning type axes. The resulting distribution of the 400-question FinanceGym subset across these axes is illustrated below.

Generated questions undergo rigorous LLM quality gates testing feasibility, institutional relevance, coherence, and groundedness. An integer linear program then selects a balanced 500-question subset. This subset passes through professional expert annotation, where annotators score feasibility and clarity and validate rubric items. The quality control metrics for this annotated pool, including expert ratings and rubric depth, are detailed in the following figure.

The evaluation contract relies on rubric-based scoring. An LLM judge evaluates agent reports against question-specific rubrics on a 5-tier scale from 0 to 4. The primary metric is the outcome score, calculated as the average fraction of rubric points earned across all questions:

During evaluation, the FAISS server strictly enforces the PIT protocol, invalidating any run that detects temporal leakage.

FinanceHarness serves as the executable environment built around this contract. It exposes a layered finance tool interface rather than a generic web search. The architecture includes a model and serving layer that hosts the orchestrator backbone and a lightweight reader, remaining opaque to the rest of the system to allow model swapping. The runtime layer acts as the control plane, managing the bounded agent loop, schema validation, and tool registries without containing model intelligence. The tool surface is tiered to keep prompts small while preserving broad capabilities, with always-loaded core actions and deferred tools loaded via a compact catalog. The system supports multiple prompt-variant modes, such as research and analytical modes, and utilizes reusable workflow skills for structured analysis. Grounding mechanisms ensure citations are composed correctly, and operational guardrails like URL pre-fetching and search budget caps manage cost and robustness. Because the harness tool calls hit the PIT search sandbox directly and uses the same rubric judge for reward signals, models trained within FinanceHarness experience the exact same tool-use distribution at test time.

Experiment

FinanceGym is constructed by mining analyst-style situations from a finance entity graph, generating unconstrained questions with pre- and post-cutoff rubrics, and filtering them through LLM quality gates and expert annotation to produce a balanced 400-question benchmark. Evaluation uses a point-in-time protocol where agents search a cutoff-filtered corpus and are scored by a rubric-based LLM judge on both retrospective synthesis and forward-looking reasoning. Experiments compare fine-tuned open-weight models, foundation models with a fixed search wrapper, and agentic search systems, revealing that backbone model choice drives more performance variation than scaffolding design, while the persistent gap between pre- and post-cutoff scores highlights the benchmark’s difficulty in anticipating future outcomes.

FinanceGym fills a gap in financial deep-research evaluation by combining long-form report generation with a reproducible, per-question publication-date cutoff corpus and a two-tier rubric that separates historical evidence retrieval from forward-looking outcome anticipation. Most existing benchmarks either lack domain specificity, rely on live web access, or do not isolate pre-cutoff and post-cutoff performance, which is central to analyst-style financial research. FinanceGym is the only benchmark that simultaneously requires long-form finance-domain reports, uses a reproducible fixed corpus, enforces per-question publication-date cutoffs, and applies a verifiable two-tier rubric. Existing finance benchmarks like FinanceBench evaluate short-form QA over filings rather than the long-form, entity-connecting research that FinanceGym targets. Other reproducible benchmarks such as BrowseComp-Plus and DeepResearchGym do not apply per-question publication-date cutoffs, while DeepScholar-Bench is tied to an evolving arXiv index rather than a controlled financial corpus. DeepResearch Bench II and ResearchRubrics provide rich rubrics but do not separate pre-cutoff evidence retrieval from post-cutoff outcome anticipation, which is the central difficulty in financial deep research.

Fine-tuned open-weight models on FinanceGym achieve overall scores from 21.2 to 28.2, with Tongyi-DR leading the group. Pre-cutoff performance is consistently several times higher than post-cutoff performance, underscoring the benchmark's difficulty on forward-looking items. A task-matched harness on a 27B backbone outperforms all open-weight models and the best agentic search system, trailing only two proprietary backbones. Among fine-tuned open-weight models, Tongyi-DR achieves the highest overall score (28.2), followed by OpenResearcher (27.2) and MiroThinker (21.2). Pre-cutoff scores are roughly 3–5 times higher than post-cutoff scores across all systems, reflecting the challenge of forward-looking financial analysis. FinanceHarness, a 27B open-weight system with a retrieval harness, surpasses the best fine-tuned open-weight model and the top agentic search system, approaching proprietary frontier performance.

Adding a naive harness improves over search-only, and the full untrained FinanceHarness yields a further gain, while GRPO fine-tuning on a separate training set provides only a small additional refinement. Pre-cutoff scores are consistently several times higher than post-cutoff scores, reflecting the benchmark's core difficulty rather than retrieval differences. The full untrained harness raises the total normalized rubric mean from 25.3 (search-only) to 32.4, with pre-cutoff scores climbing from 36.1 to 45.7. GRPO reinforcement fine-tuning adds just 0.4 points over the untrained harness, moving the total from 32.4 to 32.8, so it is treated as a refinement rather than a headline result. Across all configurations, pre-cutoff scores are several times higher than post-cutoff scores, indicating the forward-looking items remain the main challenge independent of the retrieval setup.

FinanceGym introduces a reproducible financial deep-research benchmark centered on long-form report generation, using a fixed corpus with per-question publication-date cutoffs and a two-tier rubric that separately evaluates historical evidence retrieval and forward-looking outcome anticipation. Fine-tuned open-weight models achieve modest overall scores, with pre-cutoff performance consistently several times higher than post-cutoff performance, highlighting the difficulty of forward-looking financial analysis. A task-matched retrieval harness on a 27B backbone substantially improves performance over search-only and fine-tuned models, approaching proprietary frontier results, while reinforcement fine-tuning provides only marginal additional gains. The benchmark's core challenge remains the post-cutoff, forward-looking items regardless of the retrieval configuration.