One-sentence Summary

Mercor introduces APEX–Agents, an open-source benchmark ($n = 480$n=480) for evaluating AI agents on long-horizon, cross-application tasks from investment banking, consulting, and law within realistic file-and-tool environments, testing eight agents via $\text{Pass@1}$Pass@1, where Gemini 3 Flash (Thinking=High) leads at $24.0\%$24.0%, and also releases Archipelago, its infrastructure for agent execution and evaluation.

Key Contributions

• Introduces APEX–Agents, a benchmark of 480 long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate lawyers, designed to evaluate AI agents in realistic professional work environments with files and tools.
• Presents a leaderboard evaluation of eight agents using Pass@1, showing that Gemini 3 Flash achieves the highest score at 24.0%, followed by GPT-5.2 at 23.0%, Claude Opus 4.5 at 18.4%, and Gemini 3 Pro at 18.4%, with open-source models scoring under 5%.
• Open-sources the full APEX–Agents dataset (including prompts, rubrics, gold outputs, files, and metadata) under a CC-BY license, along with Archipelago, an open-source infrastructure for agent execution and evaluation.

Introduction

If AI agents can reliably execute professional services work, the economic and social impact would be profound, reshaping knowledge work and boosting productivity. Enterprises like Box, Salesforce, and Databricks are already deploying agentic systems, yet existing agentic evals suffer from a large sim-to-real gap. These benchmarks are often narrowly scoped, contrived, and contain only simple tasks, failing to capture how professionals work day-to-day.

To address this, the authors present APEX–Agents, a new benchmark for frontier AI evaluation. The tasks were created by investment banking analysts, management consultants, and corporate lawyers, requiring agents to reason, demonstrate advanced knowledge, use multiple applications, and plan over long horizons. The benchmark was built by creating data-rich worlds based on unique project scenarios, where professionals planned work and produced customer-ready deliverables over 5-10 days. Agents were then given access to these worlds with all the data and software a human would use.

APEX–Agents contains 480 tasks across 33 worlds. Testing eight models, the authors found that Gemini 3 Flash performs best with a 24.0% Pass@1 score, followed by GPT-5.2 at 23.0%, with Claude Opus 4.5 and Gemini 3 Pro at 18.4%. Open-source models scored under 5%, highlighting a significant gap. The results show substantial headroom for improvement, as even top-performing agents score under 25% on Pass@1, with failures ranging from running out of steps to inconsistent performance across runs. The dataset and infrastructure, Archipelago, are open-sourced to enable further research.

Dataset

Dataset Composition and Sources

The APEX-Agents benchmark is built from a survey of 227 experts from the Mercor talent marketplace, covering 58 financial and investment analysts, 77 management consultants, and 92 lawyers, with an average of 10.8 years of professional experience. The survey asked participants to allocate their time across core activities, learning, admin, communications, meetings, and non-productive time. Core activities made up 47% of total time, and free-text descriptions of these activities were manually grouped into 18 categories, which directly informed task creation for the benchmark.

World Creation and Structure

• The benchmark comprises 33 worlds: 10 in investment banking, 11 in management consulting, and 12 in law.
• Each world starts from a project scenario created by experts, inspired by real professional projects, and includes context about the delivery company, customer, project goals, and constraints.
• Experts assumed in-world roles (e.g., partner, analyst, associate) and worked toward project objectives, sending emails, conducting research, and iterating on deliverables like spreadsheets, reports, and slide decks. One expert per project played the customer role.
• On average, each world contains 166 files. Of the 33 worlds, 22 involve entirely fictional entities, 9 involve real companies in fictional scenarios, and 2 mix both.
• Each world has nine applications (Calendar, Chat, Code Execution, Documents, File system, Mail, PDFs, Spreadsheets, Presentations) providing 63 tools. Two investment banking worlds also include EDGAR SEC, an equities and financial markets data app, and a fixed income markets data app, adding 187 tools. Web search is disabled to keep evaluations reproducible.

Task Design

• Worlds have between 8 and 20 tasks, with an average of 14.5 tasks per world.
• Each task is a single-turn prompt that requires using in-world files and applications. Tasks were informed by the APEX Survey and underwent multiple review rounds.
• Contributors used frontier agents (GPT-5, Claude Opus 4.5, Gemini 3 Pro) to collect outputs and adversarially iterate on task quality.
• Of 480 tasks, 422 require a message in the console as output. The remaining 58 require creating or editing spreadsheets (14 create, 16 edit), documents (20 create, 2 edit), or presentations (5 create, 1 edit).

Rubrics and Gold Outputs

• Experts created rubrics with criteria that are self-contained, short, descriptive statements graded as true or false against an agent's output. Each criterion has a grading target specifying the required output type.
• Tasks have between 1 and 10 criteria, with a mean of 4.06. Investment banking tasks have fewer criteria on average because many require a single value (e.g., an updated EBITDA).
• For each task, experts created gold outputs containing the exact information to address the prompt, in the requested output type. Experts manually graded these gold outputs against the rubric to ensure alignment.
• Each prompt and rubric is tagged with metadata, including expected output type, workflow, and estimated time for an industry professional to complete the task in the real world.

Contributors and Baselining

• 256 experts contributed to APEX-Agents, with a mean experience of 12.9 years (median 11.0). They include former consultants from BCG and McKinsey, investment bankers from Morgan Stanley and Citigroup, and corporate lawyers at Disney and other Fortune 500 companies. They created scenarios, built worlds, created tasks, audited quality, and checked gold outputs and rubrics.
• For 20% of tasks (n = 96), independent experts executed them from scratch to verify task feasibility, rubric fairness, and time estimates. Minor issues in prompts, rubrics, or metadata were found in 10% of tasks and fixed, with fixes cascaded to the rest of the dataset.
• For these sample tasks, experts estimated completion time at 1.70 hours, while the true time was 1.37 hours, an overestimate of 33%. The estimate for the whole benchmark is 1.82 hours.

Method

The authors conduct a baselining study to validate the quality and feasibility of the benchmark tasks. For 20% of the tasks, corresponding to 96 instances, experts who were not involved in the creation or review process independently executed the tasks from scratch. This procedure serves three purposes: verifying that the tasks can actually be completed, assessing the fairness of the rubric, and checking the accuracy of the time estimates provided by the original task creators. In 10% of the baselined tasks, the authors identified minor issues in the prompt, rubric, or metadata, which were subsequently fixed and the corrections cascaded to the entire dataset to ensure consistency. For the sampled tasks, the experts estimated the completion time at 1.70 hours, whereas the actual measured time was 1.37 hours. This reveals an over-estimate of 0.45 hours, corresponding to a 33% deviation, indicating that the original time estimates tend to be conservative relative to real execution times.

Experiment

The APEX-Agents benchmark was built with 33 expert-created worlds across investment banking, management consulting, and law, containing 480 long-horizon tasks graded by rubrics and gold outputs, with a baselining study confirming task feasibility and rubric fairness. Evaluation used a judge model (Gemini 3 Flash) validated at 98.5% accuracy on a ground truth set, with Pass@1 as the primary metric. Results show Gemini 3 Flash leading at 24.0% Pass@1, closely followed by GPT-5.2, while open-source agents lagged below 5%; Pass@8 scores were roughly 15 points higher, indicating capability but inconsistency. Failure analysis revealed that successful trajectories used fewer steps and tool calls, avoided doom loops, and that file-output tasks were harder, while token usage varied widely without correlating with performance.

The APEX–Agents benchmark spans three domains with 480 tasks across 33 worlds, averaging about 166 files per world and 4 evaluation criteria per task. Tasks that require file outputs are less common and tend to yield lower agent scores, while file deletion is rare and considered undesirable behavior. Investment banking has the highest proportion of file-output tasks at 16.9%, while management consulting has the lowest at 6.9%. Law tasks have the highest average estimated completion time at 2.40 hours, compared to 1.36 hours for investment banking. Across the benchmark, only 12.1% of tasks require file outputs, and agents score lower on these tasks than on console-output tasks. File deletions occur in only 0.12% of trajectories, with GPT-5.2 responsible for the most deletions.

The judge model shows high agreement with human labels, with an overall accuracy of 98.5% and balanced precision and recall around 97-98%. Errors are rare, with false positives and false negatives each occurring in about 1-2% of cases, and the model performs similarly for the agent it grades, mitigating self-preference concerns. The judge model correctly identifies most met and failed criteria, with only a small number of misclassifications in each direction. Precision and recall are both high, indicating that the judge rarely over- or under-credits criteria. The false positive rate for the judge model on the agent it also evaluates is comparable to that for other agents, suggesting limited self-preference bias.

Gemini 3 Flash leads the APEX–Agents benchmark with a Pass@1 of 24.0%, closely followed by GPT-5.2 at 23.0%, while open-source agents lag far behind with scores under 5%. Pass@8 scores are about 15 percentage points higher than Pass@1, indicating agents are capable but inconsistent. Statistical tests show the top two models are not significantly different from each other but are significantly better than all others. Closed-source agents outperform open-source agents by a large margin, with open-source models scoring below 5% on Pass@1. Pass@8 scores are roughly 15 percentage points higher than Pass@1, suggesting agents have capability but lack consistency. The difference between the two best agents, Gemini 3 Flash and GPT-5.2, is not statistically significant, but both are significantly better than all other models. Open-source agents are relatively stronger in corporate law tasks, scoring around 8%, compared to their single-digit performance in other job categories. GPT-5.2 achieves the highest Pass@8 at 40.0%, followed by Gemini 3 Pro and Gemini 3 Flash.

On APEX-Agents, closed-source models outperform open-source ones, with Gemini 3 Flash achieving the highest pass rate, followed by GPT-5.2. All agents score zero in at least 40% of runs, and timeouts are more frequent for open-source models, especially Kimi K2 Thinking. File-related tasks and unwanted file deletions are additional sources of failure. Gemini 3 Flash has the highest pass rate, followed by GPT-5.2, while open-source models like GPT-OSS-120B lag substantially. All agents fail completely in at least 40% of their runs, indicating high task difficulty. Open-source models time out much more often than closed-source ones, with Kimi K2 Thinking timing out in nearly 30% of trajectories. Unwanted file deletions are rare but occur most frequently with GPT-5.2, while some models never delete files. Tasks requiring file creation or editing yield lower scores than console-output tasks across all leading models.

Agents show wide variation in resource use, with Gemini 3 Flash consuming the most tokens and steps, while Kimi K2 Thinking also uses high resources without correspondingly better outcomes. Final answer token counts are similar across agents, indicating comparable verbosity. Gemini 3 Flash uses nearly 5 times as many tokens as GPT-5.2 and about 8 times as many as Gemini 3 Pro, along with roughly 54% more steps and 22% more tool calls than GPT-5.2. Kimi K2 Thinking averages 92 steps and 91 tool calls per task and 1.6 million tokens, yet higher resource use does not consistently lead to better output. Answer token counts are relatively uniform, ranging from 191 for Gemini 3 Pro to 472 for Claude Opus 4.5, suggesting agents do not differ much in verbosity.

The APEX-Agents benchmark evaluates agents across 480 tasks in three domains, revealing that closed-source models, led by Gemini 3 Flash and GPT-5.2, significantly outperform open-source agents, which score below 5% on Pass@1. The judge model demonstrates high reliability with 98.5% accuracy and minimal self-preference bias. File-output tasks are rare and harder, while resource usage varies widely, with Gemini 3 Flash consuming far more tokens and steps without a proportional gain in performance, and all agents fail completely in at least 40% of runs.