HyperAIHyperAI

Command Palette

Search for a command to run...

Benchmarks
LLM

SO-Arena : évaluation du spectre de capacités et de la fiabilité décisionnelle des modèles System One

Lichen Li Ziping Dong Sicheng Feng Qi Li Xinchao Wang

Résumé

Les modèles System One promettent une prise de décision rapide et généraliste sans génération longue de tokens. Cependant, comprendre leur spectre de capacités et la fiabilité de leurs décisions exige davantage que la performance agrégée sur les benchmarks actuels. Les évaluations existantes offrent souvent un aperçu limité de l'étendue des capacités des modèles et de la fiabilité de leurs décisions. Nous présentons SO-Arena, un benchmark caractérisant le spectre de capacités et la fiabilité décisionnelle des modèles System One à travers 11 domaines et cinq familles de capacités. Pour le spectre de capacités, SO-Arena propose une évaluation fondée sur des preuves grâce à des sources fiables et à une vérification manuelle des étiquettes de référence par rapport aux entrées visibles par le modèle ; une évaluation fine des capacités via des critères de difficulté spécifiques à chaque domaine ; et une évaluation interactive réaliste, où les modèles exécutent itérativement des actions dans des environnements sandbox. Les tâches à forte composante de raisonnement permettent en outre une comparaison entre System One et System Two. Au-delà de l'exactitude, SO-Arena évalue la fiabilité décisionnelle à l'aide de deux mesures complémentaires : la stabilité des décisions face à des modifications contrôlées de l'ensemble des options, et la reconnaissance de l'absence de réponse lorsqu'aucune option valide n'est disponible. Les évaluations menées sur 21 modèles révèlent que les bénéfices du passage à l'échelle et du raisonnement supplémentaire varient considérablement selon les domaines. Plus important encore, une forte performance globale n'implique pas nécessairement une prise de décision fiable : le modèle System One le plus performant ne se classe que septième en fiabilité. Ces résultats soulignent la nécessité de caractériser le spectre de capacités et la fiabilité décisionnelle comme des dimensions complémentaires, offrant ainsi une vision systématique des forces et des limites des modèles System One.

One-sentence Summary

The authors introduce SO-Arena, a benchmark that evaluates System One models across 11 domains and five capability families via evidence-grounded evaluation with reliable sources and manually verified reference labels, fine-grained domain-specific difficulty criteria, and interactive sandbox execution, while assessing decision reliability through decision stability under controlled option-set changes and missing-answer recognition when no valid option is available; across 21 models, scaling and reasoning gains vary by domain, and the best-performing System One model ranks only seventh in reliability.

Key Contributions

  • SO-Arena is a benchmark that characterizes the capability spectrum and decision reliability of System One models across 11 domains and five capability families, using evidence-grounded labels, domain-specific difficulty tiers, and interactive sandbox-based action evaluation.
  • It contributes complementary reliability measures that assess decision stability under controlled option-set changes and missing-answer recognition when no valid option is available, separating judgment reliability from accuracy.
  • Evaluations across 21 models show that scaling and added reasoning yield domain-dependent gains rather than uniform improvements, and that strong competence does not guarantee reliable decisions, with the best-performing System One model ranking only seventh in reliability.

Introduction

System Two reasoning improves language model performance in mathematics, coding, and science, but it increases inference cost because models generate extra reasoning tokens. Many routine software decisions, such as request routing, tool selection, and completion verification, can be made by System One models that predict a distribution over predefined outcomes in one forward pass, which lowers latency and gives downstream software structured outputs. Prior evaluations of these models have measured accuracy, calibration, latency, and cost, but they do not consistently combine trustworthy model-visible reference labels, difficulty-stratified tasks, interactive actor and verifier roles, and controlled option-set reliability checks. The authors introduce SO-Arena, a benchmark with 2,919 tasks across 11 domains and five capability families that unifies static and environment-grounded decisions, organizes domains into difficulty tiers, distinguishes action selection from completion verification, and reports a separate reliability score covering option reordering and missing correct answers. Evaluating 21 models, they find that strong capability and reliable decision-making do not always align.

Dataset

The authors construct a decision-making benchmark from 40 distinct public datasets, benchmarks, and environments across 11 domains. Representative sources include GoEmotions, MATH500, GPQA Diamond, Mind2Web, BFCL, and ToolSandbox. Some tasks are converted directly from source data, while others are newly derived, such as completion checks on BrowseComp-Plus search runs and VirtualHome trajectories.

Quality control and metadata

  • Source labels are kept as references and reviewed manually and with LLMs.
  • Items with questionable references are excluded.
  • Each task records its source, version, and original identifier where applicable.
  • Model-derived references are flagged in metadata.

Task composition

  • 11 domains are grouped into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion.
  • Nine domains have difficulty tiers; Sentiment & Emotion Understanding and Knowledge & Commonsense are untiered.
  • The benchmark contains 2,300 static tasks and 619 environment-based tasks.
  • Environment-based tasks split into 337 web tasks, with 100 single-step judgments and 237 reference-route tasks, plus 282 interactive tool-use episodes.
  • Table 3 summarizes per-domain sources, task counts, difficulty tiers, and decision formats.

Processing and formulation

  • Static tasks are converted into choice, yes/no-style, or score formats.
  • Choice items use multiple-choice questions as-is; classification labels become options.
  • Open-answer math problems use distractors built from wrong base-model answers, intermediate results, and rule-based near misses.
  • Binary and multi-label tasks become yes/no questions.
  • Score tasks keep the native ordinal scale.
  • Self-built tasks use execution, verified construction, or annotation as references.
  • Free-text response tasks are excluded.
  • Environment-based tasks use recorded page states or simulator execution, with multi-step web routes verified by replay.
  • No cropping strategy is described; the main transformations are format conversion and option perturbation.

How the data is used

  • The benchmark is used for evaluation, not as a training split.
  • Capability scoring treats the original task as the scoring unit and weights domains equally.
  • For tiered domains, scoring focuses on the hardest tier; Math uses hard and very hard tiers; untiered domains use all tasks.
  • Reliability evaluation shows each item in the original order and 12 reorderings, and also shows items both as-is and with the correct option removed to test recognition of a missing correct option.

Method

The authors define System One decision models as systems that map a supplied state and a decision specification to structured predictions over admissible outcomes. An instance is represented as x=(s,q,C)x = (s, q, \mathcal{C})x=(s,q,C), where sss contains the supplied information, qqq specifies the decision objective, and C={c1,…,cK}\mathcal{C} = \{c_1, \ldots, c_K\}C={c1​,…,cK​} defines the admissible outcomes. The model returns a probability distribution pθ(⋅∣x)p_\theta(\cdot \mid x)pθ​(⋅∣x) satisfying non-negativity and sum-to-one constraints. The interface supports three primary output formats: categorical selection (choice) for explicitly specified alternatives, binary judgment (noul) for yes/no questions over the outcome space {0,1}\{0, 1\}{0,1}, and ordinal assessment (score) for ordered descriptive levels where the predicted grade corresponds to the most probable level.

The construction of the SO-Arena benchmark follows a structured pipeline encompassing source collection, quality control, task organization, and evaluation.

First, tasks are drawn from 40 established public sources. A quality control phase involving human and LLM review filters out items with questionable labels. The curated tasks are then organized into domains and capability families, with difficulty tiers assigned where applicable. Finally, the evaluation phase computes capability and reliability scores based on the model's performance on these structured tasks.

The benchmark comprises 2,919 tasks spanning 11 domains grouped into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion.

Nine of the domains feature domain-relative difficulty tiers ranging from Easy to Very Hard, while two remain untiered due to limited scope for varying complexity. The tasks are formulated into two distinct settings to reflect real-world applications:

  • Static Tasks: The model judges a fixed input. Multiple-choice questions are used directly, while open-answer problems have distractors generated from base model errors or rule-based near misses. Binary judgments are posed as yes/no questions.
  • Environment-Based Tasks: The model assesses or selects actions based on observations from web or tool environments. This includes single-step web judgments, multi-step sequential decisions along reference routes, and interactive tool-use episodes where simulators execute actions and return observations for the next decision.

To assess model performance, the authors define two primary metrics: Capability and Reliability. The Capability Score C(m)C(m)C(m) averages the performance on the hardest tier of each domain, ensuring equal domain weighting so larger domains do not dominate the evaluation:

C(m)=100D∑d=1D1∣Td∣∑i∈Tdci(m)C(m) = \frac{100}{D} \sum_{d=1}^{D} \frac{1}{|\mathcal{T}_d|} \sum_{i \in \mathcal{T}_d} c_i(m)C(m)=D100​d=1∑D​∣Td​∣1​i∈Td​∑​ci​(m)

where DDD is the number of domains and Td\mathcal{T}_dTd​ contains the hardest tier tasks.

The Reliability Score R(m)R(m)R(m) combines two components to ensure robust decision-making. The first is Option Order Stability (SSS), which measures consistency when options are reordered. S(m)S(m)S(m) rescales the agreement rate A(m)A(m)A(m) across 12 reorderings against random chance pep_epe​. The second is Recognition of Missing Correct Options (NNN), which measures the model's ability to assign higher probability to the absence of a correct answer (Null) when none of the offered options is correct, quantified using the Area Under the ROC Curve (AUC). The combined reliability score is the geometric mean of these components, provided both exceed chance:

R(m)={S(m)N(m),S(m)>0 and N(m)>0,0,otherwise.R(m) = \begin{cases} \sqrt{S(m)N(m)}, & S(m) > 0 \text{ and } N(m) > 0, \\ 0, & \text{otherwise}. \end{cases}R(m)={S(m)N(m)​,0,​S(m)>0 and N(m)>0,otherwise.​

The evaluation framework utilizes these metrics to generate a comprehensive leaderboard and analyze the trade-off between performance and permutation stability across different model sizes and access types.

Experiment

The benchmark evaluates decision-making across 11 domains using 2,300 static and 619 environment-based tasks, with separate metrics for capability and reliability. Experiments on 21 models compare System One models against System Two references on performance, latency, option-order stability, and missing-option detection. System Two models lead overall but are much slower, while System One models are competitive or better on direct semantic judgment, larger backbones substantially close the gap on grounded decisions, and explicit thinking helps mainly on math and scientific reasoning. Reliability results show that option-order stability tracks performance, but sensitivity to a missing correct option varies widely and requires direct measurement.

The study evaluates 21 decision-making models spanning open-weight System One models, closed decision APIs, and closed System Two references. Performance is paired with latency, and reliability is measured through option reordering and missing correct options. The listed open-weight entries include mostly non-reasoning models at sizes up to 31B parameters, with one compact model able to switch reasoning automatically. Open-weight System One models form the largest group in the evaluation, alongside closed decision APIs and closed System Two references. Reliability is assessed through option reordering and omitted correct answers, and order stability tracks performance more closely than null discrimination does.

System Two models retain the highest overall scores, but their advantage is not uniform across domains. System One models are much faster and can match or approach System Two performance on direct semantic judgment and fine-grained grounded decision tasks, while falling far behind on multi-step derivation. Reliability is only partly explained by performance: order stability tracks performance closely, but null discrimination varies widely and must be measured directly. The two System Two reference models lead overall score, and the strongest open-weight System One models trail Luna by about ten points. Open-weight System One models answer within 0.3 seconds per task on one H100 GPU, tens of times faster than the System Two references. Higher performance does not guarantee higher reliability; the most reliable System One models rank eighth and sixth among eighteen in performance. Option order stability rises with performance, while null discrimination varies widely and is only loosely related to performance.

The study evaluates 21 decision-making models across open-weight System One models, closed decision APIs, and closed System Two references, measuring performance, latency, and reliability through option reordering and missing correct options. System Two models achieve the highest overall scores but are slower and not uniformly better, while System One models are much faster and can match or approach them on direct semantic and grounded decision tasks, though they lag substantially on multi-step derivation. Reliability is only partly tied to performance, since order stability improves with score but null discrimination varies widely, and the most reliable System One models do not rank at the top in performance.


Créer de l'IA avec l'IA

De l'idée au lancement — accélérez votre développement IA avec le co-codage IA gratuit, un environnement prêt à l'emploi et le meilleur prix pour les GPU.

Codage assisté par IA
GPU prêts à l’emploi
Tarifs les plus avantageux

HyperAI Newsletters

Abonnez-vous à nos dernières mises à jour
Nous vous enverrons les dernières mises à jour de la semaine dans votre boîte de réception à neuf heures chaque lundi matin
Propulsé par MailChimp