One-sentence Summary

Researchers at EPFL introduce blind-spots-bench, a diagnostic benchmark of 235 human-trivial tasks with a structured taxonomy and automated grading, revealing that closed-source multimodal models outperform open-weight ones by a $\approx 10\%$≈10% margin on these blind spots yet no single model dominates all task types, exposing persistent weaknesses that existing benchmarks fail to detect.

Key Contributions

• blind-spots-bench is introduced, a benchmark of 235 tasks that are simple for humans but challenging for modern AI, built from student-generated questions and annotated with structured reference solutions and a fine-grained task taxonomy.
• An automated grading pipeline is developed to evaluate a diverse set of models, including open-weight and closed-source language, vision-language, and image-generation models, enabling consistent and scalable assessment.
• Empirical analysis shows that closed-source frontier models outperform open-weight models by roughly 10%, that no single model dominates all task types, and that taxonomy-level failure analysis reveals distinct patterns of model weaknesses, with scaling model size or enabling tool use not consistently improving performance.

Introduction

Large language and vision-language models now achieve near-human or superhuman scores on many benchmarks, yet they still exhibit surprising failures on tasks that are trivial for people, such as counting objects in images or generating a string of a specific length. Existing evaluation suites typically aggregate performance into broad scores, obscuring these fine-grained reasoning gaps, and prior taxonomies lack the granularity needed to analyze specific failure modes. To systematically probe these blind spots, the authors introduce blind-spots-bench, a manually curated benchmark of 235 problems sourced from graduate students, accompanied by a structured task taxonomy and an automated grading pipeline. They evaluate 38 frontier models, uncovering persistent weaknesses in object-centric perception and highlighting divergent strengths across model families.

Dataset

The authors construct the blind-spots-bench dataset to probe reasoning failures in frontier AI models. Here is a concise overview of its sources, composition, processing, and usage.

• Source and collection

• Raw questions were collected from students in a graduate-level AI course. Each student proposed five questions that appeared easy to humans but were failed by frontier models around October 2025.

• The initial pool contained approximately 287 questions.

• Cleaning, filtering, and quality control

• A systematic pipeline removed overly difficult or duplicate entries, normalized formats, and ensured clarity.

• A three-stage review was applied: manual verification of task clarity, difficulty thresholding (removing questions too easy for models or too hard for humans), and refinement of linguistic ambiguities.

• The final dataset, blind-spots-bench, contains 235 samples.

• Annotations and metadata

• Solution annotation: Each question includes a structured reference solution specifying the expected answer, correctness conditions, and common failure modes. Failure modes were drawn from student interactions with AI agents and from annotator contributions.

• Question format: Every example is labeled as text-only, image generation, or multi-to-text (image+text input, text output). This enables assigning examples to models that support the relevant modality.

• Task taxonomy: Questions are categorized into three main types: Object-centric (e.g., attribute/pattern recognition, spatial reasoning, perceptual counting, generative counting), Abstract reasoning (e.g., geometric/graph reasoning, logical reasoning), and Language and knowledge (e.g., character-level manipulation). Sub-tasks further detail the required ability.

• Dataset composition

• By format: text-only (46.2%), image generation (35.6%), multi-to-text (18.2%).

• By sub-task frequency: spatial reasoning appears most often (53 times), followed by perceptual counting, generative counting, logical reasoning, and character-level manipulation. These frequencies reflect known weaknesses of current models and were not artificially balanced.

• How the paper uses the data

• The dataset is a zero-shot evaluation benchmark. It is not split into training/validation/test; all 235 samples serve as a test set to measure model blind spots.

• The format labels allow comparing models only on the subsets they support, while the taxonomy supports fine-grained error analysis.

• No cropping or training-specific preprocessing is applied; the benchmark is used as-is for inference and evaluation against the provided reference solutions.

Experiment

The benchmark is constructed from student-generated questions that probe blind spots of frontier models, covering object-centric, abstract reasoning, and language/knowledge tasks with annotated solutions. An automated evaluation pipeline uses a grader model to score responses across text-only, multi-to-text, and image generation formats, validated against human judgments. Results reveal that closed-source models achieve higher accuracy, though open-weight models offer better cost-effectiveness; scaling model size does not uniformly improve performance, and tool use yields mixed benefits. Fine-grained visual perception remains a major bottleneck, and different model families exhibit complementary strengths rather than a single dominant pattern across all subtask categories.

Open-weight models show steady improvement across generations, with GLM-5.2 reaching 73.8% text-only accuracy, but closed-source models Gemini-3.1-Pro and GPT-5.5 remain ahead overall, especially on multimodal problems. Cost efficiency varies widely: DeepSeek-V4-Flash achieves competitive accuracy at only 0.29 USD per 100 samples, while GLM-5.2 offers higher performance at a higher cost. Longer output tokens, as seen with Qwen3.5-397B, correlate with stronger text-only results but do not fully close the multimodal gap. DeepSeek-V4-Flash costs 0.29 USD per 100 samples, roughly a quarter of GLM-5.2's cost, while still attaining 67.6% text-only accuracy. Successive GLM versions show consistent accuracy gains, with GLM-5.2 improving to 73.8% from 61.3% in GLM-4.7, along with a jump in pass@4 from 75.9% to 84.3%.

Image-generation models exhibit a trade-off between accuracy and cost. The top Gemini models achieve the highest accuracy, with Gemini-3.1-flash-image offering a more cost-effective alternative to the pro version, while GPT models provide lower-cost options with reduced accuracy. Task analysis reveals complementary strengths: Gemini excels at abstract reasoning, GPT at language and knowledge. Gemini-3-pro-image leads in accuracy but incurs the highest generation cost per sample. Gemini-3.1-flash-image closely matches the pro model's accuracy while costing less than half. GPT-image-1.5 is less accurate than both Gemini models but costs substantially less. The cheapest model, GPT-image-1-mini, records the lowest accuracy. Gemini and GPT models show complementary strengths, with Gemini better on abstract reasoning and GPT better on language and knowledge tasks.

Enabling code execution for text-only problems changes accuracy and token usage differently across models. While some models like Gemini-3.1-pro and GLM-5.2 gain accuracy with fewer tokens, others like GPT-5.4 lose accuracy even though token count drops. The mixed results show that tool-use does not uniformly improve performance or efficiency. Gemini-3.1-pro improved accuracy by 2.78 percentage points and used 199 fewer tokens. GPT-5.4 lost 5.32 percentage points of accuracy while using 451 fewer tokens. Kimi-K2.6 required 3,314 more tokens but only gained 6.25 percentage points in accuracy. GLM-5.2 reduced token usage by 1,712 tokens with a slight accuracy increase of 1.16 percentage points.

The evaluation across text-only, multimodal, and image-generation tasks reveals that closed-source models continue to lead overall, especially on multimodal problems, while open-weight models are steadily improving with notable cost efficiency differences. Image generation exhibits a clear accuracy-cost trade-off, with Gemini models excelling at abstract reasoning and GPT models providing more affordable options stronger in language and knowledge. Enabling code execution for text-only problems yields mixed results, as accuracy and token usage changes vary widely across models, indicating that tool-use is not uniformly beneficial.