MMLU-Pro

14の科目領域にわたる12,000の大学院レベルの問題を備えたMMLUの拡張版。10個の選択肢と、より深い推論が求められる点を特徴とする。

スコア分布

非推奨のモデルを含める

ランク	モデル	スコア
51	Phi-4 Microsoft	71.4%
52	K2-V2 (Low) Institute of Foundation Models	71.3%
53	Llama 3.3 Instruct 70B Meta	71.3%
54	Command A Cohere	71.2%
55	Ministral 3 14B Mistral	69.3%
56	Llama 3.2 Instruct 90B (Vision) Meta	67.1%
57	Reka Flash 3 Reka AI	66.9%
58	Hermes 4 - Llama-3.1 70B (Non-reasoning) Nous Research	66.4%
59	Olmo 3 7B Think Allen Institute for AI	65.5%
60	NVIDIA Nemotron Nano 12B v2 VL (Non-reasoning) NVIDIA	64.9%
61	Ministral 3 8B Mistral	64.2%
62	Granite 4.0 H Small IBM	62.4%
63	Exaone 4.0 1.2B (Reasoning) LG AI Research	58.8%
64	Kimi Linear 48B A3B Instruct Kimi	58.5%
65	DeepHermes 3 - Mistral 24B Preview (Non-reasoning) Nous Research	58.0%
66	NVIDIA Nemotron 3 Nano 30B A3B (Non-reasoning) NVIDIA	57.9%
67	Jamba Reasoning 3B AI21 Labs	57.7%
68	Jamba 1.7 Large AI21 Labs	57.7%
69	Nova Micro Amazon	53.1%
70	Ministral 3 3B Mistral	52.4%
71	Olmo 3 7B Instruct Allen Institute for AI	52.2%
72	Exaone 4.0 1.2B (Non-reasoning) LG AI Research	50.0%
73	Phi-4 Multimodal Instruct Microsoft	48.5%
74	Phi-4 Mini Instruct Microsoft	46.5%
75	Llama 3.2 Instruct 11B (Vision) Meta	46.4%
76	Granite 4.0 Micro IBM	44.7%
77	Jamba 1.7 Mini AI21 Labs	38.8%
78	Molmo 7B-D Allen Institute for AI	37.1%
79	DeepHermes 3 - Llama-3.1 8B Preview (Non-reasoning) Nous Research	36.5%
80	Granite 4.0 H 1B IBM	27.7%
81	Granite 4.0 H 350M IBM	12.7%
82	Granite 4.0 350M IBM	12.4%
83	Gemma 3 270M Google	5.5%