CritPt v2

CritPt v2 is the revised version of CritPt, a benchmark designed to test LLMs on research-level physics reasoning tasks, featuring 70 composite research challenges. Models solve problems with a code-execution tool in an offline Linux sandbox.

Intelligence Index weight: 10%
Models: 110 current, 128 in total
Median: 33.5%
Range: 0.0%–76.5%

Score distribution

Lowest: 0.0%
Median: 33.5%
Highest: 76.5%

Leaderboard

Include deprecated models

Rank	Model	Score	Cost / task	Time / task	Output tokens / task
1	Claude Fable 5.2 (Max, Default Fallback) Anthropic	76.5%	$6.35	20m 0s	123K
2	Claude Fable 5.2 (Xhigh, Default Fallback) Anthropic	73.5%	$2.47	7m 48s	48.3K
3	Claude Fable 5.2 (High, Default Fallback) Anthropic	73.0%	$1.05	3m 42s	20.7K
4	GPT-6 Astra (Medium) OpenAI	72.0%	$0.41	2m 27s	6.27K
5	GPT-6 Astra (High) OpenAI	71.5%	$0.71	4m 46s	11.2K
6	GPT-6 Astra (Xhigh) OpenAI	71.0%	$0.85	5m 48s	13.5K
7	GPT-6 Astra (Max) OpenAI	70.5%	$1.14	8m 5s	18.3K
8	GPT-6.1 Sol (Max) OpenAI	69.5%	$0.23	5m 35s	18.8K
9	GPT-6.1 Sol (Xhigh) OpenAI	69.5%	$0.17	4m 2s	13.5K
10	Gemini 4 Carbon (High) Google	69.0%	$3.91	26m 33s	131K
11	Gemini 4 Carbon (Medium) Google	69.0%	$3.09	23m 21s	103K
12	GPT-6.1 Sol (High) OpenAI	68.5%	$0.13	3m 8s	10.7K
13	Claude Opus 5.5 (Max, Default Fallback) Anthropic	68.0%	$2.58	13m 51s	97.2K
14	GPT-6.1 Terra (Max) OpenAI	68.0%	$0.25	8m 43s	38.4K
15	Claude Opus 5.5 (Xhigh, Default Fallback) Anthropic	67.5%	$0.97	5m 38s	36.6K
16	GPT-6.1 Sol (Medium) OpenAI	67.0%	$0.071	1m 34s	5.55K
17	Gemini 4 Carbon (Low) Google	65.0%	$3.06	26m 4s	101K
18	GPT-6 Astra (Low) OpenAI	63.0%	$0.24	1m 29s	3.52K
19	GPT-6.1 Sol (Low) OpenAI	62.5%	$0.043	57.8s	3.26K
20	Claude Fable 5.2 (Medium, Default Fallback) Anthropic	61.5%	$0.37	1m 17s	7.01K
21	GPT-6.1 Terra (High) OpenAI	61.5%	$0.069	2m 31s	10.6K
22	GPT-6.1 Terra (Xhigh) OpenAI	60.5%	$0.12	4m 10s	18K
23	Claude Fable 5.2 (Low, Default Fallback) Anthropic	60.0%	$0.31	1m 4s	5.96K
24	Claude Opus 5.5 (High, Default Fallback) Anthropic	59.5%	$0.38	2m 17s	14.4K
25	Claude Sonnet 5.5 (Max, Default Fallback) Anthropic	59.0%	$1.93	15m 18s	141K
26	Claude Sonnet 5.5 (Xhigh, Default Fallback) Anthropic	57.5%	$0.45	4m 9s	34.7K
27	GPT-6.1 Terra (Medium) OpenAI	56.5%	$0.04	1m 28s	6.08K
28	Muse Wonder Full (Max) Meta	56.5%	—	—	65.4K
29	Claude Opus 5.5 (Medium, Default Fallback) Anthropic	54.0%	$0.25	1m 31s	9.39K
30	Gemini 4 Argon (High) Google	52.5%	$2.35	17m 35s	71.5K
31	Claude Sonnet 5.5 (High, Default Fallback) Anthropic	51.5%	$0.18	1m 37s	13.7K
32	Muse Wonder (High) Meta	51.0%	$1.17	5m 57s	43.3K
33	GPT-6.1 Terra (Low) OpenAI	50.5%	$0.018	36.4s	2.47K
34	Muse Wonder (Max) Meta	50.5%	$1.81	9m 5s	66.4K
35	Kimi K3.1 Pro Kimi	49.5%	$0.70	24m 34s	47.8K
36	Claude Haiku 5.5 (Xhigh) Anthropic	47.5%	$0.059	6m 4s	86.1K
37	Gemini 3.8 Flash (High) Google	46.5%	$1.15	9m 39s	66.5K
38	Claude Haiku 5.5 (High) Anthropic	45.0%	$0.033	3m 31s	48.1K
39	Gemini 3.8 Flash (Medium) Google	45.0%	$0.83	7m 27s	50.1K
40	Claude Haiku 5.5 (Max) Anthropic	43.5%	$0.10	9m 42s	148K
41	Grok 4.7 (Low) SpaceXAI	43.5%	$0.79	11m 40s	45.2K
42	GPT-6 Luna (Max) OpenAI	42.5%	$0.032	7m 26s	48.8K
43	Muse Ember (Max) Meta	39.0%	$0.054	2m 57s	80.7K
44	Tencent Hy4 Tencent	39.0%	$0.31	—	78.6K
45	Ling 3.1 Flash 1019 InclusionAI	38.5%	$0.32	—	157K
46	Muse Ember (Xhigh) Meta	38.5%	$0.041	2m 14s	61.3K
47	GPT-6 Luna (Xhigh) OpenAI	37.0%	$0.013	3m 11s	19.6K
48	Grok 4.7 (High) SpaceXAI	37.0%	$1.89	22m 38s	91.4K
49	Claude Opus 5.5 (Low, Default Fallback) Anthropic	36.0%	$0.12	40.5s	4.39K
50	Inkling 2 Small Thinking Machines	35.0%	$0.25	17m 44s	136K

Rank 1
Claude Fable 5.2 (Max, Default Fallback)
Anthropic
Score
76.5%
Cost / task
$6.35
Time / task
20m 0s
Output tokens / task
123K
Rank 2
Claude Fable 5.2 (Xhigh, Default Fallback)
Anthropic
Score
73.5%
Cost / task
$2.47
Time / task
7m 48s
Output tokens / task
48.3K
Rank 3
Claude Fable 5.2 (High, Default Fallback)
Anthropic
Score
73.0%
Cost / task
$1.05
Time / task
3m 42s
Output tokens / task
20.7K
Rank 4
GPT-6 Astra (Medium)
OpenAI
Score
72.0%
Cost / task
$0.41
Time / task
2m 27s
Output tokens / task
6.27K
Rank 5
GPT-6 Astra (High)
OpenAI
Score
71.5%
Cost / task
$0.71
Time / task
4m 46s
Output tokens / task
11.2K
Rank 6
GPT-6 Astra (Xhigh)
OpenAI
Score
71.0%
Cost / task
$0.85
Time / task
5m 48s
Output tokens / task
13.5K
Rank 7
GPT-6 Astra (Max)
OpenAI
Score
70.5%
Cost / task
$1.14
Time / task
8m 5s
Output tokens / task
18.3K
Rank 8
GPT-6.1 Sol (Max)
OpenAI
Score
69.5%
Cost / task
$0.23
Time / task
5m 35s
Output tokens / task
18.8K
Rank 9
GPT-6.1 Sol (Xhigh)
OpenAI
Score
69.5%
Cost / task
$0.17
Time / task
4m 2s
Output tokens / task
13.5K
Rank 10
Gemini 4 Carbon (High)
Google
Score
69.0%
Cost / task
$3.91
Time / task
26m 33s
Output tokens / task
131K
Rank 11
Gemini 4 Carbon (Medium)
Google
Score
69.0%
Cost / task
$3.09
Time / task
23m 21s
Output tokens / task
103K
Rank 12
GPT-6.1 Sol (High)
OpenAI
Score
68.5%
Cost / task
$0.13
Time / task
3m 8s
Output tokens / task
10.7K
Rank 13
Claude Opus 5.5 (Max, Default Fallback)
Anthropic
Score
68.0%
Cost / task
$2.58
Time / task
13m 51s
Output tokens / task
97.2K
Rank 14
GPT-6.1 Terra (Max)
OpenAI
Score
68.0%
Cost / task
$0.25
Time / task
8m 43s
Output tokens / task
38.4K
Rank 15
Claude Opus 5.5 (Xhigh, Default Fallback)
Anthropic
Score
67.5%
Cost / task
$0.97
Time / task
5m 38s
Output tokens / task
36.6K
Rank 16
GPT-6.1 Sol (Medium)
OpenAI
Score
67.0%
Cost / task
$0.071
Time / task
1m 34s
Output tokens / task
5.55K
Rank 17
Gemini 4 Carbon (Low)
Google
Score
65.0%
Cost / task
$3.06
Time / task
26m 4s
Output tokens / task
101K
Rank 18
GPT-6 Astra (Low)
OpenAI
Score
63.0%
Cost / task
$0.24
Time / task
1m 29s
Output tokens / task
3.52K
Rank 19
GPT-6.1 Sol (Low)
OpenAI
Score
62.5%
Cost / task
$0.043
Time / task
57.8s
Output tokens / task
3.26K
Rank 20
Claude Fable 5.2 (Medium, Default Fallback)
Anthropic
Score
61.5%
Cost / task
$0.37
Time / task
1m 17s
Output tokens / task
7.01K
Rank 21
GPT-6.1 Terra (High)
OpenAI
Score
61.5%
Cost / task
$0.069
Time / task
2m 31s
Output tokens / task
10.6K
Rank 22
GPT-6.1 Terra (Xhigh)
OpenAI
Score
60.5%
Cost / task
$0.12
Time / task
4m 10s
Output tokens / task
18K
Rank 23
Claude Fable 5.2 (Low, Default Fallback)
Anthropic
Score
60.0%
Cost / task
$0.31
Time / task
1m 4s
Output tokens / task
5.96K
Rank 24
Claude Opus 5.5 (High, Default Fallback)
Anthropic
Score
59.5%
Cost / task
$0.38
Time / task
2m 17s
Output tokens / task
14.4K
Rank 25
Claude Sonnet 5.5 (Max, Default Fallback)
Anthropic
Score
59.0%
Cost / task
$1.93
Time / task
15m 18s
Output tokens / task
141K
Rank 26
Claude Sonnet 5.5 (Xhigh, Default Fallback)
Anthropic
Score
57.5%
Cost / task
$0.45
Time / task
4m 9s
Output tokens / task
34.7K
Rank 27
GPT-6.1 Terra (Medium)
OpenAI
Score
56.5%
Cost / task
$0.04
Time / task
1m 28s
Output tokens / task
6.08K
Rank 28
Muse Wonder Full (Max)
Meta
Score
56.5%
Cost / task
—
Time / task
—
Output tokens / task
65.4K
Rank 29
Claude Opus 5.5 (Medium, Default Fallback)
Anthropic
Score
54.0%
Cost / task
$0.25
Time / task
1m 31s
Output tokens / task
9.39K
Rank 30
Gemini 4 Argon (High)
Google
Score
52.5%
Cost / task
$2.35
Time / task
17m 35s
Output tokens / task
71.5K
Rank 31
Claude Sonnet 5.5 (High, Default Fallback)
Anthropic
Score
51.5%
Cost / task
$0.18
Time / task
1m 37s
Output tokens / task
13.7K
Rank 32
Muse Wonder (High)
Meta
Score
51.0%
Cost / task
$1.17
Time / task
5m 57s
Output tokens / task
43.3K
Rank 33
GPT-6.1 Terra (Low)
OpenAI
Score
50.5%
Cost / task
$0.018
Time / task
36.4s
Output tokens / task
2.47K
Rank 34
Muse Wonder (Max)
Meta
Score
50.5%
Cost / task
$1.81
Time / task
9m 5s
Output tokens / task
66.4K
Rank 35
Kimi K3.1 Pro
Kimi
Score
49.5%
Cost / task
$0.70
Time / task
24m 34s
Output tokens / task
47.8K
Rank 36
Claude Haiku 5.5 (Xhigh)
Anthropic
Score
47.5%
Cost / task
$0.059
Time / task
6m 4s
Output tokens / task
86.1K
Rank 37
Gemini 3.8 Flash (High)
Google
Score
46.5%
Cost / task
$1.15
Time / task
9m 39s
Output tokens / task
66.5K
Rank 38
Claude Haiku 5.5 (High)
Anthropic
Score
45.0%
Cost / task
$0.033
Time / task
3m 31s
Output tokens / task
48.1K
Rank 39
Gemini 3.8 Flash (Medium)
Google
Score
45.0%
Cost / task
$0.83
Time / task
7m 27s
Output tokens / task
50.1K
Rank 40
Claude Haiku 5.5 (Max)
Anthropic
Score
43.5%
Cost / task
$0.10
Time / task
9m 42s
Output tokens / task
148K
Rank 41
Grok 4.7 (Low)
SpaceXAI
Score
43.5%
Cost / task
$0.79
Time / task
11m 40s
Output tokens / task
45.2K
Rank 42
GPT-6 Luna (Max)
OpenAI
Score
42.5%
Cost / task
$0.032
Time / task
7m 26s
Output tokens / task
48.8K
Rank 43
Muse Ember (Max)
Meta
Score
39.0%
Cost / task
$0.054
Time / task
2m 57s
Output tokens / task
80.7K
Rank 44
Tencent Hy4
Tencent
Score
39.0%
Cost / task
$0.31
Time / task
—
Output tokens / task
78.6K
Rank 45
Ling 3.1 Flash 1019
InclusionAI
Score
38.5%
Cost / task
$0.32
Time / task
—
Output tokens / task
157K
Rank 46
Muse Ember (Xhigh)
Meta
Score
38.5%
Cost / task
$0.041
Time / task
2m 14s
Output tokens / task
61.3K
Rank 47
GPT-6 Luna (Xhigh)
OpenAI
Score
37.0%
Cost / task
$0.013
Time / task
3m 11s
Output tokens / task
19.6K
Rank 48
Grok 4.7 (High)
SpaceXAI
Score
37.0%
Cost / task
$1.89
Time / task
22m 38s
Output tokens / task
91.4K
Rank 49
Claude Opus 5.5 (Low, Default Fallback)
Anthropic
Score
36.0%
Cost / task
$0.12
Time / task
40.5s
Output tokens / task
4.39K
Rank 50
Inkling 2 Small
Thinking Machines
Score
35.0%
Cost / task
$0.25
Time / task
17m 44s
Output tokens / task
136K

Data from Artificial Analysis

Command Palette

CritPt v2

Score distribution

Leaderboard