CWE-Bench-AA

An implementation of Collinear AI's CWE-bench, a defensive cybersecurity benchmark testing whether coding agents can audit a real open-source codebase, find a vulnerability in a described area of concern, and patch it without breaking legitimate behavior.

Models: 19 current, 25 in total
Median: 55.8%
Range: 26.7%–69.2%

Score distribution

Lowest: 26.7%
Median: 55.8%
Highest: 69.2%

Leaderboard

Include deprecated models

Rank	Model	Score
1	DeepSeek V4.1 Flash 1017 (Max) DeepSeek	69.2%
2	Grok 4.7 (Xhigh) SpaceXAI	68.3%
3	GPT-6 Sol (Daybreak Blue, max) OpenAI	64.7%
4	GPT-6 Astra (Max) OpenAI	63.3%
5	MiMo-V2.6-Pro Xiaomi	63.3%
6	Kimi K3.1 Pro Kimi	60.8%
7	Claude Opus 5.5 (Max, Default Fallback) Anthropic	58.3%
8	Mistral Large 4 (High) Mistral	58.3%
9	GPT-6 Luna (Max) OpenAI	56.7%
10	GLM-5.3 (Max) Z AI	55.8%
11	GLM-5.3-Flash Z AI	55.8%
12	Gemini 3.8 Flash (High) Google	52.5%
13	Step 5 Preview StepFun	52.5%
14	Mistral Large 4 (Medium) Mistral	50.0%
15	Beam 1.0 Reflection	49.2%
16	MiniMax-M3 MiniMax	41.7%
17	Qwen3.8 2.4T A95B Alibaba	28.3%
18	Nemotron 3 Ultra 550B A55B (Reasoning) NVIDIA	27.5%
19	Qwen3.8 27B (Xhigh) Alibaba	26.7%

1
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek
69.2%
2
Grok 4.7 (Xhigh)

Data from Artificial Analysis

Command Palette

CWE-Bench-AA

Score distribution

Leaderboard