Security bugs: which models catch them
Of the 17 models ranked on security bugs, 5 models tied for the most: 89% of known defects. 11 of the other 12 models have 95% intervals that overlap theirs. Gemini 3.5 Flash-Lite caught the fewest, 11% [0%–33%]. Together they caught 65% of security bugs, against 60% for the same 17 across the whole benchmark.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Claude Opus 5.5 | 89%[67%–100%] | 90% | −1 pt |
| GPT-6.1 Sol | 89%[67%–100%] | 89% | 0 pts |
| Gemini 4 Argon | 89%[67%–100%] | 82% | +7 pts |
| Claude Sonnet 5.5 | 89%[67%–100%] | 77% | +12 pts |
| GPT-5.6 Sol | 89%[67%–100%] | 72% | +17 pts |
| GPT-6 Luna | 78%[44%–100%] | 67% | +11 pts |
| Claude Haiku 5.5 | 78%[44%–100%] | 64% | +14 pts |
| GLM-5.3 Flash (Workers AI) | 70%[41%–93%] | 61% | +9 pts |
| Gemini 3.8 Flash | 67%[33%–100%] | 68% | −1 pt |
| GLM-5.3 (Workers AI) | 67%[41%–89%] | 61% | +6 pts |
| DeepSeek V4 Pro (Workers AI) | 64%[41%–87%] | 54% | +10 pts |
| Gemini 3.1 Pro | 56%[22%–89%] | 67% | −11 pts |
| DeepSeek V4 Flash (Workers AI) | 52%[26%–78%] | 49% | +3 pts |
| Kimi K2.7 Code (Workers AI) | 50%[25%–75%] | 38% | +12 pts |
| GPT-OSS 120B (Workers AI) | 39%[13%–67%] | 40% | −1 pt |
| Gemma 4 26B A4B (Workers AI) | 33%[0%–67%] | 26% | +7 pts |
| Gemini 3.5 Flash-Lite | 11%[0%–33%] | 11% | 0 pts |
Security bugs were the easiest of the 6 kinds of bug for these models.
GPT-5.6 Sol gains the most here: 89%, against 72% over every bug.
Gemini 3.1 Pro drops the most: 56%, against 67% over every bug.
A security bug is one an attacker can exploit: untrusted input that reaches a query, a path or a header without validation, an authentication or authorization check that can be skipped, a limit that can be bypassed.
The change “[Security] Allow multiple OIDC discovery endpoints” in symfony/symfony, a change of over 300 lines that landed in October 2025, introduced a security bug in PHP code that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, Claude Sonnet 5.5, GPT-5.6 Sol and Claude Haiku 5.5 caught it; DeepSeek V4 Pro (Workers AI), DeepSeek V4 Flash (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Gemini 3.8 Flash, GPT-6 Luna, Gemini 3.1 Pro, GLM-5.3 Flash, GLM-5.3, GPT-OSS 120B (Workers AI), Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.
Every model that reviewed “Properly handle redirects in prerendered pages” (remix-run/react-router) caught its bug.