Python code review: which models catch the bugs
Of the 17 models ranked on Python code, 4 models tied for the most known bugs: 100%. 9 of the other 13 models have 95% intervals that overlap theirs. Gemini 3.5 Flash-Lite caught the fewest, 17% [0%–50%]. Together they caught 71% of known bugs in Python code, against 60% for the same 17 across the whole benchmark.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Claude Opus 5.5 | 100% | 90% | +10 pts |
| GPT-6.1 Sol | 100% | 89% | +11 pts |
| GPT-5.6 Sol | 100% | 72% | +28 pts |
| GPT-6 Luna | 100% | 67% | +33 pts |
| Gemini 4 Argon | 83%[50%–100%] | 82% | +1 pt |
| Gemini 3.8 Flash | 83%[50%–100%] | 68% | +15 pts |
| GLM-5.3 Flash (Workers AI) | 78%[50%–100%] | 61% | +17 pts |
| GPT-OSS 120B (Workers AI) | 78%[33%–100%] | 40% | +38 pts |
| DeepSeek V4 Pro (Workers AI) | 72%[50%–100%] | 54% | +18 pts |
| Claude Sonnet 5.5 | 67%[0%–100%] | 77% | −10 pts |
| Gemini 3.1 Pro | 67%[50%–100%] | 67% | 0 pts |
| Kimi K2.7 Code (Workers AI) | 62%[20%–100%] | 38% | +24 pts |
| DeepSeek V4 Flash (Workers AI) | 61%[50%–67%] | 49% | +12 pts |
| GLM-5.3 (Workers AI) | 58%[25%–100%] | 61% | −3 pts |
| Gemma 4 26B A4B (Workers AI) | 44%[0%–67%] | 26% | +18 pts |
| Claude Haiku 5.5 | 39%[0%–67%] | 64% | −25 pts |
| Gemini 3.5 Flash-Lite | 17%[0%–50%] | 11% | +6 pts |
Python was the second-easiest of the 12 languages for these models.
GPT-OSS 120B (Workers AI) gains the most here: 78%, against 40% over every bug.
Claude Haiku 5.5 drops the most: 39%, against 64% over every bug.
The change “keep websocket compression state across interleaved control frames” in aio-libs/aiohttp, a change of under 100 lines that landed in July 2026, introduced a logic bug that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, GPT-5.6 Sol, Gemini 3.8 Flash, GPT-6 Luna and DeepSeek V4 Pro (Workers AI) caught it; Gemini 3.1 Pro, GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Flash (Workers AI), GPT-OSS 120B (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Claude Sonnet 5.5, Claude Haiku 5.5, Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.