Logic bugs: which models catch them
Of the 17 models ranked on logic bugs, GPT-6.1 Sol caught the most: 100% of known defects. Claude Opus 5.5, Gemini 4 Argon and Gemini 3.8 Flash have 95% intervals that overlap its own. Gemini 3.5 Flash-Lite caught the fewest, 11% [0%–29%].
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| GPT-6.1 Sol | 100% | 89% | +11 pts |
| Claude Opus 5.5 | 90%[74%–100%] | 90% | 0 pts |
| Gemini 4 Argon | 89%[71%–100%] | 82% | +7 pts |
| Gemini 3.8 Flash | 82%[61%–100%] | 68% | +14 pts |
| GPT-5.6 Sol | 71%[50%–93%] | 72% | −1 pt |
| Claude Sonnet 5.5 | 64%[36%–86%] | 77% | −13 pts |
| GPT-6 Luna | 64%[36%–86%] | 67% | −3 pts |
| GLM-5.3 Flash (Workers AI) | 63%[43%–81%] | 61% | +2 pts |
| Gemini 3.1 Pro | 61%[39%–82%] | 67% | −6 pts |
| DeepSeek V4 Pro (Workers AI) | 55%[35%–74%] | 54% | +1 pt |
| GLM-5.3 (Workers AI) | 52%[32%–71%] | 61% | −9 pts |
| DeepSeek V4 Flash (Workers AI) | 50%[30%–70%] | 49% | +1 pt |
| Claude Haiku 5.5 | 46%[21%–71%] | 64% | −18 pts |
| Kimi K2.7 Code (Workers AI) | 44%[27%–62%] | 38% | +6 pts |
| GPT-OSS 120B (Workers AI) | 38%[19%–58%] | 40% | −2 pts |
| Gemma 4 26B A4B (Workers AI) | 24%[10%–42%] | 26% | −2 pts |
| Gemini 3.5 Flash-Lite | 11%[0%–29%] | 11% | 0 pts |
On average these models caught 59% of logic bugs, less often than the 60% the same 17 caught across the whole benchmark: the third-easiest of the 6 kinds of bug.
Gemini 3.8 Flash gains the most here: 82%, against 68% over every bug.
Claude Haiku 5.5 drops the most: 46%, against 64% over every bug.
A logic bug gives the wrong result on an ordinary input: an inverted condition, an off-by-one, a missed case, a calculation on the wrong value.
The change “Fix DataClassEq comparator” in kotest/kotest, a change of 100 to 300 lines that landed in April 2026, introduced a logic bug in Kotlin code that a later upstream commit fixed.
GPT-6.1 Sol, Gemini 4 Argon and Claude Sonnet 5.5 caught it; GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Pro (Workers AI), DeepSeek V4 Flash (Workers AI), Kimi K2.7 Code (Workers AI) and Gemma 4 26B A4B caught it in part; Claude Opus 5.5, GPT-5.6 Sol, Gemini 3.8 Flash, GPT-6 Luna, Claude Haiku 5.5, Gemini 3.1 Pro, GPT-OSS 120B (Workers AI) and Gemini 3.5 Flash-Lite missed it.