Error handling bugs: which models catch them
Of the 17 models ranked on error-handling bugs, Claude Opus 5.5, GPT-6.1 Sol and Claude Sonnet 5.5 tied for the most: 80% of known defects. Every other model's 95% interval overlaps theirs. Gemini 3.5 Flash-Lite caught the fewest, 20% [0%–60%].
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Claude Opus 5.5 | 80%[40%–100%] | 90% | −10 pts |
| GPT-6.1 Sol | 80%[40%–100%] | 89% | −9 pts |
| Claude Sonnet 5.5 | 80%[40%–100%] | 77% | +3 pts |
| Claude Haiku 5.5 | 73%[33%–100%] | 64% | +9 pts |
| Gemini 3.1 Pro | 70%[30%–100%] | 67% | +3 pts |
| GLM-5.3 Flash (Workers AI) | 70%[30%–100%] | 61% | +9 pts |
| GLM-5.3 (Workers AI) | 65%[25%–100%] | 61% | +4 pts |
| Gemini 4 Argon | 60%[20%–100%] | 82% | −22 pts |
| GPT-5.6 Sol | 60%[20%–100%] | 72% | −12 pts |
| GPT-6 Luna | 60%[20%–100%] | 67% | −7 pts |
| GPT-OSS 120B (Workers AI) | 60%[27%–93%] | 40% | +20 pts |
| DeepSeek V4 Flash (Workers AI) | 53%[13%–93%] | 49% | +4 pts |
| Gemma 4 26B A4B (Workers AI) | 53%[13%–93%] | 26% | +27 pts |
| DeepSeek V4 Pro (Workers AI) | 50%[20%–80%] | 54% | −4 pts |
| Kimi K2.7 Code (Workers AI) | 40%[18%–62%] | 38% | +2 pts |
| Gemini 3.8 Flash | 30%[0%–70%] | 68% | −38 pts |
| Gemini 3.5 Flash-Lite | 20%[0%–60%] | 11% | +9 pts |
On average these models caught 59% of error-handling bugs, less often than the 60% the same 17 caught across the whole benchmark: the third-hardest of the 6 kinds of bug.
Gemma 4 26B A4B gains the most here: 53%, against 26% over every bug.
Gemini 3.8 Flash drops the most: 30%, against 68% over every bug.
An error-handling bug is a failure path that does the wrong thing: an error swallowed or escalated where it should not be, a fallback that hides the fault, cleanup that is skipped when a call fails.
The change “reorganize nushell configuration” in nushell/nushell, a change of over 300 lines that landed in July 2026, introduced an error-handling bug in Rust code that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol and Claude Sonnet 5.5 caught it; Gemini 3.1 Pro, GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Pro (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Gemini 4 Argon, GPT-5.6 Sol, Gemini 3.8 Flash, GPT-6 Luna, Claude Haiku 5.5, DeepSeek V4 Flash (Workers AI), GPT-OSS 120B (Workers AI), Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.