Resource handling bugs: which models catch them
Of the 17 models ranked on resource-handling bugs, Claude Opus 5.5, GPT-6.1 Sol and Claude Sonnet 5.5 tied for the most: 100% of known defects. 8 of the other 14 models have 95% intervals that overlap theirs. Gemma 4 26B A4B and Gemini 3.5 Flash-Lite tied for the fewest, 0%.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Claude Opus 5.5 | 100% | 90% | +10 pts |
| GPT-6.1 Sol | 100% | 89% | +11 pts |
| Claude Sonnet 5.5 | 100% | 77% | +23 pts |
| GPT-5.6 Sol | 67%[0%–100%] | 72% | −5 pts |
| GPT-6 Luna | 67%[0%–100%] | 67% | 0 pts |
| Claude Haiku 5.5 | 67%[0%–100%] | 64% | +3 pts |
| GLM-5.3 (Workers AI) | 64%[25%–100%] | 61% | +3 pts |
| Gemini 3.8 Flash | 50%[0%–100%] | 68% | −18 pts |
| Gemini 3.1 Pro | 50%[0%–100%] | 67% | −17 pts |
| GLM-5.3 Flash (Workers AI) | 44%[0%–67%] | 61% | −17 pts |
| Kimi K2.7 Code (Workers AI) | 41%[33%–50%] | 38% | +3 pts |
| Gemini 4 Argon | 33%[0%–100%] | 82% | −49 pts |
| GPT-OSS 120B (Workers AI) | 33%[0%–100%] | 40% | −7 pts |
| DeepSeek V4 Pro (Workers AI) | 28%[0%–50%] | 54% | −26 pts |
| DeepSeek V4 Flash (Workers AI) | 22%[0%–67%] | 49% | −27 pts |
| Gemma 4 26B A4B (Workers AI) | 0% | 26% | −26 pts |
| Gemini 3.5 Flash-Lite | 0% | 11% | −11 pts |
On average these models caught 51% of resource-handling bugs, less often than the 60% the same 17 caught across the whole benchmark: the hardest of the 6 kinds of bug.
Claude Sonnet 5.5 gains the most here: 100%, against 77% over every bug.
Gemini 4 Argon drops the most: 33%, against 82% over every bug.
A resource-handling bug holds something too long or releases it too early: a file, socket, task or lock that is never released, memory used after it is freed, a cache or queue that grows without bound.
The change “Event Thread could stall when not notified of new queries on existing connections that are in a bad state” in c-ares/c-ares, a change of 100 to 300 lines that landed in September 2025, introduced a resource-handling bug in C code that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol and Claude Sonnet 5.5 caught it; Gemini 3.8 Flash, Gemini 3.1 Pro, GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Pro (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Gemini 4 Argon, GPT-5.6 Sol, GPT-6 Luna, Claude Haiku 5.5, DeepSeek V4 Flash (Workers AI), GPT-OSS 120B (Workers AI), Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.