API misuse bugs: which models catch them
Of the 17 models ranked on API-misuse bugs, GPT-6.1 Sol and Gemini 4 Argon tied for the most: 93% of known defects. 8 of the other 15 models have 95% intervals that overlap theirs. Gemma 4 26B A4B and Gemini 3.5 Flash-Lite tied for the fewest, 7%.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| GPT-6.1 Sol | 93%[80%–100%] | 89% | +4 pts |
| Gemini 4 Argon | 93%[80%–100%] | 82% | +11 pts |
| Claude Opus 5.5 | 87%[60%–100%] | 90% | −3 pts |
| Gemini 3.1 Pro | 87%[60%–100%] | 67% | +20 pts |
| Claude Sonnet 5.5 | 60%[0%–100%] | 77% | −17 pts |
| DeepSeek V4 Pro (Workers AI) | 60%[20%–100%] | 54% | +6 pts |
| Gemini 3.8 Flash | 57%[0%–100%] | 68% | −11 pts |
| GPT-5.6 Sol | 53%[0%–100%] | 72% | −19 pts |
| GPT-6 Luna | 53%[0%–100%] | 67% | −14 pts |
| Claude Haiku 5.5 | 53%[0%–100%] | 64% | −11 pts |
| GLM-5.3 (Workers AI) | 45%[25%–60%] | 61% | −16 pts |
| GLM-5.3 Flash (Workers AI) | 43%[25%–55%] | 61% | −18 pts |
| DeepSeek V4 Flash (Workers AI) | 37%[25%–60%] | 49% | −12 pts |
| Kimi K2.7 Code (Workers AI) | 34%[13%–60%] | 38% | −4 pts |
| GPT-OSS 120B (Workers AI) | 22%[0%–50%] | 40% | −18 pts |
| Gemma 4 26B A4B (Workers AI) | 7%[0%–20%] | 26% | −19 pts |
| Gemini 3.5 Flash-Lite | 7%[0%–20%] | 11% | −4 pts |
On average these models caught 52% of API-misuse bugs, less often than the 60% the same 17 caught across the whole benchmark: the second-hardest of the 6 kinds of bug.
Gemini 3.1 Pro gains the most here: 87%, against 67% over every bug.
Gemma 4 26B A4B drops the most: 7%, against 26% over every bug.
An API-misuse bug calls a library or runtime the wrong way: an option that does not do what the caller assumes, a documented contract the caller breaks, an argument of the wrong type or unit.
The change “Add the option to define custom cipher suites for backend serversTransport” in traefik/traefik, a change of over 300 lines that landed in January 2026, introduced an API-misuse bug in Go code that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, Gemini 3.8 Flash, Claude Haiku 5.5, Gemini 3.1 Pro and DeepSeek V4 Pro (Workers AI) caught it; GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Flash (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Claude Sonnet 5.5, GPT-5.6 Sol, GPT-6 Luna, GPT-OSS 120B (Workers AI), Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.