Kotlin code review: which models catch the bugs
Of the 17 models ranked on Kotlin code, GPT-6.1 Sol and Gemini 4 Argon tied for the most known bugs: 100%. 12 of the other 15 models have 95% intervals that overlap theirs. Gemini 3.8 Flash caught the fewest, 0%.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| GPT-6.1 Sol | 100% | 89% | +11 pts |
| Gemini 4 Argon | 100% | 82% | +18 pts |
| GLM-5.3 Flash (Workers AI) | 89%[67%–100%] | 61% | +28 pts |
| GLM-5.3 (Workers AI) | 72%[50%–100%] | 61% | +11 pts |
| Claude Opus 5.5 | 67%[0%–100%] | 90% | −23 pts |
| Claude Sonnet 5.5 | 67%[0%–100%] | 77% | −10 pts |
| GPT-5.6 Sol | 67%[0%–100%] | 72% | −5 pts |
| GPT-6 Luna | 67%[0%–100%] | 67% | 0 pts |
| Claude Haiku 5.5 | 67%[0%–100%] | 64% | +3 pts |
| Gemini 3.1 Pro | 67%[0%–100%] | 67% | 0 pts |
| DeepSeek V4 Flash (Workers AI) | 61%[33%–100%] | 49% | +12 pts |
| Gemma 4 26B A4B (Workers AI) | 56%[33%–100%] | 26% | +30 pts |
| GPT-OSS 120B (Workers AI) | 44%[0%–100%] | 40% | +4 pts |
| Gemini 3.5 Flash-Lite | 33%[0%–100%] | 11% | +22 pts |
| DeepSeek V4 Pro (Workers AI) | 30%[14%–50%] | 54% | −24 pts |
| Kimi K2.7 Code (Workers AI) | 14%[13%–14%] | 38% | −24 pts |
| Gemini 3.8 Flash | 0% | 68% | −68 pts |
On average these models caught 59% of known bugs in Kotlin code, less often than the 60% the same 17 caught across the whole benchmark: the sixth-hardest of the 12 languages.
Gemma 4 26B A4B gains the most here: 56%, against 26% over every bug.
Gemini 3.8 Flash drops the most: 0%, against 68% over every bug.
The change “KTOR-9171 Redesign ByteReadChannel.readUTF8Line API” in ktorio/ktor, a change of over 300 lines that landed in January 2026, introduced a logic bug that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, GPT-5.6 Sol, GPT-6 Luna, Claude Haiku 5.5, Gemini 3.1 Pro and GLM-5.3 Flash caught it; GLM-5.3, DeepSeek V4 Pro (Workers AI), DeepSeek V4 Flash (Workers AI), GPT-OSS 120B (Workers AI), Kimi K2.7 Code (Workers AI) and Gemma 4 26B A4B caught it in part; Claude Sonnet 5.5, Gemini 3.8 Flash and Gemini 3.5 Flash-Lite missed it.