Java code review: which models catch the bugs
Of the 17 models ranked on Java code, Gemini 4 Argon caught the most known bugs: 83% [63%–100%]. 12 of the other 16 models have 95% intervals that overlap its own. Gemini 3.5 Flash-Lite caught the fewest, 5% [0%–15%]. Together they caught 52% of known bugs in Java code, against 60% for the same 17 across the whole benchmark.
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Gemini 4 Argon | 83%[63%–100%] | 82% | +1 pt |
| Claude Opus 5.5 | 78%[55%–100%] | 90% | −12 pts |
| Gemini 3.1 Pro | 78%[55%–100%] | 67% | +11 pts |
| GPT-6.1 Sol | 70%[25%–100%] | 89% | −19 pts |
| GPT-5.6 Sol | 69%[38%–100%] | 72% | −3 pts |
| GPT-6 Luna | 65%[25%–100%] | 67% | −2 pts |
| DeepSeek V4 Flash (Workers AI) | 55%[29%–83%] | 49% | +6 pts |
| DeepSeek V4 Pro (Workers AI) | 49%[30%–63%] | 54% | −5 pts |
| GLM-5.3 (Workers AI) | 46%[29%–63%] | 61% | −15 pts |
| GPT-OSS 120B (Workers AI) | 46%[25%–62%] | 40% | +6 pts |
| Claude Sonnet 5.5 | 45%[0%–90%] | 77% | −32 pts |
| GLM-5.3 Flash (Workers AI) | 45%[29%–61%] | 61% | −16 pts |
| Gemini 3.8 Flash | 43%[0%–85%] | 68% | −25 pts |
| Claude Haiku 5.5 | 40%[0%–80%] | 64% | −24 pts |
| Kimi K2.7 Code (Workers AI) | 31%[6%–55%] | 38% | −7 pts |
| Gemma 4 26B A4B (Workers AI) | 30%[0%–75%] | 26% | +4 pts |
| Gemini 3.5 Flash-Lite | 5%[0%–15%] | 11% | −6 pts |
Java was the fourth-hardest of the 12 languages for these models.
Gemini 3.1 Pro gains the most here: 78%, against 67% over every bug.
Claude Sonnet 5.5 drops the most: 45%, against 77% over every bug.
The change “Add core API to enable Kotlin singleton mocking” in mockito/mockito, a change of over 300 lines that landed in February 2026, introduced an API-misuse bug that a later upstream commit fixed.
Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, Claude Sonnet 5.5, GPT-5.6 Sol, GPT-6 Luna and Gemini 3.1 Pro caught it; GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Pro (Workers AI), DeepSeek V4 Flash (Workers AI), GPT-OSS 120B (Workers AI) and Kimi K2.7 Code (Workers AI) caught it in part; Gemini 3.8 Flash, Claude Haiku 5.5, Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.