Concurrency bugs: which models catch them
Of the 17 models ranked on concurrency bugs, Claude Opus 5.5 caught the most: 94% [81%–100%] of known defects. 9 of the other 16 models have 95% intervals that overlap its own. Kimi K2.7 Code (Workers AI) caught the fewest, 11% [3%–24%].
| Model | Caught here | Every bug | Difference |
|---|---|---|---|
| Claude Opus 5.5 | 94%[81%–100%] | 90% | +4 pts |
| Gemini 4 Argon | 90%[75%–100%] | 82% | +8 pts |
| Gemini 3.1 Pro | 90%[75%–100%] | 67% | +23 pts |
| Claude Sonnet 5.5 | 83%[58%–100%] | 77% | +6 pts |
| Gemini 3.8 Flash | 79%[50%–100%] | 68% | +11 pts |
| Claude Haiku 5.5 | 75%[50%–100%] | 64% | +11 pts |
| GPT-5.6 Sol | 73%[44%–96%] | 72% | +1 pt |
| GLM-5.3 (Workers AI) | 72%[54%–89%] | 61% | +11 pts |
| GPT-6.1 Sol | 71%[38%–96%] | 89% | −18 pts |
| GPT-6 Luna | 71%[38%–96%] | 67% | +4 pts |
| DeepSeek V4 Flash (Workers AI) | 54%[33%–79%] | 49% | +5 pts |
| GLM-5.3 Flash (Workers AI) | 53%[33%–75%] | 61% | −8 pts |
| DeepSeek V4 Pro (Workers AI) | 50%[28%–72%] | 54% | −4 pts |
| GPT-OSS 120B (Workers AI) | 42%[24%–63%] | 40% | +2 pts |
| Gemma 4 26B A4B (Workers AI) | 22%[3%–47%] | 26% | −4 pts |
| Gemini 3.5 Flash-Lite | 13%[0%–38%] | 11% | +2 pts |
| Kimi K2.7 Code (Workers AI) | 11%[3%–24%] | 38% | −27 pts |
On average these models caught 61% of concurrency bugs, more often than the 60% the same 17 caught across the whole benchmark: the second-easiest of the 6 kinds of bug.
Gemini 3.1 Pro gains the most here: 90%, against 67% over every bug.
Kimi K2.7 Code (Workers AI) drops the most: 11%, against 38% over every bug.
A concurrency bug shows only when work overlaps: a data race on shared state, a missing or misplaced lock, an ordering assumption that holds on one thread and breaks on several.
The change “Store cached child els in Attributes, not field” in jhy/jsoup, a change of under 100 lines that landed in April 2025, introduced a concurrency bug in Java code that a later upstream commit fixed.
Claude Opus 5.5, Gemini 4 Argon, GPT-5.6 Sol, Gemini 3.1 Pro, GLM-5.3 Flash, GLM-5.3, DeepSeek V4 Pro (Workers AI), DeepSeek V4 Flash (Workers AI) and GPT-OSS 120B (Workers AI) caught it in part; GPT-6.1 Sol, Claude Sonnet 5.5, Gemini 3.8 Flash, GPT-6 Luna, Claude Haiku 5.5, Kimi K2.7 Code (Workers AI), Gemma 4 26B A4B and Gemini 3.5 Flash-Lite missed it.