Claude Haiku 5.5 vs Gemini 3.5 Flash-Lite
Reading a 26.2K-token pull request costs $0.0079 on Gemini 3.5 Flash-Lite, 3.0× the $0.0026 it costs on Claude Haiku 5.5. Above 100K tokens long-context rates apply, and Gemini 3.5 Flash-Lite becomes the cheaper one. Both sit in the economy price tier, so the choice between them is about review quality more than budget. In the benchmark Claude Haiku 5.5 caught 53 percentage points more of the known defects than Gemini 3.5 Flash-Lite (95% interval +37 pts to +67 pts, a − b).
Side by side
| Figure | Claude Haiku 5.5 | Gemini 3.5 Flash-Lite |
|---|---|---|
| Price tier | Economy | Economy |
| Bugs caughtshare of known bugs | 64% (better) | 11% |
| False alarmsper clean PR | 1.4 | 0.4 (better) |
| Cost per reviewmeasured | $0.0038 (better) | $0.0059 |
| Cost per bug caught | $0.0064 (better) | $0.0528 |
| Exact linecatches on the expected lines | 82% | 83% (better) |
| Review timemean | 15s (better) | 103s |
| Typical review calllist price | $0.0038 (better) | $0.0139 |
| Input / 1M tokens | $0.10 (better) | $0.30 |
| Output / 1M tokens | $0.50 (better) | $2.50 |
| Context window | 1M | 1M |
- Above 100K tokens, long-context rates mean Gemini 3.5 Flash-Lite is the cheaper of the two.
Head to head
The known bugs both were scored on, by who caught each in at least one run.
By language
| Model | All | TypeScript6 bugs | Go4 bugs | Java4 bugs | Rust4 bugs | C#3 bugs | JavaScript3 bugs | Kotlin3 bugs | PHP3 bugs | Python3 bugs | Ruby3 bugs | Swift3 bugs | C++2 bugs | C1 bug |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Haiku 5.5 | 64% | 83% | 75% | 40% | 75% | 67% | 33% | 67% | 100% | 39% | 67% | 67% | 50% | 0% |
| Gemini 3.5 Flash-Lite | 11% | 33% | 0% | 5% | 0% | 0% | 0% | 33% | 0% | 17% | 0% | 0% | 50% | 0% |
Of the 42 known bugs both were scored on, both caught 6 and neither caught 14. Claude Haiku 5.5 caught 22 bugs that Gemini 3.5 Flash-Lite missed: 3 in Go, 3 in PHP, 3 in Rust, 3 in TypeScript, 2 in C#, 2 in Ruby, 2 in Swift, 1 in Java, 1 in JavaScript, 1 in Kotlin and 1 in Python.
On pull requests with no known bug, Claude Haiku 5.5 raised 1.4 false alarms per review and Gemini 3.5 Flash-Lite 0.4. A measured review cost $0.0038 on Claude Haiku 5.5, 35% less than $0.0059 on Gemini 3.5 Flash-Lite.
| Pull request | Language · kind | Claude Haiku 5.5 | Gemini 3.5 Flash-Lite |
|---|---|---|---|
| MudBlazor/MudBlazorRefactor: MudDebouncedInput, ThrottleDispatcher and DebounceDispatcher | C# · Concurrency | 3/3 | 0/3 |
| OrchardCMS/OrchardCoreSimplify using `StringValues` in Liquid templates | C# · Security | 1/1 | 0/1 |
| better-auth/better-authfix(saml): IdP-Initiated Callback Routing | TypeScript · Security | 1/1 | 0/2 |
| colinhacks/zodfeat(v4): parse input containing reference cycles | TypeScript · Resource handling | 1/1 | 0/1 |
| denoland/denoperf(libs/core): poll only ready handles and yield between I/O batches | Rust · Concurrency | 1/1 | 0/1 |
| doctrine/ormFix collection filtering API for IN/NOT IN comparisons that require type conversions | PHP · Logic | 1/1 | 0/1 |
| go-gitea/giteaperf(gitdiff): optimize inline diff highlighting using cache | Go · Error handling | 1/1 | 0/1 |
| golang/gocrypto/tls: add LocalCertificate to ConnectionState | Go · Logic | 1/1 | 0/1 |
| groue/GRDB.swiftDatabaseQueue and DatabaseSnapshot reads can access task locals | Swift · Concurrency | 1/1 | 0/1 |
| iBotPeaches/Apktoolrefactor: Improve renaming/injection of resources in stripped APKs | Java · Security | 1/1 | 0/1 |
| ktorio/ktorKTOR-9171 Redesign ByteReadChannel.readUTF8Line API | Kotlin · Logic | 1/1 | 0/1 |
| laravel/framework[12.x] Colocate Container build functions with the `Buildable` interface | PHP · Error handling | 1/1 | 0/1 |
| nodejs/undicifeat: add configurable maxPayloadSize for WebSocket | JavaScript · Security | 1/1 | 0/1 |
| pointfreeco/swift-composable-architectureRemove child store from parent when invalidated | Swift · Resource handling | 1/1 | 0/1 |
| rack/rackApply bytesize and number of param limits in QueryParser | Ruby · Security | 1/1 | 0/1 |
| rustls/rustlsAffix `Protocol` at top of client handshake | Rust · Logic | 1/1 | 0/1 |
| socketry/asyncBarrier wait in completion order + `Queue#close`. | Ruby · Concurrency | 1/1 | 0/1 |
| symfony/symfony[Security] Allow multiple OIDC discovery endpoints | PHP · Security | 1/1 | 0/1 |
| tokio-rs/tokiotime: move lazy-registration state into `Sleep` | Rust · Logic | 1/1 | 0/1 |
| traefik/traefikAdd the option to define custom cipher suites for backend serversTransport | Go · API misuse | 1/1 | 0/1 |
| trpc/trpcfix(server): pass maxDurationMs timeout signal to subscription handlers | TypeScript · Concurrency | 1/1 | 0/1 |
| django/djangoFixed #36226 -- Accepted str or bytes for password and salt in password hashers. | Python · Error handling | 2/3 | 0/1 |
Defects caught over all runs: a PR reviewed three times with one known bug shows 3/3 when it was caught every time.
Every bug each was scored on: Claude Haiku 5.5 and Gemini 3.5 Flash-Lite.