# Recall is not enough: measuring false positives in code review

URL: https://codepanel.ai/blog/measuring-false-positives-in-code-review

How CodePanel counts false alarms: clean and trap fixtures, blocking recall, and precision from verdicts the author agent cannot give itself.

A reviewer that flags every line catches every bug, so recall alone cannot tell a good reviewer from a noisy one. We count false positives directly, as warnings raised on changes we know are correct, and track blocking recall: bugs caught by a finding that actually stops the merge. In production, precision counts only verdicts the author agent cannot give itself.

## Why recall alone rewards noise

Recall is the share of known defects a review catches, and CodePanel's scorer counts a catch at any severity. A reviewer that posts fifty findings, most of them wrong, scores well. The cost lands on the reader. In [AI code review](/ai-code-review) where a coding agent is the reader, every [false positive](/glossary/false-positive) at warning or above is work: the agent changes correct code to make the finding go away, or disputes it, and the pull request takes another round. So every eval run reports recall, blocking recall and false positives side by side, and a change is read on all three.

## Clean fixtures: every warning is a false positive

A clean fixture is a correct change from CodePanel's own history, checked against what later commits did to it. Any finding of warning or critical severity on it is a false positive; suggestions are counted apart, as noise rather than a claim that something is broken. The report gives false positives per clean fixture (FP/clean) with a 95% interval resampled over fixtures.

Clean fixtures are most useful when they look like bugs. Two of the newest are reads that look unbounded but are capped by a named limit, the pattern behind a cluster of findings agents had rated down.

## Trap fixtures: correct code that older models flag

A trap fixture is a correct change that uses an API newer than many models' training data. Three of ours:

- Convex calls of the form `ctx.db.get("table", id)`, the current recommended form, which a model trained on the older one-argument API may flag.
- A GitHub request for `application/vnd.github.raw+json`, the documented raw media type, which older models may "correct" to a `v3` form.
- Provider responses read for newer fields: Anthropic's `refusal` stop reason, skipped thinking blocks, Gemini's thought-token count.

Each fixture's notes record the current documentation it was checked against. A false positive here means the model is reciting rather than reading, and those findings are the hardest to dismiss: specific, confident and wrong.

## Bug fixtures have false positives too

On a bug fixture, a finding that matches nothing expected is "unlabeled". It may be noise or a real problem the fixture never listed, so it is read by hand, not counted as wrong. Two lists sharpen the count. `acceptable` holds real problems the pull request was not responsible for: a finding about one is neither a catch nor noise. `falseClaims` holds claims the code contradicts that a review is likely to make, such as calling a test payload missing when triage only skipped it as too large. A warning matching one is a definite false positive. Catches are matched first, so a correct finding is never counted against the review.

## Blocking recall: the catches that stop a merge

Blocking recall counts only the catches made by a [blocking finding](/glossary/blocking-finding), one the [merge gate](/glossary/merge-gate) blocks on at its default level: today a critical or a warning that is not rated low confidence. It is replayed from the judge verdicts recall already has, so it costs no extra judge calls, and it is never above recall. Every blocking-recall figure in this post predates 2026-10-23, when every warning still blocked, so it is not directly comparable with runs since.

It matters most for changes that demote findings. Our `changed-lines` variant moves a warning more than five lines from anything the change touched down to a suggestion. Recall cannot see the cost, since it counts the catch either way, and FP/clean shows only the gain, since suggestions are not counted. The cost is a bug still reported but no longer stopping the merge, and only blocking recall shows it. In a paired run on 2026-10-16 (53 fixtures, 2 repeats, under the old every-warning rule), blocking recall moved +2pt [-3, +8]: no sign of that cost, but no interval excluded zero, so the switch stayed off.

## What the default panel numbers show

These runs chose the models CodePanel seeds by default: the codepanel suite with 36 bug and 13 clean fixtures at the time, a single Sonnet judge, 2 to 4 repeats per panel.

| Panel | Recall | Blocking recall | FP/clean | FP/trap | Cost a round |
| --- | --- | --- | --- | --- | --- |
| All Haiku reviewers, Sonnet orchestrator (the default) | 47% | 37% | 0.48 | 0.5 | $0.045 |
| All Haiku, Haiku orchestrator | 49% | 39% | 0.85 | 1.0 | $0.009 |
| Haiku and Sonnet reviewers, Sonnet orchestrator | 47% | 34% | 0.42 | 0.9 | $0.110 |
| All Sonnet | 52% | 37% | 0.31 | 0.8 | $0.143 |

- **The orchestrator is where precision comes from.** Moving it to Haiku kept recall and cut the round's cost by four fifths, but FP/clean rose from 0.48 to 0.85: paired over the 13 clean fixtures, +0.37 [+0.04, +0.75], sign-flip p about 0.11, not significant, with three fixtures behind most of it. The orchestrator stayed on Sonnet as a judgement on that estimate.
- **Sonnet reviewers mostly buy precision.** The all-Sonnet recall gain, +5.4pt [-3.5, +15.7], is not established. Its FP/clean is the lowest here, at about three times the cost.
- **Trap numbers are a warning light, not a result.** FP/trap is an unpaired mean over a few fixtures, untested. It doubled with the Haiku orchestrator: a reason to look, not a finding.

The judge shared a family with every panel here, which is why later eval runs use judges from several families and leave out each judge's vote on its own family's findings.

## Precision needs verdicts the author agent cannot give itself

After a change ships, the natural measure is precision: of the findings posted, how many were right. The trap is where verdicts come from. A coding agent rates the findings it acted on, and it "fixes" what it is told to fix, false positives included. Any rate built on its ratings or fixes goes up when the agent complies more, not when the review gets better.

So CodePanel's verified precision counts only verdicts the author agent cannot produce: a dispute the orchestrator upheld, revised or withdrew after reading the code, and a trusted person's rating from the review page or a `/feedback` reply. It reports a 95% Wilson interval, and no figure for a segment with fewer than 20 verdicts.

That figure has its own bias: agents dispute the findings they already doubt, so it is the precision of a challenged finding, not a posted one. Our answer is a fixed weekly sample of posted findings, stratified by severity, that a person rates, with draws aimed at buckets that look bad kept out of the headline number.

## The bottleneck is clean fixtures

False positives on a correct change are rare events, counted in fractions per fixture. Our `precision` variant moved FP/clean by -0.06 [-0.17, +0.00] (2026-10-15, 24 fixtures, 3 repeats), and `changed-lines` by -0.15 [-0.58, +0.19] (2026-10-16, 53 fixtures, 2 repeats, on a different panel). Both point the right way and both intervals touch or cross zero, so both switches are off. A dozen or so clean fixtures cannot resolve changes that small. More repeats on them help a little; more clean fixtures help most, and the clean set is one of the parts of the suite we are growing now.

The [guide to reducing false positives](/guides/reducing-false-positives) covers the fixes, and the [methodology page](/benchmarks/methodology) shows how the public benchmark counts false alarms on clean open-source changes. To try CodePanel on your own pull requests, ask your coding agent to set it up from the [quickstart](/docs/quickstart).