Reducing false positives in AI code review

Most false positives in AI code review come from five causes: the reviewer cannot see enough code, it knows an older version of your libraries, it inflates severity, it does not know your conventions, or it reviews code the pull request did not write. Each has a specific fix. The most effective single change is a filtering step that checks every finding against the code before anything is posted. The second is a clear rule for what counts as a warning.

Why false positives cost more than missed bugs

A missed bug costs what it would have cost without the reviewer. A false alarm costs more than reading time. A coding agent told to fix a non-problem changes correct code to satisfy it, the next round reviews that change, and the loop spends a round on nothing. An agent that learns to rebut everything is worse: it will rebut the real finding on line 40 too. People who read the review learn to skim it for the same reason.

So measure noise directly. Run your reviewer on changes you know are correct, and count findings of warning severity or worse. Every one is a false positive. On a well-tuned setup that number should be close to zero per change.

Cause 1: the reviewer cannot see enough

A reviewer that sees only the diff will flag a null dereference when the null check is ten lines above the hunk, or a missing import that is already at the top of the file. These findings are confident and wrong.

Fix: send the code around each change, not only the diff. Whole files when they are small; a window of lines around each hunk when they are large. CodePanel sends a changed file whole up to 24,000 characters, otherwise about 40 lines around each change, for up to 25 files.

Also number the lines. A reviewer counting offsets from @@ hunk headers gets line numbers wrong often enough to matter. A gutter with each line's number in the new revision lets the model copy the number instead of computing it.

Cause 2: outdated library knowledge

A model trained before your framework's latest release will flag new, correct APIs as errors. These are the hardest false positives to dismiss, because they are specific and sound authoritative: "useActionState is not a React export; use useFormState."

Fixes:

• Put the versions and the newer APIs you rely on in your repository rules.
• Tell reviewers to mark claims about library behaviour as low confidence unless the code shown settles them.
• In the filtering step, keep such a finding only if the risk is real either way, and say it needs checking against the library's docs.

When you evaluate a reviewer, include "trap" changes: correct code that uses APIs newer than the model's training data. False positives on traps tell you how much a model recites rather than reads.

Cause 3: severity inflation

"No tests cover this" is often true. It is rarely a warning: nothing fails because of it. When the first real evaluation of CodePanel's pipeline was run, that finding was the most common warning on correct pull requests.

Fix: define severity by what goes wrong at runtime. CodePanel's orchestrator applies this rule:

Use critical or warning only when you can name what fails, for whom, and how, from the code as it stands. A gap that only makes a future mistake more likely (missing or thin tests, missing docs, naming, duplication) is a suggestion.

This does not hide test gaps. It moves them to a severity a coding agent may skip, and keeps warnings for things that break.

Cause 4: the reviewer does not know your conventions

Your codebase stores money in integer cents. Your migrations are append-only. Your auth helper must be called before any write. A reviewer that does not know this will both miss real violations and flag correct code that looks unusual.

Fix: write the rules down where every reviewer reads them. Short, checkable, one per line:

- Money is stored and computed in integer cents. Flag any float arithmetic on amounts.
- Every mutation that takes a repoId must call requireRepoOwnership first.
- Files under convex/migrations/ are append-only. Flag an edit to an existing migration.
- We use the React 19 form APIs (useActionState). Do not suggest useFormState.

Rules can also say what not to flag. The last line above prevents a false positive rather than catching a bug.

In CodePanel, a file at .codepanel/rules.md is read on every review, from the branch the pull request targets, never from the pull request itself. A pull request cannot loosen the rules it is reviewed under. See Repo context and custom rules.

Cause 5: reviewing code the pull request did not write

When an author merges main into their branch, the next push contains everyone else's recent changes. A naive reviewer reviews all of it and reports other people's bugs on this pull request.

Fixes:

• After the first round, review only the range since the last reviewed commit.
• Exclude code a base-branch merge brought in. A push that only merges the base can skip review entirely.
• Never mark a finding resolved because a round did not look at its code. Resolve it only when a round re-examined that line and did not report it again.

The filtering step

Reviewers are most useful when they report everything they notice, including low-confidence findings. A separate step then decides what holds up. That split matters: a reviewer told to be careful misses things, and a filter told to be thorough lets things through.

The filter needs four things:

1. The code, for every file a finding names. Without it, the filter judges findings by how convincing they sound.
2. An instruction to drop what the code contradicts: the case is handled, the variable is checked a few lines up, the reviewer misread a removed line as current, the code it describes does not exist.
3. Permission to correct a real finding's line number rather than drop it.
4. The same rules the reviewers had, so it judges findings against your conventions.

CodePanel's orchestrator works this way. Its prompt grows with what was reported, not with the size of the change.

Make dismissals stick

A false positive that comes back every push is worse than one that appears once. When someone dismisses a finding as wrong, record why, and suppress the same finding in later rounds rather than reopening it.

CodePanel's dismiss reasons are false positive, wrong location, not worth fixing, duplicate and other. A dismissed finding stays closed and stops counting toward a merge gate.

A dismissal covers one pull request. The same false positive often returns on the next pull request that touches the file. CodePanel's verdict memory shows the orchestrator earlier verdicts on the changed files, with their reasons, so it can drop a finding someone already refuted if the reason still holds in the code. See verdict memory.

Rate findings so you can see the trend

You cannot reduce what you do not count. After acting on a review, the coding agent rates each finding: right or wrong, and if wrong, why. It is the best-placed rater, because it has just checked every finding against the code. A one-line reply on the pull request is enough:

/feedback F2 down false-positive the null check is on line 31 --agent claude-code
/feedback F3 up fixed --agent claude-code

Over a month, ratings show which models and roles produce the noise. If one reviewer's findings are dismissed far more often than the others', change its model or rewrite its instructions. CodePanel's repository Reviewers tab shows, per model, findings reported, dismissed by reason and resolved. See Findings and feedback.

Narrow the scope

Some files should never be reviewed: lock files, generated code, minified bundles, snapshot files. Findings there are noise by definition. Some reviewers should not run on every change: a UI reviewer on a pull request that touches no UI files can only produce noise. CodePanel skips its Design & UX reviewer on rounds with no UI changes, at no cost.

A checklist