# How to evaluate an AI code reviewer

URL: https://codepanel.ai/guides/evaluating-ai-code-reviewers

Test an AI code reviewer on your own code: known-defect fixtures, clean changes, recall and false positives, line accuracy, repeats and judge bias.

To evaluate an AI code reviewer, run it on pull requests where you already know the answer: changes that introduced a real bug, and changes that were correct. Score how many known bugs it finds, how many findings it raises on correct code, and whether it points at the right lines. Run every case several times, put a confidence interval on every number, and compare two reviewers on the same cases rather than by their separate averages. A couple of days of work gets you an answer that applies to your codebase, which no public leaderboard can.

## Build fixtures from your own history

A fixture is a pull request with a known answer. Your repository's history is full of them.

**Bug fixtures.** Find commits that fixed a bug. `git log --grep='fix'` is a start; better, follow the links from closed bug reports. For each fix, find the commit that introduced the defect (`git blame` on the fixed lines, at the parent of the fix). The introducing commit is the "pull request" the reviewer sees. The fix tells you where the bug is and what it was.

**Clean fixtures.** Find changes that were correct: commits whose lines nothing later fixed or reverted. Any finding of warning severity or worse on a clean fixture is a false positive. Clean fixtures are the most direct measure of how noisy a reviewer is, and the ones people most often skip.

**Trap fixtures.** Correct changes that use library APIs newer than the models' training data. Models that recite old documentation flag them. Useful if you work on fast-moving frameworks.

For each bug fixture, record:

| Field | Example |
| --- | --- |
| Files before and after | Whole file contents, so the reviewer can see context |
| PR title and description | Paraphrased; must not hint at the bug |
| Expected defect | `src/billing.ts`, lines 88 to 91, "refund amount uses float arithmetic" |
| Also acceptable | Other real problems in the same change, so a reviewer that finds them is not scored as noisy |

Two rules matter more than the rest. Never hint at the bug in the description: "Fix rounding in refunds" gives the answer away. And prefer real defects to planted ones. Planted bugs are too easy; every reviewer scores near 100% and you learn nothing.

## What to measure

| Metric | Definition | Why |
| --- | --- | --- |
| Recall | Per bug fixture, caught ÷ expected defects, then averaged over fixtures | How much it finds |
| False positives per clean fixture | Warning-or-worse findings on correct changes | How much noise it makes |
| Line accuracy | Of caught defects, share within a few lines of the expected range | Whether the coding agent can find it |
| Unlabeled findings | Findings on bug fixtures that match nothing expected | May be noise, may be real bugs you missed |
| Cost and time | Tokens times rates; wall-clock per fixture | What it takes to run |

Average recall per fixture, not per defect. Otherwise a fixture with five defects counts five times as much as one with one.

Fixtures give you the offline numbers; [code review metrics](/guides/code-review-metrics) covers which of them to watch on live pull requests.

Read the unlabeled findings by hand. Some will be real problems your fixture did not list. Add those to "also acceptable" and rescore, or you penalise the reviewer for being right.

## Matching findings to defects

A finding catches a defect if it is in the same file and describes the same root cause. "Same root cause" is a judgement. "Refund uses floats" and "rounding error in refund calculation" are the same bug in different words; "refund has no test" is not.

You can judge by hand for a small set. For more, use a model as the judge, with care:

- **Ask a narrow question.** "Does this finding describe the same root cause as this defect?" Not "how good is this review?"
- **Use judges from several model families and take the majority.** Judges favour output from their own family, a documented effect. A panel where no single family decides reduces it.
- **Do not credit "unsure."** It counts as a miss.
- **Check the judge against people.** Label a sample of judged pairs yourself and measure agreement. If the judge disagrees with you often, fix the judge prompt before trusting any number it produces.

Location is a separate check from the judge. A finding on the right root cause but 30 lines away is a catch with poor placement. Count both.

## Repeats and confidence intervals

Run every fixture at least three times. The same model on the same input gives different findings run to run, and you need to know how large that variation is before you believe a difference.

Then put an interval on every number. The right unit is the fixture: average each fixture over its repeats, then bootstrap over fixtures (resample fixtures with replacement, 10,000 times, take the 2.5th and 97.5th percentiles). Resampling individual runs treats repeats as independent evidence and makes intervals look narrower than they are.

Intervals are wide with few fixtures. With 20 bug fixtures, a 95% interval on recall is commonly 15 to 20 points either side. That is not a flaw in the method; it is how much 20 examples can tell you. It means a 5-point difference between two reviewers on 20 fixtures is noise.

## Comparing two reviewers

Compare on the same fixtures, paired. Fixtures vary in difficulty far more than reviewers do, so two separate averages with overlapping intervals hide real differences, and two separate averages that do not overlap can still mislead.

For each fixture, take reviewer A's recall minus reviewer B's. Bootstrap the mean of those differences for an interval, and use a paired permutation test (randomly flip the sign of each fixture's difference) for a p-value. If the interval on the difference spans zero, you have not shown a difference. If you compare many reviewers, correct for multiple comparisons.

A worked reading:

| | Recall | FP per clean fixture |
| --- | --- | --- |
| Reviewer A | 58% [44–71%] | 0.4 [0.1–0.8] |
| Reviewer B | 64% [50–77%] | 1.3 [0.7–2.0] |
| B − A (paired) | +6 pts [−3 to +15] | +0.9 [+0.3 to +1.6] |

B's higher recall is not established: the paired interval includes zero. B's extra noise is established. On this evidence, A is the better reviewer. These numbers are illustrative, not from any real run.

## Contamination

Public repositories are in models' training data, and so are their bug fixes. A model may "find" a bug because it has seen the fix, not because it reviewed the code. Two defences:

- **Prefer fresh fixtures**, introduced after the models' training cutoffs, and report older fixtures separately.
- **Keep a held-out set** that is never published and never used to tune prompts. If a reviewer does much better on public fixtures than held-out ones, suspect memorisation or overfitting.

Your own private repository is a natural held-out set. Fixtures built from it have not been trained on, unless your code is public.

## Running it fairly

- Use the reviewer's real prompts and context, the same as production. A benchmark of a stripped-down pipeline measures something else.
- Fix the model version and reasoning effort, and record them.
- A run where the reviewer errored is not a clean review with zero findings. Count errors separately and leave them out of the scores.
- Record cost from actual token counts, not estimates.
- Never edit a fixture's expected answer after seeing results to make a change look better.

## How large a set do you need?

To tell apart two reviewers that differ by 15 points of recall, a few dozen bug fixtures is enough. To see 5 points, you need hundreds. Start with 20 to 30 bug fixtures and 5 to 10 clean ones from your own repository. That is enough to catch a reviewer that is clearly noisy or clearly blind on your code, which is the decision most teams need to make.

CodePanel's public benchmark applies this method to single models: real defects and clean changes from permissively licensed open-source projects, a three-family judge panel, three repeats, bootstrap intervals over fixtures, paired permutation tests and a held-out suite. The [methodology page](/benchmarks/methodology) describes every step. The harness and fixtures are private, to keep the fixtures out of code that models train on, so what carries over to your own repository is the method on this page, not the tool.