# How to tell if an AI code review change actually helps

URL: https://codepanel.ai/blog/how-to-tell-if-an-ai-code-review-change-helps

Paired runs over the same fixtures, intervals resampled over fixtures, a sign-flip test and Holm. The pipeline changes we measured and did not ship.

Run the old and new pipeline on the same fixtures, average each fixture over its repeats, and read the difference fixture by fixture, with an interval resampled over fixtures. Ship a quality change only when that interval excludes zero in the change's favour and a paired sign-flip test agrees; ship a cost change only when it rules out a large quality loss. In October 2026 that rule kept three plausible changes out of CodePanel's production reviews.

This post is about changes to our own review pipeline, not about ranking models; [how we built the benchmark](/blog/how-the-benchmark-is-built) covers that. The statistics are shared, so the words here go on the decisions.

## Every change is a switch, measured against the old pipeline

A change to what [AI code review](/ai-code-review) shows a model goes in behind a switch, off by default, with an eval variant that turns it on. For a prompt change, a test checks that with the switch off the prompts are byte-identical to before, so "off" really is the old pipeline.

The eval runs `current` and the variant over the same fixtures with the same judges, normally in one run. A fixture is a pull request with a known answer: a commit from CodePanel's history that introduced a bug a later commit fixed, or a change that was correct.

## Fixtures are the unit, not runs

The same pipeline on the same fixture gives a different review each time, so fixtures run more than once, and the repeats are averaged into one number per fixture before anything else. The interval then comes from resampling fixtures, 10,000 times with a fixed seed, so the same results file always gives the same interval.

Counting each repeat as a data point would make the interval look tighter than the evidence. A bug the panel cannot see stays invisible on the third try. Miller's "Adding Error Bars to Evals" (Anthropic, 2024) makes the general point: sampling a question again shrinks the noise within that question, not the spread between questions. For recall, more fixtures buy power and more repeats mostly do not.

So the harness states power before it spends anything. A run can name the smallest recall change it is meant to detect, and it is refused, before any model call, when its fixtures and repeats cannot see a change that size.

## Pairing removes how hard each fixture is

Some fixtures are caught by every variant, some by none. That spread is far larger than a prompt change's effect, and two separate averages bury the change under it. So the eval report pairs: for each fixture, the variant's score minus the base's, and an interval on the mean of those differences, for recall, blocking recall and false positives per clean fixture. A difference whose interval spans zero is not a win, however good the mean looks.

## The p-value comes from flipping signs, and it is the one we read

The interval is a percentile bootstrap; the p-value comes from a paired sign-flip test. If a change does nothing, each fixture's difference is as likely to be positive as negative, so the test flips signs and asks how often a total as large as the real one turns up. Up to 16 fixtures it counts every assignment exactly; above that it draws 10,000 seeded ones. We use it because a percentile bootstrap is overconfident with few fixtures.

Two results from October show why:

| Change | Paired difference [95%] | Sign-flip p |
| --- | --- | --- |
| `consistent-prompts`: recall | -9pt [-19, -0] | ≈0.08 |
| Haiku orchestrator in place of Sonnet: false positives per clean fixture | +0.37 [+0.04, +0.75] | ≈0.11 |

The Haiku orchestrator's bootstrap interval excludes zero, so the bootstrap alone would call it a real rise in false positives; the sign-flip test says it is not established. That is a genuine disagreement, and we read the sign-flip test, because the percentile bootstrap is the one that runs overconfident on few fixtures. The consistent-prompts upper bound rounds to zero at the precision shown, so there the two tests agree closely enough: the drop is not established. Both still decided something, for other reasons. Consistent prompts was meant to raise recall and pointed the other way, so there was nothing to ship. The Haiku orchestrator cut a round from $0.045 to $0.009, and we kept Sonnet anyway, because every false positive costs the author's coding agent a fix or a dispute: a judgement on the point estimate, not a significant result.

## Holm when several questions share a run

Three tests at p < 0.05 give much more than a 5% chance of at least one false "significant". Holm's correction handles that: sort the p-values, multiply the smallest by the number of tests, the next by one fewer, and never let an adjusted value fall below the one before it.

CodePanel applies it in two places. The weekly drift run reruns the production panel, compares it with the week before on recall, blocking recall and false positives per clean fixture, and corrects across the three. The benchmark corrects across every pair of models it ranks. The eval report does not: it prints each variant against the base uncorrected, so several variants sharing one base should be adjusted by hand before anyone reads a winner from them.

## Changes we did not ship

Each variant was paired against `current`; FP/clean is warning-or-worse findings per clean fixture. A single Sonnet judge scored all three runs.

| Change (run date) | Panel; fixtures; repeats | Recall | Blocking recall | FP/clean |
| --- | --- | --- | --- | --- |
| `precision` (2026-10-15) | Sonnet and Haiku reviewers, Opus orchestrator; 24; 3 | +0pt [-10, +11] | not recorded | -0.06 [-0.17, +0.00] |
| `consistent-prompts` (2026-10-16) | Mixed economy reviewers, Sonnet orchestrator; 53; 2 | -9pt [-19, -0] | -4pt [-10, +1] | +0.00 [-0.19, +0.15] |
| `changed-lines` (2026-10-16) | Mixed economy reviewers, Sonnet orchestrator; 53; 2 | +4pt [-3, +11] | +2pt [-3, +8] | -0.15 [-0.58, +0.19] |

**Precision rules.** Six orchestrator rules, each drawn from a cluster of findings coding agents had rated down in production: reads a named limit already bounds, triggers no caller can produce, behaviour the change says is deliberate, and so on. Recall held and false positives moved the right way, but the interval touches zero. Clean fixtures average well under one false positive each, so a few dozen cannot show a drop that small. Off.

**Consistent prompts.** The prompts contradicted themselves: each role ended "report only concrete problems" while the task under it said "report every issue, do not filter". The variant resolved that toward reporting anything that could be a defect. Recall fell. The orchestrator dropped 27% of reviewer catches against 14% for `current`, and run-to-run spread rose from 1.6 to 4.6 points: asking reviewers for more gave the precision stage more to throw away. Off.

**Changed lines.** A finding more than five lines from anything the change touched is marked pre-existing and demoted from warning to suggestion, unless it is a critical security finding. This one is code, not prompt, and it came from production too: many down-rated findings sat on code the pull request never wrote. Recall and false positives cannot show its cost, because recall counts a catch at any severity and FP/clean ignores suggestions. The cost is a caught bug that no longer stops the merge, which only blocking recall shows. Blocking recall did not fall, but no interval excludes zero. Off until a run on a larger clean set.

## One change shipped on cost, not quality

Cacheable prompts put the material reviewers share first and each role's instructions last, so reviewers on one model and effort read one prompt cache. It could not make reviews better; the risk was making them worse, with the role's instructions now after a long diff. So we read the result differently: did the quality interval rule out a large loss, and how much did it save?

On a panel with all four reviewers on Sonnet (2026-10-16, 53 fixtures, 2 repeats), recall moved -1pt [-8, +7] and FP/clean -0.04 [-0.12, +0.00], while reviewer cost within a round fell 34%, from $0.1029 to $0.0675. That run rules out a recall loss above about 8 points, not every loss. On the all-Haiku panel then in production the saving was smaller, 17% of reviewer cost, because cheaper shared tokens save less. It became the production default that day.

## What a null result tells you

Each interval says how big an effect the run could have missed. The precision run rules out a recall drop larger than 10 points; it cannot rule out any smaller drop. Two of the three decisions above stalled on the same thing, too few clean fixtures to see a small drop in false positives, and that is the part of the suite we are growing.

The [methodology page](/benchmarks/methodology) applies the same method to the public benchmark, and the [evaluation guide](/guides/evaluating-ai-code-reviewers) shows how to run it on your own repository. To try CodePanel's reviews, ask your coding agent to set it up from the [quickstart](/docs/quickstart).