How we built the code review benchmark

The CodePanel benchmark measures one thing: given a real pull request that introduced a real bug, does a model, reviewing on its own, find it? And given a correct pull request, does it stay quiet? Every number it publishes carries a 95% confidence interval, every comparison between two models is a paired test on the same fixtures, and three judges from three model families decide what counts as a catch. This post explains how it is built, and why its results will get sharper with every release.

Real defects, not planted ones

Every bug fixture is a commit from a permissively licensed open-source project that introduced a defect, confirmed by a later commit that fixed it. The model sees what a reviewer would have seen at the time: the title, a description paraphrased from the upstream pull request, and the changed files. It never sees the fix, the commit messages or our notes.

We do not plant bugs. Planted bugs are easy: they are self-contained and look like bugs. In our own pipeline evals, planted bugs pushed every variant close to 100%, which told us nothing. Real defects are subtle in the ways that matter: a check that runs after the write, a boundary that is off by one only on the last page, a retry that re-sends a charge.

Clean fixtures are correct upstream commits whose lines nothing later fixed or reverted. Any warning-or-worse finding on one is a false positive. A reviewer that flags everything would ace recall and fail here.

Each fixture is checked by a blind audit before it counts. An auditor sees the fixture without our notes, re-derives every claim from upstream, and records a verdict.

One model, the production pipeline

Each model runs as a single reviewer in the Bugs & Security role, with the same prompts, diff format, surrounding code and context packing that CodePanel uses in production. No orchestrator, no other reviewers. The point is to compare models, and a panel would blur which model found what.

A model that fails on a fixture is not scored as a clean review with zero findings. What happens instead depends on whose failure it was. If the model timed out, ran out of output or answered with something unreadable, the round counts as a miss, and the model's completion rate is published next to its recall: a reviewer that cannot finish a review has not found the bug. If the call failed outside the model, on a billing refusal, a rate limit or a dropped connection, the round is left out as if it never ran, and a model with more than one round in five lost that way is not ranked at all.

Three judges, no single family in charge

A finding catches a defect when it is in the same file and names the same root cause. "Refund uses float arithmetic" and "rounding error in refund calculation" are the same bug; "refund has no test" is not. That judgement is made by models.

Model judges have a known bias: they favour output from their own family. So the judge is a panel of three, one each from Anthropic, OpenAI and Google, and every judge is also a contestant. A model is never judged by its own maker: that judge's vote is left out, and the other two must both agree. A model from any other maker keeps all three votes and needs two. A split or an "unsure" is not credited. Each model's recall with all three judges is published beside it, so the effect of the rule is visible.

Every judge vote is kept, so we publish how often the judges agree with each other. The next step is human labels on a stratified sample of judged pairs, published as agreement between each judge and people.

Intervals on everything

Run the same model on the same fixture twice and you get two different reviews. The benchmark can run every model on every fixture more than once, averages each fixture over its runs, and then resamples fixtures 10,000 times to put a 95% interval on recall and false positives.

The unit of resampling is the fixture, not the run. Treating repeats as independent evidence would make the intervals look tighter than the data supports.

Two models are compared on the fixtures both were run on, paired, and the headline recall of every ranked model is computed on the same fixtures: the ones all of them ran. Fixtures differ in difficulty far more than models do, and pairing removes that. The p-value comes from an exact paired permutation test, corrected for the number of pairs compared. A model's rank is one plus the number of models significantly better than it, so models the data cannot separate share a rank. The site only calls a gap a lead when that test says so.

Contamination

Public bugs and their fixes end up in training data. A model that has seen the fix may "find" the bug from memory. Two controls:

• Fresh fixtures. The headline counts only defects introduced on or after 2025-01-01. Older fixtures still run and are reported separately.
• A canary string in every public fixture, so training pipelines that honour canaries drop them, and a model that can complete the string has likely trained on the suite.

Why the results will grow

A benchmark like this gets better the same way any measurement does: more data narrows the intervals. The intervals are honest about what the fixtures can separate, and in early runs we expect several models to share a rank. That is the method working, not failing: it will not declare a winner the data cannot support.

The work from here is mostly fixtures. Real defects from upstream history, each with an introducing commit, a fixing commit and a blind audit. As projects fix new bugs, fresh defects become available that no current model can have seen. Each one added narrows the intervals for every model, and the methodology page states how small a difference the current set can detect, so you can see the effect directly.

The model list moves too. When a provider releases a newer model in a family we benchmark, a test fails until the list is updated, so the leaderboard does not quietly fall behind.

Checking a score

The harness, fixtures and judge prompt are private, which keeps the fixtures out of public code that models train on. A real run needs Anthropic, OpenAI and Google keys, because the judges need all three. A publish writes a transcript of every finding, catch and judge vote alongside the results, so we can re-check or re-judge any number.

The methodology page has every constant and limit. The benchmarks hub shows the models and their prices. If you want to run the same method on your own code, the evaluation guide walks through it.