Code review metrics: what to track when agents write the PRs
Code review metrics for agent-written PRs and model reviewers: precision, blocking recall, fix vs dispute rate, rounds to green and cost per PR.
Code review metrics are numbers that tell you whether review catches defects, how fast it does, and what it costs. When agents write and fix pull requests, keep four classic ones (time to first review, rounds per pull request, change size, escaped defects) and add five for model reviewers: finding precision, blocking recall, fix and dispute rate, rounds to green, and cost per pull request.
| Metric | What it tells you | Measured from | Main trap |
|---|---|---|---|
| Time to first review | Whether the author agent waits | PR timestamps | Counting hours when the agent needs minutes |
| Review rounds per PR | How long the loop runs | Round history | Fewer rounds because the reviewer saw less |
| Change size | Whether review can see the change | Diff stats | Counting docs and data as code |
| Escaped defects | What review missed | Issue tracker | Blaming review for what it never saw |
| Finding precision | How much of the review is true | Verdicts you trust | Letting the author agent score itself |
| Blocking recall | Real defects that stop a merge | Fixtures with known bugs | Measuring it on live pull requests |
| Fix rate and dispute rate | How findings are answered | Ratings and disputes | A 100% fix rate looks like quality |
| Rounds to green | How soon the gate clears | Round history | Pushing fixes one at a time |
| Cost per PR | What the loop spends | Spend records | Dividing by rounds, not pull requests |
Time to first review: minutes, not days
Definition. Time from a pull request being ready to the first review landing, taken from GitHub timestamps.
For people this is a day-scale number. For a coding agent it is minutes, because the session that opened the pull request is usually still open and waiting. A review that arrives after the session ends is acted on by a new session with no memory of the change.
The trap. Measuring from "opened" counts time a draft sat idle. CodePanel holds each push for 60 seconds so a burst of commits gets one review (triggering reviews); that wait is deliberate and belongs in the number.
Review rounds per PR
Definition. How many review rounds a pull request goes through before it merges. Compute the mean, the 90th percentile and the share that took five or more. CodePanel's get_quality tool returns exactly these under convergence.rounds: mean, p90 and fiveOrMore, for pull requests merged in the window (up to the newest 60), and marks a segment insufficientData below 20 pull requests.
The trap. Fewer rounds is good only while precision holds. A reviewer that sees less, or reports less, also produces fewer rounds. Read this number next to finding precision, never alone. Also split rounds by what started them: a push, a manual /review, or a base-branch merge. CodePanel reports that split as roundsByTrigger, because a base merge is a round the author did not choose.
Change size
Definition. Lines changed and files touched per pull request. Compute additions plus deletions from the diff, and report the median.
Size matters more for a model than for a person, because a model has a context window and a person has patience. CodePanel sends a changed file whole up to 24,000 characters and otherwise about 40 lines around each change, for up to 25 files. A 40-file pull request is partly reviewed from the diff alone. As of 17 October 2026 the median merged pull request on CodePanel's own repository changed 434.5 lines across 12 files, which is far past the sizes studied for human review (the evidence is in best practices).
The trap. Additions plus deletions count documents, snapshots and data files. Report size with generated and lock files excluded, or the median says more about your fixtures than your changes.
Escaped defects
Definition. Defects found after merge that review could have caught. Trace each production bug to the pull request that introduced it (git blame at the fix's parent) and ask whether the reviewer saw those lines.
It matters most and is hardest to collect: it arrives weeks late, needs a judgement per bug, and get_quality does not report it. Treat it as a quarterly audit.
The trap. A defect in code the reviewer was never shown is a context failure, not a judgement failure. Sort escaped defects by whether the lines were in the reviewer's view; the fixes differ.
Finding precision
Definition. Of the findings a reviewer reports, the share that are true. Compute true positives divided by true positives plus false positives, over findings that have a verdict you trust.
CodePanel's get_quality counts only verdicts that did not come from the author agent alone. A finding counts as a true positive when a person rates it up, or when the agent disputed it and the orchestrator, rereading the code, upheld it or revised it. Counting a revised finding as right is the lenient choice: the problem was real, but the finding may have overstated it. get_quality reports disputesUpheld and disputesRevised apart under truePositives, so you can recompute precision counting only findings that were right as stated. It counts as a false positive when a person rates it false_positive or not_worth_fixing, or when the orchestrator withdrew it after a dispute. Precision comes with a 95% interval, and stays null until a row has 20 verdicts. It is broken out by model, role, severity and confidence, which is where it becomes useful. The merge gate page reports that of 72 low-confidence warnings that got a checkable verdict on its busiest repository in the 14 days to 23 October 2026, 66 were false positives. An overall figure would have hidden that; the warning/low row shows it, and the default gate no longer blocks on them.
The trap. Who gave the verdict. An author agent that fixes every finding has a perfect "rated up" rate whether the findings were right or not. CodePanel keeps those ratings in a separate unverified block and counts them toward no precision figure. If you build your own, do the same. See reducing false positives for what moves this number.
Blocking recall
Definition. Of the real defects in a set of changes, the share caught by a finding strong enough to block the merge. It differs from plain recall, which counts a catch at any severity: a defect reported as a suggestion is caught but does not stop anything.
Compute it on fixtures: pull requests where you know the defect. CodePanel's eval harness scores a defect as blocking-caught only if the matching finding is one the default merge gate blocks on. It is hard to measure on live pull requests, because it needs labels for the defects a reviewer missed, and those are rarely collected. That is what the evaluation guide is for, and the benchmarks apply it to single models.
The harness tracks it separately so a change that lowers noise by demoting findings shows what it costs. In CodePanel's own paired run on 3 November 2026 (50 bug fixtures, two repeats), putting a performance reviewer in the Tests & Maintainability seat changed blocking recall by 0 points, with an interval of -5 to +5.
The trap. An interval that wide bounds the change to about five points either way, so a three-point loss is not ruled out. Report the interval with the number.
Fix rate against dispute rate
Definition. Of the findings raised, how many the author agent fixed, and how many it disputed with a reason.
CodePanel records both. An agent disputes a finding by rating it down with a reason such as false_positive and a note citing a file and line; the orchestrator then reads the code and answers withdrawn, revised or upheld. The get_quality unverified block carries addressedByCommit (a commit edited the finding's lines), upByAgents and downByAgents.
Read them as a pair. A high fix rate with almost no disputes means good findings or an agent that does as it is told. Many disputes mostly upheld means the agent is rebutting real problems; mostly withdrawn means the reviewer is noisy. The outcome is the evidence, not the rate.
The trap. Treating "resolved" as "right". A finding that closes after a commit touched its lines was addressed, not proved correct.
Rounds to green
Definition. How many rounds before the merge gate first clears, and how often it clears and then goes red again.
get_quality reports firstGreen: the share of pull requests green after round one, the mean round at which they first went green, and how many merged without ever being green after a round. It also reports afterGreen: rounds that began on a green gate, and how many of them (reBlocked) raised a new warning or worse. The gate state is rebuilt from findings and ratings, so the red share is a lower bound.
The trap. Every push is a round. An agent that pushes after each fix pays for a round per push; fixing everything open and pushing once is the cheapest improvement to this number.
Cost per PR
Definition. What review spends on one pull request, across all its rounds. CodePanel's get_spend returns this month's cost on a repository: totalUsd, split into ownKeyUsd and creditsUsd, with a breakdown by model and role and the failed calls included. Divide by merged pull requests for a per-PR figure. A per-PR limit caps the review calls one pull request can make, and calls whose output could not be used still count toward it (budgets and limits).
The cost guide works through a full pull request, and model pricing lists the rates.
The trap. Cheapest per round is not cheapest per pull request. A cheaper reviewer with lower precision costs extra rounds, and each one is paid for twice: once for the findings, once for the fixes they trigger. Compare cost per merged pull request at equal precision.
Which three to start with
Start with finding precision, rounds to green and cost per PR: is the review true, does it converge, what does it cost. All three come from get_quality and get_spend over MCP, or GET /quality and GET /spend over HTTP (API reference). Add blocking recall when you change a model or a role. How findings are answered is in findings and feedback; how the loop runs is on the AI code review page.