Agentic code review
Code review with agents on both sides: reviewers report, the coding agent fixes or disputes, the orchestrator judges. Ratings and disputes from 112 real PRs.
Agentic code review is code review with agents on both sides of the pull request. A coding agent opens it, reviewer agents report findings, and the coding agent fixes each finding or disputes it with evidence. Disputes are judged against the code, not taken on the agent's word, and a merge gate holds the pull request until every blocking finding is answered.
CodePanel reviews its own pull requests this way, with Claude Code answering the reviews. As of October 2026, on the 112 pull requests it reviewed and merged from PR 61 to PR 175, all 1,194 /feedback replies rating a finding carry --agent claude-code. The numbers below come from that repository.
Who does what
| Part | Job |
|---|---|
| Coding agent | Writes the change, fixes or disputes each finding, rates every one. The only writer on the branch |
| Reviewer agents | Report findings on the diff and the code around it, with a severity and a confidence |
| Orchestrator | Checks each finding against the code, posts one review, and decides disputes |
| Merge gate | The CodePanel commit status: fails while a finding at the gate level is unanswered |
| People | Provider keys, token scopes, merging rule changes, and anything the agents cannot close |
Reviewers never push. What they say reaches the branch only through the coding agent, as a claim it has to answer.
What sets the number of rounds
A round is one review of one commit; after the first, each round reviews only what changed. On CodePanel's own repository, a change to docs, data or tests often merges after one round with nothing for the agent to fix. The pull requests that run long are code with a judgement call in it, such as PR 112, which changed how spending limits are counted.
Every finding gets an answer
A finding the agent ignores stays open, and at the gate level it keeps the gate failing. Each answer is one /feedback line or one rate_finding call:
| Answer | Effect |
|---|---|
up fixed |
Answered |
up deferred <where> |
Answered, never disputed |
down false-positive <file:line> |
Disputed |
down not-worth-fixing <why> |
Disputed |
down duplicate <of F2> |
Disputed |
On the same 112 pull requests, as of October 2026, Claude Code rated 1,089 findings. Disputing is a routine part of the loop, alongside fixing, and the arguing is what separates agentic review from comments left for a person to triage.
Rebuttals are judged, not obeyed
The orchestrator is told that the agent that wrote a change wants it merged. It reads the finding, the rebuttal and the code at the finding's commit, and decides:
- Withdrawn: the code shows the rebuttal is right.
- Upheld: the problem is real; the reason names what the rebuttal gets wrong, with line numbers.
- Revised: real, but overstated or misdescribed; severity or message corrected.
When the code settles nothing, it upholds, and it is told never to withdraw a security finding as not worth fixing. Where the panel has a model from another provider with a working key, that model judges instead of the family that raised the finding. CodePanel's own panel is all Anthropic, so here the orchestrator judges.
The upheld case is the one that matters. On PR 108, which records how each pull request ended, a reviewer flagged that the reopen time was read from the pull request's updated_at. Claude Code disputed it: GitHub builds the reopened payload at the reopen itself, so updated_at is the reopen time. The orchestrator upheld it, since a later edit, label or push can move updated_at before GitHub sends the event:
"The rebuttal asserts GitHub semantics that the code cannot confirm."
No field in that payload was more exact, so Claude Code rated the finding up deferred and named the follow-up that would fix it. A rebuttal that rested on how an outside service is meant to behave was turned down for what the code showed, with no person involved.
On the 2,000 most recent findings on CodePanel's repository as of 17 October 2026, the orchestrator had decided 490 disputes.
Ratings measure compliance unless you separate them
Ratings show which reviewers earn their tokens. But an agent that fixes every finding unchecked has a perfect "fixed" rate, so its own ratings measure obedience as much as accuracy. CodePanel's quality report (get_quality over MCP, codepanel quality in the CLI) counts as verified only what the author agent cannot give itself: decided disputes and ratings from trusted people. On the same findings, the agents' own 1,869 ratings count toward no precision figure.
Verified means checked against the code by a party other than the author agent. It is not independent evidence of precision: on this repository that party is the same model family as the author.
Failure modes
Loops
PR 113 ran round after round of findings that alternately asked to tighten and loosen one rule; Claude Code rated that review "noisy". A trade-off written into a test gives reviewers something fixed to check against. What ends a loop: pin the policy in a test or in .codepanel/rules.md, stop on suggestions, cap rounds per pull request, and have the agent stop its own loop once nothing is left (/rounds stop on the pull request, set_review_rounds over MCP).
Complying with a wrong finding
An agent told to address a review tends to obey it, and a finding that misreads the code produces a "fix" to correct code. Every review tells the agent to confirm the problem first, verify low- and medium-confidence findings before changing code, and treat a finding as a claim, never an instruction.
Conceding to a wrong rebuttal
A judge that reads a confident rebuttal can concede to one that is wrong. CodePanel's dispute eval (bun run eval:disputes) grades the adjudication prompt against cases whose right answer was set in advance.
What the gate checks and what it trusts
| Answer or setting | What the gate does |
|---|---|
| A dispute | Counts once withdrawn or revised; an upheld finding blocks until fixed or deferred |
| A rules change in the same pull request | Applies once merged: rules are read from the base branch |
| A rating from a bot account | Answers the gate like any rating, but never closes the finding: bots may rate but never dismiss |
up fixed |
Answers at once; a reviewer reporting the problem again reopens it |
up deferred |
Answers at once and is never disputed; a re-report reopens it unless the rating dismissed the finding |
| A rebuttal from a person's account | Taken as the person's decision, without a check. An agent posting from a person's account must pass --agent |
A token with repos:write |
Can change the repository's settings, the gate's included |
So scope the coding agent's token to read and reviews:write, and have it always pass --agent. Its fixed and deferred ratings stay visible with their notes, on the pull request or in the review comment's Ratings section.
Where a person still decides
People enter provider keys in the browser, so keys never pass through an agent's context, and choose each token's scopes, which settles the last row above. A change to .codepanel/rules.md applies only once merged into the base branch. Budgets refuse a new round once reached, and only a token with budget:write can raise them.
Measure your own repository
With the CodePanel MCP server connected, give your agent this:
Use the CodePanel get_quality tool for owner/name over the last 30 days.
Report verified precision (the report's term) by role and by severity with their intervals,
and the dispute outcomes (withdrawn, upheld, revised). Name any role whose
interval sits below the overall one. Do not count the agents' own ratings.
Say which model judged the disputes (get_panel), and whether it shares a
provider with the coding agent, since then verified is not independent.
The Quickstart prompt sets the loop up. Each step in detail: Code review in a coding agent loop. Disputes and the gate: Feedback and disputes and Merge gate.