Reviewing AI-generated code
What goes wrong in code a coding agent wrote, with real findings from CodePanel's own pull requests, a table of what to check, and rules an agent can paste.
Reviewing AI-generated code comes down to checking claims. Agent-written code reads fluently and passes its own tests, so its defects sit where the code's claims (comments, tests, the pull request description, assumptions about outside APIs and packages) part from what the code does. Check each claim against the code, and let the author agent answer each finding.
CodePanel reviews its own repository, where Claude Code writes the pull requests and answers the reviews: as of 17 October 2026, Claude Code wrote 147 of the 150 pull requests merged from PR 61 to PR 213, and answered the reviews on them. This page is the reviewer's side of that loop. The loop itself is in Agentic code review and Code review in a coding agent loop.
What outside research shows, and what it does not
None of three studies settles how often agent code is wrong today.
| Study | What it found | What it does not show |
|---|---|---|
| Pearce et al., IEEE S&P 2022 | About 40% of 1,689 programs, from 89 scenarios, were vulnerable | The scenarios were built around high-risk weaknesses from MITRE's Top 25, and the tool was a 2021 completion model. It is not a base rate for AI code |
| Spracklen et al., USENIX Security 2025 | Across 16 models and 576,000 code samples, the average rate of invented package names was at least 5.2% for commercial models and 21.7% for open-source ones. In a 500-prompt sample, 43% of invented names recurred in all 10 runs of a prompt | The commercial models were GPT-3.5 Turbo, GPT-4 and GPT-4 Turbo, chosen in January 2024, and the rates depend on the model |
| Watanabe et al., arXiv 2509.14745 (preprint, v3, February 2026) | 475 of 567 Claude Code pull requests merged (83.8%) against 516 of 567 human ones (91.0%) in the sample, a significant gap | One agent, 157 projects, and a preprint. A merge rate is not a quality measure |
The second row matters most for reviewers: an invented package name that returns run after run is one someone can register.
Where it went wrong on our own pull requests
Each finding below is from a pull request in CodePanel's private repository. Claude Code wrote the code, CodePanel's reviewers reported the finding, and Claude Code answered it.
The comment promised what the code did not do
On PR 203, which hides duplicate model pages from search, finding F1 read:
The hosted-copy rule never checks benchmark results, though the header comment says pages become indexable again by themselves when a result lands.
It continued: "The catalog test's expectedDuplicate omits the result check too, so nothing catches it." The code and the test shared one omission, so the test passed. Claude Code rated the finding fixed: a hosted copy with its own benchmark result is no longer treated as a duplicate.
The test suite passed. A reviewer reading the comment against the code caught it.
The same default, twice in two days
On PR 106, F4 said:
estimateCost(...) ?? 0books an unpriced model as free.
Claude Code rated it fixed. The next day, on PR 194, F1 found estimateCost(...) ?? 0 in an unrelated file, and this time cited the repository's own convention: "the pricing.json convention is that null means not applicable, never zero." The fix returned null where estimateCost does, and the table shows a dash.
A defect that comes back belongs in a rule the reviewers read on every pull request, not in whatever the author agent remembers. The last section has one.
The reviewer was wrong
On PR 71, a deploy script sent a version-override header to Cloudflare. A reviewer flagged, at low confidence, that the header format "has not been run against real Cloudflare." Claude Code disputed the finding: the value matched the form on Cloudflare's version-overrides documentation, and the pull request already called out the first live run. The orchestrator withdrew it: the header as built matched the documented form the rebuttal cited, and "A wrong format would fail safe."
A claim about an outside service gets the same treatment from either side: check it against the vendor's own page. False positives cost an agent more than a person, since an agent that trusts every finding edits working code to make it go away. Agentic code review covers how disputes are judged.
What to check in agent-written code
| Check | What to look for | How to check it from the diff |
|---|---|---|
| Claim against behavior | A comment, docstring or pull request description that says what the code does, when, or what a flag guards | Read each claim, then the lines it describes. A claim with no matching branch, or a branch the claim never mentions, is a finding |
| Tests the same agent wrote | A test whose expected value is computed with the implementation's own helper or constant, a stub that ignores its input, a test name that promises more than the body checks | Google's reviewer guidance says "Tests do not test themselves". Ask what a wrong implementation would do to this test. If nothing changes, the test is a copy |
| Assumptions about outside APIs | A header format, status code, rate limit or default taken from memory | Find the vendor's page the author relied on. The description should link it |
| New dependencies | A package in the manifest or lockfile the description does not name | Confirm the name exists on the registry, is the package meant, and is not a near-miss spelling. Check whether an installed package already does the job |
| Scope | Files changed that the description never mentions | Compare the file list with the description. Unmentioned behavior changes deserve their own sentence or their own pull request |
| Conventions the agent was told | A rule in CLAUDE.md or AGENTS.md the diff breaks |
Reviewers read those files only if the repository turns that on (off by default in CodePanel). Put checkable rules in .codepanel/rules.md |
| Fallbacks and dead paths | ?? 0, ?? "", an empty catch, a branch no caller reaches |
Ask what happens when the value is missing. A fallback that turns "unknown" into a plausible value is the ?? 0 defect |
CodePanel's starter reviewers are prompted for several of these rows. The Tests & Maintainability reviewer looks for "tests that cannot fail" and the Docs & Best Practices reviewer for "a README, guide or comment that no longer matches the code." Those are the first two rows. How often each model catches real defects is on the benchmarks page, not here.
What a second model family adds, and what it does not
The usual case for a panel is that models from different families make different mistakes. CodePanel's own eval did not detect that gain: on TypeScript pull requests from this project, with one judge, an eval-only panel with reviewers on three providers (not the starter panel) caught about as many defects as four Haiku reviewers (Reviewers). The defects above were found by reviewers reading claims against the code, and kept by an orchestrator that checks each finding against it; neither depends on the model family. CodePanel's own panel is Anthropic models, the same family as the author. A reviewer on another provider still keeps the panel running when one provider is down. One model or a panel of reviewers? covers the trade-offs, and AI code review the whole setup.
Rules an agent can paste
A defect keeps coming back when the reviewers do not know the repository's lesson. .codepanel/rules.md is where it goes, read on every review from the base branch; Writing review rules that agents follow covers how to write rules. These are aimed at agent-written code. A ## heading that is a file pattern scopes the rules under it (Repo context). Adjust the paths to your repository.
- A comment or docstring that states behavior must match the code in the same diff. Flag a comment the code contradicts or does not implement.
- A test must assert a literal expected value or one derived independently. Flag a test that computes its expectation with the same helper or constant as the code under test.
- The pull request description names every behavior change. Flag a change in the diff that the description does not mention.
- Flag a fallback that turns a missing value into a plausible one (`?? 0`, `|| ""`, an empty `catch`) unless a comment says why.
## `package.json`
- Flag a new dependency the pull request description does not name, or one that duplicates a package already installed.
## src/pricing/**
- A null price means unknown, never zero. Flag `?? 0` or `|| 0` on a price or cost. Show a dash.
Rate what comes back in the same loop: /feedback F1 up fixed --agent claude-code, or down false-positive <reason> with the file and line (Feedback and disputes).
To try it with your agent, run the Quickstart prompt and then add the rules above.