TL;DR: AI writes code faster than teams can read it, and it fails in its own way — hallucinated imports, happy-path-only logic, and security holes hiding under clean, confident-looking code. Review AI-generated PRs against the ticket and the runtime, not against how plausible the diff looks: run a fixed checklist, verify every API call exists, trace the unhappy paths, and confirm the change actually works when the app runs. That last step is where screencli comes in — it opens your preview deploy on every PR and clicks through the change before your review starts.
Why reviewing AI-generated PRs is different
The volume went up and the failure modes changed at the same time. Teams with heavy AI adoption merge about 98% more pull requests while PR review time climbs roughly 91%, according to Faros AI telemetry across 10,000+ developers — so reviewers see more, larger diffs with no author to ask about intent (see the AI code review bottleneck for the full data).
The harder problem is what you're reviewing. Human bugs are usually random. AI bugs are systematic and wear a disguise: the code reads idiomatically, passes shallow tests, and looks like diligence. Veracode's 2025 GenAI Code Security Report, which tested over 100 models across 80 tasks, found 45% of AI-generated code introduced an OWASP Top 10 vulnerability — and security pass rates have stayed essentially flat near 55% across successive model generations, even as syntax correctness climbed past 95%.
So the reviewer's job shifts. You're not just judging quality; you're actively hunting for confident-looking mistakes a human wouldn't make.
The AI-generated PR review checklist
Run these seven checks in order. The early ones are cheap and catch the most common failures, so a bad PR fails fast.
| # | Check | What you're looking for | Time |
|---|---|---|---|
| 1 | Scope | The diff changes only what the ticket asked for — no unrequested "improvements" or stray files | 30s |
| 2 | Real APIs only | Every import, method, and config key exists in the exact version your lockfile pins | 2m |
| 3 | Unhappy paths | Empty input, null, one-item lists, unicode, network/timeout failures are handled | 3m |
| 4 | Test honesty | Tests would actually fail if the feature broke — not assert True or self-fulfilling mocks |
2m |
| 5 | Security paths | Auth, crypto, input handling, and secrets reviewed line by line, not skimmed | 3m |
| 6 | Reuse & fit | No duplicated utilities, no over-engineered abstractions, matches existing patterns | 2m |
| 7 | It actually runs | The change works when a user touches it — the flow completes, nothing unrelated broke | — |
1. Scope: does this PR do only what was asked?
AI over-builds. Open the linked ticket first and confirm the diff maps to its acceptance criteria — nothing more. Extra files, speculative config, and "while I was here" refactors are a flag, not a bonus. If they're there, ask why or request they move to a separate PR.
2. Real APIs only: does everything it calls actually exist?
This is the most AI-specific failure and the easiest to miss. Models invent methods that look right and import packages that were renamed or never existed. Roughly one in five AI-suggested dependencies in Python and JavaScript is a non-existent package name — a slopsquatting risk, since attackers register those hallucinated names as malware. Verify every new import and method call against the version in your lockfile. Jump to the definition; if it isn't in the type defs or source, you've found a hallucination.
3. Unhappy paths: what breaks at the edges?
AI-generated code handles the common case and assumes the edges away — one analysis attributes about 60% of AI code faults to silent logic failures that pass tests but break on unusual inputs. Mentally trace the three most likely edge cases: empty input, null, a single-element list, unicode, a network call that fails halfway. If the PR doesn't handle them and the tests don't cover them, flag it.
4. Test honesty: would these tests fail if the code were wrong?
Self-confirming tests are a signature AI pattern. When a model writes the code and its tests in one shot, the tests exercise the paths the model imagined and skip the cases it got wrong. Watch for hardcoded expected values that happen to match output, assert True, tests that assert only "no exception thrown," and tests that verify a mock instead of real behavior.
5. Security paths: read auth and crypto line by line
A Stanford study (Perry et al.) found developers using AI assistants wrote less secure code while being more confident it was secure — the exact trap a reviewer has to counter. Treat any security-adjacent change with extra suspicion: string-concatenated SQL, weak hashes (MD5/SHA1), missing input validation on user-facing endpoints, hardcoded or logged secrets, and overly permissive CORS or file permissions. Never trust AI on authentication or authorization.
6. Reuse and fit: is this the simplest change that fits the codebase?
AI is great at generating new code and poor at reusing existing code. GitClear's analysis of 211 million changed lines found copy-pasted code rose from 8.3% to 12.3% while refactoring dropped sharply. Search for an existing utility before accepting a new one, and cut abstractions and error handling written for impossible cases.
7. It actually runs: does the change work when a user touches it?
The diff can read perfectly and still ship a dead button. This is the check a text-based review structurally cannot make — covered next.
The failure modes to hunt for
Most AI-generated bugs fall into five recurring classes. Naming them makes them easier to spot:
- Hallucinated APIs and packages — calls to functions or imports that don't exist; the names look exactly real.
- Plausible-but-wrong logic — reads idiomatically, solves a subtly different problem (pagination that breaks past page 10, a retry loop that never backs off).
- Silent edge-case errors — happy path works; empty, null, duplicate, or timeout inputs misbehave.
- Scope creep — changes beyond the task that look like diligence.
- Self-confirming tests — the model tests its own assumptions, so green means little.
A formal-verification study (arXiv, April 2026) drives the point home: 55.8% of AI-generated artifacts contained a formally proven vulnerability, and standard SAST tools missed 97.8% of them. Tellingly, the same models identified their own vulnerable output as insecure 78.7% of the time when asked to review it — the flaw is in unreviewed generation, which is exactly why the review step matters.
What a checklist can't catch: behavioral failures
A reviewer reading a diff — human or AI — is reasoning about code that has never run. You can verify every import exists, trace every edge case on paper, and still not know whether the button works, whether the flow completes, or whether the change quietly broke an unrelated screen. Static AI reviewers share the same ceiling: they inspect code, they never execute it.
That's the gap behind the scariest AI bugs. Plausible-but-wrong logic and silent edge-case failures are, by definition, the ones that read fine and only reveal themselves at runtime. The only way to close that gap is to run the change and watch what happens — see AI code review that actually uses the app and how to test AI-generated code before you merge it.
Where screencli fits in your review
screencli is the reviewer built for that last check. You install the screencli GitHub App and pick which repos it watches. When a PR opens, screencli reads the diff, plans the browser checks that matter for that change, runs them against your preview deployment (Vercel, Netlify, Cloudflare Pages — it reads the URL from your GitHub deployments and never touches production), and posts a pass/fail comment with a recording — before your review even starts.
You don't write test files. The AI decides what to exercise based on what changed, then shows you a video of the flow completing or the exact step where it broke. Pair it with your checklist and a static reviewer: the checklist and the static tool catch code-level issues, and screencli catches the behavioral failures they can't see.
Install the screencli GitHub App and your next pull request gets an AI-driven browser test on your preview deploy — automatically.
Frequently asked questions
How do you review an AI-generated pull request? Review it against the ticket, not against how confident the code looks. Run a fixed checklist: confirm scope matches the ticket, verify every import and method call exists in your locked versions, trace the unhappy paths, check that tests would actually fail if the feature broke, scan security-sensitive paths line by line, and confirm the change behaves correctly when the app actually runs.
Why is reviewing AI-generated code harder than reviewing human code? The surface is polished but the failure modes are different and hidden. AI code reads idiomatically and passes shallow tests, yet 45% of it introduces an OWASP Top 10 vulnerability and security pass rates have stayed flat near 55% across model generations. Reviewers also face more, larger PRs with no author to ask about intent.
What should I look for first in an AI-generated PR? Scope and reality. Confirm the PR changes only what the ticket asked for, then verify every dependency and method call exists in the version your lockfile pins — roughly one in five AI-suggested packages is a non-existent, potentially typosquatted name.
Can automated tools review AI-generated code for me? Partly. Deterministic gates and AI reviewers catch style, obvious bugs, and known vulnerability patterns cheaply and should run before a human looks. But static tools read code without running it, so they miss behavioral failures — a broken button, a dead flow, a regression elsewhere. Behavioral review that runs the app closes that gap.
What are the most common AI code failure modes to check for? Hallucinated APIs and packages, silent edge-case errors, scope creep, self-confirming tests, and plausible-but-wrong logic that reads correctly but solves a subtly different problem.
Last updated 2026-07-13. Sources: Veracode — 2025 GenAI Code Security Report · Cloud Security Alliance — AI codegen vulnerability debt · GitHub Docs — Review AI-generated code · Sentry — Reviewing AI-generated code playbook · Faros AI — The AI Productivity Paradox. Figures are the latest available as of publication; exact ratios vary by team, language, and methodology.