Building your own AI code review agent: a sandbox and a webhook first, then fix what goes wrong with prompts, a historical PR test set, model inversion and cheaper models

The starting point

1. Get a sandbox (several vendors sell them, or use Firecracker).
2. Put an agent in it. He recommends a model-independent open-source harness such as OpenCode or Pi rather than being tied to Codex or Claude Code.
3. Connect GitHub or GitLab so a pull-request webhook sends each new PR to the agent.
4. Prompt it to review the PR and post feedback back through the platform's review API.

That works, and feedback arrives in about five minutes. Then five things go wrong.

What goes wrong, and his fixes

• It misses bugs, and nitpicks. Fix the prompt: name specific bug types (build the catalogue from your own past incidents and outages) and tell it exactly what a nitpick is so it skips them. Nitpicks matter because coding agents that read review comments tend to act on all of them, which he says produces needlessly defensive code. A small prompt change can swing the number of comments a lot, and a reviewer can show bugs present but never absent.
• You cannot tell if it got better. Keep a dataset of historical PRs that caused outages and measure how often each version of the reviewer catches them. He says only about one in a thousand PRs has a catastrophic bug, so rare misses go unnoticed for a long time, and the dataset has to keep updating because agents start failing in new ways.
• Same-model blind spots. He reports that Claude models are roughly three times more likely to produce off-by-one errors than GPT models, while GPT models are more likely to produce auth bypasses. Review code with a different model family from the one that wrote it ("model inversion"); he reports a 10 percent higher catch rate. To know which model wrote a PR, look for fingerprints such as a codex/ branch-name prefix or a "Co-authored-by: Claude" line, which he says identify the author about 80 percent of the time; the rest needs heuristics or telemetry.
• Missing context. Grep-and-read-files is imprecise and expensive. Build an internal knowledge base that catalogues the code by area (for authentication: how it works, the flows, a diagram, every file involved), and add tickets and docs from Jira, Linear, Confluence or Notion for intent. Greptile calls its version a living wiki.
• Code that runs wrong though it reads right. Test end to end in a sandbox that spins up your stack with the PR's dependencies, using a vision model to click through documented flows. You need a dev box, a simple build script and a map of flows such as sign-up and auth.
• Cost. He says teams see $3 to $25 per PR, and that even after all the improvements it can still cost $3 to $20. Use frontier models for the critical steps only (about 20 percent of Greptile's share) and small models for context finding and exploratory reads; Greptile uses 20 subtasks, each with its own model, and he thinks a simple swap gets 70 to 80 percent of the way. A cheap decision model can re-rank which context to load into the expensive cache.

Gotchas & Caveats

• The statistics (1 PR in 1,000, three times, 10 percent, 80 percent, costs) come from Greptile's own data and experience and are not independently checked.
• The talk ends by saying the alternative to building is to use Greptile's product, so the advice is partly a pitch.
• Swapping in smaller models without a good dataset can quietly make the reviewer worse, and you would find out only when it misses an incident.
• No code or configuration is shown.

Key Takeaways

• Build a test set from your own incidents before tuning prompts, or you cannot tell whether a change helped.
• Review with a different model family from the one that wrote the code.
• Define "nitpick" in the prompt, because downstream coding agents will otherwise act on every comment.