Published: 2026-08-14
Deep dive

System design for a multi-agent PR reviewer: the five-move loop

Chapters / key moments (click to jump — plays here on the page)

This freeCodeCamp course opens by telling you to close it if what you wanted was "send the diff to an API with a prompt asking what's wrong" — there are, in the instructor's estimate, four hundred tutorials that already do that, and none of them is a production system. What follows instead is a repeatable five-move design loop that you run once per component, and the whole architecture falls out of a single word: selectivity. The valuable part for anyone building agents is that the loop is domain-independent — the PR reviewer is the worked example, not the point.

Source video

"System Design for AI Agents – Building a Multi-Agent PR Reviewer" by freeCodeCamp — Watch on YouTube →

The reframe the design hangs on

The problem is not "build an agent that reviews code automatically." It is selectivity: which findings does the agent take, and which ones does scarce human judgment get spent on? Every architectural decision downstream is an answer to that question. The motivating observation is that a senior reviewer with twenty PRs to get through does not review the twentieth the way they reviewed the first — fatigue, inconsistency, missed detail. That is the gap the system is filling, not raw throughput.

Step-by-Step Breakdown

Move 1 — Map the mess
Before any technology, document what happens today, with no agent in the picture. Remove the component and watch the human: what notifies them a PR exists (a GitHub notification, a Discord or Slack alert, someone asking directly), what knowledge base they recall, which architecture decision records and docs they open, what micro-decisions they make without noticing. Write down every one of those minute steps. Then mark which are mechanical, which need human judgment, and where the human process breaks. The instructor is explicit that there is no clever shortcut here — AI system design is manual observation work, and skipping it is why most agent architectures are guesses.
Move 2 — Define a precise trigger and a structured output
Vague triggers produce vague systems. "A claim comes in" is not a trigger; "a claim arrives at this email address carrying the keyword claims" is. For the PR reviewer, the trigger is a GitHub webhook firing when a pull request is posted, and the output is a structured review posted back. Getting the output shape explicit up front — reviewed across readability, security, test coverage and architectural fit — is what makes the rest designable.
Move 3 — Assign each step a component type
Walk the mapped human steps and give each one a mechanism. Tools, APIs and webhooks for fetching and detecting. An LLM only where you genuinely need to read unstructured input or generate language. Deterministic ML where you want a stable score rather than a plausible one. Retrieval for codebase context — not just the code touched by the diff but the code the diff will impact, pulled semantically. And an explicit human checkpoint wherever safety, financial or legal consequences land. Note the direction of travel: the four reviewer agents (security, quality and correctness, testing, documentation) come from how a human reviewer thinks in separate concerns, not from a decision to use four agents. Each finding one produces must carry a rationale and a confidence.
Move 4 — Decide how much autonomy, deliberately
Full automation is not a default you get for free. Three factors set the level. Consequence of error: a wrong style comment is annoying, a missed SQL injection is dangerous. Reversibility: an auto-posted review can be disputed and removed; a merged migration cannot easily be un-run. System maturity: new systems need more override, proven ones earn less. The stated golden rule is that anything touching financial, legal or health data does not get full autonomy, full stop — because 99 right out of 100 is worthless if the hundredth carries an enormous consequence. Start with more human involvement than you think you need and reduce it as the system earns trust, rather than removing it after an expensive mistake teaches you to.
Move 5 — Assume everything breaks
For each component ask one question — what can go wrong — from two directions. The engineering direction: GitHub retries the same delivery, someone who is not GitHub sends you a request, you have ten seconds to acknowledge and your agent takes ninety. The LLM direction: the model hallucinates, the retriever pulls the wrong slices. Sort what you find into a 2×2 of what you know you don't know, what you would recognise on sight, and what you have never considered. Then repeat the entire loop for the next component — trigger, orchestrator, retriever, events, gates — and by the fifth pass the architecture has written itself, fault tolerance included.
Only then build, under a harness
Coding starts after the design, and runs inside what the course calls a genesis ritual: context and invariants written down, an explicit definition of done, exactly what each milestone outputs, five gates, and an independent verifier that checks whether the work was actually done. The loop then drives the agent against that spine. The framing is that you are directing the agent rather than depending on it to think — you keep the harness and stay able to say what your agent is doing at any point. What ships is a baseline to extend: parallel multi-agents, a human approval queue, a full trace viewer and a cost dashboard showing the real economics of the LLM calls.

Common Errors & Fixes Covered

Error: hallucination — the model states something false in a place that matters

Why it happens: Inherent to the mechanism. The instructor's line is that hallucination is a feature rather than a bug, but that the consequence is severe when it lands in a security finding.

Fix: Design against it structurally rather than hoping: require a citation on every finding, add a fact-check layer, attach a confidence score to each output, and route anything below a threshold to a human reviewer.

Error: model drift — strong on your eval set, degrading in the world

Why it happens: The world moves after you build. The canonical example is spam detection: the system was excellent when trained, then attackers changed strategy and the distribution shifted underneath it. The LLM-era version is the same shape, expressed through prompts.

Fix: A monitoring dashboard, alert thresholds, periodic retraining and periodic prompt updates, plus a rules-based fallback for the common cases.

Error: tool or API timeout takes the whole system down

Why it happens: A single hard dependency — an embedding provider going down, GitHub's API or webhooks unavailable — propagates into total failure when nothing catches it.

Fix: Timeouts and retries, graceful degradation on partial data, a circuit breaker on a dead service, and the ability to switch to an alternative embedding model. Components should fail gracefully rather than stopping the system.

Error: webhook delivery assumptions that don't hold

Why it happens: Three separate problems named for the same component. GitHub retries the same delivery, so naive handling double-processes. Requests can arrive that did not come from GitHub. And the acknowledgement window is about ten seconds while the review agent needs roughly ninety.

Fix: Verify the request signature, make delivery handling idempotent so a retry is harmless, and acknowledge immediately while processing asynchronously.

Error: orchestration deadlock between parallel agents

Why it happens: Two agents run in parallel and a merger combines their results. If one returns nothing, the merger gets a single input; if both fail, it gets none. Either way the merger fails and the pipeline dead-ends.

Fix: Design the merger for missing inputs explicitly rather than assuming both branches deliver.

Error: feedback-loop poisoning

Why it happens: Two failure directions. The agent ignores feedback and never improves; or it absorbs bad feedback — a very junior reviewer's poor-quality correction gets stored and learned from as if authoritative.

Fix: A minimum evidence threshold before stored feedback is acted on, and decay for old feedback that is no longer helping, so stale embeddings stop steering the system.

Error: the human escalation queue exceeds human capacity

Why it happens: A human checkpoint is only real if a human can reach it. One reviewer cannot process a thousand escalations a day, so an escalate-everything design silently becomes an ignore-everything design.

Fix: Queue prioritisation against business importance, and genuine capacity planning — decide what volume of escalation your humans can actually absorb and design the thresholds to hit it.

Error: the "almost right" problem — 90% correct, 10% subtly wrong

Why it happens: The hardest failure to notice, because the output looks right. Nothing errors, nothing alerts.

Fix: Flag low-confidence output for review, run random audits on a regular cadence, rotate reviewers and swap models, and deliberately test with new and wrong inputs to see whether performance holds.

Gotchas & Caveats

Never post a security vulnerability publicly in an open-source PR comment. Called out explicitly as a design requirement, not an afterthought: if the agent finds a serious vulnerability on a public repository, it must not describe it in the open PR where anyone can read and exploit it. Route it privately — a direct Slack message or equivalent — behind the scenes.
The five moves run per component, not once for the system. Trigger, orchestrator, retriever, events and gates each get the full loop. Running it once at the top level is the shortcut that produces the architecture you were trying to avoid.
The autonomy spectrum has more than two settings. Full automation for routine, reversible, low-stakes work. Human-reviews-output where reputational stakes exist — the system drafts and waits. Human-handles-exceptions, where the system takes the easy cases and anomalies or low-confidence results go to a person; this is the mode chosen for the PR reviewer. Human-decides-system-prepares, where the system gathers all context and scores and a person makes the call — the standard shape for lending and other financial decisions. And fully human with AI assistance.
Every finding carries a rationale and a confidence, or the selectivity design collapses. Confidence is what the routing threshold reads; rationale is what lets a reviewer check the agent rather than trust it. They are load-bearing, not metadata.
Needing a lot of human involvement early is a good sign, not a failure. The instructor's argument is that it forces you to watch your agent consciously, and you then reduce involvement as the system matures — which is cheaper than discovering the right level through an expensive mistake.

Key Takeaways

Start from the human process, not the tech stack: observe and write down every micro-decision a senior reviewer makes, including the unconscious ones.
A precise trigger and an explicit structured output are what make the middle of the system designable at all.
Component type follows from the job — LLM for unstructured language, deterministic ML for stable scores, retrieval for impacted code, human checkpoint for consequential calls.
Set autonomy by consequence of error, reversibility and system maturity; financial, legal and health domains never get full autonomy.
Sweep every component for failures in both the engineering and the LLM direction, and sort them by what you know you don't know versus what you have never considered.
Each named failure mode has a designed mitigation — citations and confidence thresholds for hallucination, circuit breakers for dead services, idempotency and async acknowledgement for webhooks, evidence thresholds and decay for feedback.
Build under a harness with invariants, a definition of done, per-milestone outputs, gates and an independent verifier — so you are directing the agent rather than depending on it.
The loop generalises: point it at an incident agent or any other domain and the same five moves apply.