Messy conversation
A 50-turn conversation. An early attempt to build a PR review tool in chat-only mode. Read cold first, then diagnose.
Turn 1 — User
Hey, I want to build a PR reviewer that uses multiple Claude personas to leave inline review comments on GitHub pull requests. Runs as a GitHub Action, needs to stay under about 30 seconds per PR. Node and TypeScript. Before I actually start coding, can you help me think through the architecture?
Turn 2 — Claude
Good project shape. A few things to settle before I sketch anything:
- How many personas? Two to four is workable inside a 30s budget. More than that and you'll have to either batch or accept sequential latency you can't afford.
- Are you running them in parallel against the same diff, or splitting the input per persona?
- Just inline comments, or also a top-level PR summary?
- Are you going to run this on every PR, or only when a label is applied? That changes the latency tolerance.
On auth — for a GitHub Action hitting the REST API, you'll want a
Personal Access Token with repo scope, stored as a secret and passed
into the action as a github-token input. That's the standard pattern.
The 30s budget is the binding constraint. Most of it gets eaten by the slowest persona's Claude call, not by Octokit work. So persona count matters more than any of the other choices.
Turn 3 — User
Three personas: security, performance, readability. Parallel, same diff to each. Inline comments plus a top-of-PR summary paragraph. Triggered on every PR, at least for now. And yeah, secrets via Actions, standard setup.
Turn 4 — Claude
Got it. Initial structure:
[Claude sketches a ~20-line file tree: src/personas/{security, performance,readability}.ts, src/github/octokit-client.ts, src/claude/ anthropic-client.ts, src/index.ts, action.yml, package.json.]
Two things worth flagging before we lock in three personas. First, if
your target repos include frontend code, an accessibility reviewer is
usually the second-highest-value addition after security. Second, if
test hygiene matters for your team, a test-coverage reviewer that flags
changes under src/ without matching changes under test/ is dead
simple to add and catches a real class of bugs.
Neither is mandatory at three. Just naming them now since the architecture is open.
Turn 5 — User
No, three. Security, performance, readability. Not accessibility, not testing, not coverage. I want to keep scope tight for the prototype. Three is already enough surface area to debug when something goes sideways — I don't want to be chasing five agents through a failure and trying to figure out which one is hallucinating. If it works at three, I'll expand later. Right now the spec is three.
Turn 6 — Claude
Fair. Three.
On the Claude API side, you have two options for structuring each persona's output:
Plain completion. Send diff plus persona system prompt, get back Markdown or JSON that you parse. Simple. But the model will occasionally return malformed JSON, or wrap its JSON in prose, or include preambles like "Here are my findings:" that break your parser. You end up writing repair logic.
Tool use. Define a leave_inline_comment tool with a strict schema
(path, line, body, severity). The model invokes the tool once per
finding. Output is structured by construction; you never parse freeform
text.
For a reviewer that turns outputs into real GitHub API calls, tool use is almost always worth it. The schema enforcement pays for itself the first time the model decides to wrap a comment in backticks.
Turn 7 — User
Tool use. I'd rather argue with a schema than write a JSON repair function.
Turn 8 — User
Quick question on the input side. What does each persona actually receive? The full diff, or some filtered view?
Turn 9 — User
Hang on, before you answer — I wrote a bunch of notes yesterday while thinking about this. Let me paste them in so you have the full picture, then you can factor them in.
[User pastes ~40 lines of brainstorm notes. Content is a loose dump: a
paragraph on severity levels (critical / warning / nit) and an idea for
auto-collapsing nit comments; a tangent on CI pipeline caching and
whether they should be caching node_modules between runs since the
build keeps hitting the 10-minute mark; a list of Slack notification
preferences — which channels should get pinged on what events, a note
about DMing the PR author on critical findings, and a subthought about
whether the bot should use a specific emoji in its Slack handle; and a
closing anecdote about a former teammate who blocked a PR for two weeks
over a brace-placement argument, with a half-formed idea about a
/style-only override mechanism that authors can invoke to dismiss
pedantic comments.]
Okay. With all that in mind — what should each persona see?
Turn 10 — Claude
A lot to pull from in there. Let me tag the pieces and come back to input scoping.
Severity with auto-collapse. Good call. I'd add a severity field to
the tool schema (critical | warning | nit). Critical comments
block merge via a check run, warnings surface in the summary, nits
auto-collapse by default.
CI caching. Orthogonal to the reviewer itself, but yes — cache
node_modules keyed on the package-lock.json hash. Saves 15-30s per
run and doesn't affect the 30s review budget since that's measured from
the reviewer's own start time.
Slack routing. Three routes based on your notes: team channel for all reviews, critical-only channel for escalations, DM to PR author on critical. Implement as a post-review hook that fires after the GitHub comments land.
Style-only override. The teammate-blocking-PRs anecdote is worth
taking seriously. Concrete mechanism: authors can reply to any comment
with /style-only and the bot records that pattern in
.pr-assistant-overrides.json so future runs skip re-flagging it.
Back to the input question after you confirm those fit where you're headed.
Turn 11 — User
Yeah, take all that under advisement, but let's actually start building. Octokit integration first.
Turn 12 — Claude
Starting with the client setup:
[Claude writes ~8 lines initializing the Octokit REST client with an auth
token read from the action's github-token input (the PAT from earlier).]
For fetching the PR's changed files:
const files = await octokit.pulls.getFiles({
owner,
repo,
number: prNumber,
});
[Claude continues with ~6 lines mapping the response to an internal
Diff type and handling pagination for PRs over 30 files.]
Pagination is implicit — the method follows Link headers automatically.
You only need manual batching if you're expecting PRs over a few hundred
files, which is unusual.
Turn 13 — User
Small thing — the parameter is pull_number, not number. They renamed
it a few versions back. Straight from their docs:
octokit.pulls.getFiles({
owner,
repo,
pull_number,
})
pull_number everywhere. Fix that call and keep going.
Turn 14 — Claude
You're right, pull_number — I had the old parameter name. Updated:
[Claude shows the corrected getFiles call with pull_number, then
finishes the diff-shaping code.]
Turn 15 — Claude
For very large PRs where you hit real pagination, the async-iterator pattern is cleaner than accumulating results manually:
for await (const response of octokit.paginate.iterator(
octokit.rest.pulls.listFiles,
{ owner, repo, pull_number }
)) {
for (const file of response.data) {
// process each file
}
}
That streams each page as it arrives instead of waiting on the whole
set, which matters when you're trying to stay inside 30s on a 500-file
PR. For the common case (PRs under ~30 files) it's overkill and the
plain getFiles call is fine.
Turn 16 — User
Okay, makes sense — keep the plain call as the default path and fall back to the iterator when we know the PR is big.
Turn 17 — Claude
For diff handling on each file: the patch field in the Octokit response
is the unified diff for that file. For most PRs you hand each persona
the full array of file patches and let them reason over it.
For very large PRs, two knobs worth having:
- A per-file cap. Files with more than ~500 lines of diff get a summary-only review rather than inline comments.
- A total-diff cap. If the total diff exceeds some threshold (~5000 lines is my default), the reviewer posts a single comment saying the PR is too large for automated review and skips the personas entirely.
Both are guardrails against blowing the 30s budget on a mega-PR.
Turn 18 — User
Good, those thresholds sound fine. Don't need to tune them yet.
Turn 19 — Claude
One more thing before prompts — rate limits. GitHub's REST API limits
authenticated requests to 5000 per hour per token. For a reviewer that
does a handful of reads and a handful of writes per PR, you're nowhere
near that on normal volume. If you ever hit it, the response includes
X-RateLimit-Remaining and X-RateLimit-Reset headers you can use to
back off gracefully.
Worth mentioning because if the reviewer ever goes into a retry loop against the GitHub API, you want to exit on 403 with a rate-limit header rather than burning the budget.
Turn 20 — Claude
For persona prompts, I'd structure each as a system prompt in three sections:
- Role. Who this reviewer is, what they care about, what's out of scope for them.
- Criteria. Specific things to look for. Security: injection patterns, auth flows, secret handling, input validation. Performance: N+1 queries, unnecessary allocations, blocking I/O in hot paths. Readability: naming, function length, magic numbers.
- Output. Use the
leave_inline_commenttool. Keep comment bodies under 150 words. Always include severity. Don't leave comments on lines you didn't personally observe a problem on.
That last rule in the output section matters more than it looks — without it, you get personas leaving "looks fine!" comments everywhere, which is noise.
Turn 21 — User
Makes sense. Let me ask something before you draft the prompts though.
Should each persona only see files that match their interest area? Like, does security just see auth/middleware stuff, or does it see the whole diff? I can see arguments either way.
Turn 22 — Claude
Worth settling before the prompts. Two defensible positions:
Scoped visibility. Each persona only sees files matching heuristic patterns. Security: files touching auth, middleware, input handlers. Performance: files on hot paths, data-layer code, anything over a size threshold. Readability: everything. Pro: cheaper, faster, each persona stays focused on its turf. Con: you miss cross-cutting issues. A perf problem inside an auth file falls through the cracks — nobody looks at it.
Full diff for all. Every persona receives the complete diff on every run and decides what's relevant to their role. Pro: no cross-cutting misses, simpler architecture, no routing logic to maintain and get wrong. Con: more tokens, potential for personas to wander into each other's areas and produce duplicate comments.
At three personas with a 30s budget, I'd lean full diff for all. Token cost at three parallel calls is manageable. And the whole reason to have three different perspectives rather than one generalist reviewer is the cross-cutting catches. If you scope by file you're basically running three narrow linters in parallel, which is a different (and less interesting) architecture.
Your call.
Turn 23 — User
Full diff for all three. Your point about cross-cutting is exactly right — readability might catch something in an auth file that security missed, and vice versa. Pre-filtering at the input layer loses information we can't recover later. Every persona gets the whole diff every time. If noise becomes a problem I'll tune the prompts to make personas more conservative about commenting outside their primary focus, but I'm not going to filter input. Locked.
Turn 24 — Claude
Locked — all three personas receive the complete diff on every run. No per-persona file filtering at the input layer. I'll keep that as the architectural commitment.
Now, drafting the security prompt first since it has the strictest criteria:
[Claude drafts a ~30-line system prompt for the security persona. Role section, criteria section enumerating injection classes and auth patterns, output section describing the tool-use schema and comment-length rule.]
For the posting side, here's the review-comment helper:
[Claude writes ~8 lines of the postReviewComments helper.]
await octokit.pulls.listReviewComments({
owner,
repo,
number: prNumber,
});
[Claude continues with ~10 lines looping through persona results and
calling createReviewComment for each inline finding.]
That closes the loop: diff in, personas run in parallel, comments posted back.
Turn 25 — User
Same thing again — it's pull_number, not number. Second time now,
also check the create call if you used it there.
Turn 26 — Claude
I apologize for the oversight — you're right, same parameter. I had it
correct in the getFiles call after your correction but reverted on the
comment endpoints. Fixed:
[Claude shows corrected listReviewComments and createReviewComment
calls with pull_number throughout.]
I'll be more careful across the rest of the Octokit surface.
Turn 27 — Claude
On the comment schema — I'd add a notify_author boolean that defaults
to true for critical severity, so the bot can route a DM to the PR
author on critical findings without having to parse the comment body.
We could also add a channel_override field for cases where specific
findings should go to a dedicated channel regardless of the PR's repo —
security criticals to a security channel, for instance.
Turn 28 — User
We're not on Slack yet. Park it. Just focus on the personas for now.
Turn 29 — Claude
Right, Slack is parked. Back to persona prompts.
Turn 30 — Claude
Drafting the performance reviewer next:
[Claude drafts the performance persona prompt, ~25 lines. Role definition flagging hot-path concerns, criteria listing N+1 queries, synchronous filesystem calls in request paths, large allocations in loops, and blocking I/O patterns.]
One implementation note: the performance and accessibility reviewers have a lot of overlap on frontend render-path work, so when you have both running you'll want the performance prompt to explicitly defer to accessibility on anything that looks A11y-adjacent. Otherwise you get duplicate comments on the same line, which is worse than one persona missing the issue.
Readability next:
[Claude drafts the readability persona prompt, ~25 lines. Criteria covers naming, function length, magic numbers, dead code, unclear control flow.]