Our engineers use AI on more than 90% of the pull requests they merge, and they often hand whole tasks to a coding agent, such as resolving merge conflicts, fixing a failing build or addressing review comments. We're building a platform to run agents like these in the cloud, started by events instead of by a person at a laptop. As agents take on more of this work, the review before a merge decides what gets shipped, and an AI reviewer becomes one of the checks we rely on. Before letting agents act on its findings, we had to know how many of them were worth acting on.
We wanted to replace our AI code reviewer because most of us thought it was noise. Before trying alternatives, we measured it, and the data disagreed.
Its findings led to a code change 60% of the time, against 63% for human reviewers. It also flagged nearly eight times as many real bugs as all human reviewers combined. Almost a third of its comments got no change and no reply, compared with about one in eight for humans.
Our assumption was wrong, and we only found that out by measuring. Here’s the metric we used, how we checked it, and how we’re piloting the AI reviewers on our shortlist.
One metric for every reviewer
Whatever we recommended had to hold up with the engineering team, so we picked one primary metric and treated the rest as guardrails: cost per pull request had to stay below a cap, and no AI reviewer could block a merge. With one number, we’d know at the end whether the pilot worked. With several, each could tell a different story.
The metric is the actioned-finding rate: the share of findings, meaning review threads, that lead to a change before the pull request merges. A finding counts as actioned when the code it flagged, as it read when the comment was posted, no longer appears anywhere in the merged file.
A finding counts as actioned when the code it flags no longer appears in the merged fileThe metric is crude, on purpose. Flagged lines sometimes change for unrelated reasons, which inflates the rate, but the effect should be about the same for every reviewer. Because it only needs a pull request’s history, we scored our existing AI reviewer and the human reviewers on a month of past pull requests before the pilot began.
The baseline
| Reviewer | Actioned-finding rate (script) | Checked by Claude |
|---|---|---|
| Existing AI reviewer | 60% | 57% |
| All human reviewers combined | 63% | 61% |
Because decisions would rest on this number, we checked it by going through every finding from that month, with Claude reading what happened after each comment in the commits and replies. The result was close to the script’s, and most of the real bugs the AI reviewer flagged got fixed.
The human numbers need some context. People usually review after the AI reviewer has run, so the easy problems are often fixed before a person looks. We treat the human numbers as a reference point and don’t use them to rank reviewers.
We can’t say why so many of its comments went unanswered, though some people may simply distrust advice from a machine. Even if we keep our current AI reviewer, we can now tell the team how well it works, and anyone can check the numbers.
Shortlisting on paper
Running an AI reviewer on real pull requests takes weeks, and a new vendor means a security review and a contract, so we narrowed the field on paper first. We scored each tool against fourteen criteria, six of them gates: fail any one and a tool is out. The gates cover rules versioned in our repository, findings measurable from GitHub data alone, the ability to read code outside the diff, independent review lanes, restraint (quiet on a clean run, capped output, no repeats), and, if anything needs hosting, running on our cloud platform. We skipped vendor benchmarks, because a benchmark built on someone else’s rules says little about how a tool would do on ours.
Running the pilot
We first made the pilot opt-in, with a label that ran every tool on a pull request. Hardly anyone used it. Now each pull request gets one AI reviewer automatically and keeps it until it merges, which sent about eight times as many pull requests through the pilot each week as with the label. Once a pull request is approved, the author gets a one-click survey about the review.
The one-click survey that appears on a pull request once it is approvedThe line-change rule can’t tell us whether a comment was right, so a second script asks a small, fast model to judge each finding: was it addressed, and was it an important bug, a missing test, documentation, or a nitpick? When we checked a sample by hand, it agreed with us on almost all of them.
One tool in the pilot is our own: GitHub Actions jobs that call Claude with our prompts. Its instructions live as Markdown files in our repository, so changing how it reviews takes a normal pull request. Its most important rule is to comment only on things worth reading.
Learnings
Our pilot is still running, and so far the tools land within a few points of each other on the actioned-finding rate, so the choice will likely come down to cost as well as how much work a tool takes to run and tune. These lessons apply to whichever tool we end up with:
Measure the tool you already have before you look at others.
Start with a simple metric that treats every reviewer the same, and check it by hand before you base decisions on it.
Decide in advance when the evaluation ends and what each result means. Ours ends after a hundred pull requests per tool or four weeks, whichever comes first: at least 50% and ahead of the existing reviewer means we expand, 30–50% means we revise and try again, and below 30% means we stop.
Keep the scope small. Answer one question well before moving to the next.
As agents write more of our code and the tools change, we expect to run this measurement again. We built this evaluation while building Gigs OS, which made TIME’s list of Best Inventions of 2026. If you want to work on both, you can take the first step at gigs.com/careers.





