We measured three AI code review bots across 315 pull requests

Trying a code review bot is easy: install it, read what it writes on two pull requests, say "not bad." Measuring one is hard, because only you know whether what it found was really a bug, and you never know whether what it missed was there at all. On August 30 we added a third bot to our repository, and from that day on three bots reviewed the same pull requests, at the same time, under the same conditions. Thirty-five days later we had 315 pull requests with bot activity, 1,070 inline findings and one question: which of them finds the critical bugs?

The short answer: CodeRabbit in absolute numbers, Seer in precision per finding. But that answer held for the first half of the measurement, not the second. After day seventeen CodeRabbit's review capacity shrank, it stopped looking at half the pull requests, and in that period Seer overtook it on critical findings. Sourcery's low volume was not a matter of looking badly; it was a matter of not looking at more than half the pull requests at all. The long answer follows, including the two places where the measurement fooled us.

The first version of this article was published on Medium with the data from the first 18 days. Here it is extended to 35 days with the same method, which is why the numbers differ from that version.

The setting

The repository is a multi-tenant operations platform on Next.js and PostgreSQL: a customer portal, invoicing, card and balance payments, domain and server orders, an accounting integration, and an AI agent that manages servers. Some of the bugs here are denominated in money, others in tenant boundaries. That is the context in which we judged whether a bot's "critical" was really critical.

Within the window, 315 pull requests with bot activity were opened and 293 commits reached the main branch. The median pull request changes 533 lines (the mean is 5,238, because a few are enormous), and the median pull request goes from open to merge in 86 minutes. The pull requests were written by one developer and an AI coding agent; alongside the bots there is an internal review chain (a per-task agent review and a whole-branch review before merge). The bots sit next to that chain, not in place of it.

The three bots: CodeRabbit (added on August 30, which is where the window starts), Sentry's Seer (its pull request review feature) and Sourcery. All three run automatically on every pull request. Quotas and review limits vary by plan tier; the limits quoted below are the ones we saw on ours.

Method

The data came from the GitHub API: every pull request's inline comments, review rounds and conversation comments, plus the check runs on each pull request's final commit. We separated the bots by user name and classified every comment by the bot's own markers.

A finding is an inline comment that the bot itself marks as a finding. CodeRabbit puts a category and severity header on each one ("Functional Correctness | Major") and a hidden marker in the body. Sourcery labels each comment issue (bug_risk), issue (security), suggestion and so on. Seer opens each finding with Bug: and assigns a level such as Severity: HIGH. Bulk lists inside summary comments were not counted; CodeRabbit's summaries also carry 66 nitpicks and 78 "outside the diff" items, which are outside this number.

Fixed means the bot's own stamp. All three bots mark their own comment when a push containing a fix arrives: CodeRabbit writes "Addressed in commits", Seer "Resolved in", Sourcery "Addressed in". We did not mark anything by hand. The fix counts are therefore a lower bound: a fix that lands in another pull request, or one the bot cannot match, is not counted.

Reviewed means the bot actually reviewed that pull request. Appearing on it is not enough; a bot that left a quota or limit notice and moved on counts as not having looked. Because Seer leaves no comment on a clean run, we read the pull requests it reviewed from the check runs.

For critical, the bots' scales do not line up: CodeRabbit almost never uses its top level, Seer's CRITICAL means something else, Sourcery has no severity scale. So we ran a second measurement: we read, one by one, the fixed findings on the money and identity paths (billing, payments, accounting, authentication, authorization) and counted them by our own definition. Our definition of critical: money moving wrongly, a tenant boundary crossed, consent and legal records, data loss, outage. That count is subjective, which is why we give it as an approximation.

The limits are plain: one repository, one team, 35 days, the bots' own stamps, our own reading of critical. This is a field measurement, not a benchmark.

Some things are deliberately outside the measurement. Semgrep, GitGuardian and Dependabot also run on the same pull requests, but they are not part of this comparison, because they answer a different question. The three bots ask "is this change wrong?" and answer by reading intent; Semgrep looks for a known bad pattern, GitGuardian for a secret, Dependabot for a dependency with a known vulnerability. They are rule-based and deterministic, and their output is a pass/fail gate; their silence really does mean clean, within the reach of their rules. Measures such as "finding count", "fixed stamp" or results that change from round to round do not fit them. The two classes are layers, not rivals: one of CodeRabbit's "critical" findings said that a private key embedded in a test file was failing the Semgrep gate; it carried the scanner's finding along with its context.

The numbers

All 35 days:

Severity mix, with the bot's fixed stamp in parentheses:

In its own taxonomy CodeRabbit labelled 223 of its findings functional correctness, 159 data integrity, 123 security and privacy, 75 stability, 49 maintainability and 6 performance.

Note the rejection counts: low for all three, but not equal. We measured and rejected 12 of Seer's 235 findings in the thread (5 percent), 7 of CodeRabbit's 635 (1 percent), and 1 of Sourcery's 200. Eight of Seer's rejected findings fall in the second period; we describe one of them below.

Two periods: how a plan limit reordered the ranking

We first wrote up the measurement at the end of the first 18 days. Over the next 17 days the picture changed, and what changed it was not the bots' intelligence but their capacity.

The pull request cadence was the same in both periods: about nine a day. What changed was CodeRabbit's capacity. In the first days the limit notice said "you've used all 8 included reviews currently available" and reopened two to four minutes later; it appeared on six pull requests, and on five of them the review continued shortly after. After September 17 the number of included reviews in the notice dropped to two or four, and the waits we saw grew to 24 to 51 minutes. In the second period 103 of 143 pull requests got a "Review limit reached" notice, and CodeRabbit never looked at more than half of them (74). The notice does not retry by itself; when capacity opens you have to trigger it by hand, and @coderabbitai review treats that commit as already reviewed and does nothing, so it takes @coderabbitai full review. Every push also consumes capacity; for a team that pushes three or four rounds to one pull request in response to bot findings, the allowance runs out fast.

The result: the bot that led on critical findings by about a factor of two in the first period fell behind Seer in the second. Neither the quality of the code nor the bot's ability changed; half the pull requests went through unreviewed. Over the same period the median time to merge rose from 59 minutes to 250. Part of that is our rule of rebasing and rerunning CI before every merge, part is waiting for review capacity; we cannot separate the two. Past its limit CodeRabbit offers usage-based billing at $0.25 per file; we did not enable it.

Seer's precision dropped in the second period too. Its fix rate on merged pull requests fell from 81 percent to 69, and the findings we explicitly rejected in the thread went from 4 to 8. One of the rejected ones was labelled CRITICAL: it said a Zod schema required fields that CPU and memory data do not have, so parsing would always fail. In the schema those fields were .nullish(); we measured against the real data the provider returned that day, and parsing passed. Another said next/dynamic does not accept a component directly; Next's own loader explicitly handles both forms. In the first period Seer's severity label was a reliable signal; in the second, acting on a HIGH without measuring it first would have been wasted work. Sourcery behaved the same in both periods: accurate where it had budget, an 80 percent fix rate on merged pull requests in the second period, but unable to look at 85 of 131 pull requests because of its budget.

The lesson: the ranking of the first 18 days was a measurement, not a property. A review bot's value is its ability times the share of pull requests it actually looks at; the second factor depends on the plan tier and on your own push habits, and it changes silently.

Who found the critical bugs

We read the fixed findings on the money and identity paths one by one, for both periods:

• CodeRabbit: roughly 31 critical (25 in the first period, 6 in the second).
• Seer: roughly 19 critical (12 and 7).
• Sourcery: roughly 13 critical (10 and 3); some of them are security findings outside the money and identity paths (cross-tenant memory leakage, the shell separator).

So over the full 35 days CodeRabbit leads in absolute count. Seer leads in density: the highest share of critical per finding, and a 75 percent fix rate. Sourcery, where it looked, sits just behind Seer in critical findings per finding; its problem is coverage. And the table above shows that this ranking is not fixed.

The cases tell it better than the counts. All of them were fixed; none of them reached production.

The card was charged, and nothing was recorded. In the saved-card payment flow, the card is charged at the provider and then the payment row is written to the database. If the write failed, the finally block released the charge lock, and what remained was neither a record nor a retry path: money taken from the customer, no trace of it in the system. Seer marked this CRITICAL. In the same file CodeRabbit found something else: an expired lock handle could release a newer lock acquired after it. Two bots saw two different holes in one module.

The same service was invoiced twice. When a customer with a managed contract also buys a self-invoicing service, the contract renewal cron has to exclude that service's line item. While a new service type was being added, the exclusion query had not been updated; Seer caught it in the renewal route. This is the "sibling forgotten" class of bug: a fix applied in one place, the second place carrying the same rule skipped.

Absence of consent was treated as consent. The code that sends ad conversions sent them whenever marketing consent was not false. null, meaning no decision had been recorded, passed that condition. Seer marked it HIGH; for us it is critical, because it is a legal record. Another from the same class arrived in the second period: because the order route did not pass the request's language along, a customer who accepted the English consent text had their consent recorded with a Turkish language stamp. All three bots found that one independently.

The confirmation screen showed the list price while the discounted amount was charged. In the domain order flow, after a discount is applied the charge is made at the discounted amount, but the pricing object returned to the customer carried the values computed before the discount. We measured a divergence of up to ninety-nine percent between the amount shown on the confirmation screen and the amount charged. CodeRabbit found this in its first round, labelled Major.

A refunded invoice opened a paid period. In the path that resumes suspended services when a payment arrives, the status of the pending invoice for corporate e-mail and traffic-management services was read first, and the service was updated by a separate write afterwards. If that invoice was refunded between the two steps, the service still resumed, and a paid period was recorded against a refunded invoice. CodeRabbit found it in both services and asked for the read and the write to happen in one locked transaction; Sourcery separately showed another race on the pending renewal marker in the same file.

A plan document was running a migration against production. An implementation plan contained the step "run drizzle:migrate to verify"; the top of the same document said that this machine's database connection is production. CodeRabbit put the two paragraphs side by side and gave it a Major. Not a code bug, but an instruction that would have cost more than one had it been followed.

The hole in the shield. We have a scan test that prevents identity rows (user tables carrying the 2FA secret) from leaking to the client. Sourcery showed that the test's regex only matched .select().from(users) separated by whitespace, and that an intervening comment or a schema alias walked past it. The shield existed; it could be walked around.

Cross-tenant memory leakage, failing open. In the memory layer of the server-managing agent, if the query that reads the company's server inventory failed it returned an empty list, and the filter read the empty list as "no scope" and injected other servers' memories into this server's task. One of Sourcery's issue (security) findings, fixed. Another from the same class: the URL grammar for a curl command executed over a remote shell accepted &; in a shell, & is a command separator.

We write in commit bodies whose finding we are closing. Of 293 commits, 90 credit CodeRabbit, 60 credit Seer, 42 credit Sourcery.

Silence means three different things

The most useful result of this measurement is not a bug count but a behavioural difference.

Seer says nothing when it is clean. Seer did not appear on 194 of 315 pull requests. The check runs showed "No issues found" on 179 of them and no run at all on 15. On 54 of those 179 clean pull requests CodeRabbit had a fixed-stamped finding; on 36 of them, Sourcery did. Seer saying "clean" does not mean the pull request is clean; it means Seer found nothing in that round. We saw more than once that re-triggering it on the same commit produced new findings; on the ad-conversion pull request, two of four Seer rounds carried findings and two were empty.

Sourcery never looked at 166 pull requests. Most of Sourcery's review comments were this sentence: "this account has used its review budget of 2,500,000 diff characters for the last 7 days." A few pull requests also hit the single-PR size limit (500,000 diff characters). So the Sourcery that appeared on 302 pull requests actually reviewed 136. On 31 of those 136 it posted no finding; on 5 of those 31 CodeRabbit, and on 5 Seer, had a fixed-stamped finding. Sourcery's real weakness is coverage, not accuracy. Because the budget is a rolling seven-day window, a large pull request eats the following days' reviews as well.

CodeRabbit's silence comes in two kinds. On 57 of the 233 pull requests it reviewed it left no finding; on 12 of those 57 Seer had a finding, on 8 Sourcery did. That is the most reliable silence of the three. But on 78 pull requests the reason for the silence was not cleanliness but capacity: a limit notice, a draft pull request, or one not targeting the main branch. The notice is visible on the pull request but easy to miss, and a pull request that looks reviewed but was not is the most expensive silence of all.

The count of pull requests where only one bot found anything summarises the table: CodeRabbit 44, Sourcery 20, Seer 16. CodeRabbit and Seer together carry most of the coverage; but when CodeRabbit's capacity shrank, the pull requests where Sourcery or Seer alone found something went up.

The measurement's own mistakes. Our first reading said "Sourcery did not hit its quota in this window." Our classifier was searching for the string "rate limit"; Sourcery says "review budget." A second mistake the same day: Sourcery's security findings begin with 🚨 issue (security), the emoji did not pass the regex, and those findings went uncounted. When you measure a bot's absence, search for the bot's own refusal sentence; when you count findings, read the bot's current format. We got both wrong and corrected them. The numbers in this article are the corrected ones.

Character portraits

Seer: a nose for money. Where it is best is state machines: partial success, retries, idempotency, two currencies added together by accident. It gave three consecutive HIGH findings in the split-payment logic of the accounting integration, and all three were real. It also found a pre-existing bug that was not in the diff; we say so explicitly because some bots only look at changed lines. Its weaknesses: a slow queue (p90 of 74 minutes, close to our median time to merge), silence when clean, different results from round to round on the same commit, and visibly more false positives in the second period. It barely looks at test files; of its 176 fixed-stamped findings, two are in a test.

CodeRabbit: volume and breadth. It looks outside the diff (78 "outside the diff" items), reads documents, plans and runbooks (101 of its 386 fixed-stamped findings are under docs and scripts), and verifies its claims by running scripts against the repository; under many of its comments there is a "Supported by static analysis" block with the rg commands it ran. It tags security findings with CWE identifiers. It has the fewest false positives of the three. The price is reading volume, and capacity: it can only produce that volume on the pull requests it actually reviews. And a warning about labels: Minor sometimes carries money. Code that read only the first page of the accounting API and never saw an open invoice on the next page, and two parallel --execute processes applying the same delta twice, drifted between Minor and Major. Read the body, not the label.

Sourcery: the shield's shield. It is the best of the three at finding gaps in regression and scan tests and at finding swallowed errors (catch(() => {}), an empty catch block, a cron whose alarm never fires). Its issue (security) class carries real findings: cross-tenant memory leakage, the shell separator, an unaudited IDOR attempt. In 35 days we rejected exactly one of its findings in the thread. Its Reviewer's Guide arrives within seconds and is a good map for starting to read a pull request. The price is the budget.

What it costs

What we pay the three bots per month: CodeRabbit $60, Sentry $79, Sourcery $30; $169 in total. Sentry's fee covers the whole error-monitoring platform, and Seer review is one part of it; its cost per finding is therefore an upper bound. Plan tier and seat count change the price; these are our own invoices. The 35-day window is 1.15 months, and the fees are prorated accordingly.

CodeRabbit is still the cheapest per finding and per critical, but in the first 18 days it was $1.44 per critical; when capacity shrank, the same fee bought fewer pull requests. Sourcery looks cheapest per pull request, but $30 bought 136 of 302 pull requests; when the budget runs out it is not money that is missing, it is the look. More coverage means a higher plan or usage-based billing; either decision should be made against what a bug the bot finds would cost us. Seer's $4.78 per critical counts the rest of the Sentry bill as zero; divided by its real share it would be lower.

All three together: $195 for 35 days, roughly 63 critical findings, roughly $3.10 per critical. The cost of one unrecorded card charge, or one invoice issued twice, is a multiple of that.

What we do now

Five rules came out of this.

1. Two bots are mandatory. CodeRabbit and Seer together carry most of the coverage. Sourcery is a third pair of eyes while it has budget.
2. Track capacity as a metric. Count each week how many pull requests each bot actually reviewed. Do not treat a pull request with a limit notice as ready to merge; trigger full review when capacity opens, and if it does not arrive, merge knowing that.
3. On money and identity pull requests, trigger Seer once more. Even when the check run says clean. The second round sees what the first did not. But do not fix a HIGH or a CRITICAL without measuring it either.
4. Do not read silence as clean. It means three different things across three bots: a budget or limit refusal, a clean run, never ran. Only the check run and the bot's own message tell them apart.
5. The bots do not replace the review chain. There were bugs our internal chain found that all three bots missed, and bugs the bots found that the internal chain missed. Both are necessary, neither is sufficient alone.

And one point of merge discipline: not a single finding in the window arrived after a merge, because we wait for the bots. Given Seer's p90, that sometimes means more than an hour. An hour's wait is cheaper than an unrecorded card charge.

Repeat it on your own repository

The measurement is a few hundred lines of script and the GitHub API.

1. Choose the window: from the first bot's first comment to today. If it spans weeks, split it into periods; a single average hides a break like the one we saw.
2. For each pull request, fetch pulls/{n}/comments (inline), pulls/{n}/reviews, issues/{n}/comments and the final commit's commits/{sha}/check-runs.
3. Separate comments by the bot's user name. Recognise findings by the bot's own marker: for CodeRabbit the cr-indicator-types marker in the body and the severity header, for Sourcery the **issue (...)** label (allow an emoji prefix), for Seer Bug: and Severity:.
4. Read fixes from the bot's stamp; do not mark them yourself. Keep pull requests that are not merged yet out of the fix rate.
5. Search for a bot's absence with the bot's own refusal sentence: "review budget", "Review limit reached", "Draft PR not reviewed", "No issues found" in the check run.
6. Join with pull request metadata: lines changed, opened and merged times. Findings per 1,000 lines and latency come from there.
7. For severity, do not trust the bots' scales; write your own definition of critical and read the findings.

The number that will help you most is probably the one that helped us: the coverage asymmetry, which bot never looked at which pull request. Saying "this bot finds less" without that number is doing the measurement in the bot's place. And that number changes over time; measuring it once and stopping would be mistaking our first 18 days for a permanent ranking.