Benchmark
How your bots compare with the same bots in similar-sized repositories, and when Limn refuses to compare.
On this page
The Benchmark tab of Bots Monitoring compares each of your review bots with the same bot running in other teams’ repositories of a similar size. It answers questions your own data cannot: is CodeRabbit this noisy for everyone, or only for us? Do other teams act on more of what Copilot says? Where a fair comparison is not possible, Limn says so and shows no number.
#Where the comparison comes from
The comparison data ships inside Limn. It was measured from about 2,200 other repositories and is grouped by bot and by repository size (how many pull requests a repository merges in 14 days). Limn places your repositories in those groups on your computer. Nothing about your repositories is sent anywhere to make the comparison.
Each figure is measured the way the comparison data measures it, which is not always the way the ROI tab does. For example, “Acted on” here is counted differently from Used on the ROI tab, so the two need not match. The ⓘ button beside the heading, How Benchmark works, lists how each figure is measured and its limits.
#What you see
On the Bots Monitoring page there is one card per bot, for the whole workspace.
Your team used 52% of what it said. Teams with similar repos used 74%.
| Repository | Size group | Acted on |
|---|---|---|
| acme/api | Band 3 of 10 | 25% |
| acme/payments | Band 7 of 10 | 59% |
| acme/web | Band 6 of 10 | 52% |
- Cost: the price you entered, then Per merged PR and Per comment your team used, each marked with where it comes from, for example “From your own data”.
- One plain sentence comparing you with similar teams, for example “Your team used 52% of what it said. Teams with similar repos used 74%.”
- Where these numbers come from: one row per repository the bot works in, with its size group and your team’s acted-on rate there.
A repository’s own Bots tab shows the full detail for that repository: each figure against the spread of similar repositories, and any finding that stands out.
Findings that stand out
Limn raises a finding only when a figure is far from typical and the gap is large, so a healthy repository is not flagged just for being at one end of the range. The findings are:
- “Comments far more than it does for other teams”
- “Your team uses far less of it than other teams do”
- “Someone gets to it far later than on similar repos”
- “Repeats another bot on the same lines”
Each finding comes with a suggested action. No AI writes them.
#When Limn refuses to compare
A comparison against the wrong group is worse than none, so Limn shows a reason instead of a number. The reasons you may see include:
| What it says | What it means |
|---|---|
| We have never measured this bot | The comparison data has no record of this product. |
| Too few teams run this bot to compare against | The group is too small to be fair. |
| No team we measured runs this bot on a repo this size | There is no group of your size for it. |
| This repo is too new to compare | Limn has watched it for under 14 days, so its figures would be too low. |
| Nothing merged here recently | There is no recent work to measure. |
| Price entered, but the seat count is missing | A per-seat price needs the number of seats to give a monthly cost. |
Severity and category comparisons are not offered yet, because the comparison data has no severity scores. The ⓘ button lists them as not compared yet, rather than showing your figures against nothing.
#What you can do
- Open Bots Monitoring and choose Benchmark.
- Enter each bot’s price in Settings → Bots to see the cost lines.
- Open a repository’s page and its Bots tab for the per-repository detail.
#Why it helps
Your own figures on ROI cannot tell you what “good” looks like. Benchmark gives you a fair reference, and refuses where the reference would mislead.