ROI

The ROI tab is the first tab of Bots Monitoring. It has one row per review bot in the workspace. Each row says how much the bot said in the window you picked, how much of it your team used, how much was left unanswered, how serious its findings were and, if you entered its price, what each useful thread cost. A verdict sums it up.

#What you see

The columns

• Threads, Comments and Comments/PR: how much the bot says, and how much it says on each pull request it comments on.
• Used: threads the team used. A thread counts as used if it was resolved, a person replied after the bot’s last comment, or a later commit touched its file. The commit test is a guess: the commit may have changed something else.
• Not addressed: still unresolved, nobody replied and no later commit touched the file. A thread counts as Unanswered only once it is older than 36 hours, so a bot is not marked down for threads nobody has reached yet.
• Merged anyway: pull requests that merged with at least one of the bot’s threads not addressed.
• Time to address, Oldest not addressed and % not addressed.
• Overlap: the share of the bot’s threads within three lines of another bot’s thread in the same file. A high figure means two bots say the same thing.
• Severity mix, High, Nits, Not addressed by severity and Inflation: from Limn’s own severity model (see Severity scoring). These columns appear once comments have been scored.
• $ per used thread: the bot’s monthly price divided by the used threads it produces in a month, at this window’s rate.

#The verdicts

• Rarely used: at least 10 threads, under 30% used, and at least half unanswered.
• Tune: at least 5 threads and under 60% used, or at least 20 scored findings of which 70% or more are nits.
• Keep: everything else.

For a priced bot marked Rarely used, a line under the table says what turning it off would save in a year (its monthly price times 12). Tuning suggestions under the table name the bot and the paths where most of its comments go unused, so you can narrow what it reviews.

#Set a bot’s price

1. Open Settings → Bots.
2. Find the bot’s card and enter its price, per month or per seat.
3. Return to Bots Monitoring. The $ per used thread column now shows a figure for that bot.

A price is set per workspace, and another workspace can hold a different price for the same bot. The two are never added together. The cost column shows only on the Bots Monitoring page, not on a single repository’s Bots tab, because one repository’s share of a workspace price is not a real cost.

#Workspace charts

At the bottom of the tab, open the charts for Thread volume (weekly, for the last 12 weeks), Bot effectiveness (the share of threads used, by verdict) and Used vs not addressed (by bot, for this window).

#Why it helps

A bot’s vendor tells you how many comments it wrote. This tab tells you how many your team used, at what price, and how much went unread. That is the figure you need when a renewal comes round. To see how the same bot does for other teams, open Benchmark.