Replies to findings
When somebody replies to a finding, Limn reads the reply and says whether the finding still stands.
On this page
When someone answers one of your Limn's findings on GitHub, that answer deserves a reply, not another full review. The replies pass reads just the replies to Limn's own findings, at the current commit, and decides for each one whether the finding still stands. It is short and cheap: a smaller model with a firm cost cap.
#What it judges
refunds/policy.py:31Reply acceptedThe API rejects negative amounts before this runs: see validate_refund().
Only replies by a person on a thread or comment of one of your Limn's posted findings, newer than that finding's last judgement. Your own replies count. Replies from bots, and anything Limn posted, do not. A thread that was resolved after the reply is still judged: resolving alone does not settle a point.
Comments on other people's threads are a different pass, Check threads. See Replies pass or Check threads below.
#The outcomes
| Outcome | What it means | What happens next |
|---|---|---|
| Reply accepted | The finding was wrong, or the reply settles it. | The finding is settled for good and never raised again. |
| Fix later | The reply says it will be dealt with in later work, and that is reasonable. | Settled for this pull request. A blocker can never be deferred. |
| Pushed back | The point still stands. | The thread stays open; the next full review raises it again. |
#When it starts
By itself, when auto review is on for the workspace: after 5 quiet minutes on the pull request, at most 5 a day per pull request, within your spend budget. It never runs while a full review is running, because a full review checks the replies itself. There is no button for it.
While it runs, the AI Review tab shows Checking replies with Cancel. If it fails, the tab shows the error. Each pass is listed in Reviews on this PR as Replies checked; open it to see the judgements. It is not a review: the next full review still follows up the latest full review.
#What it posts
With posting off for the workspace, it posts nothing: each reply it would make is a draft with a Post reply button.
With posting on for the workspace, it may write:
- One pushback per thread, ever, in the model's own words, on an open thread. Limn never resolves a thread after pushing back. On a resolved thread it posts a Still open: reply instead.
- A one-sentence acknowledgement, then a resolve, for an accepted reply, where Limn is allowed to resolve: your own pull request, or another pull request after an automatic review while posting is on. Otherwise nothing is posted and the finding is settled in Limn only.
#The verdict
The replies pass can change the review's verdict, through the same rules as a full review:
- A disputed blocker can turn the verdict into Request changes.
- Once nothing blocking is left, and nothing your workspace's approval rules forbid, it can approve. The approval is held, with a reason, if a teammate's Limn has since judged one of the settled points still open.
It posts its verdict as one short review only when it rests on an automatic review and posting is on. See Verdicts and posting to GitHub.
#Replies pass or Check threads
| Replies pass | Check threads | |
|---|---|---|
| Judges | Replies on your Limn's own findings | Everyone else's threads: people, review bots, teammates' Limn |
| Starts | A reply on a Limn finding (by itself) | A new comment in an open thread; or Check threads |
| Daily limit (automatic) | 5 per pull request | 30 per workspace |
| Posts | With posting on: one pushback per thread, or an acknowledgement and a resolve | Nothing |
| Verdict | Can request changes or approve | Changes no verdict by itself |
Warning: amount can be negative here, which would pay the customer.