Repository navigation
Regression threshold against a tag or branch #569
Description
Thanks for working on this tool. It is awesome :)
This is a feature request.
The regression threshold is nice.
I am worried about regression slippage across a series for commits.
Each commit might only introduce an 1% regression ... but when compounded they can result with significant regression, that are not detected by the automatic PR threshold.
I know that setting up the "long term" comparison target is not that easy as comparing with the main branch.
But maybe this can be implemented by adding a separate regression comparisong against a predefined branch.
The simplest option is comparing with an arbitrary other branch or tag.
Developers can change their release process, to create a branch or tag called stable and then the PR regressions with compare with main branch but also with this other stable branch.
Another option is to look at all the tags, extract semantic versioning or calendar versioning from then and compare based on latest tag.
Read The Docs, does something similar based on semantic versioning to show the documentation for the latest stable version.
Thanks again!
Activity
art049 commentedon Oct 7, 2026
Hi, thanks for the kind words.
Just to make sure I understand correctly, you'd want to have a second comparison target in the PRs that would be the previous release? (or simply an older ref than the base of the branch)
I think we couldn't really derive a status check on top of it though. Because you might have a 20% regression on a bench that is planned and until you released a new version all the pr checks would fail.
Except maybe if we move the acknowledgement we currently have to also support those new comparisons.
adiroiban commentedon Oct 7, 2026
Hi. Yes.
As an example. Version 1.0 was released in July 2026, and we will use it as the baseline.
Over the months of August and September development work was done for a future release 2.0 , targeted for October.
All the PRs for the development of release 2.0 have an 1% regression.
The thing is that when we aggregate those 40 PRs, we can end up with a big regression.
Well... if there is a 20% regression for a PR on a planned work, that work needs to be fixed before the future release :)
I fell that the "acknowledgement" is something that should be considered an exception and used only when hardware is changed or compiler is changed.
You want to fix the regression and the performance issue in the first place, rather than ignoring it.