Repository navigation
CI perf (settings): 90-min merge-queue check timeout let one stuck check freeze main for 77 min #1670
Description
The merge queue's check_response_timeout_minutes is 90 (ruleset 24668460). Merge-group CI takes far less than that. So when a required check never reports, the queue stays frozen for about 1.5 hours before it gives up.
What happened on 2026-10-22. The clippy check of queue head #1653 stayed in_progress after its job and its run had both finished successfully. Details are in #1660. Nothing merged from 17:38 to 18:55 UTC. Seven green entries waited behind it, then all of them were rebuilt after the timeout evicted #1653.
Merge-group CI durations today (CI workflow, success, created to completed):
| run | PR | duration |
|---|---|---|
| 39820221573 | #1651 | 8.5 min |
| 39820490126 | #1656 | 12.2 min |
| 39822385680 | #1663 | 12.3 min |
| 39822379065 | #1642 | 13.2 min |
| 39822377061 | #1640 | 13.7 min |
| 39822369515 | #1652 | 13.8 min |
| 39822372512 | #1653 | 14.1 min |
| 39824290078 | #1667 | 14.7 min |
| 39826164711 | #1664 | 10.1 min |
| 39826160345 | #1658 | 11.3 min |
The longest is under 15 minutes.
Suggestion (needs a repo admin): lower check_response_timeout_minutes from 90 to 45. That is about 3x the slowest run above, which leaves room for a runner backlog (see #1512) and still cuts a stall like today's roughly in half. Before changing it, check that no merge-group tier runs the ~45-minute Gradle legs that scripts/merge-queue-fail-fast.py mentions. If one does, 60 is the safer value.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 22, 2026
[agent] Triaged (CI lane): p3, needs a repo admin, because the fix is a ruleset setting (ruleset 24668460) and agents don't edit rulesets.
On the question at the end: on
origin/main(8ede754), the ~45-min Gradle legs thatscripts/merge-queue-fail-fast.py:7mentions are the macOS/Windows ones. Those jobs (e2e-build-windows,.github/workflows/ci.yml:1099, ande2e-build-macos) are gated onvars.CI_SCOPE == 'full' || schedule || workflow_dispatch, so lean merge-group runs never start them. The largesttimeout-minuteson a job that does run in the lean merge group istestat 50 (ci.yml:354). Thee2e-buildjob is 45 (ci.yml:1006), and its e2e legs come after it.So 45 is fine for every merge-group run in the table above. 60 is the value that still covers the worst case where a lean job runs up to its own timeout. Either is a big improvement on 90. Leaving this for an admin.
Generated by Claude Code
mikolalysenko commentedon Oct 22, 2026
CI profiler, 2026-10-22 20:16 UTC: what the 17:38–18:55 freeze cost, from the jobs API
b12ecc1c8,clippycompletedsuccessat 17:27:37 andci-okat 17:38:21 (run 39822372512, attempt 1). The queue still removed the entry at 18:55:24, which is 90 min after the group was created at 17:24:17. A re-run at 18:40 (attempt 2,ci-ok18:48:50) didn't unstick it either.ci-okin p50 11.3 / p90 13.7 min, max 14.2 (not counting the stuck run). That is still well under the 45 min proposed above.At 45 min, this stall would have ended around 18:09 instead of 18:55, so about 46 min sooner for each of the 7 waiting PRs. The rebuild cost would have been about the same.
Generated by Claude Code