Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-23 20:16 UTC)
Window: 2026-10-22 20:16 → 2026-10-23 20:16. 298 workflow runs (all listed, 3 pages). Since the 16:16 run: 2 merge_group CI runs, both green; 2 PRs merged (#1695, #1680), 2 push-to-main CI runs, 6 full PR runs (one a re-run attempt) plus 5 stubs. No nightly, no dispatch, no failures, 0 evictions.
Depot slow starts are back: 48% of jobs this slice
- 94 of 197 jobs (48%) waited over 15 s between runner start and their first step (up from 4% in 12:18–16:16). In merge_group run 39998858554 (Fix yarn installs told to upgrade with npm (#1629) #1680), all 26 sampled jobs were slow (30–93 s), and the Gradle e2e leg's
Run e2e teststook 374 s vs 244–264 s on fast hosts. That run'sci-okcame at 10.03 min, vs 7.35 for Fix duplicate references request on hosted rescans (#1606, #1607) #1695's run (39998197443) 5 min earlier. One push run'srelease-readinesswaited 300 s for its first step (40000300562). - Queue wait itself stayed at 0.02 (max 0.08), so this is host speed, not capacity. Tracked in #1512; its daily fresh-numbers comment lands at the 00:16 run (last one 10-23 00:18).
Critical path after #1697
- #1697 (open, Ready for review,
ci-perf) splits the Gradle e2e leg. In its PR run 40005592193 the two new legs took 3.77 and 2.73 min, vs 4.58–4.92 min for the combined leg on fast hosts this slice (7.40 on a slow host). Once it lands,coverageis the whole critical path: 5.80–6.17 min on 16-core Depot, of whichRun tests with coverageis 324–345 s (n=5 this slice). Shaving 2 more minutes would mean splitting coverage; not filed yet (no concrete split with an evidenced saving).
Sample: all 298 runs in the 24h window listed (3 pages). Jobs (filter=all) for all 9 non-stub CI runs since 16:16. 1 job log (coverage tail; report section only). 2 PR timelines. 1 ruleset read. ~20 REST requests. GraphQL is blocked here.
| metric | 24h |
|---|---|
| PRs merged | 20 (4 on 10-22 20:16–00:16, 1 at ~01:00, 13 in the 10-23 14:38–14:57 burst, 2 at 17:57 / 18:04) |
| Merge queue enqueue → merge | since 16:16: 8.1 (#1695), 10.75 (#1680), 0 evictions. Earlier slice: p50 12.8 / p90 15.0 (n=13, burst) |
merge_group created → ci-ok (success) |
7.35 / 10.03 (n=2; 10.03 on slow hosts). All with #1682 (n=5): 6.50, 6.73, 7.35, 7.58, 10.03 |
PR run created → ci-ok |
p50 6.8 / p90 9.3 (n=3 full first-attempt runs: 6.57, 6.80, 9.90) |
push to main created → ci-ok |
4.72 / 7.52 (n=2; both slow-start, 7.52 includes a 300 s host start) |
| Nightly | 10-23 failed (carried). Fixes #1686 and #1690 merged; the 10-24 nightly is the first test |
| Linux queue wait p50 / p90 | Depot 0.02 / 0.02 (n=197, max 0.08); start-up gap over 15 s in 48% (94/197) |
| Windows queue wait p50 / p90 | 0.02 / 0.03 (n=2 windows-compile-check) |
| macOS queue wait p50 / p90 | 0.03 / 0.05 (carried; no macOS jobs since 08:16) |
| Job-min/day (Linux / Windows / macOS) | ~3.0k / 0.20k / 0.13k (est.: 10-22 16:16–20:16 freeze slice (~2.97k L / 0.135k W) rolled out of the window; +322 L / 5.5 W measured since 16:16) |
Merge-queue settings (ruleset 24668460, re-read this run): build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH. Unchanged.
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-23 20:16 | 20 PRs merged (2 since 16:16: #1695, #1680; 0 evictions, 0 mg failures; the 10-22 freeze aftermath rolled out of the window) | 8.1, 10.75 since 16:16 (n=2) | 7.35 / 10.03 (n=2; 10.03 on slow hosts) | 6.8 / 9.3 (n=3 full runs, 6.57–9.90) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n=197, max 0.08) | ~3.0k / 0.20k / 0.13k (est.; since 16:16 mg 96 L / 5.5 W in 2 runs, push 32 L in 2 runs, PR ~194 L in 5 full runs; Depot slow-start 48% (94/197), mg #1680 100%) |
| 2026-10-23 16:16 | 41 PRs merged (13 since 12:18, all 14:38–14:57 in one burst; 0 evictions, 0 mg failures) | 12.8 / 15.0 since 12:18 (n=13; first 5 entries 8.1–8.9, entries 6–12 12.2–17.4 waiting 4.3–9.4 min for a build slot) | 7.68 (n=13, p90 8.13); after #1682 6.50 / 6.73 | 8.3 / 9.1 (n=5 full runs, 6.97–9.07) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n=603, max 0.22) | ~5.6k / 0.33k / 0.13k (est.; since 12:18 mg 509 L / 32 W in 13 runs, push 92 L in 9 runs, PR ~88 L in 5 full runs; Depot slow-start 4% (26/616)) |
| 2026-10-23 12:18 | 28 PRs merged (0 since 00:56; queue empty; no nightly or dispatch in slice) | none since 00:56; 24h 133 / 175 carried | none since 04:16 (last 7.9) | 7.7 / 8.1 (n=4 full runs since 08:16, 7.30–8.12) | 0.03 / 0.05 (carried) | 0.02 / 0.03 (Depot, n=116, max 0.05) | ~5.1k / 0.30k / 0.13k (est.; since 08:16 PR ~130 L in 4 full runs + 7 stubs; Depot slow-start 0% (0/116)) |
| 2026-10-23 08:16 | 28 PRs merged (0 since 04:16; queue empty; nightly failed on Windows shards + gem old-Bundler legs, fixes #1686/#1687 open) | none since 04:16; 24h 133 / 175 carried | none since 04:16 (last 7.9) | 7.8 / 10.9 (n=6 full runs since 04:16, 6.58–10.85; 6.58 on #1682's branch) | 0.03 / 0.05 (nightly + dispatch, n=58) | 0.02 / 0.03 (Depot, n=655, max 0.03) | ~5.2k / 0.35k / 0.18k (est.; since 04:16 PR 339 L, nightly 246 / 75 / 57, CI dispatch on #1686 251 / 79 / 69; Depot slow-start 2% (14/655)) |
| 2026-10-23 04:16 | 28 PRs merged (1 since 00:16: #1675; 0 evictions, 0 mg failures) | 8.45 since 00:16 (n=1); 24h 133 / 175 carried | 7.9 (n=1, fast host) | 8.0 / 9.6 (n=5 full runs since 00:16, 7.73–9.57) | 0.05 / 0.06 (carried) | 0.02 / 0.02 (Depot, n=183, max 0.03) | ~4.6k / 0.26k / 0.08k (est.; since 00:16 PR 174 L in 5 full runs, mg 42 L / 2.3 W in 1 run incl. fail-fast watcher, push 8 L; Depot slow-start 6% (10/183), all in one PR run, mg 0/30) |
| 2026-10-23 00:16 | 27 PRs merged (4 since 20:16: #1646, #1668, #1673, #1674; 0 evictions, 0 mg failures) | 8.7–10.2 since 20:16 (n=4); 24h 133 / 175 carried | 8.2 (n=4, 8.10–9.47) | 10.4 / 16.2 (n=7 full runs since 20:16) | 0.05 / 0.06 (carried) | 0.02 / 0.02 (Depot, n=440, max 0.08) | ~4.4k / 0.26k / 0.08k (est.; since 20:16 PR 433 L, mg 179 L / 11 W in 4 runs, push 23 L; Depot slow-start 46% (195/425), mg 66%) |
| 2026-10-22 20:16 | 23 PRs merged 16:50–20:02 (queue resumed after 20.9 h quiet); 3 evictions with a known cause (stuck check #1670, lost Depot runner #1512, yarn corepack fetch #1671) | 133 / 175 first enqueue (n=15; 11–43 before the 17:38 freeze) | 11.3 (n=36, p90 13.7) | 12.8 / 13.1 (n=5 full runs since 16:16, range 9.97–13.25) | 0.05 / 0.06 (carried) | 0.02 / 0.02 (Depot, n=1,706, max 0.45) | ~4.0k / 0.25k / 0.08k (est.; since 16:16 mg 2,433 L / 135 W in 49 runs, ~1.0k (39%) wasted: 632 in 13 cancelled/failed runs plus 369 in 7 green runs discarded after the freeze; PR 287 L; push 249 L in 17 runs; Depot slow-start 56% (971/1,741)) |
| 2026-10-22 16:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (20.3 h quiet; 25 non-draft PRs open, 24 Ready for review) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 12:20: 9.1 / 10.5 (n=4, range 7.80–10.82; slow hosts make the Gradle e2e leg the critical path) | 0.05 / 0.06 (carried) | 0.01 / 0.02 (Depot, n=116, max 0.02) | ~2.2k / 0.16k / 0.08k (est.; PR 170 L in 4 full runs since 12:20 (33.2 / 42.4 / 47.9 / 46.9); Depot slow-start 44% (51/116), 59% (51/87) since 13:59, max gap 60 s) |
| 2026-10-22 12:20 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (16.4 h quiet; 22+ non-draft PRs open, 20 Ready for review) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 08:16: 7.35 / 7.67 / 8.13 (n=4, fast hosts; coverage 5.75–6.16) | 0.05 / 0.06 (CI dispatch, n=4) | 0.01 / 0.01 (Depot, n=207, max 0.03) | ~2.2k / 0.16k / 0.08k (est.; PR 132 L in 4 full runs since 08:16; CI dispatch on ci-janitor/uv-settings-test-windows 85 L / 47 W / 53 M, cancelled; Depot slow-start 2% (2/116), max gap 18 s) |
| 2026-10-22 08:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (12.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 04:16: 7.28 / 7.83 / 8.17 (n=3, fast hosts) | 0.03 / 0.05 (10-22 nightly) | 0.01 / 0.02 (Depot, n=178, max 0.02) | ~2.0k / 0.11k / 0.03k (est.; PR 95 L in 3 full runs since 04:16; nightly 10-22 failed (Windows shard 1, #1663) 87 / 61 / 32; Depot slow-start 0% (0/87)) |
| 2026-10-22 04:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (8.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 00:16: 7.73 (n=1, fast host; coverage 5.98) | 0.04 / 0.05 (carried) | 0.02 / 0.02 (Depot, n=29, max 0.02) | ~2.0k / 0.10k / 0.04k (est.; PR 1 full run ~33 L + 10 stubs/audits since 00:16; Depot slow-start 0% (0/29)) |
| 2026-10-22 00:16 | 7 PRs merged in the window (18:59–19:57 on 10-21); 0 merge_group runs since (4.3 h quiet) | 20.9 / 70.2 (n=7, carried) | 9.0 (n=8, carried) | since 20:16: 9.01–11.23 (n=4; 1 Setup PHP failure, re-run passed at 14.13) |
0.04 / 0.05 (carried) | 0.02 / 0.02 (Depot, n=145, max 0.33) | ~2.1k / 0.10k / 0.04k (est.; PR 252 L in 4 runs since 20:16; Depot slow-start 65% (94/145)) |
| 2026-10-21 20:16 | 7 PRs merged 18:59–19:57 (queue reopened at 18:47 after 23.7 h idle; all 7 had been evicted in the 10-20 stall, so ~26 h of extra latency each; 1 lost-Depot-runner eviction (#1593), 1 real test failure (#1632)) | 20.9 / 70.2 today's first enqueue (n=7) | 9.0 (n=8, p90 10.8) | 11.0 / 11.8 (n=10 since 16:16, range 7.92–13.85; slow hosts) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot, n=755, max 0.9) | ~2.0k / 0.10k / 0.04k (est.; since 16:16 mg 595 L / 43 W in 13 runs, PR 520 L, push 66 L; Depot slow-start 47% (353/748), 19h 52%) |
| 2026-10-21 16:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (21.2 h quiet; 23 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried); since 12:22: 8.98–10.52 (n=4, exact; coverage 7.6–8.9 on slow hosts) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot; since 12:22 max 2 s, n=145) | ~7.3k / 0.49k / 0.12k (mg carried; PR +195 L in 5 runs since 12:22; Depot slow-start 25% (36/145), one run 62% (18/29)) |
| 2026-10-21 12:22 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (17.3 h quiet; 22 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried); since 08:16: 7.88 (n=1, exact) | 0.04 / 0.05 (10-21 nightly, carried) | 0.02 / 0.02 (Depot; since 08:16 max 1 s, n=29) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried; PR +33 L in 1 full run since 08:16 plus 8 stubs/audits; Depot slow-start 0/29 in the new run) |
| 2026-10-21 08:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (13.2 h quiet; 21 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried, n=14); since 04:16: 7.28, 8.43 (n=2, exact) | 0.04 / 0.05 (10-21 nightly, green, 23.3 min) | 0.02 / 0.02 (Depot; since 04:16 max 1 s) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried; PR +66 L in 2 full runs since 04:16 (32, 34); 10-21 nightly 85 L / 54 W / 39 M; Depot slow-start 0 in the new sample) |
| 2026-10-21 04:16 | 10 PRs merged in the window, all 10-20 18:38–19:02; 0 merge_group runs since 19:02 (9.2 h quiet) | 172 / 191 first enqueue (n=10, carried: no new merges) | 11.4 (n=16, carried) | 10.9 / 15.8 24h (carried, n=14); since 00:16: 7.20–8.38 (n=4 full runs, exact) | 0.08 / 0.13 (nightly + 1 CI dispatch only, carried) | 0.02 / 0.02 (Depot; since 00:16 max 1 s, n=116) | ~7.3k / 0.49k / 0.12k (mg 4,876 L / 337 W carried, 84% wasted in the 10-20 stall; PR +130 L in 4 full runs since 00:16 (31.6–33.6 each) plus 7 stubs; Depot slow-start 0% (0/116) since 00:16) |
| 2026-10-21 00:16 | 10 PRs merged in the window, all 18:38–19:02; 0 merge_group runs since 19:02 (5.2 h quiet; 15 non-draft PRs open) | 172 / 191 first enqueue (n=10, carried: no new merges) | 11.4 (n=16, range 7.4–12.6; slow hosts) | 10.9 / 15.8 (n=14 full runs; since 20:16: 7.4–9.98, one 15.75 with a 148 s Depot start gap) | 0.08 / 0.13 (nightly + 1 CI dispatch only) | 0.02 / 0.02 (Depot, n=4,387, max 0.8) | ~7.2k / 0.49k / 0.12k (mg 4,876 L / 337 W in 120 runs, 4,087 L (84%) in 104 cancelled/failed runs; PR ~1.77k L (since 20:16 exact, earlier ×3); push 42 L; nightly 264 / 80 / 58; CI dispatch on #1647 280 / 71 / 64; Depot slow-start 57% (2,231/3,926) in 24h, 23% (79/348) since 20:16) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~14 | #1670 (settings) 90-min check timeout let a stuck check freeze main for 77 min | ~46 min sooner recovery per stuck-check incident, for every waiting entry (7 on 10-22) | No recurrence since 10-22 20:16 (15/15 mg runs green through 10-23 20:16). Triaged p3 and needs a repo admin. Profiler ROI estimate: 46 MQ-min × 0.3 ÷ 1. |
| ~11 (was ~1.0) | #1248 (settings) merge-queue build concurrency | 4–9 min per entry past the 5th in a burst; 44 entry-min in the 10-23 14:30 burst | No burst since 16:16 (2 entries, no slot wait). Fresh numbers posted at 16:16. 7 of 13 entries waited 4.3–9.4 min for a slot while Depot queue wait stayed at 0.02. Needs a human decision. ROI: ~44 MQ-min/day in bursts × 0.25 ÷ 1. |
| ~2.5 | #1512 (settings) slow Depot hosts | ~2–4 min per merge on a slow host, plus lost-runner evictions | 16:16–20:16: 48% slow-start (94/197), up from 4%; #1680's merge_group run 100% slow, ci-ok 10.03 vs 7.35. Intermittent and recurring. Last commented 10-23 00:18; next fresh numbers at the 10-24 00:16 run. Needs a human to raise it with Depot. |
Closed since last run: none. Filed: none. Commented: none.
Open ci-perf / ci-janitor PRs: #1697 (ci-perf, Ready for review): split the Gradle e2e leg; PR run shows legs of 3.77 / 2.73 min vs 4.6–4.9 combined, ~0.45 min/run expected off ci-ok (up to ~3 on slow hosts). #1639 (ci-janitor, agent:needs-human), redundant with main's uv pin.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
| #1682 start coverage and e2e-build without waiting on clippy (#1681) | 10-23 14:54 | Delivered. In merge_group runs, coverage now starts at 0.20–0.30 min (was 1.43–1.47). ci-ok with #1682: 6.50, 6.73, 7.35, 7.58, 10.03 (n=5; p50 7.35, the 10.03 on 100%-slow hosts) vs 7.53–8.15, p50 7.76 before. On fast hosts ~1.1 min saved against 1.3 expected. |
| #1686 Windows nightly path asserts (ci-janitor) | 10-23 14:38 | It passed both Windows shards in its dispatch run. The 10-24 nightly decides. |
| #1690 gem old-Bundler legs (follow-up to #1687) | 10-23 14:54 | The 10-24 nightly decides. |
| #1692 compatibility workflows off Node 20 pins (ci-janitor) | 10-23 14:47 | Maintenance; no CI-time effect expected. |
| #1673 retry the yarn-classic corepack probe (#1671) | 10-23 00:13 | Before: 2 of 40 tests failed on one DNS timeout and evicted #1653 (~94 job-min). After the merge: 0 yarn-classic failures in 28 jobs (7 full runs through 04:16, then 21/21 green since). Holding. |
| #1663 fix uv settings test on Windows nightly (ci-janitor) | 10-22 19:29 | Did not make the Windows nightly green. The 10-23 nightly failed both Windows shards (cli --test user_patches, core --lib, path-form asserts). Follow-up #1686 passed both shards in its dispatch run. |
| #1652 keep transport cause in registry errors | 10-22 17:38 | Diagnostics; no CI-time effect expected. |
| #1654 retry crates.io in coverage's cargo e2e | 10-22 17:12 | 0 infra coverage failures in 53 merge-group + 19 push runs + 10 PR runs since 04:16. The one 04:16–08:16 failure was a PR's own new test. The 10-22 21:25 PR failure is carried. Holding. |
| #1643 fix update-lock flake | 10-22 17:09 | 0 recurrences since. Too early to call. |
| #1647 pin floating toolchains (#1645) | 10-22 17:03 | No toolchain-drift red main since (7 h). Needs a week to judge. |
| #1659 runner image's PHP in composer e2e | 10-22 17:03 | 0 Setup PHP failures since (was 1 re-run on 10-21). Composer leg p50 unchanged. |
#1666 skip cargo shapes on failed --precise pin |
10-22 17:01 | Robustness; no CI-time effect expected. |
#1620 ci-watch.sh works without GraphQL (ci-janitor) |
10-20 19:02 | Tooling; no CI-time effect expected. |
| #1634 keep reuse self-test notices out of CI logs (ci-janitor) | 10-20 19:02 | Log hygiene; no CI-time effect expected. |
| #1590 move action pins off deprecated Node 20 (#1588, ci-janitor) | 10-17 14:47 | Maintenance. No measurable change, as expected. |
| #1585 core covgap tests join their suites (#1089) | 10-15 18:58 | No measurable change; it claimed no saving. |
Method notes
- GraphQL and
search/issuesare blocked in this environment. Merge-queue state comes from REST timelines of merged PRs. "First enqueue" means the PR's firstadded_to_merge_queueevent in the window. "Slot wait" = lastadded_to_merge_queue→ that entry's merge_groupCIruncreated_at. - Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_atof the latest attempt. - "Discarded green runs" = successful merge-group runs whose entries were rebuilt on a new base before merging.
- This run (10-23 20:16): the 24h window held 298 runs, all listed. Job-minutes since 16:16 are exact. The rest of the 24h is carried from earlier runs. PR push→ci-ok excludes re-run attempts (their
created_atis the first attempt's).
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code