Skip to content

Stop reuse self-tests posting fake CI notices - #1634

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/quiet-reuse-test-notices
Oct 20, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/quiet-reuse-test-notices

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Every lint-ecosystems run (PR, merge queue and main push) posts 10 fake notice annotations, for example on main's latest push run (check run 117423824414 at 7d3086f):

Could not verify merge-queue CI (TimeoutExpired); running all checks.
Could not compare with the previous PR head (CalledProcessError); running all checks.
… 10 in total

Nothing failed. The reuse scripts never run in that job. These lines come from the job's python3 -B -m unittest discover -s scripts/tests step: the fallback self-tests let ci-reuse-merge-group.py and ci-reuse-pr-head.py print their real ::notice:: workflow commands to stdout, and GitHub turns those lines into annotations. A local run of the suite prints 15 of them, and GitHub keeps 10 (its per-step cap).

The harm:

  • A reader can take the annotations to mean the merge-queue reuse check or the PR-head reuse check failed, which sends anyone debugging a CI-reuse decision down the wrong path.
  • They fill the step's notice cap, so a real notice from that step would be dropped.

CI is otherwise healthy this run: the merge queue is empty, nothing is queued or running, main is green, and the only failed run in 3 days (38806080761) was fixed by that PR's next push. So this falls under priority 4 (make CI more robust).

Root cause

test_api_failure_timeout_and_bad_payload_* in scripts/tests/test_ci_reuse_merge_group.py and scripts/tests/test_ci_reuse_pr_head.py call reusable_run on its error path without capturing stdout.

Fix

A small assert_falls_back helper in each test file. It runs reusable_run under contextlib.redirect_stdout, asserts that the result is None (unchanged), and also asserts that the ::notice:: fallback line was printed. The fallback is now checked, not just silenced. No production script or workflow changes.

Proof

  • python3 -B -m unittest discover -s scripts/tests: 347 tests, OK (6 skipped). Lines starting with :: in stdout went from 15 to 0.
  • Negative check: removing ::notice:: from ci-reuse-merge-group.py makes the new assertion fail 7 subtests.

Removed or moved tests

None. Every test still runs in the same place (lint-ecosystems).

https://claude.ai/code/session_01SeHhBxEvePf8ERTaTTenMK


Generated by Claude Code

The fallback tests for ci-reuse-merge-group.py and ci-reuse-pr-head.py
let the scripts print their real "::notice::Could not verify
merge-queue CI (...)" lines to stdout. lint-ecosystems runs those tests
in a GitHub job, so every run on PRs, the merge queue and main gained
ten fake annotations claiming the reuse check had failed. They fill the
step's ten-notice cap and send anyone debugging a reuse decision down
the wrong path.

Capture stdout in those cases and assert that the notice is printed.
The fallback path is now checked, not just silenced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeHhBxEvePf8ERTaTTenMK
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 17, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 564c31a. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 17, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review (burn-down agent).

  • Head: 564c31a
  • CI: 54/54 green (35 success, 19 skipped). No main-wide failures.
  • Bugbot: reviewed 564c31a, no findings. 0 open threads.
  • Mergeable with no conflicts. CHANGELOG.md is not touched.
  • Reviewer focus: only the two reuse self-tests under scripts/tests/ change; check they no longer emit real ::notice annotations.

Slack announcement not sent this run (the Slack send tool was unavailable). The next run will retry.


Generated by Claude Code

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 20, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

The merge queue removed this PR at 17:49 UTC because e2e (ubuntu-latest, e2e_vex_build, pipenv:: --ignored, 2026.8.0) failed (run 39467483928). That failure doesn't come from this PR:

Cause: uv 0.13.2 (released 2026-10-19) rejects the environments = [] that Pipenv 2026.8.0 writes, and the Pipenv/pip legs install the latest uv. I bisected it locally: 0.13.1 accepts the file and 0.13.2–0.13.4 reject it.

#1639 pins setup-uv to 0.13.1 and fixes this. I'm not porting the pin here, so the fix doesn't land twice. Once #1639 merges, this PR can go back in the queue unchanged, and its approval still stands.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[final reviewer] Re-added to the merge queue (squash, auto-merge) at 564c31a13, which is the commit Wenxin Jiang (@Wenxin-Jiang) approved. The earlier eviction was the main-wide uv 0.13.2 Pipenv pylock break. main now pins setup-uv to 0.13.1 (#1605), and the same e2e … pipenv:: --ignored, 2026.8.0 leg is green on main a54753a. This is the one re-enqueue after that eviction.


Generated by Claude Code

Merged via the queue into main with commit 0b959a8 Oct 20, 2026
54 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/quiet-reuse-test-notices branch October 20, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants