[Hackathon] fix(ci): keep ci run --dry-run free of GitHub lookup and workflow output writes #555

Verified against (commit SHA or released version)

Rechecked on commit a8f8cca244b4daf1b5d37ec3fe7c2c4fff2492a3 (package version 0.15.0), Windows, Node 24.18.0. Initially reproduced on release commit 8a2361316a6b439250d95ba9ed619713aac70d3f; the relevant application code is unchanged between those commits.

Discord Username / User ID

iamharrie

What does this improvement do?

Keep ci run --dry-run --output json usable without a GitHub token and free of GitHub lookups and workflow-output writes.

Under a deployment_status event, runCiRun resolves live GitHub context before downstream dry-run handling. With no GitHub token, the preview fails validation. With a token, it performs the commit-to-PR lookup. If that returns no PR, or if a pull_request event is a draft, the live skip branch writes zero counters into GITHUB_OUTPUT and returns empty stdout instead of the JSON preview.

This matters for validating a generated gate before credentials are configured: the preview's behavior depends on the surrounding Actions event and can modify outputs consumed by later steps. No test is dispatched in the reproduction, and I am not claiming that a dry run spends test credits.

Actual executable results, using a loopback HTTP fixture and fake GitHub token, no TestSprite credentials:

All runs used an explicit --target-url. DOCUMENTATION.md describes credentialless previews and explicit targets as offline, with no writes. The no-write and usable-preview problems also occur on the draft-PR branch without any network lookup.

Reproduction

Build the specified upstream checkout with npm ci --ignore-scripts --no-audit --no-fund and npm run build. Save the following as repro-ci-dry-run.mjs in that checkout and run node repro-ci-dry-run.mjs. It uses only Node built-ins. It starts a loopback server standing in for GitHub and runs the actual compiled CLI as a child process; no TestSprite account or hosted execution is used. Its assertions pin the observed defect, not a fixed result.

import assert from 'node:assert/strict';
import { spawn } from 'node:child_process';
import { createServer } from 'node:http';
import { mkdtempSync, writeFileSync, readFileSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';

const cli = resolve(process.argv[2] ?? 'dist/index.js');
const dir = mkdtempSync(join(tmpdir(), 'testsprite-dry-run-'));
const event = join(dir, 'event.json');
const output = join(dir, 'output.txt');
writeFileSync(event, JSON.stringify({ deployment: { sha: 'a'.repeat(40), ref: 'feature/example' } }));
let calls = 0;
const server = createServer((_req, res) => {
  calls++;
  res.writeHead(200, { 'content-type': 'application/json' });
  res.end('[]');
});
await new Promise(done => server.listen(0, '127.0.0.1', done));
const base = `http://127.0.0.1:${server.address().port}`;
const env = Object.fromEntries(Object.entries(process.env)
  .filter(([key]) => !/^(TESTSPRITE_|GITHUB_)/.test(key)));
delete env.NODE_OPTIONS;
Object.assign(env, {
  USERPROFILE: dir, TESTSPRITE_NO_TELEMETRY: '1', TESTSPRITE_NO_UPDATE_NOTIFIER: '1',
  TESTSPRITE_NO_SKILL_WARNING: '1', GITHUB_ACTIONS: 'true',
  GITHUB_EVENT_NAME: 'deployment_status', GITHUB_REPOSITORY: 'example/app',
  GITHUB_EVENT_PATH: event, GITHUB_OUTPUT: output, GITHUB_API_URL: base,
  GITHUB_TOKEN: 'fake-local-fixture-token',
});
const child = spawn(process.execPath, [cli, '--output', 'json', '--dry-run',
  '--endpoint-url', base, 'ci', 'run', '--project', 'project-demo',
  '--target-url', 'https://preview.example.invalid', '--max-concurrency', '5',
  '--no-execution-report'], { env, stdio: ['ignore', 'pipe', 'pipe'], windowsHide: true });
let stdout = '';
let stderr = '';
child.stdout.on('data', data => { stdout += data; });
child.stderr.on('data', data => { stderr += data; });
const watchdog = setTimeout(() => child.kill(), 20_000);
try {
  const exitCode = await new Promise((done, fail) => {
    child.on('error', fail); child.on('close', done);
  });
  const stepOutputs = existsSync(output) ? readFileSync(output, 'utf8') : null;
  console.log(JSON.stringify({ exitCode, calls, stdout, stderr, stepOutputs }, null, 2));
  assert.equal(exitCode, 0);
  assert.equal(calls, 1);
  assert.equal(stdout.trim(), '');
  assert.match(stepOutputs, /total=0/);
} finally {
  clearTimeout(watchdog);
  server.closeAllConnections();
  await new Promise(done => server.close(done));
}

Actual output includes:

{
  "exitCode": 0,
  "calls": 1,
  "stdout": "",
  "stepOutputs": "total=0\npassed=0\nfailed=0\nskipped=0\ntimedOut=0\n"
}

The stderr field contains the Skipping: commit ... belongs to no pull request notice. To check the no-token path, remove the fake GITHUB_TOKEN assignment: it exits 5 with VALIDATION_ERROR instead of rendering the preview.

Details / implementation notes

Source:

• runCiRun, ci.ts:1330 calls readGithubRunContext before the downstream dry-run branch. The environment precheck already skips dry-run reads, but this GitHub lookup does not.
• skipCiRun, ci.ts:1379 calls the real step-output writer on draft/no-PR paths.

Suggested approach: handle simulation before live event lookup/skip paths, reuse the existing canned batch preview, preserve local flag validation, and leave output/summary files untouched. Local payload metadata can be shown where useful, but an unresolved deployment-to-PR association should not require credentials or skip the preview. Happy to follow the maintainers' preferred policy.

Regression cases should cover deployment and draft/open-PR events, with/without a token, and assert zero GitHub requests, unchanged output files, and a usable JSON preview. Normal non-dry-run skip behavior should remain unchanged. The existing GitHub-context, CI-run, and flaky suites pass (115 tests); these additional cases are not covered there.

Related work / scope: #372 concerns subprocess effects in ci connect --dry-run and ci doctor --dry-run. This report concerns direct lookup and file writes in the new ci run path. If you prefer one dry-run umbrella, I can contribute the reproduction there. #550 and #551 cover closed-PR selection and lookup timeout; I independently reproduced those but am not filing duplicates or bundling their fixes here.

No implementation PR yet: following CONTRIBUTING.md, I will wait for triage and an eligible claim before source changes. The reproduction is self-contained here; reviewers do not need access to my private research repository.

Confirmations

• I searched existing issues and disclosed related reports above.
• I supplied my Discord identity for reward coordination.