⚡ Copilot Token Optimization2026-10-15 — Build Test Suite #10020

Target Workflow: Build Test Suite (.github/workflows/build-test.md)

Source report: #10019
Estimated cost per run: ~$25.55 (AIC 25.2–25.9)
Total tokens per run: ~11K fresh (input ~7.5K + output ~3.7K); ~600K cache-read + ~59K cache-write per run
Cache hit rate: ~91% (cache_read / (cache_read + cache_write + input))
LLM turns: 11–12 (model: claude-sonnet-5.5)

Current Configuration

Note: the report's "Turns" column is 0 (telemetry gap); turns here come from agent-session.jsonl (assistant.turn_start).

Trace Analysis (hypotheses only)

Successful runs analysed: 38652921568 (12 non-final calls) and 38593075773 (11 non-final calls).

• Repeated identical call: run 38652921568 calls 3→4 and run 38593075773 calls 2→3 run which bun deno dotnet ... cargo ... once inside a compound command and again alone with identical arguments and no intervening write. Hypothesis: the compound command's output was truncated/unclear, forcing a repeat.
• Retry after failure (Java): in both runs, the Maven step fails (~/.m2 not writable) and is retried with -Dmaven.repo.local=/tmp/gh-aw/agent/m2 (run 38652921568 calls 9→12; run 38593075773 calls 9→11). Hypothesis: the prompt should supply the repo-local flag up front.
• Retry after failure (.NET): repeated dotnet invocations with progressively added env vars (DOTNET_CLI_HOME, XDG_RUNTIME_DIR, TMPDIR) (run 38652921568 calls 6–8; run 38593075773 calls 7–8). Hypothesis: the env vars could be provided once in the prompt/pre-step.
• Broad exploration later superseded: the first call is a multi-ecosystem clone/script that is then followed by separate narrower calls per ecosystem.
• No evidence was found of results ignored in the final response.

Empirical tool-call cap

Only 2 usable successful runs (n=2; p50=11, p95=12 by nearest rank), which is fewer than the required 3. No --max-num-tool-calls recommendation yet; more successful runs are needed. Supporting runs: 38652921568, 38593075773.

Recommendations

1. Move deterministic setup/build work to pre-agent steps: (highest impact)

Estimated savings: ~60–70% of the ~25 AIC/run (roughly 8–9 of 11–12 turns, ~400K cache-read tokens)

The agent merely runs fixed clone/install/build/test commands and summarises them. Run them in steps: writing results to /tmp/gh-aw/build-test/results.json (set TMPDIR, DOTNET_CLI_HOME, XDG_RUNTIME_DIR, -Dmaven.repo.local there too), then the agent only reads the results and posts the comment:

steps:
  - name: Run build matrix
    run: bash scripts/ci/build-test-matrix.sh > /tmp/gh-aw/build-test/results.md

Prompt body then reduces to: "Read /tmp/gh-aw/build-test/results.md and post it via add_comment."

Caveat: pre-agent steps run outside the AWF firewall, so firewall-allowlist coverage would no longer be tested. If that is the point of this workflow, instead keep the agent but see #2–#4.

2. Bake known workarounds into the prompt and batch commands

Estimated savings: ~3–4 turns/run (~25–30%)

Add to the prompt: use -Dmaven.repo.local=/tmp/gh-aw/agent/m2, export DOTNET_CLI_HOME/XDG_RUNTIME_DIR/TMPDIR=/tmp/gh-aw/agent up front, run all ecosystems in a single scripted bash call, and do not repeat which.

3. Drop the unused github tool

Estimated savings: ~10K tokens/turn schema overhead (~20–30K tokens of cache-read per run)

No GitHub MCP calls were made in either run. Remove the github: block under tools: (or restrict with toolsets: [pull_requests] if keeping). Comments/labels go through safe-outputs.

4. Trim the prompt

Estimated savings: ~1–2K tokens/turn

Collapse the per-ecosystem sections (206 lines) into a compact table of repo, path, and command.

5. Network trimming

Low token impact. Verify jsr.io and dl.deno.land are actually needed by the Deno tests before keeping them.

Expected Impact

Implementation Checklist

• Decide whether pre-agent execution (Improve links in readme to AW project #1) is acceptable for firewall coverage
• Add env/Maven workarounds and single-script batching to the prompt (Secret proxying #2)
• Remove or restrict github: tool (feat: add integration test for rostbuness #3)
• Condense the prompt (fix: add missing Docker image pulls to robustness test workflow #4)
• Recompile: gh aw compile .github/workflows/build-test.md
• Post-process: npx tsx scripts/ci/postprocess-smoke-workflows.ts
• Verify CI passes on PR
• Compare token usage on new run vs baseline; revisit tool-call cap once ≥3 successful runs exist

Generated by Daily Copilot Token Optimization Advisor · copilot · auto · 23.9 AIC · ⊞ 11.3K · ◷