Repository navigation
Nudge agents to add a regression test when fixing a verified app security finding - #8927
Merged
Merged
Conversation
…rity finding The remediation guidance said to make the smallest safe changes, but never mentioned tests, so a fixed finding could silently come back. It now asks for a deterministic regression test that reproduces the finding where the app has a suitable test setup, run after the fix. Co-authored-by: Qlaw <noreply@qlaw.quick.shopify.io>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The guidance is narrowly scoped, both copies match, and no blocking issues remain.
0 open findings
What changed in this PR
Updates app security remediation guidance to help prevent verified findings from recurring.
Changes:
- Directs agents to add or update a deterministic regression test where suitable test infrastructure exists, then run it after the fix.
- Synchronizes the embedded instructions without changing check prompts or versions.
| File | Description |
|---|---|
| packages/app/src/cli/services/app-security-engine/INSTRUCTIONS.md | Adds regression-test guidance for verified fixes. |
| packages/app/src/cli/services/app-security-engine/checks/embedded.ts | Updates the embedded instructions to match. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
dmerand
approved these changes
Oct 15, 2026
jplhomer
enabled auto-merge
October 20, 2026 13:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY are these changes introduced?
The app security remediation guidance told agents to make the smallest safe changes when fixing findings, but never mentioned tests — so a fixed verified finding could silently come back.
WHAT is this pull request doing?
Adds one sentence to Section 6 of the app security agent instructions: when fixing a verified finding, add or update a deterministic regression test that reproduces it where the app has a suitable test setup, and run that test after the fix. Regenerates the embedded instructions copy. No changeset, matching recent instructions-only changes to this file, and no check prompts or versions change.
How to manually test your changes?
Run
shopify app security checkon an app, ask the agent to fix a verified finding, and confirm it adds or updates a regression test and runs it after the fix.Checklist
pnpm changeset addPR authored by Qlaw