Skip to content

fix: vendor hosted patches directly from lockfiles - #1648

Open
Mikola Lysenko (mikolalysenko) wants to merge 6 commits into
mainfrom
fix/vendor-lockfile-clean-checkout
Open

Mikola Lysenko (mikolalysenko) wants to merge 6 commits into
mainfrom
fix/vendor-lockfile-clean-checkout

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

socket-patch vendor could ignore hosted lockfile pins when an agent manifest existed, and an unrelated Python environment could reject a pinned patch as not installed. Select hosted patches from the lockfiles first, download their prebuilt artifacts into .socket/vendor/, and rewrite the dependency files without requiring installed packages.

Reuse the existing lockfile readers and vendoring backend. Hosted conversion skips installed-package discovery and a redundant wiring scan, while reusing the restored-lock preview and failure rollback. Preserve the legacy manifest fallback when no hosted pins are in scope, and update CLI help and documentation.

Validation on current main (dc0dc755d):

  • 334 focused tests passed: 71 vendor unit tests and 263 integration/parser tests covering clean npm/Cargo/Python checkouts, ambient virtualenvs, Bun locks, pnpm refusals, dry runs, revert, and rollback.
  • cargo clippy --workspace --all-features -- -D warnings passed.
  • cargo fmt --all -- --check and git diff --check passed.

Draft pending /code-review high, which AGENTS.md requires before marking a PR ready for review. That command is not available in the publishing session.


Note

Medium Risk
Changes how vendor selects patches and sources artifacts for hosted projects; incorrect behavior could mis-wire lockfiles or skip patches, though the change is scoped to vendoring with broad test coverage.

Overview
socket-patch vendor now treats hosted lockfile pins as the primary input, even when .socket/manifest.json exists or nothing is installed. Hosted conversion runs the eject path: fetch patch records from the API, restore upstream lock metadata for reversal, download prebuilt patched artifacts into .socket/vendor/, and rewire locks—without crawling node_modules/site-packages or using the manifest to pick patches.

The vendoring engine skips installed-package discovery and per-package hosted takeover during eject (upstream restore already happened in one transaction). Dry-run reuses the takeover backend preview on in-memory restored lock text so refusals like unsupported pnpm catalog entries match wet runs. --offline on hosted eject fails up front with offline_eject_unavailable instead of per-pin redirect_revert_failed. Manifest-only vendoring remains when no hosted pins are in scope.

CLI help, CLI_CONTRACT.md, and docs/usage.md describe lockfile-first behavior; integration tests cover manifest+hosted coexistence, fresh checkouts, ambient virtualenvs, and updated event codes (eject_planned vs legacy takeover advisories).

Reviewed by Cursor Bugbot for commit 25a3728. Configure here.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] I ran /code-review high on 25a3728, which AGENTS.md requires before this leaves draft. One finding needs a product decision before this PR can be ready, so I'm leaving it as a draft and labelling it agent:needs-human.

Question for a human: when .socket/manifest.json exists and at least one hosted pin is in scope, should vendor (a) vendor only the hosted lockfile pins, as it does now, (b) vendor the union of the manifest and the hosted pins, or (c) refuse and name the manifest patches it would skip?

As written (vendor.rs:991-1017), vendor returns from run_eject(pins) before the manifest arm runs. Any manifest patch without a hosted pin is skipped silently and the run exits 0. Example: the manifest lists pkg:npm/a@1 applied in place, and package-lock.json pins pkg:npm/b@1 hosted. Before this PR both were vendored; now only b is, and a fresh checkout builds a without its patch. Related outcomes that follow from the same choice:

  • A contested hosted entry now refuses the run (hosted_wiring_contested) before the manifest check. That blocks manifest-only vendoring even when the contested package is not in the manifest.
  • If the manifest and the lockfile pin the same purl at different UUIDs, the selection flips between runs. Run 1 ejects the lockfile UUID; run 2 has no hosted pins left, takes the manifest arm and re-vendors the manifest UUID.
  • --vex after an eject still reads the existing manifest, so manifest patches that aren't installed can turn a successful eject into exit 1.

Other findings for the author, not verified in depth:

  1. run_eject runs the takeover_dry_refusal(..., Some(&service), ...) preview for every pin on wet runs too (vendor.rs:1844). Each preview fetches the archive, so every artifact is granted twice, and a --dry-run that used to make no service requests now downloads every archive.
  2. The eject path never collects vlt_heal::lock_targets, so a vlt project whose .vlt store still holds the hosted bytes no longer gets healed and gets no redirect_vlt_reinstall_required warning.
  3. Installed-package discovery is skipped under eject. A pypi release with two hosted ?artifact_id= variant pins now always fails vendor_variant_ambiguous; before, the installed variant decided it.
  4. takeover_dry_refusal returns None if any file in the shared multi-pin plan has no staged text (for example bun.lockb). One such pin turns off the preview for every pin.
  5. The doc comment on run_eject still describes a per-package takeover that the eject no longer runs.

CI on the current head is green for the draft subset (6 passed, 31 skipped). The full matrix and Bugbot haven't run yet.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] Burn-down agent: I ran /code-review high on 25a37284 (AGENTS.md requires it before this leaves draft). This needs a product decision before anyone fixes code, so I'm adding agent:needs-human and leaving the draft open.

Question: when .socket/manifest.json exists and the lockfiles have at least one hosted pin, should plain vendor:

  • (a) eject only the hosted pins and stop, as this PR does now, or
  • (b) vendor the union, like main does today: hosted pins from the lockfile, plus manifest patches that have no hosted pin?

With (a), manifest patches that have no hosted pin are skipped. The run still exits 0, so a fresh checkout installs those packages unpatched. Projects that use a manifest also hit new refusals: hosted_wiring_contested, offline_eject_unavailable, and patch_fetch_failed for local-only records.

Confirmed failures (local cargo test -p socket-patch-cli --test mode_migration_cargo on this head: 27 passed, 3 failed):

  • double_takeover_a_b_a_preserves_lock_originals
  • hosted_then_vendored_takeover_leaves_pure_vendored
  • vendor_over_unrestorable_hosted_pin_is_refused

The review also flags the same takeover expectations in mode_migration_npm.rs:621 and e2e_golang_hosted_state.rs:534. CI shows green only because every test job is skipped while the PR is a draft.

Other review findings to handle once (a) or (b) is chosen:

  • The takeover_dry_refusal preflight downloads each prebuilt artifact, including on --dry-run. A wet run then downloads it a second time.
  • The dry-run preview drops PENDING/UNAVAILABLE backend codes. That gives eject_planned on the dry run but eject_incomplete on the wet run.
  • CLI_CONTRACT.md:176 and the run_eject doc comment still describe the per-purl takeover.
  • discover_wiring now runs twice on the manifest path.
  • In vendor_records_reusing, whether EjectCapture is present now quietly turns off discovery, alias resolution and takeover. An explicit flag would be clearer.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Question for a human: when .socket/manifest.json exists and at least one hosted pin is in scope, should vendor (a) vendor only the hosted lockfile pins, as it does now, (b) vendor the union of the manifest and the hosted pins, or (c) refuse and name the manifest patches it would skip?

(a) is the correct approach for now. If a user wants to vendor the manifest patches they should do a full scan with vendor mode and apply the edits in the lockfiles.

A plain `vendor` with a hosted pin in scope now ejects exactly the
lockfile's hosted patch, even when `.socket/manifest.json` names another
patch for the same purl; vendoring manifest patches takes a full
`scan --mode vendored`. The hosted→vendored and A→B→A tests expected the
old per-purl takeover onto the manifest's patch.

Point the eject at the mock API, serve the granted crate (with the
sha512 the artifact download checks), and assert the new contract: the
hosted UUID is vendored and attested, the manifest-only UUID is neither
vendored, reported nor attested, and the manifest is left byte-identical.
The fail-closed test now covers both refusals: `--offline` is refused up
front (`offline_eject_unavailable`), and an index that cannot re-resolve
the crates.io entry still yields `redirect_revert_failed` with the
checkout remedy, nothing touched.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: picked this up now that the product question is answered (17:43 comment: (a), vendor ejects only the hosted lockfile pins). No code pushed this run; here is the state for whoever continues.

Why CI is red on 25a3728 (5 jobs: coverage, yarn-classic, cargo lock-v4, bun e2e, ci-ok): the existing takeover tests encode the old behavior. They stage a manifest patch (UUID_V) over a live hosted pin (UUID_H) and expect vendor to do the per-package manifest takeover. With (a), vendor goes straight to run_eject, fetches the hosted record for UUID_H, and the mocks don't serve it, so it refuses with eject_refused: not every hosted patch record could be fetched.

Reproduced locally on this head:

  • mode_migration_npm: classic_hosted_then_vendored_takeover_round_trips_to_registry
  • mode_migration_bun: bun_dry_run_previews_match_wet_outcomes, bun_hosted_then_vendored_takeover_round_trips_to_registry
  • mode_migration_cargo: double_takeover_a_b_a_preserves_lock_originals, hosted_then_vendored_takeover_leaves_pure_vendored, vendor_over_unrestorable_hosted_pin_is_refused
  • mode_migration_pypi: pipenv_hosted_to_vendored_keeps_the_requirements_patched, pipenv_vendor_wires_the_exported_requirements_too, requirements_editable_beside_hosted_pin_unwinds, requirements_sole_hosted_pin_unwinds
  • mode_migration_vlt and vendor_eject pass. coverage reports about 22 failures in total, so other suites are affected too (e2e_golang_hosted_state.rs:534 was flagged earlier).

Next step: rewrite each of these as an eject. Mount the hosted record endpoint for UUID_H, and assert that UUID_H ends up vendored, that a manifest-only patch is left alone, and that --revert round-trips. Don't keep the UUID_V takeover expectations: (a) makes them obsolete. Then work through the earlier /code-review high findings, which still apply: the takeover_dry_refusal preflight double-downloads (including on --dry-run); PENDING/UNAVAILABLE codes are dropped in the dry preview; vlt_heal targets aren't collected on eject; pypi ?artifact_id= variant ambiguity; the bun.lockb staged-text gap disables the preview for every pin; and the stale docs (CLI_CONTRACT.md:176, the run_eject doc comment).

Not labeled; state set to ready so the next run continues.


Generated by Claude Code

With a hosted pin in scope, plain `vendor` ejects the lockfile's hosted
patch and ignores `.socket/manifest.json`; the bun hosted→vendored and
dry-run tests still expected the manifest's patch (uuid_v) to take over
the hosted pin, with the per-purl takeover advisories.

Point `vendor_cmd` at the mock API so the eject can fetch the record,
expect the hosted uuid to be vendored (detached ledger entry, VEX on the
view's CVE), and assert that the manifest-only patch is not previewed,
reported or vendored and that the manifest is left byte-identical. The
dry run now previews the eject (`eject_planned`) instead of the old
`vendor_would_revert_redirect` advisory.

Co-Authored-By: Claude <noreply@anthropic.com>
The classic and berry hosted→vendored legs expected plain `vendor` to
take over the hosted pin with the staged manifest's patch (UUID_V) and
announce `vendor_takeover_reverted_redirect`. With a hosted pin in scope
`vendor` now ejects the lockfile's hosted patch and ignores the manifest.

Point the vendor runs at the mock API (the eject fetches the record from
`patches/view`), expect UUID_H and its CVE in the manifest-less VEX
matrices with UUID_V as the never-attested patch, and replace the
takeover-advisory check with the eject contract: the manifest-only patch
is neither reported nor vendored, and the manifest is not rewritten.

Co-Authored-By: Claude <noreply@anthropic.com>
The test staged a manifest naming UUID_V over a hosted go.mod pin and
expected plain `vendor` to take the pin over with the manifest's patch.
With a hosted pin in scope `vendor` now ejects the lockfile's hosted
patch (UUID_H) and ignores the manifest; the run reached the public
proxy for the record and failed.

Point the run at the mock API, serve the hosted patch's prebuilt module
from the artifact-service mock, and expect the go.mod replace to name
UUID_H. Assert the manifest-only patch is neither reported nor vendored
and the manifest is left byte-identical; the existing upstream-restore
checks (no hosted replace, go.sum back to its pre-redirect bytes, no
ledger) are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
The "Takeover reconciliation" contract and the `run_eject` doc comment
still described plain `vendor` as a per-purl takeover onto the
manifest's patches. With a hosted pin in scope it ejects exactly the
lockfile's hosted pins as one planned transition and ignores the
manifest; say so, point manifest vendoring at `scan --mode vendored`,
and keep the per-purl takeover wording for `scan` / `get --mode
vendored` and the no-hosted-pin manifest path.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: following the maintainer's answer (keep (a): vendor only the hosted lockfile pins), I pushed bdce77b. It changes tests and docs only; product behavior is unchanged.

  • mode_migration_cargo, mode_migration_bun, mode_migration_npm and e2e_golang_hosted_state now expect (a):
    • The hosted pin (UUID_H) is vendored and attested.
    • The manifest-only patch (UUID_V) is never vendored or reported.
    • The manifest is left untouched.
    • The vendor runs point at the mock API. For cargo, the granted artifact is served with sha512.
  • vendor_over_unrestorable_hosted_pin_is_refused now checks both refusals: offline_eject_unavailable for --offline and redirect_revert_failed online.
  • CLI_CONTRACT.md and the run_eject doc comment now describe the eject instead of the per-purl takeover.
  • Local results: cargo 30/30, bun 35/35 (bun 1.4.2), npm 32/32 (yarn 1.22.22), golang 29/29.
  • Not run locally: the Berry leg (no yarn 4 here) and the cargo lock-v4 leg.
  • Not checked: the earlier coverage log listed ~21 more failing tests that I couldn't see, so CI on this head is the real check.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants