diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 863dbcbca..e9e9e764d 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -979,7 +979,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem | Key | Shape | Meaning | |---|---|---| | `error` | `{code, message}` | Only on `status: "error"` (v5.0, MAJOR: was a string). Codes: `manifest_not_found`, `manifest_invalid`, `manifest_unreadable`, `patch_not_found`, `path_glob_no_match`, `hosted_wiring_contested`, `vendor_ledger_missing`, `rollback_failed`, `lock_held` / `lock_io`, and `path_glob_invalid` (a usage error, exit 2). Per-result `results[*].error` stays a string. | -| `warnings` | `[{code, detail}]` | Run-level warnings, now populated (previously always empty): `reinstall_required`, `hosted_state_not_preservable`, `out_of_scope_copies_restored`, `vendor_state_unreadable`, `cleanup_failed`, `manifest_write_failed`, `legacy_redirect_ledger_kept`, the upstream-restore advisories (`npm_allow_remote_left`, `pnpm_trust_lockfile_left`, `maven_trusted_checksums_left`, `nuget_default_config_left`, `upstream_uv_override_removed`, `upstream_registry_fallback`, `upstream_pnpm_tarball_setting_guessed`, `upstream_gem_stale_cache`), `ownership_not_restored` (a restored file whose ownership could not be put back — see the apply warnings), `rollback_record_superseded` (a manifest record superseded by a live hosted pin, left to the hosted leg — see Manifest cleanup), plus vendored/hosted leg advisories. New codes are additive (MINOR) | +| `warnings` | `[{code, detail}]` | Run-level warnings, now populated (previously always empty): `reinstall_required`, `hosted_state_not_preservable`, `out_of_scope_copies_restored`, `vendor_state_unreadable`, `cleanup_failed`, `manifest_write_failed`, `legacy_redirect_ledger_kept`, the upstream-restore advisories (`npm_allow_remote_left`, `pnpm_trust_lockfile_left`, `maven_trusted_checksums_left`, `nuget_default_config_left`, `upstream_uv_override_removed`, `upstream_registry_fallback`, `upstream_pnpm_tarball_setting_guessed`, `upstream_gem_stale_cache`, `yarn_berry_node_gyp_unresolved`), `ownership_not_restored` (a restored file whose ownership could not be put back — see the apply warnings), `rollback_record_superseded` (a manifest record superseded by a live hosted pin, left to the hosted leg — see Manifest cleanup), plus vendored/hosted leg advisories. New codes are additive (MINOR) | | `vendored` | `[purl]` | **Meaning narrowed (MAJOR)**: vendor-owned purls the run did NOT act on — today exactly the corrupt-vendor-ledger skip. | | `vendoredReverted` | `[purl]` | Ledger entries cleanly reverted this run (unwired + artifact deleted + entry dropped; previewed on dry-run) | | `vendoredPreserved` | `[purl]` | `--preserve-state`: unwired with artifact + ledger entry kept | @@ -1284,6 +1284,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `upstream_registry_fallback` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore: a yarn berry, pnpm or vlt entry is restored from the version document of the registry the project resolves it against (`.yarnrc.yml` `npmRegistryServer`, the pnpm lock's sibling `.npmrc` `registry` / `@scope:registry` or pnpm-workspace.yaml `registry` / `registries`, vlt's node registry); that registry could not be read (e.g. it needs credentials), so the default registry's document was used and the restored tarball URL may not be the mirror's. | | `upstream_gem_stale_cache` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#1260): a restored gem's `-.gem` is still in Bundler's cache dir (`cache_path`, default `vendor/cache`, resolved as for the Gem stale-install guard) and its sha256 is not the upstream one (the restored `CHECKSUMS` entry, else the rubygems.org compact index), or it could not be checked (`--offline`, a registry error). Bundler installs from that dir first, so a `bundle cache` taken while the hosted pin was live makes every later install fail on the upstream checksum (exit 37) or, on bundler < 2.6 frozen installs, keep installing the patched bytes. The detail names the file. Remedy: delete it, then run `bundle cache` to cache the upstream gem in its place (or `bundle install` if the project does not commit its cache; with the cache dir committed, a frozen install reads only the cache). Read-only: the restore never deletes it. Not raised for an archive whose sha256 matches upstream. | | `upstream_pnpm_tarball_setting_guessed` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#902): nothing showed which pnpm wrote a hosted `pnpm-lock.yaml` (no unpinned registry entry that shows the setting, no `node_modules/.modules.yaml` install record, no package.json `packageManager` pin; for a Rush lock, no rush.json `pnpmVersion`), so its entries were restored with or without `tarball:` by pnpm 10's reading of `lockfileIncludeTarballUrl` (pnpm-workspace.yaml, else `.npmrc` `lockfile-include-tarball-url`), and pnpm 9 (which reads only `.npmrc`) or pnpm >= 11 (which reads only pnpm-workspace.yaml) would have read it the other way. The detail names the lock, the setting followed, the pnpm that disagrees and the entries. Remedy: pin the pnpm (package.json `packageManager`, or reinstall so the install record names it; rush.json `pnpmVersion` for Rush), or give the two files the same value so every pnpm reads it alike; a rollback or remove can then be redone by restoring the lock from version control and re-running. Not raised when evidence decided, when both files read the same on every pnpm, or when the tarball is one pnpm records regardless. | +| `yarn_berry_node_gyp_unresolved` | rollback/remove `warnings[]`; vendor advisory event (takeover) | upstream restore (#737): yarn's npm resolver gives the restored registry entry an implicit `node-gyp: "npm:latest"` dependency (the version document runs `node-gyp` in a script without declaring it), which the hosted pin dropped because yarn never gives a tarball-locator entry one, and the lock no longer has a `node-gyp@npm:latest` entry (the pin dropped the subtree only node-gyp reached). The entry is restored without it; a plain `yarn install --immutable` accepts that, a hardened or `--refresh-lockfile` install reports the lock modified. Remedy: run `yarn install` once. Not raised while another entry keeps `node-gyp@npm:latest` in the lock: the dependency is then restored too. | | `legacy_redirect_ledger_kept` | rollback `warnings[]` (+ remove stderr) | v5.0: a pre-v5 `.socket/vendor/redirect-state.json` could not be deleted once no hosted pin was left; the file is inert (never read for planning). Never flips the exit. | | `vendor_stale_artifact_removed` | `removed` | vendor / scan `--mode vendored`: re-vendor under a newer patch uuid removed the previous uuid's orphaned artifact dir. | | `vendor_unsupported_ecosystem` | `skipped` | vendor: no vendor backend for this purl's ecosystem (jsr). | @@ -1380,6 +1381,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `pypi_hatch_stale_install` | `skipped` (warning) | vendor (hatch): an existing Hatch environment of the project (found under Hatch's data dir, `dirs.env.virtual` or an explicit env `path`) still holds the upstream release; Hatch keeps it on the next `hatch run`, so the detail names the env and the `hatch env remove ` / `hatch env prune` remedy. | | `pypi_pipenv_installer_unknown` | `skipped` (warning) | vendor (pipenv): no `pipenv` answered on PATH; the vendored references assume Pipenv 2018 or later (7–11 cannot consume them — use hosted mode there); `SOCKET_PIPENV_MAJOR` pins the release. | | `vendor_lock_entry_relocked` | revert `warnings[]` | vendor `--revert` / rollback (pipenv): a relock regenerated the wired entry to a registry reference, or removed it; the record is retired (artifact removed, ledger entry dropped) instead of drift-kept. | +| `vendor_lock_entry_pruned_kept` | revert `warnings[]` | vendor `--revert` / rollback (yarn berry, #737): vendoring dropped a lock entry only the patched package's implicit `node-gyp` dependency reached (or one descriptor of a shared entry), and the lock has re-added or changed that entry since; it is left as it is (not drift: the artifact is still removed). Run `yarn install` if the lock is incomplete. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | | `pypi_poetry_integrity_unverified` | `skipped` (warning) | vendor (pypi / poetry): the lock was written by Poetry < 1.4 (0.12 `[metadata.hashes]`, lock 1.0/1.1, or a 2.0 lock without a `@generated by Poetry X.Y.Z` header — 1.3 wrote those). That installer does not verify local wheel hashes (the committed wheel bytes are the protection) and does not replace an already-installed package at the same version; recreate the virtualenv or `pip uninstall` the package before `poetry install`, or upgrade Poetry. | | `redirect_poetry_stale_install_risk` | `redirect.warnings[]` (warning) | scan `--mode hosted` (poetry): same writer test as above — a warm virtualenv keeps the upstream package after the redirect on Poetry < 1.4 (1.4+ re-installs from the new source); fresh installs pick up the patched wheel. Emitted once per rewritten lock, only on the run that rewrites it. | diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index 799267eeb..8f89b2b53 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -631,6 +631,11 @@ async fn mock_reference_with_berry(server: &MockServer) { } async fn mock_reference_with_berry_url(server: &MockServer, hosted_url: &str) { + mock_reference_with_berry_sha512(server, hosted_url, PATCHED_SHA512).await; +} + +/// [`mock_reference_with_berry_url`] granting a tarball of `sha512`. +async fn mock_reference_with_berry_sha512(server: &MockServer, hosted_url: &str, sha512: &str) { Mock::given(method("POST")) .and(path(format!("/v0/orgs/{ORG}/patches/package"))) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ @@ -641,7 +646,7 @@ async fn mock_reference_with_berry_url(server: &MockServer, hosted_url: &str) { "purl": PURL, "artifacts": [ { "kind": "tarball", "url": hosted_url, - "integrity": { "sha512": PATCHED_SHA512 } }, + "integrity": { "sha512": sha512 } }, { "kind": "yarn-berry-zip", "url": "http://patch.test/berry.zip", "integrity": { "yarnBerry10c0": BERRY_CHECKSUM } } ], @@ -861,6 +866,129 @@ async fn scan_redirect_rewrites_crlf_and_bom_yarn_berry_locks_and_rollback_resto } } +/// #737: yarn's npm resolver gives a registry entry whose scripts run +/// node-gyp (the registry injects `install: node-gyp rebuild` for any +/// package with a `binding.gyp`) an implicit `node-gyp: "npm:latest"` +/// dependency, which the tarball-locator pin never gets: the pin drops it, +/// and with it every entry only node-gyp reached. Rollback re-resolves the +/// registry entry and puts the dependency back while the lock still +/// resolves node-gyp; once it does not, it warns that `yarn install` must +/// re-add it. +#[tokio::test] +#[serial] +async fn yarn_berry_pin_drops_the_implicit_node_gyp_and_rollback_restores_it() { + let server = MockServer::start().await; + mock_discovery(&server).await; + let hosted_url = HOSTED_URL.replace("http://patch.test", &server.uri()); + let tarball = upstream_tarball(); + mock_reference_with_berry_sha512( + &server, + &hosted_url, + &vlt_hosted_common::sha512_sri(&tarball), + ) + .await; + mock_view(&server).await; + // The served tarball's own manifest (the pin reads it): no node-gyp. + Mock::given(method("GET")) + .and(path(hosted_url.trim_start_matches(&server.uri()))) + .respond_with(ResponseTemplate::new(200).set_body_bytes(tarball.clone())) + .mount(&server) + .await; + mock_npm_registry_scripts( + &server, + &vlt_hosted_common::sha512_sri(&tarball), + Some(tarball), + serde_json::json!({ "install": "node-gyp rebuild" }), + ) + .await; + + let node_gyp = "\"node-gyp@npm:latest\":\n version: 13.1.0\n \ + resolution: \"node-gyp@npm:13.1.0\"\n dependencies:\n \ + nopt: \"npm:^10.0.0\"\n checksum: 10c0/aa\n languageName: node\n \ + linkType: hard\n\n\"nopt@npm:^10.0.0\":\n version: 10.0.1\n \ + resolution: \"nopt@npm:10.0.1\"\n checksum: 10c0/bb\n \ + languageName: node\n linkType: hard\n"; + let other_gyp = "\n\"other-gyp@npm:1.0.0\":\n version: 1.0.0\n \ + resolution: \"other-gyp@npm:1.0.0\"\n dependencies:\n \ + node-gyp: \"npm:latest\"\n checksum: 10c0/cc\n \ + languageName: node\n linkType: hard\n"; + for shared in [true, false] { + let label = if shared { "shared" } else { "sole" }; + let tmp = tempfile::tempdir().unwrap(); + write_berry_project_spelled(tmp.path(), |t| { + let mut t = t + .replace( + &format!("resolution: \"{NAME}@npm:{VERSION}\"\n"), + &format!( + "resolution: \"{NAME}@npm:{VERSION}\"\n dependencies:\n \ + node-gyp: \"npm:latest\"\n" + ), + ) + .replace( + "\n\"consumer@workspace:.\"", + &format!("\n{node_gyp}\n\"consumer@workspace:.\""), + ); + if shared { + t = t.replace( + &format!(" {NAME}: \"npm:^{VERSION}\"\n"), + &format!(" {NAME}: \"npm:^{VERSION}\"\n other-gyp: \"npm:1.0.0\"\n"), + ); + t.push_str(other_gyp); + } + t + }); + let lock_path = tmp.path().join("yarn.lock"); + let pristine = std::fs::read_to_string(&lock_path).unwrap(); + + let env = run_redirect_subprocess_with( + tmp.path(), + &server.uri(), + &["--patch-server-url", &server.uri()], + ); + assert_eq!(env["redirect"]["redirected"], 1, "{label}: {env:#}"); + assert!(warning_codes(&env).is_empty(), "{label}: {env:#}"); + let lock = std::fs::read_to_string(&lock_path).unwrap(); + assert!( + lock.contains(&format!( + "\"{NAME}@{hosted_url}\":\n version: {VERSION}\n \ + resolution: \"{NAME}@{hosted_url}\"\n checksum: {BERRY_CHECKSUM}\n" + )), + "{label}: the pin has no node-gyp dependency: {lock}" + ); + assert_eq!( + lock.contains("\"node-gyp@npm:latest\":") && lock.contains("\"nopt@npm:^10.0.0\":"), + shared, + "{label}: node-gyp's subtree stays only while another entry reaches it: {lock}" + ); + + let (code, env) = rollback_json_with_origin(tmp.path(), &server, &server.uri()); + assert_eq!(code, Some(0), "{label}: rollback: {env:#}"); + let restored = std::fs::read_to_string(&lock_path).unwrap(); + let checksum = berry_checksum_of(&restored); + let pristine = pristine.replace( + &format!("10c0/{}", "3".repeat(128)), + &format!("10c0/{checksum}"), + ); + let warned = env.to_string().contains("yarn_berry_node_gyp_unresolved"); + if shared { + assert_eq!(restored, pristine, "{label}: byte-exact rollback"); + assert!(!warned, "{label}: {env:#}"); + } else { + assert_eq!( + restored, + pristine + .replace( + " dependencies:\n node-gyp: \"npm:latest\"\n checksum: 10c0/", + " checksum: 10c0/" + ) + .replace(&format!("{node_gyp}\n"), ""), + "{label}: the registry entry, minus the dependency the lock cannot resolve" + ); + assert!(warned, "{label}: {env:#}"); + } + } +} + /// #632: a dependency declared through a yarn catalog (`"catalog:"`) is /// matched by yarn's `resolutions` before the catalog is expanded, so the /// hosted pin must also route `@catalog:`; `rollback` must drop every @@ -1802,12 +1930,23 @@ fn hosted_unwind_json( /// `tarball` served at the document's `dist.tarball` (a yarn berry restore /// downloads it to recompute the zip checksum). async fn mock_npm_registry(server: &MockServer, integrity: &str, tarball: Option>) { + mock_npm_registry_scripts(server, integrity, tarball, serde_json::json!({})).await; +} + +/// [`mock_npm_registry`] whose version document carries `scripts`. +async fn mock_npm_registry_scripts( + server: &MockServer, + integrity: &str, + tarball: Option>, + scripts: serde_json::Value, +) { let tarball_path = format!("/npm-registry/{NAME}/-/{NAME}-{VERSION}.tgz"); Mock::given(method("GET")) .and(path(format!("/npm-registry/{NAME}/{VERSION}"))) .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "name": NAME, "version": VERSION, + "scripts": scripts, "dist": { "tarball": format!("{}{tarball_path}", server.uri()), "integrity": integrity, diff --git a/crates/socket-patch-core/src/formats/yarn/berry_entry.rs b/crates/socket-patch-core/src/formats/yarn/berry_entry.rs index 0a0d95b5a..f0ad5d2f3 100644 --- a/crates/socket-patch-core/src/formats/yarn/berry_entry.rs +++ b/crates/socket-patch-core/src/formats/yarn/berry_entry.rs @@ -10,6 +10,14 @@ //! install, so `yarn install --immutable` fails with YN0028. //! //! [`render_pinned_entry`] is the one place both writers build that entry. +//! +//! One difference is not a spelling: yarn's npm resolver adds an implicit +//! `node-gyp: "npm:latest"` dependency to a registry entry whose manifest +//! runs `node-gyp` in a script without declaring it (the registry injects +//! `install: node-gyp rebuild` for any package shipping a `binding.gyp`). +//! The tarball and `file:` resolvers read the tarball's manifest as is and +//! add nothing, so the pin drops that dependency (#737), and the entries +//! only it reached go with it (see [`super::berry_prune`]). use serde_json::Value; use std::collections::BTreeMap; @@ -58,6 +66,96 @@ fn unscoped(name: &str) -> &str { } } +/// The package name yarn's npm resolver adds as an implicit dependency. +const NODE_GYP: &str = "node-gyp"; + +/// The implicit dependency's line in an entry's `dependencies:` map. +const IMPLICIT_NODE_GYP_LINE: &str = " node-gyp: \"npm:latest\""; + +/// Whether `manifest` declares `node-gyp` itself (yarn reads +/// `optionalDependencies` into `dependencies`). +fn declares_node_gyp(manifest: &Value, fields: &[&str]) -> bool { + fields.iter().any(|field| { + manifest + .get(field) + .and_then(Value::as_object) + .is_some_and(|deps| deps.contains_key(NODE_GYP)) + }) +} + +/// Whether yarn's npm resolver gives the registry version document +/// `manifest` the implicit `node-gyp: "npm:latest"` dependency: a script +/// mentions `node-gyp` (the registry's injected `node-gyp rebuild`, or a +/// `node-gyp-build` hook), and the manifest declares no `node-gyp` +/// dependency or peer dependency of its own. +pub(crate) fn registry_adds_node_gyp(manifest: &Value) -> bool { + !declares_node_gyp( + manifest, + &["dependencies", "optionalDependencies", "peerDependencies"], + ) && manifest + .get("scripts") + .and_then(Value::as_object) + .is_some_and(|scripts| { + scripts + .values() + .any(|script| script.as_str().is_some_and(|s| s.contains(NODE_GYP))) + }) +} + +/// Whether an entry's lines (key first, or body only) carry the implicit +/// `node-gyp: "npm:latest"` dependency line. +pub(crate) fn has_implicit_node_gyp>(lines: &[S]) -> bool { + lines + .iter() + .any(|l| l.as_ref().trim_end_matches('\r') == IMPLICIT_NODE_GYP_LINE) +} + +/// `lines` (an entry, key first) with the implicit `node-gyp: "npm:latest"` +/// dependency added in yarn's (sorted) place, opening a `dependencies:` map +/// in yarn's field order when the entry has none. Unchanged when the entry +/// already depends on `node-gyp`. +pub(crate) fn with_implicit_node_gyp(lines: &[String]) -> Vec { + if lines.iter().any(|l| dependency_name(l) == Some(NODE_GYP)) { + return lines.to_vec(); + } + let mut fields = split_fields(lines.iter().skip(1).map(String::as_str)); + match fields.iter_mut().find(|(name, _)| name == "dependencies") { + Some((_, deps)) => { + let at = deps + .iter() + .skip(1) + .position(|l| dependency_name(l).is_some_and(|n| n > NODE_GYP)) + .map_or(deps.len(), |i| i + 1); + deps.insert(at, IMPLICIT_NODE_GYP_LINE.to_string()); + } + None => fields.push(( + "dependencies".to_string(), + vec![ + " dependencies:".to_string(), + IMPLICIT_NODE_GYP_LINE.to_string(), + ], + )), + } + fields.sort_by(|(a, _), (b, _)| field_order(a).cmp(&field_order(b))); + let mut out: Vec = lines.iter().take(1).cloned().collect(); + out.extend(fields.into_iter().flat_map(|(_, lines)| lines)); + out +} + +/// The dependency name a `dependencies:` map line names (` name: range`, +/// the name unquoted). +fn dependency_name(line: &str) -> Option<&str> { + let rest = line.strip_prefix(" ")?; + if rest.starts_with([' ', '\t']) { + return None; + } + let name = match rest.strip_prefix('"') { + Some(quoted) => quoted.split('"').next()?, + None => rest.split(':').next()?, + }; + (!name.is_empty()).then_some(name) +} + /// What a pin sets in the entry; everything else carries over from the /// registry entry. pub(crate) struct Pin<'a> { @@ -67,30 +165,34 @@ pub(crate) struct Pin<'a> { pub resolution: &'a str, /// The new `checksum:` value. `None` keeps the registry entry's line. pub checksum: Option<&'a str>, - /// The tarball manifest's `bin` (see [`manifest_bin`]). `None` keeps the - /// registry entry's `bin:` section; an empty map drops it. - pub bin: Option<&'a BTreeMap>, + /// The served tarball's own `package.json`. Its `bin` (see + /// [`manifest_bin`]) replaces the registry entry's `bin:` section (an + /// empty map drops it), and the registry's implicit `node-gyp` + /// dependency is dropped unless the manifest declares one. `None` + /// keeps both registry fields. + pub manifest: Option<&'a Value>, } /// The pinned entry's lines (no line endings), built from the registry /// entry's body lines (`registry_body`: every line after its key) the way /// yarn writes it: the pin's resolution, checksum and `bin:` in place of the -/// registry's, `languageName` and `linkType` defaulted when the registry -/// entry has none, and every field in yarn's order. +/// registry's, the registry's implicit `node-gyp` dependency dropped, +/// `languageName` and `linkType` defaulted when the registry entry has +/// none, and every field in yarn's order. pub(crate) fn render_pinned_entry>( registry_body: &[S], pin: &Pin<'_>, ) -> Vec { - let mut fields: Vec<(String, Vec)> = Vec::new(); - for line in registry_body.iter().map(AsRef::as_ref) { - match field_name(line) { - Some(name) => fields.push((name.to_string(), vec![line.to_string()])), - // A deeper-indented line belongs to the field above it; one - // before any field rides at the head of the body. - None => match fields.last_mut() { - Some((_, lines)) => lines.push(line.to_string()), - None => fields.push((String::new(), vec![line.to_string()])), - }, + let mut fields = split_fields(registry_body.iter().map(AsRef::as_ref)); + if let Some(manifest) = pin.manifest { + if !declares_node_gyp(manifest, &["dependencies", "optionalDependencies"]) { + for (name, lines) in fields.iter_mut() { + if name == "dependencies" { + lines.retain(|l| l.trim_end_matches('\r') != IMPLICIT_NODE_GYP_LINE); + } + } + // A map left with only its header line is no map at all. + fields.retain(|(name, lines)| name != "dependencies" || lines.len() > 1); } } let mut set = |name: &str, lines: Vec| { @@ -106,7 +208,8 @@ pub(crate) fn render_pinned_entry>( if let Some(checksum) = pin.checksum { set("checksum", vec![format!(" checksum: {checksum}")]); } - if let Some(bin) = pin.bin { + if let Some(bin) = pin.manifest.map(manifest_bin) { + let bin = &bin; let mut lines = Vec::new(); if !bin.is_empty() { lines.push(" bin:".to_string()); @@ -127,6 +230,23 @@ pub(crate) fn render_pinned_entry>( out } +/// An entry body's lines grouped by the field they belong to, in order. +fn split_fields<'a>(body: impl Iterator) -> Vec<(String, Vec)> { + let mut fields: Vec<(String, Vec)> = Vec::new(); + for line in body { + match field_name(line) { + Some(name) => fields.push((name.to_string(), vec![line.to_string()])), + // A deeper-indented line belongs to the field above it; one + // before any field rides at the head of the body. + None => match fields.last_mut() { + Some((_, lines)) => lines.push(line.to_string()), + None => fields.push((String::new(), vec![line.to_string()])), + }, + } + } + fields +} + /// A body line's field name when it opens a field (two-space indent). fn field_name(line: &str) -> Option<&str> { let rest = line.strip_prefix(" ")?; @@ -191,7 +311,7 @@ mod tests { key_line: "\"a@https://h/a-1.0.0.tgz\":", resolution: "a@https://h/a-1.0.0.tgz", checksum: Some("10c0/ab"), - bin: None, + manifest: None, }, ); assert_eq!( @@ -225,7 +345,7 @@ mod tests { key_line: "k:", resolution: "r", checksum: None, - bin: Some(&BTreeMap::new()), + manifest: Some(&json!({})), }, ); assert_eq!( @@ -251,7 +371,7 @@ mod tests { key_line: "k:", resolution: "r", checksum: Some("c"), - bin: None, + manifest: None, }, ); assert_eq!( @@ -260,6 +380,123 @@ mod tests { ); } + /// #737: real yarn 4.12.0 output. The registry entry of nan@2.22.0 + /// carries the npm resolver's implicit `node-gyp: "npm:latest"`; the + /// entry yarn writes for its tarball URL (here pinned through + /// `resolutions`) has none, because the tarball resolver reads the + /// manifest as is. bufferutil keeps its declared dependency. + #[test] + fn issue_737_pin_drops_the_registry_implicit_node_gyp() { + let nan = body( + " version: 2.22.0\n resolution: \"nan@npm:2.22.0\"\n dependencies:\n \ + node-gyp: \"npm:latest\"\n checksum: 10c0/d5\n languageName: node\n \ + linkType: hard", + ); + let url = "https://registry.npmjs.org/nan/-/nan-2.22.0.tgz"; + let key = format!("\"nan@{url}\":"); + let resolution = format!("nan@{url}"); + let manifest = json!({"name": "nan", "version": "2.22.0", "scripts": { + "rebuild-tests": "node-gyp rebuild --directory test"}}); + let pin = Pin { + key_line: &key, + resolution: &resolution, + checksum: Some("10c0/d5"), + manifest: Some(&manifest), + }; + assert_eq!( + render_pinned_entry(&nan, &pin).join("\n"), + format!( + "{key}\n version: 2.22.0\n resolution: \"{resolution}\"\n \ + checksum: 10c0/d5\n languageName: node\n linkType: hard" + ) + ); + + let bufferutil = body( + " version: 4.0.8\n resolution: \"bufferutil@npm:4.0.8\"\n dependencies:\n \ + node-gyp: \"npm:latest\"\n node-gyp-build: \"npm:^4.3.0\"\n checksum: 10c0/36\n \ + languageName: node\n linkType: hard", + ); + let manifest = json!({"name": "bufferutil", "scripts": {"install": "node-gyp-build"}, + "dependencies": {"node-gyp-build": "^4.3.0"}}); + let out = render_pinned_entry( + &bufferutil, + &Pin { + manifest: Some(&manifest), + ..pin + }, + ); + assert_eq!( + &out[3..5], + [" dependencies:", " node-gyp-build: \"npm:^4.3.0\""] + ); + assert!(!has_implicit_node_gyp(&out), "{out:?}"); + + // A manifest declaring node-gyp keeps the line; no manifest keeps + // the registry entry as is. + let declared = json!({"dependencies": {"node-gyp": "latest"}}); + for manifest in [Some(&declared), None] { + let out = render_pinned_entry( + &nan, + &Pin { + key_line: &key, + resolution: &resolution, + checksum: None, + manifest, + }, + ); + assert!(has_implicit_node_gyp(&out), "{out:?}"); + } + } + + /// The restore side of #737: yarn's npm resolver adds the dependency + /// for any script mentioning node-gyp unless the document declares + /// one, and the restored entry gets it back in yarn's place. + #[test] + fn registry_implicit_node_gyp_round_trips() { + assert!(registry_adds_node_gyp( + &json!({"scripts": {"install": "node-gyp rebuild"}}) + )); + assert!(registry_adds_node_gyp( + &json!({"scripts": {"install": "node-gyp-build"}, "dependencies": {"node-gyp-build": "^4"}}) + )); + assert!(!registry_adds_node_gyp( + &json!({"scripts": {"test": "tap"}}) + )); + assert!(!registry_adds_node_gyp(&json!({}))); + for field in ["dependencies", "optionalDependencies", "peerDependencies"] { + assert!(!registry_adds_node_gyp(&json!({ + "scripts": {"install": "node-gyp rebuild"}, field: {"node-gyp": "^10"} + }))); + } + + let lines = |text: &str| text.lines().map(str::to_string).collect::>(); + let bare = lines( + "\"nan@npm:2.22.0\":\n version: 2.22.0\n resolution: \"nan@npm:2.22.0\"\n \ + checksum: 10c0/d5\n languageName: node\n linkType: hard", + ); + assert_eq!( + with_implicit_node_gyp(&bare).join("\n"), + "\"nan@npm:2.22.0\":\n version: 2.22.0\n resolution: \"nan@npm:2.22.0\"\n \ + dependencies:\n node-gyp: \"npm:latest\"\n checksum: 10c0/d5\n \ + languageName: node\n linkType: hard" + ); + let deps = lines( + "k:\n version: 4.0.8\n dependencies:\n \"@a/b\": \"npm:1\"\n \ + node-gyp-build: \"npm:^4.3.0\"\n checksum: c", + ); + let out = with_implicit_node_gyp(&deps); + assert_eq!( + &out[2..6], + [ + " dependencies:", + " \"@a/b\": \"npm:1\"", + " node-gyp: \"npm:latest\"", + " node-gyp-build: \"npm:^4.3.0\"" + ] + ); + assert_eq!(with_implicit_node_gyp(&out), out, "already present"); + } + #[test] fn manifest_bin_reads_like_yarn() { assert_eq!( diff --git a/crates/socket-patch-core/src/formats/yarn/berry_prune.rs b/crates/socket-patch-core/src/formats/yarn/berry_prune.rs new file mode 100644 index 000000000..1a58ff50c --- /dev/null +++ b/crates/socket-patch-core/src/formats/yarn/berry_prune.rs @@ -0,0 +1,252 @@ +//! Dropping the berry lock entries a pin leaves unreachable. +//! +//! Yarn writes only the entries the project's dependency graph reaches. A +//! pin whose entry depends on less than the registry entry it replaced +//! (the npm resolver's implicit `node-gyp`, #737) cuts edges, and every +//! entry only those edges reached is one yarn deletes on the next install, +//! so `yarn install --immutable` fails YN0028 while it stays. +//! +//! [`prune_unreferenced`] walks out from the cut descriptors and drops what +//! nothing else references. It errs on keeping: an entry the walk cannot +//! prove unreferenced (a workspace, a `patch:` entry, a package a +//! `resolutions` selector names, or one whose descriptor any other entry +//! mentions) stays. + +use std::collections::BTreeSet; + +use super::patterns::{split_berry_key_patterns, split_pattern}; +use super::stanzas::stanza_key; + +/// One entry [`prune_unreferenced`] changed: its text before, and after +/// (`None` when the whole entry went). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct Pruned { + pub before: String, + pub after: Option, +} + +/// The descriptors (`name@range`) an entry's `dependencies:` map names. +pub(crate) fn dependency_descriptors(stanza: &str) -> Vec { + let mut out = Vec::new(); + let mut in_deps = false; + for line in stanza.lines().skip(1) { + if let Some(field) = line.strip_prefix(" ").filter(|r| !r.starts_with(' ')) { + in_deps = field.trim_end() == "dependencies:"; + continue; + } + if !in_deps { + continue; + } + let Some(rest) = line.strip_prefix(" ").filter(|r| !r.starts_with(' ')) else { + continue; + }; + let (name, range) = match rest.strip_prefix('"') { + Some(quoted) => match quoted.split_once("\": ") { + Some((name, range)) => (name, range), + None => continue, + }, + None => match rest.split_once(": ") { + Some((name, range)) => (name, range), + None => continue, + }, + }; + let range = range.trim_end(); + let range = range + .strip_prefix('"') + .and_then(|r| r.strip_suffix('"')) + .unwrap_or(range); + out.push(format!("{name}@{range}")); + } + out +} + +/// The descriptors `before` depends on that `after` (the same entry, +/// rewritten) no longer does: the edges a rewrite cut. +pub(crate) fn cut_descriptors(before: &str, after: &str) -> Vec { + let kept: BTreeSet = dependency_descriptors(after).into_iter().collect(); + dependency_descriptors(before) + .into_iter() + .filter(|d| !kept.contains(d)) + .collect() +} + +/// Drop from `stanzas` every entry (or key descriptor of a shared entry) +/// that only the `cut` descriptors reached, walking on through what each +/// dropped entry depended on. `protected` names packages a `resolutions` +/// selector routes, which are never dropped. Returns what changed, in +/// order; a changed key is also pushed to `moved` (the stanza view sorts +/// it back into place). +pub(crate) fn prune_unreferenced( + stanzas: &mut Vec, + cut: Vec, + protected: &BTreeSet, + moved: &mut Vec, +) -> Vec { + let mut out = Vec::new(); + let mut queue = cut; + let mut seen = BTreeSet::new(); + while let Some(descriptor) = queue.pop() { + if !seen.insert(descriptor.clone()) { + continue; + } + let Some((name, _)) = split_pattern(&descriptor) else { + continue; + }; + if protected.contains(name) || referenced(stanzas, &descriptor) { + continue; + } + let Some(idx) = stanzas.iter().position(|s| { + stanza_key(s).is_some_and(|k| split_berry_key_patterns(k).contains(&descriptor)) + }) else { + continue; + }; + let key = stanza_key(&stanzas[idx]).unwrap_or_default(); + let patterns = split_berry_key_patterns(key); + if patterns + .iter() + .any(|p| p.contains("@workspace:") || p.contains("@patch:") || p.contains("@portal:")) + { + continue; + } + let before = stanzas[idx].clone(); + let rest: Vec<&String> = patterns.iter().filter(|p| **p != descriptor).collect(); + if rest.is_empty() { + stanzas.remove(idx); + queue.extend(dependency_descriptors(&before)); + out.push(Pruned { + before, + after: None, + }); + } else { + let new_key = format!( + "\"{}\"", + rest.iter() + .map(|p| p.as_str()) + .collect::>() + .join(", ") + ); + let body = before.split_once('\n').map_or("", |(_, body)| body); + let after = if body.is_empty() { + format!("{new_key}:") + } else { + format!("{new_key}:\n{body}") + }; + stanzas[idx] = after.clone(); + moved.push(new_key); + out.push(Pruned { + before, + after: Some(after), + }); + } + } + out +} + +/// Whether any entry of `stanzas` still reaches `descriptor`: a +/// `dependencies:` map naming it, or any mention of it in another entry's +/// text (a `patch:` locator embeds its base descriptor, `:` spelled +/// `%3A`). The entry keyed by it does not count. +fn referenced(stanzas: &[String], descriptor: &str) -> bool { + let encoded = descriptor.replace(':', "%3A"); + stanzas.iter().any(|s| { + let own = stanza_key(s) + .is_some_and(|k| split_berry_key_patterns(k).iter().any(|p| p == descriptor)); + if dependency_descriptors(s).iter().any(|d| d == descriptor) { + return true; + } + !own && stanza_key(s).is_some() && s.contains(&encoded) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn stanzas(text: &str) -> Vec { + text.split("\n\n").map(str::to_string).collect() + } + + #[test] + fn descriptors_read_quoted_and_bare_names() { + let s = "\"tar@npm:^7.5.7\":\n version: 7.5.22\n dependencies:\n \ + \"@isaacs/fs-minipass\": \"npm:^4.0.0\"\n chownr: \"npm:^3.0.0\"\n \ + peerDependencies:\n x: ^1\n checksum: c"; + assert_eq!( + dependency_descriptors(s), + ["@isaacs/fs-minipass@npm:^4.0.0", "chownr@npm:^3.0.0"] + ); + } + + #[test] + fn walk_drops_only_what_nothing_else_reaches() { + let mut lock = stanzas( + "__metadata:\n version: 8\n\n\ + \"a@npm:^1\":\n version: 1.0.0\n dependencies:\n shared: \"npm:^1\"\n\n\ + \"app@workspace:.\":\n version: 0.0.0-use.local\n dependencies:\n \ + a: \"npm:^1\"\n shared: \"npm:^1.1\"\n\n\ + \"gone@npm:latest\":\n version: 2.0.0\n dependencies:\n \ + leaf: \"npm:^1\"\n shared: \"npm:^1\"\n prot: \"npm:^1\"\n\n\ + \"leaf@npm:^1\":\n version: 1.0.0\n\n\ + \"prot@npm:^1\":\n version: 1.0.0\n\n\ + \"shared@npm:^1, shared@npm:^1.1\":\n version: 1.1.0", + ); + let mut moved = Vec::new(); + let protected = BTreeSet::from(["prot".to_string()]); + let pruned = prune_unreferenced( + &mut lock, + vec!["gone@npm:latest".to_string()], + &protected, + &mut moved, + ); + assert_eq!(pruned.len(), 2, "{pruned:?}"); + let keys: Vec<&str> = lock.iter().filter_map(|s| stanza_key(s)).collect(); + assert_eq!( + keys, + [ + "__metadata", + "\"a@npm:^1\"", + "\"app@workspace:.\"", + "\"prot@npm:^1\"", + "\"shared@npm:^1, shared@npm:^1.1\"" + ] + ); + assert!(moved.is_empty()); + + // A shared key loses only the descriptor nothing reaches any more. + let mut lock = stanzas( + "\"app@workspace:.\":\n dependencies:\n x: \"npm:^1.1\"\n\n\ + \"x@npm:^1, x@npm:^1.1\":\n version: 1.1.0", + ); + let pruned = prune_unreferenced( + &mut lock, + vec!["x@npm:^1".to_string()], + &BTreeSet::new(), + &mut moved, + ); + assert_eq!(lock[1], "\"x@npm:^1.1\":\n version: 1.1.0"); + assert_eq!(pruned[0].after.as_deref(), Some(lock[1].as_str())); + assert_eq!(moved, ["\"x@npm:^1.1\""]); + } + + #[test] + fn patch_and_workspace_entries_keep_their_bases() { + let mut lock = stanzas( + "\"r@npm:^1\":\n version: 1.0.0\n\n\ + \"r@patch:r@npm%3A^1#optional!builtin\":\n version: 1.0.0\n\n\ + \"w@workspace:w\":\n version: 0.0.0", + ); + let before = lock.clone(); + let pruned = prune_unreferenced( + &mut lock, + vec![ + "r@npm:^1".to_string(), + "r@patch:r@npm%3A^1#optional!builtin".to_string(), + "w@workspace:w".to_string(), + ], + &BTreeSet::new(), + &mut Vec::new(), + ); + assert!(pruned.is_empty()); + assert_eq!(lock, before); + } +} diff --git a/crates/socket-patch-core/src/formats/yarn/blocks.rs b/crates/socket-patch-core/src/formats/yarn/blocks.rs index 0cabd1463..a29fa27a7 100644 --- a/crates/socket-patch-core/src/formats/yarn/blocks.rs +++ b/crates/socket-patch-core/src/formats/yarn/blocks.rs @@ -269,14 +269,18 @@ pub(crate) fn berry_lock_locks(lock: &str, name: &str, version: &str) -> bool { }) } -/// The entries of the berry lock `lock` that carry a `bin:` map: the only -/// ones whose pin needs the served tarball's own package.json (#718). -/// Scanned once per lock, so the per-dep check in -/// [`berry_pin_needs_manifest`] only walks these (usually none). +/// The entries of the berry lock `lock` that carry a `bin:` map (#718) or +/// the npm resolver's implicit `node-gyp` dependency (#737): the only ones +/// whose pin needs the served tarball's own package.json. Scanned once per +/// lock, so the per-dep check in [`berry_pin_needs_manifest`] only walks +/// these (usually none). pub(crate) fn berry_bin_entries(lock: &str) -> Vec { scan_blocks(lock) .into_iter() - .filter(|block| block.lines.iter().skip(1).any(|l| is_body_field(l, "bin"))) + .filter(|block| { + block.lines.iter().skip(1).any(|l| is_body_field(l, "bin")) + || super::berry_entry::has_implicit_node_gyp(&block.lines) + }) .collect() } diff --git a/crates/socket-patch-core/src/formats/yarn/mod.rs b/crates/socket-patch-core/src/formats/yarn/mod.rs index 5e6dba3fd..143819ee0 100644 --- a/crates/socket-patch-core/src/formats/yarn/mod.rs +++ b/crates/socket-patch-core/src/formats/yarn/mod.rs @@ -7,7 +7,8 @@ //! * where yarn 1 installs a block's copy from ([`source`]); //! * the stanza view the hosted berry writers re-key and re-order //! entries in ([`stanzas`]); -//! * the berry pinned-entry renderer ([`berry_entry`]). +//! * the berry pinned-entry renderer ([`berry_entry`]) and the walk that +//! drops the entries a pin leaves unreachable ([`berry_prune`]). //! //! The vendored backends (`vendor::yarn_classic_lock`, //! `vendor::yarn_berry_lock`), the hosted rewriters and restorers @@ -16,6 +17,7 @@ pub(crate) mod berry_entry; pub mod berry_gates; +pub(crate) mod berry_prune; pub(crate) mod blocks; pub(crate) mod patterns; pub(crate) mod source; diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs index e1f023d25..d8f95e0db 100644 --- a/crates/socket-patch-core/src/hosted/engine.rs +++ b/crates/socket-patch-core/src/hosted/engine.rs @@ -1080,10 +1080,11 @@ pub fn wheel_metadata_unavailable(dep: &DepOverride, detail: &str) -> SkippedPat /// The npm candidates whose yarn berry pin needs the served tarball's own /// `package.json`, in candidate order: yarn builds a tarball entry's `bin:` /// from that manifest, not from the registry metadata the locked `npm:` -/// entry came from, and the two spell bin paths differently (#718). Only an -/// entry the pin would re-key that carries a `bin:` map needs it (see -/// `berry_pin_needs_manifest`; a fork alias never counts), so a berry -/// project without bins fetches nothing. +/// entry came from, and the two spell bin paths differently (#718); only the +/// npm resolver adds an implicit `node-gyp` dependency (#737). Only an entry +/// the pin would re-key that carries a `bin:` map or that dependency needs it +/// (see `berry_pin_needs_manifest`; a fork alias never counts), so a berry +/// project with neither fetches nothing. pub fn yarn_berry_manifest_targets<'a>( candidates: &'a [Candidate], files: &BTreeMap, diff --git a/crates/socket-patch-core/src/patch/redirect/mod.rs b/crates/socket-patch-core/src/patch/redirect/mod.rs index b916e1196..5f36a4174 100644 --- a/crates/socket-patch-core/src/patch/redirect/mod.rs +++ b/crates/socket-patch-core/src/patch/redirect/mod.rs @@ -65,7 +65,8 @@ use crate::formats::gem::{lock_lists_direct_dependency, locked_specs as gem_lock #[cfg(test)] use crate::formats::pnpm::hosted::pnpm_unrewritten_instances; use crate::formats::pnpm::plan_hosted; -use crate::formats::yarn::berry_entry::{manifest_bin, render_pinned_entry, Pin}; +use crate::formats::yarn::berry_entry::{render_pinned_entry, Pin}; +use crate::formats::yarn::berry_prune::{cut_descriptors, prune_unreferenced}; pub(crate) use crate::formats::yarn::is_berry_lock; pub mod gradle; #[cfg(test)] @@ -4204,6 +4205,9 @@ fn rewrite_yarn_berry_with_manifests( // entries by key, so each one moves to its sorted position at the end. let mut moved_keys: Vec = Vec::new(); let mut changed = false; + // The dependency edges the pins cut (the registry entry's implicit + // `node-gyp`, #737): the entries only they reached go too. + let mut cut: Vec = Vec::new(); for dep in &npm { let fname = full_name(dep); // The API hands the prefixed `10c0/`; a yarn 4.0.x lock spells @@ -4517,17 +4521,17 @@ fn rewrite_yarn_berry_with_manifests( // The entry yarn writes for those resolutions: the key, resolution // and checksum change, `bin:` comes from the served tarball's own // package.json when the caller fetched it (#718; yarn builds a - // tarball entry from it, not from the registry metadata), every + // tarball entry from it, not from the registry metadata), the npm + // resolver's implicit `node-gyp` dependency goes (#737), every // other field carries over, and all of them sit in yarn's order // (#697). let new_key = format!("\"{fname}@{}\"", dep.artifact_url); let key_line = format!("{new_key}:"); let resolution = format!("{fname}@{}", dep.artifact_url); - let tarball_bin = manifests + let tarball_manifest = manifests .get(&dep.artifact_url) .and_then(|text| serde_json::from_str::(text).ok()) - .filter(Value::is_object) - .map(|manifest| manifest_bin(&manifest)); + .filter(Value::is_object); let body: Vec<&str> = block.lines().skip(1).collect(); let rewritten = render_pinned_entry( &body, @@ -4535,10 +4539,11 @@ fn rewrite_yarn_berry_with_manifests( key_line: &key_line, resolution: &resolution, checksum: checksum.as_deref(), - bin: tarball_bin.as_ref(), + manifest: tarball_manifest.as_ref(), }, ) .join("\n"); + cut.extend(cut_descriptors(&block, &rewritten)); for (selector, original) in pin.apply(manifest_obj, &dep.artifact_url) { manifest_changed = true; result.edits.push(FileEdit { @@ -4575,6 +4580,42 @@ fn rewrite_yarn_berry_with_manifests( .confirmed_yarn_berry_uuids .insert(dep.patch_uuid.clone()); } + if changed && !cut.is_empty() { + let protected: std::collections::BTreeSet = manifest + .as_ref() + .and_then(|m| m.get("resolutions")) + .and_then(Value::as_object) + .map(|table| { + table + .keys() + .filter_map(|sel| { + crate::formats::yarn::patterns::resolution_selector_target(sel) + .map(str::to_string) + }) + .collect() + }) + .unwrap_or_default(); + for pruned in prune_unreferenced(&mut blocks, cut, &protected, &mut moved_keys) { + let key = stanza_key(&pruned.before) + .map(|k| k.trim_matches('"').to_string()) + .unwrap_or_default(); + result.edits.push(FileEdit { + path: "yarn.lock".into(), + kind: "redirect_yarn_berry_entry_pruned".into(), + action: if pruned.after.is_some() { + "rewritten" + } else { + "removed" + } + .into(), + key: Some(key), + original: Some(Value::String(doc.on_disk(&pruned.before).into_owned())), + new: pruned + .after + .map(|after| Value::String(doc.on_disk(&after).into_owned())), + }); + } + } if changed { doc.stanzas = blocks; result @@ -4596,6 +4637,7 @@ fn rewrite_yarn_berry_with_manifests( result.files.remove("yarn.lock"); result.edits.retain(|e| { e.kind != "redirect_yarn_berry_entry" + && e.kind != "redirect_yarn_berry_entry_pruned" && e.kind != "redirect_yarn_berry_resolution" }); for dep in &npm { @@ -22133,6 +22175,119 @@ packages: ); } + /// #737: a yarn 4.12.0 project depending on nan@2.22.0 and + /// bufferutil@4.0.8 (fixtures are yarn's own output). The registry + /// entries carry the npm resolver's implicit `node-gyp: "npm:latest"`, + /// which yarn never gives a tarball-locator entry, and with it gone the + /// whole node-gyp subtree is unreachable. Each pin must match the lock + /// yarn writes for the same `resolutions`, byte for byte. + #[test] + fn issue_737_pin_drops_the_implicit_node_gyp_and_the_subtree_only_it_reached() { + const BEFORE: &str = + include_str!("../../../tests/fixtures/yarn-berry-node-gyp-before.lock"); + const NAN_ONLY: &str = + include_str!("../../../tests/fixtures/yarn-berry-node-gyp-nan-only.lock"); + const BOTH: &str = include_str!("../../../tests/fixtures/yarn-berry-node-gyp-both.lock"); + fn checksum_of(lock: &str, key: &str) -> String { + let at = lock.find(key).expect(key); + let line = lock[at..] + .lines() + .find(|l| l.starts_with(" checksum: ")) + .unwrap(); + line[" checksum: ".len()..].to_string() + } + let nan_url = berry_hosted_url("nan", "nan", "2.22.0"); + let bu_url = berry_hosted_url("bufferutil", "bufferutil", "4.0.8"); + let nan = berry_override( + "nan", + "2.22.0", + &nan_url, + &checksum_of(BEFORE, "\"nan@npm:2.22.0\":"), + ); + let bu = berry_override( + "bufferutil", + "4.0.8", + &bu_url, + &checksum_of(BEFORE, "\"bufferutil@npm:4.0.8\":"), + ); + let entries = berry_bin_entries(BEFORE); + assert!(berry_pin_needs_manifest(&entries, &nan)); + assert!(berry_pin_needs_manifest(&entries, &bu)); + let mut manifests = BTreeMap::new(); + manifests.insert( + nan_url.clone(), + r#"{"name":"nan","version":"2.22.0","scripts":{"rebuild-tests":"node-gyp rebuild --directory test"}}"# + .to_string(), + ); + manifests.insert( + bu_url.clone(), + r#"{"name":"bufferutil","version":"4.0.8","scripts":{"install":"node-gyp-build"},"dependencies":{"node-gyp-build":"^4.3.0"}}"# + .to_string(), + ); + let ours = |lock: &str| { + lock.replace("https://registry.npmjs.org/nan/-/nan-2.22.0.tgz", &nan_url) + .replace( + "https://registry.npmjs.org/bufferutil/-/bufferutil-4.0.8.tgz", + &bu_url, + ) + }; + let files = berry_files(BEFORE.to_string(), berry_manifest()); + + // nan alone: bufferutil still reaches node-gyp, so only the + // dependency line goes. + let r = rewrite_registry_redirect_with_python_metadata( + &files, + std::slice::from_ref(&nan), + &manifests, + ); + assert!(r.warnings.is_empty(), "{:?}", r.warnings); + assert_eq!(r.files["yarn.lock"], ours(NAN_ONLY)); + assert!(!r + .edits + .iter() + .any(|e| e.kind == "redirect_yarn_berry_entry_pruned")); + + // Both: nothing reaches node-gyp any more, and yarn drops its + // subtree (shared entries such as minipass stay while reached). + let r = rewrite_registry_redirect_with_python_metadata( + &files, + &[nan.clone(), bu.clone()], + &manifests, + ); + assert!(r.warnings.is_empty(), "{:?}", r.warnings); + assert_eq!(r.files["yarn.lock"], ours(BOTH)); + let pruned: Vec<&FileEdit> = r + .edits + .iter() + .filter(|e| e.kind == "redirect_yarn_berry_entry_pruned") + .collect(); + // 20 entries go; minipass's shared key first loses one descriptor + // (one edit), then the other (a second). + assert_eq!( + pruned.iter().filter(|e| e.action == "removed").count(), + 20, + "{pruned:#?}" + ); + assert_eq!( + pruned + .iter() + .filter(|e| e.action == "rewritten") + .filter_map(|e| e.key.as_deref()) + .collect::>(), + ["minipass@npm:^7.0.4, minipass@npm:^7.1.2"], + "{pruned:#?}" + ); + assert!(pruned.iter().all(|e| e.original.is_some())); + + // Re-running on the pinned lock is a no-op. + let pinned = berry_files( + r.files["yarn.lock"].clone(), + r.files["package.json"].clone(), + ); + let again = rewrite_registry_redirect_with_python_metadata(&pinned, &[nan, bu], &manifests); + assert!(!again.files.contains_key("yarn.lock"), "{:?}", again.files); + } + /// Only an entry the berry pin would re-key, with a `bin:` map, needs the /// served manifest: a fork alias (`left-pad@npm:other@…`) at the same /// version never queues a fetch whose failure would drop the patch. diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/client.rs b/crates/socket-patch-core/src/patch/redirect/upstream/client.rs index 1c055b800..a753729c8 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/client.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/client.rs @@ -21,6 +21,10 @@ pub(crate) struct NpmDist { pub integrity: Option, /// The hex sha1 `dist.shasum`. pub shasum: Option, + /// Whether yarn's npm resolver gives this version the implicit + /// `node-gyp: "npm:latest"` dependency (see + /// `formats::yarn::berry_entry::registry_adds_node_gyp`, #737). + pub node_gyp: bool, } /// A Go module version's two go.sum hashes. @@ -295,6 +299,7 @@ impl UpstreamClient { tarball, integrity: str_field("integrity"), shasum: str_field("shasum"), + node_gyp: crate::formats::yarn::berry_entry::registry_adds_node_gyp(&doc), }) } @@ -840,6 +845,7 @@ mod tests { tarball: format!("{}/archive.tgz", server.uri()), integrity: registry_sri, shasum: registry_sha1, + node_gyp: false, }), ); for _ in 0..2 { @@ -900,6 +906,7 @@ mod tests { tarball: format!("{}/archive.tgz", server.uri()), integrity: Some("sha512-other".into()), shasum: None, + node_gyp: false, }), ); assert!( diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs index c3d0d26e7..e1e64d7f4 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/npm.rs @@ -743,6 +743,32 @@ async fn restore_berry( { lines = pinned; } + // The npm resolver's implicit `node-gyp` dependency, which the pin + // dropped (#737), comes back while the lock still holds the entry + // it resolves to; without it only `yarn install` can re-resolve + // that subtree. + if dist.node_gyp && !crate::formats::yarn::berry_entry::has_implicit_node_gyp(&lines) { + let resolvable = blocks.iter().any(|b| { + stanza_key(b).is_some_and(|k| { + split_berry_key_patterns(k) + .iter() + .any(|p| p == "node-gyp@npm:latest") + }) + }); + if resolvable { + lines = crate::formats::yarn::berry_entry::with_implicit_node_gyp(&lines); + } else { + result.warnings.push(( + "yarn_berry_node_gyp_unresolved", + format!( + "{name}@{version}: yarn gives the restored registry entry an \ + implicit `node-gyp` dependency that {rel} no longer resolves; run \ + `yarn install` once to add it back (until then a hardened or \ + `--refresh-lockfile` install reports the lockfile as modified)" + ), + )); + } + } if let Some(key) = key { lines[0] = format!("{key}:"); moved.push(key); @@ -2407,6 +2433,7 @@ mod tests { tarball: tarball.to_string(), integrity: None, shasum: None, + node_gyp: false, }, from_project, }; diff --git a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs index 13596dfb2..1b0cffc94 100644 --- a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs @@ -44,10 +44,12 @@ use serde_json::Value; use sha2::{Digest, Sha256, Sha512}; use crate::constants::SOCKET_DIR; -use crate::formats::yarn::berry_entry::{manifest_bin, render_pinned_entry, Pin}; +use crate::formats::yarn::berry_entry::{render_pinned_entry, Pin}; use crate::formats::yarn::berry_gates::{self, BerryGate, Yarnrc, SUPPORTED_CACHE_KEY}; +use crate::formats::yarn::berry_prune::{cut_descriptors, prune_unreferenced}; use crate::formats::yarn::blocks::{berry_field, block_eol, replace_block, scan_blocks, LockBlock}; use crate::formats::yarn::patterns::{pattern_real_name, split_berry_key_patterns, split_pattern}; +use crate::formats::yarn::stanzas::{stanza_key, BerryStanzas}; use crate::manifest::schema::PatchRecord; use crate::patch::apply::PatchSources; use crate::utils::fs::{ @@ -83,6 +85,11 @@ static PKG_JSON_MEMO: ParseMemo = ParseMemo::new(); /// Wiring kinds this backend owns. const KIND_RESOLUTION: &str = "yarn_berry_resolution"; const KIND_LOCK_ENTRY: &str = "yarn_berry_lock_entry"; +/// An entry the pin left unreachable and vendoring dropped (or a shared +/// key that lost a descriptor, #737): `original` holds the entry's lines, +/// `new` the rewritten lines (absent when the whole entry went). Revert +/// puts the original back. +const KIND_LOCK_ENTRY_PRUNED: &str = "yarn_berry_lock_entry_pruned"; /// Vendor one installed npm package into a yarn-berry (4.x, cacheKey 10c0) /// project. Same contract as [`super::npm_lock::vendor_npm`]: refuse-early, @@ -340,14 +347,14 @@ impl NpmLockBackend for YarnBerryBackend { let lock_key = format!("\"{name}@file:./{rel_tgz}::locator={locator}\""); // Yarn builds a `file:` entry from the tarball's own package.json, // whose `bin` keeps its published spelling where the registry - // entry's is normalized (#718). A tarball without a readable - // manifest keeps the registry's map (yarn cannot install it either - // way). - let tarball_bin = crate::patch::package::read_archive_bytes_to_map(&tgz_bytes) + // entry's is normalized (#718), and which never gets the npm + // resolver's implicit `node-gyp` dependency (#737). A tarball + // without a readable manifest keeps the registry's fields (yarn + // cannot install it either way). + let tarball_manifest = crate::patch::package::read_archive_bytes_to_map(&tgz_bytes) .ok() .and_then(|members| serde_json::from_slice::(members.get(PACKAGE_JSON)?).ok()) - .filter(Value::is_object) - .map(|manifest| manifest_bin(&manifest)); + .filter(Value::is_object); // The exact entry yarn 4 emits for a resolutions-driven `file:` // tarball (the B3 fixture shape), fields in yarn's order (#697). let lock_key_line = format!("{lock_key}:"); @@ -357,7 +364,7 @@ impl NpmLockBackend for YarnBerryBackend { key_line: &lock_key_line, resolution: &resolution, checksum: Some(&checksum), - bin: tarball_bin.as_ref(), + manifest: tarball_manifest.as_ref(), }, ); @@ -385,12 +392,41 @@ impl NpmLockBackend for YarnBerryBackend { let new_pkg_bytes = JsonLayout::of(&String::from_utf8_lossy(&pkg_bytes)) .render(&new_pkg) .map_err(|e| format!("cannot serialize {PACKAGE_JSON}: {e}"))?; - let new_lock_text = replace_block( + let mut new_lock_text = replace_block( &lock_text, target, &new_lines, block_eol(&lock_text, target), ); + // The entries only the dropped dependency edges reached go too, or + // yarn deletes them on the next install (#737). + let pruned = { + let cut = cut_descriptors(&target.lines.join("\n"), &new_lines.join("\n")); + if cut.is_empty() { + Vec::new() + } else { + let mut doc = BerryStanzas::parse(&new_lock_text); + let mut moved = Vec::new(); + let protected = new_pkg + .get("resolutions") + .and_then(Value::as_object) + .map(|table| { + table + .keys() + .filter_map(|sel| { + crate::formats::yarn::patterns::resolution_selector_target(sel) + .map(str::to_string) + }) + .collect() + }) + .unwrap_or_default(); + let pruned = prune_unreferenced(&mut doc.stanzas, cut, &protected, &mut moved); + if !pruned.is_empty() { + new_lock_text = doc.render(&moved); + } + pruned + } + }; commit_pair( project_root, &new_pkg_bytes, @@ -429,6 +465,22 @@ impl NpmLockBackend for YarnBerryBackend { new: Some(lines_to_json(&new_lines)), }, ]; + let stanza_lines = + |text: &str| -> Vec { text.lines().map(str::to_string).collect() }; + let mut wiring = wiring; + wiring.extend(pruned.into_iter().map(|p| { + WiringRecord { + file: YARN_LOCK.to_string(), + kind: KIND_LOCK_ENTRY_PRUNED.to_string(), + action: WiringAction::Rewritten, + key: stanza_key(&p.before).map(str::to_string), + original: Some(lines_to_json(&stanza_lines(&p.before))), + new: p + .after + .as_deref() + .map(|after| lines_to_json(&stanza_lines(after))), + } + })); Ok(Some(NpmCommit { wiring, yarn_berry10c0: packed @@ -714,6 +766,10 @@ pub async fn revert_yarn_berry_opts( Ok(mut text) => { let mut changed = false; for rec in lock_recs { + if rec.kind == KIND_LOCK_ENTRY_PRUNED { + changed |= restore_pruned_entry(&mut text, rec, &mut outcome.warnings); + continue; + } changed |= revert_recorded_block( &mut text, rec, @@ -873,6 +929,74 @@ pub async fn revert_yarn_berry_opts( // ───────────────────────────── revert internals ───────────────────────────── +/// Put back an entry vendoring pruned (see [`KIND_LOCK_ENTRY_PRUNED`]): +/// the rewritten shared entry returns to its original lines, a removed one +/// is inserted where yarn sorts it. Already restored is silent; anything +/// else (the entry was re-resolved since) is left alone with a warning — +/// the user's lock, not the artifact, is at stake, so it is not drift. +fn restore_pruned_entry( + text: &mut String, + rec: &WiringRecord, + warnings: &mut Vec, +) -> bool { + let Some(original) = rec + .original + .as_ref() + .and_then(super::yarn_classic_lock::json_to_lines) + .map(|lines| lines.join("\n")) + else { + return false; + }; + let new = rec + .new + .as_ref() + .and_then(super::yarn_classic_lock::json_to_lines) + .map(|lines| lines.join("\n")); + let mut doc = BerryStanzas::parse(text); + if doc.stanzas.contains(&original) { + return false; + } + let Some(key) = stanza_key(&original).map(str::to_string) else { + return false; + }; + let patterns = split_berry_key_patterns(&key); + let live = |s: &String| { + stanza_key(s).is_some_and(|k| { + split_berry_key_patterns(k) + .iter() + .any(|p| patterns.contains(p)) + }) + }; + match new { + Some(new) => match doc.stanzas.iter().position(|s| *s == new) { + Some(at) => doc.stanzas[at] = original, + None => { + warnings.push(VendorWarning::new( + "vendor_lock_entry_pruned_kept", + format!( + "lock entry `{key}` changed since vendoring dropped a descriptor \ + from it; left alone (run `yarn install` if it is incomplete)" + ), + )); + return false; + } + }, + None if doc.stanzas.iter().any(live) => { + warnings.push(VendorWarning::new( + "vendor_lock_entry_pruned_kept", + format!( + "lock entry `{key}`, which vendoring dropped, was re-added since; \ + left alone" + ), + )); + return false; + } + None => doc.stanzas.push(original), + } + *text = doc.render(&[key]); + true +} + /// Remove our resolutions entry iff the live value still points into our /// uuid dir; drop the `resolutions` table when that leaves it empty (we only /// ever ADD entries — an empty table would be vendor residue). @@ -2237,6 +2361,71 @@ __metadata: assert!(!text.contains("left-pad: bin/cli.js"), "{text}"); } + /// #737: yarn's npm resolver gives a registry entry whose scripts run + /// node-gyp an implicit `node-gyp: "npm:latest"` dependency; the `file:` + /// entry yarn builds from the tarball has none, and the entries only + /// node-gyp reached go with it (yarn drops them on the next install, so + /// `--immutable` fails YN0028 while they stay). Revert puts all of it + /// back byte for byte. + #[tokio::test] + async fn issue_737_implicit_node_gyp_and_its_subtree_are_dropped_and_reverted() { + let lock = B3_BEFORE_LOCK + .replace( + " resolution: \"left-pad@npm:1.3.0\"\n checksum:", + " resolution: \"left-pad@npm:1.3.0\"\n dependencies:\n \ + node-gyp: \"npm:latest\"\n checksum:", + ) + .replace( + "\"vendor-spike@workspace:.\":", + "\"node-gyp@npm:latest\":\n version: 13.1.0\n resolution: \ + \"node-gyp@npm:13.1.0\"\n dependencies:\n nopt: \"npm:^10.0.0\"\n \ + shared: \"npm:^1.0.0\"\n checksum: 10c0/aa\n languageName: node\n \ + linkType: hard\n\n\"nopt@npm:^10.0.0\":\n version: 10.0.1\n resolution: \ + \"nopt@npm:10.0.1\"\n checksum: 10c0/bb\n languageName: node\n \ + linkType: hard\n\n\"shared@npm:^1.0.0\":\n version: 1.0.0\n resolution: \ + \"shared@npm:1.0.0\"\n checksum: 10c0/cc\n languageName: node\n \ + linkType: hard\n\n\"vendor-spike@workspace:.\":", + ) + .replace( + " left-pad: \"npm:1.3.0\"\n", + " left-pad: \"npm:1.3.0\"\n shared: \"npm:^1.0.0\"\n", + ); + let fx = fixture_with(B3_BEFORE_PKG, &lock).await; + let (result, entry, _) = expect_done(fx.vendor(false).await); + assert!(result.success, "{:?}", result.error); + let entry = entry.unwrap(); + + let text = tokio::fs::read_to_string(fx.lock_path()).await.unwrap(); + let (hash6, checksum) = fx.packed_berry_facts().await; + let expected = spike_after_lock(&hash6, &checksum).replace( + "\"vendor-spike@workspace:.\":", + "\"shared@npm:^1.0.0\":\n version: 1.0.0\n resolution: \"shared@npm:1.0.0\"\n \ + checksum: 10c0/cc\n languageName: node\n linkType: hard\n\n\ + \"vendor-spike@workspace:.\":", + ); + let expected = expected.replace( + " left-pad: \"npm:1.3.0\"\n", + " left-pad: \"npm:1.3.0\"\n shared: \"npm:^1.0.0\"\n", + ); + assert_eq!(text, expected); + let pruned: Vec<&WiringRecord> = entry + .wiring + .iter() + .filter(|w| w.kind == KIND_LOCK_ENTRY_PRUNED) + .collect(); + assert_eq!(pruned.len(), 2, "{pruned:#?}"); + + let outcome = revert_yarn_berry(&entry, fx.root(), false).await; + assert!(outcome.success, "{:?}", outcome.error); + assert!(outcome.warnings.is_empty(), "{:?}", outcome.warnings); + assert_eq!( + tokio::fs::read_to_string(fx.lock_path()).await.unwrap(), + lock, + "yarn.lock restored byte-for-byte, pruned entries included" + ); + assert_eq!(tokio::fs::read(fx.pkg_path()).await.unwrap(), fx.pkg_bytes); + } + #[tokio::test] async fn commit_pair_unwinds_package_json_when_the_lock_write_fails() { let tmp = tempfile::tempdir().unwrap(); diff --git a/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-before.lock b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-before.lock new file mode 100644 index 000000000..0d081a53b --- /dev/null +++ b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-before.lock @@ -0,0 +1,226 @@ +# This file is generated by running "yarn install" inside your project. +# Manual changes might be lost - proceed with caution! + +__metadata: + version: 8 + cacheKey: 10c0 + +"@isaacs/fs-minipass@npm:^4.0.0": + version: 4.0.1 + resolution: "@isaacs/fs-minipass@npm:4.0.1" + dependencies: + minipass: "npm:^7.0.4" + checksum: 10c0/c25b6dc1598790d5b55c0947a9b7d111cfa92594db5296c3b907e2f533c033666f692a3939eadac17b1c7c40d362d0b0635dc874cbfe3e70db7c2b07cc97a5d2 + languageName: node + linkType: hard + +"abbrev@npm:^5.0.0": + version: 5.0.0 + resolution: "abbrev@npm:5.0.0" + checksum: 10c0/8e88f5c798ea4562d28c5a3e9ad69e3879890bc5d695d8f2dffb8609be4c890aacc8f80ef4553fdd2c6a62d70c2ce8bc57b38074e383beb7487bdafa9ed42ea5 + languageName: node + linkType: hard + +"app@workspace:.": + version: 0.0.0-use.local + resolution: "app@workspace:." + dependencies: + bufferutil: "npm:4.0.8" + nan: "npm:2.22.0" + languageName: unknown + linkType: soft + +"bufferutil@npm:4.0.8": + version: 4.0.8 + resolution: "bufferutil@npm:4.0.8" + dependencies: + node-gyp: "npm:latest" + node-gyp-build: "npm:^4.3.0" + checksum: 10c0/36cdc5b53a38d9f61f89fdbe62029a2ebcd020599862253fefebe31566155726df9ff961f41b8c97b02b4c12b391ef97faf94e2383392654cf8f0ed68f76e47c + languageName: node + linkType: hard + +"chownr@npm:^3.0.0": + version: 3.0.0 + resolution: "chownr@npm:3.0.0" + checksum: 10c0/43925b87700f7e3893296c8e9c56cc58f926411cce3a6e5898136daaf08f08b9a8eb76d37d3267e707d0dcc17aed2e2ebdf5848c0c3ce95cf910a919935c1b10 + languageName: node + linkType: hard + +"env-paths@npm:^2.2.0": + version: 2.2.1 + resolution: "env-paths@npm:2.2.1" + checksum: 10c0/285325677bf00e30845e330eec32894f5105529db97496ee3f598478e50f008c5352a41a30e5e72ec9de8a542b5a570b85699cd63bd2bc646dbcb9f311d83bc4 + languageName: node + linkType: hard + +"exponential-backoff@npm:^3.1.1": + version: 3.1.3 + resolution: "exponential-backoff@npm:3.1.3" + checksum: 10c0/77e3ae682b7b1f4972f563c6dbcd2b0d54ac679e62d5d32f3e5085feba20483cf28bd505543f520e287a56d4d55a28d7874299941faf637e779a1aa5994d1267 + languageName: node + linkType: hard + +"fdir@npm:^6.5.0": + version: 6.5.0 + resolution: "fdir@npm:6.5.0" + peerDependencies: + picomatch: ^3 || ^4 + peerDependenciesMeta: + picomatch: + optional: true + checksum: 10c0/e345083c4306b3aed6cb8ec551e26c36bab5c511e99ea4576a16750ddc8d3240e63826cc624f5ae17ad4dc82e68a253213b60d556c11bfad064b7607847ed07f + languageName: node + linkType: hard + +"graceful-fs@npm:^4.2.6": + version: 4.2.11 + resolution: "graceful-fs@npm:4.2.11" + checksum: 10c0/386d011a553e02bc594ac2ca0bd6d9e4c22d7fa8cfbfc448a6d148c59ea881b092db9dbe3547ae4b88e55f1b01f7c4a2ecc53b310c042793e63aa44cf6c257f2 + languageName: node + linkType: hard + +"isexe@npm:^4.0.0": + version: 4.0.0 + resolution: "isexe@npm:4.0.0" + checksum: 10c0/5884815115bceac452877659a9c7726382531592f43dc29e5d48b7c4100661aed54018cb90bd36cb2eaeba521092570769167acbb95c18d39afdccbcca06c5ce + languageName: node + linkType: hard + +"minipass@npm:^7.0.4, minipass@npm:^7.1.2": + version: 7.1.3 + resolution: "minipass@npm:7.1.3" + checksum: 10c0/539da88daca16533211ea5a9ee98dc62ff5742f531f54640dd34429e621955e91cc280a91a776026264b7f9f6735947629f920944e9c1558369e8bf22eb33fbb + languageName: node + linkType: hard + +"minizlib@npm:^3.1.0": + version: 3.1.0 + resolution: "minizlib@npm:3.1.0" + dependencies: + minipass: "npm:^7.1.2" + checksum: 10c0/5aad75ab0090b8266069c9aabe582c021ae53eb33c6c691054a13a45db3b4f91a7fb1bd79151e6b4e9e9a86727b522527c0a06ec7d45206b745d54cd3097bcec + languageName: node + linkType: hard + +"nan@npm:2.22.0": + version: 2.22.0 + resolution: "nan@npm:2.22.0" + dependencies: + node-gyp: "npm:latest" + checksum: 10c0/d5d31aefdb218deba308d44867c5f432b4d3aabeb57c70a2b236d62652e9fee7044e5d5afd380d9fef022fe7ebb2f2d6c85ca3cbcac5031aaca3592c844526bb + languageName: node + linkType: hard + +"node-gyp-build@npm:^4.3.0": + version: 4.8.4 + resolution: "node-gyp-build@npm:4.8.4" + bin: + node-gyp-build: bin.js + node-gyp-build-optional: optional.js + node-gyp-build-test: build-test.js + checksum: 10c0/444e189907ece2081fe60e75368784f7782cfddb554b60123743dfb89509df89f1f29c03bbfa16b3a3e0be3f48799a4783f487da6203245fa5bed239ba7407e1 + languageName: node + linkType: hard + +"node-gyp@npm:latest": + version: 13.1.0 + resolution: "node-gyp@npm:13.1.0" + dependencies: + env-paths: "npm:^2.2.0" + exponential-backoff: "npm:^3.1.1" + graceful-fs: "npm:^4.2.6" + nopt: "npm:^10.0.0" + proc-log: "npm:^7.0.0" + semver: "npm:^7.3.5" + tar: "npm:^7.5.7" + tinyglobby: "npm:^0.2.12" + undici: "npm:^8.4.1" + which: "npm:^7.0.0" + bin: + node-gyp: bin/node-gyp.js + checksum: 10c0/c6693e52e0e111b21f5cd8ff303b41b51f40338adcc79ac5c6a02399eba7039d50dc816dcafcc485870291a641f7cb75db8887a9bdd0de537fe635a3a81cba5b + languageName: node + linkType: hard + +"nopt@npm:^10.0.0": + version: 10.0.1 + resolution: "nopt@npm:10.0.1" + dependencies: + abbrev: "npm:^5.0.0" + bin: + nopt: bin/nopt.js + checksum: 10c0/980d89257f9587f3e1f77877ddbf905d6aa3b738ec33e49a4fa1a059a0dd82eb28063982b150654a7ae9de386f2ead60e56172db7d37cf56de545f7392a2a26a + languageName: node + linkType: hard + +"picomatch@npm:^4.0.4": + version: 4.0.7 + resolution: "picomatch@npm:4.0.7" + checksum: 10c0/beb6ae02c43ae44e84883b90830196d9046b1726ead292adcf7f57945e0bb0d992d68563d87e03b484b6f3c9a5c6defda7523477f047d7f0e663f126cc01787f + languageName: node + linkType: hard + +"proc-log@npm:^7.0.0": + version: 7.0.0 + resolution: "proc-log@npm:7.0.0" + checksum: 10c0/b89c2d862604f35fec795477b0c7e376feab3ba0d4f4d291c4e959567442697cf451ac557d0623c1cc38af45a78128b983410f397a10c5d3a67f76c33de4754b + languageName: node + linkType: hard + +"semver@npm:^7.3.5": + version: 7.8.5 + resolution: "semver@npm:7.8.5" + bin: + semver: bin/semver.js + checksum: 10c0/b1f3127a5be8125a94f37188b361c212466c292c6910adce3ec106cff5dc211ccaedc4739c11bb70fda59d6fc1f040a9bca289f4e093451521a2372e5231fe0c + languageName: node + linkType: hard + +"tar@npm:^7.5.7": + version: 7.5.22 + resolution: "tar@npm:7.5.22" + dependencies: + "@isaacs/fs-minipass": "npm:^4.0.0" + chownr: "npm:^3.0.0" + minipass: "npm:^7.1.2" + minizlib: "npm:^3.1.0" + yallist: "npm:^5.0.0" + checksum: 10c0/1311f6be85a8157ac4c9147bae43e13923d2a1aae15e4aa1bd5239e4e03d2cf53cfe103dde7f35832fbb4c938b042856bc8e9a0afd29abd05e2d1608788c4fea + languageName: node + linkType: hard + +"tinyglobby@npm:^0.2.12": + version: 0.2.17 + resolution: "tinyglobby@npm:0.2.17" + dependencies: + fdir: "npm:^6.5.0" + picomatch: "npm:^4.0.4" + checksum: 10c0/7f7bb0f197c88bc4b20c231e0deca4240ca3bf313a88f5a7fee93a872b84966a4d50220947c0455ad07a60b3b360961c5b7fd979222aeb716a9f99b412002e4c + languageName: node + linkType: hard + +"undici@npm:^8.4.1": + version: 8.11.2 + resolution: "undici@npm:8.11.2" + checksum: 10c0/106f6991178fb7562fd90cdec7993c5f807a9428682306091f0c4b27fc474ec291f868e29972c6252988bf061ad193df943b3b99e775be157a28d92bbb95699d + languageName: node + linkType: hard + +"which@npm:^7.0.0": + version: 7.0.0 + resolution: "which@npm:7.0.0" + dependencies: + isexe: "npm:^4.0.0" + bin: + node-which: bin/which.js + checksum: 10c0/ca0b54f198f78bbc4b7c02e34bda8d335cb352e0adb4cbca1c37b1a957af3a879a82c4c27ca6525bc942f548d8b64f816ef6528360af9f3de55ffb9b979b620d + languageName: node + linkType: hard + +"yallist@npm:^5.0.0": + version: 5.0.0 + resolution: "yallist@npm:5.0.0" + checksum: 10c0/a499c81ce6d4a1d260d4ea0f6d49ab4da09681e32c3f0472dee16667ed69d01dae63a3b81745a24bd78476ec4fcf856114cb4896ace738e01da34b2c42235416 + languageName: node + linkType: hard diff --git a/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-both.lock b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-both.lock new file mode 100644 index 000000000..3d7f5a526 --- /dev/null +++ b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-both.lock @@ -0,0 +1,42 @@ +# This file is generated by running "yarn install" inside your project. +# Manual changes might be lost - proceed with caution! + +__metadata: + version: 8 + cacheKey: 10c0 + +"app@workspace:.": + version: 0.0.0-use.local + resolution: "app@workspace:." + dependencies: + bufferutil: "npm:4.0.8" + nan: "npm:2.22.0" + languageName: unknown + linkType: soft + +"bufferutil@https://registry.npmjs.org/bufferutil/-/bufferutil-4.0.8.tgz": + version: 4.0.8 + resolution: "bufferutil@https://registry.npmjs.org/bufferutil/-/bufferutil-4.0.8.tgz" + dependencies: + node-gyp-build: "npm:^4.3.0" + checksum: 10c0/36cdc5b53a38d9f61f89fdbe62029a2ebcd020599862253fefebe31566155726df9ff961f41b8c97b02b4c12b391ef97faf94e2383392654cf8f0ed68f76e47c + languageName: node + linkType: hard + +"nan@https://registry.npmjs.org/nan/-/nan-2.22.0.tgz": + version: 2.22.0 + resolution: "nan@https://registry.npmjs.org/nan/-/nan-2.22.0.tgz" + checksum: 10c0/d5d31aefdb218deba308d44867c5f432b4d3aabeb57c70a2b236d62652e9fee7044e5d5afd380d9fef022fe7ebb2f2d6c85ca3cbcac5031aaca3592c844526bb + languageName: node + linkType: hard + +"node-gyp-build@npm:^4.3.0": + version: 4.8.4 + resolution: "node-gyp-build@npm:4.8.4" + bin: + node-gyp-build: bin.js + node-gyp-build-optional: optional.js + node-gyp-build-test: build-test.js + checksum: 10c0/444e189907ece2081fe60e75368784f7782cfddb554b60123743dfb89509df89f1f29c03bbfa16b3a3e0be3f48799a4783f487da6203245fa5bed239ba7407e1 + languageName: node + linkType: hard diff --git a/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-nan-only.lock b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-nan-only.lock new file mode 100644 index 000000000..320a733f2 --- /dev/null +++ b/crates/socket-patch-core/tests/fixtures/yarn-berry-node-gyp-nan-only.lock @@ -0,0 +1,224 @@ +# This file is generated by running "yarn install" inside your project. +# Manual changes might be lost - proceed with caution! + +__metadata: + version: 8 + cacheKey: 10c0 + +"@isaacs/fs-minipass@npm:^4.0.0": + version: 4.0.1 + resolution: "@isaacs/fs-minipass@npm:4.0.1" + dependencies: + minipass: "npm:^7.0.4" + checksum: 10c0/c25b6dc1598790d5b55c0947a9b7d111cfa92594db5296c3b907e2f533c033666f692a3939eadac17b1c7c40d362d0b0635dc874cbfe3e70db7c2b07cc97a5d2 + languageName: node + linkType: hard + +"abbrev@npm:^5.0.0": + version: 5.0.0 + resolution: "abbrev@npm:5.0.0" + checksum: 10c0/8e88f5c798ea4562d28c5a3e9ad69e3879890bc5d695d8f2dffb8609be4c890aacc8f80ef4553fdd2c6a62d70c2ce8bc57b38074e383beb7487bdafa9ed42ea5 + languageName: node + linkType: hard + +"app@workspace:.": + version: 0.0.0-use.local + resolution: "app@workspace:." + dependencies: + bufferutil: "npm:4.0.8" + nan: "npm:2.22.0" + languageName: unknown + linkType: soft + +"bufferutil@npm:4.0.8": + version: 4.0.8 + resolution: "bufferutil@npm:4.0.8" + dependencies: + node-gyp: "npm:latest" + node-gyp-build: "npm:^4.3.0" + checksum: 10c0/36cdc5b53a38d9f61f89fdbe62029a2ebcd020599862253fefebe31566155726df9ff961f41b8c97b02b4c12b391ef97faf94e2383392654cf8f0ed68f76e47c + languageName: node + linkType: hard + +"chownr@npm:^3.0.0": + version: 3.0.0 + resolution: "chownr@npm:3.0.0" + checksum: 10c0/43925b87700f7e3893296c8e9c56cc58f926411cce3a6e5898136daaf08f08b9a8eb76d37d3267e707d0dcc17aed2e2ebdf5848c0c3ce95cf910a919935c1b10 + languageName: node + linkType: hard + +"env-paths@npm:^2.2.0": + version: 2.2.1 + resolution: "env-paths@npm:2.2.1" + checksum: 10c0/285325677bf00e30845e330eec32894f5105529db97496ee3f598478e50f008c5352a41a30e5e72ec9de8a542b5a570b85699cd63bd2bc646dbcb9f311d83bc4 + languageName: node + linkType: hard + +"exponential-backoff@npm:^3.1.1": + version: 3.1.3 + resolution: "exponential-backoff@npm:3.1.3" + checksum: 10c0/77e3ae682b7b1f4972f563c6dbcd2b0d54ac679e62d5d32f3e5085feba20483cf28bd505543f520e287a56d4d55a28d7874299941faf637e779a1aa5994d1267 + languageName: node + linkType: hard + +"fdir@npm:^6.5.0": + version: 6.5.0 + resolution: "fdir@npm:6.5.0" + peerDependencies: + picomatch: ^3 || ^4 + peerDependenciesMeta: + picomatch: + optional: true + checksum: 10c0/e345083c4306b3aed6cb8ec551e26c36bab5c511e99ea4576a16750ddc8d3240e63826cc624f5ae17ad4dc82e68a253213b60d556c11bfad064b7607847ed07f + languageName: node + linkType: hard + +"graceful-fs@npm:^4.2.6": + version: 4.2.11 + resolution: "graceful-fs@npm:4.2.11" + checksum: 10c0/386d011a553e02bc594ac2ca0bd6d9e4c22d7fa8cfbfc448a6d148c59ea881b092db9dbe3547ae4b88e55f1b01f7c4a2ecc53b310c042793e63aa44cf6c257f2 + languageName: node + linkType: hard + +"isexe@npm:^4.0.0": + version: 4.0.0 + resolution: "isexe@npm:4.0.0" + checksum: 10c0/5884815115bceac452877659a9c7726382531592f43dc29e5d48b7c4100661aed54018cb90bd36cb2eaeba521092570769167acbb95c18d39afdccbcca06c5ce + languageName: node + linkType: hard + +"minipass@npm:^7.0.4, minipass@npm:^7.1.2": + version: 7.1.3 + resolution: "minipass@npm:7.1.3" + checksum: 10c0/539da88daca16533211ea5a9ee98dc62ff5742f531f54640dd34429e621955e91cc280a91a776026264b7f9f6735947629f920944e9c1558369e8bf22eb33fbb + languageName: node + linkType: hard + +"minizlib@npm:^3.1.0": + version: 3.1.0 + resolution: "minizlib@npm:3.1.0" + dependencies: + minipass: "npm:^7.1.2" + checksum: 10c0/5aad75ab0090b8266069c9aabe582c021ae53eb33c6c691054a13a45db3b4f91a7fb1bd79151e6b4e9e9a86727b522527c0a06ec7d45206b745d54cd3097bcec + languageName: node + linkType: hard + +"nan@https://registry.npmjs.org/nan/-/nan-2.22.0.tgz": + version: 2.22.0 + resolution: "nan@https://registry.npmjs.org/nan/-/nan-2.22.0.tgz" + checksum: 10c0/d5d31aefdb218deba308d44867c5f432b4d3aabeb57c70a2b236d62652e9fee7044e5d5afd380d9fef022fe7ebb2f2d6c85ca3cbcac5031aaca3592c844526bb + languageName: node + linkType: hard + +"node-gyp-build@npm:^4.3.0": + version: 4.8.4 + resolution: "node-gyp-build@npm:4.8.4" + bin: + node-gyp-build: bin.js + node-gyp-build-optional: optional.js + node-gyp-build-test: build-test.js + checksum: 10c0/444e189907ece2081fe60e75368784f7782cfddb554b60123743dfb89509df89f1f29c03bbfa16b3a3e0be3f48799a4783f487da6203245fa5bed239ba7407e1 + languageName: node + linkType: hard + +"node-gyp@npm:latest": + version: 13.1.0 + resolution: "node-gyp@npm:13.1.0" + dependencies: + env-paths: "npm:^2.2.0" + exponential-backoff: "npm:^3.1.1" + graceful-fs: "npm:^4.2.6" + nopt: "npm:^10.0.0" + proc-log: "npm:^7.0.0" + semver: "npm:^7.3.5" + tar: "npm:^7.5.7" + tinyglobby: "npm:^0.2.12" + undici: "npm:^8.4.1" + which: "npm:^7.0.0" + bin: + node-gyp: bin/node-gyp.js + checksum: 10c0/c6693e52e0e111b21f5cd8ff303b41b51f40338adcc79ac5c6a02399eba7039d50dc816dcafcc485870291a641f7cb75db8887a9bdd0de537fe635a3a81cba5b + languageName: node + linkType: hard + +"nopt@npm:^10.0.0": + version: 10.0.1 + resolution: "nopt@npm:10.0.1" + dependencies: + abbrev: "npm:^5.0.0" + bin: + nopt: bin/nopt.js + checksum: 10c0/980d89257f9587f3e1f77877ddbf905d6aa3b738ec33e49a4fa1a059a0dd82eb28063982b150654a7ae9de386f2ead60e56172db7d37cf56de545f7392a2a26a + languageName: node + linkType: hard + +"picomatch@npm:^4.0.4": + version: 4.0.7 + resolution: "picomatch@npm:4.0.7" + checksum: 10c0/beb6ae02c43ae44e84883b90830196d9046b1726ead292adcf7f57945e0bb0d992d68563d87e03b484b6f3c9a5c6defda7523477f047d7f0e663f126cc01787f + languageName: node + linkType: hard + +"proc-log@npm:^7.0.0": + version: 7.0.0 + resolution: "proc-log@npm:7.0.0" + checksum: 10c0/b89c2d862604f35fec795477b0c7e376feab3ba0d4f4d291c4e959567442697cf451ac557d0623c1cc38af45a78128b983410f397a10c5d3a67f76c33de4754b + languageName: node + linkType: hard + +"semver@npm:^7.3.5": + version: 7.8.5 + resolution: "semver@npm:7.8.5" + bin: + semver: bin/semver.js + checksum: 10c0/b1f3127a5be8125a94f37188b361c212466c292c6910adce3ec106cff5dc211ccaedc4739c11bb70fda59d6fc1f040a9bca289f4e093451521a2372e5231fe0c + languageName: node + linkType: hard + +"tar@npm:^7.5.7": + version: 7.5.22 + resolution: "tar@npm:7.5.22" + dependencies: + "@isaacs/fs-minipass": "npm:^4.0.0" + chownr: "npm:^3.0.0" + minipass: "npm:^7.1.2" + minizlib: "npm:^3.1.0" + yallist: "npm:^5.0.0" + checksum: 10c0/1311f6be85a8157ac4c9147bae43e13923d2a1aae15e4aa1bd5239e4e03d2cf53cfe103dde7f35832fbb4c938b042856bc8e9a0afd29abd05e2d1608788c4fea + languageName: node + linkType: hard + +"tinyglobby@npm:^0.2.12": + version: 0.2.17 + resolution: "tinyglobby@npm:0.2.17" + dependencies: + fdir: "npm:^6.5.0" + picomatch: "npm:^4.0.4" + checksum: 10c0/7f7bb0f197c88bc4b20c231e0deca4240ca3bf313a88f5a7fee93a872b84966a4d50220947c0455ad07a60b3b360961c5b7fd979222aeb716a9f99b412002e4c + languageName: node + linkType: hard + +"undici@npm:^8.4.1": + version: 8.11.2 + resolution: "undici@npm:8.11.2" + checksum: 10c0/106f6991178fb7562fd90cdec7993c5f807a9428682306091f0c4b27fc474ec291f868e29972c6252988bf061ad193df943b3b99e775be157a28d92bbb95699d + languageName: node + linkType: hard + +"which@npm:^7.0.0": + version: 7.0.0 + resolution: "which@npm:7.0.0" + dependencies: + isexe: "npm:^4.0.0" + bin: + node-which: bin/which.js + checksum: 10c0/ca0b54f198f78bbc4b7c02e34bda8d335cb352e0adb4cbca1c37b1a957af3a879a82c4c27ca6525bc942f548d8b64f816ef6528360af9f3de55ffb9b979b620d + languageName: node + linkType: hard + +"yallist@npm:^5.0.0": + version: 5.0.0 + resolution: "yallist@npm:5.0.0" + checksum: 10c0/a499c81ce6d4a1d260d4ea0f6d49ab4da09681e32c3f0472dee16667ed69d01dae63a3b81745a24bd78476ec4fcf856114cb4896ace738e01da34b2c42235416 + languageName: node + linkType: hard diff --git a/docs/ecosystems.md b/docs/ecosystems.md index 759d7d182..38619f6d1 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -153,7 +153,15 @@ The backticked slug in each row is the value `-e`/`--ecosystems` accepts (e.g. `"left-pad@catalog:"`). Yarn then fetches it without npm registry credentials and hardened mode accepts it. The re-keyed entry is written the way yarn writes it, with its fields in yarn's order and its `bin:` map taken from the served - tarball's own `package.json`. For an entry that has a `bin:` map, the scan + tarball's own `package.json`. Yarn's npm resolver gives a package that runs + `node-gyp` in a script (every package shipping a `binding.gyp`: nan, + bufferutil, utf-8-validate, …) an implicit `node-gyp: "npm:latest"` + dependency, which yarn never gives a tarball entry, so the pin drops it and, + when nothing else needs node-gyp, the lock entries only it reached; vendored + mode does the same, and `vendor --revert` puts them back. Hosted `rollback` + puts the dependency back while the lock still resolves node-gyp; otherwise it + warns `yarn_berry_node_gyp_unresolved` (run `yarn install` once). For an entry + that has a `bin:` map or that implicit dependency, the scan downloads the served tarball to read it. If that download fails, the patch is skipped as `npm_manifest_unavailable`. A user-authored `resolutions` entry for the package is never overwritten (`redirect_yarn_berry_resolutions_conflict`), and