mirror of
https://github.com/microsoft/vscode.git
synced 2026-10-28 13:26:52 +08:00
OSS: Generate ThirdPartyNotices via Component Governance + Helper Scripts (#322410)
This commit is contained in:
Ben Villalobos authored and GitHub committed
1 parent
2311aaab86
commit
fd396ed9a2
19 files changed
+5180
-129
No files matched your search
@@ -0,0 +1,321 @@
|
||||
---
|
||||
applyTo: 'build/azure-pipelines/oss/**'
|
||||
---
|
||||
|
||||
# VS Code OSS Third-Party-Notices pipeline
|
||||
|
||||
This directory contains the thin VS Code-specific layer around Component Governance (CG) for producing OSS third-party notices. The goal is to replace the legacy custom OSS tool with CG plus a gap-filling scanner, so release owners do not need to babysit `ThirdPartyNotices.txt` every release. The generated output should be at least as good as the legacy tool: CG provides the base NOTICE, the local scanner fills known CG gaps, and human-authored overrides cover the small set that automation cannot safely resolve.
|
||||
|
||||
## Architecture
|
||||
|
||||
The product-quality pipeline in `build/azure-pipelines/product-quality-checks.yml` is the main CI entry point.
|
||||
|
||||
1. `notice@0` generates the CG base NOTICE at `ThirdPartyNotices.generated.txt`.
|
||||
2. `scan-licenses.ts` scans local sources CG misses and writes `ThirdPartyNotices.extensions.txt`, plus sibling index files used by the merge step.
|
||||
3. `merge-notices.ts` merges the CG output and scanner output, then applies `cglicenses.json` overrides through `apply-overrides.ts`.
|
||||
4. `check-pr-dependencies.ts` is used by `pr-oss-check.yml` as a PR-time gate. It blocks dependency additions that have no license source.
|
||||
5. If CG is down or `notice@0` emits an empty/non-trivial failure output, the pipeline substitutes the last good `ThirdPartyNotices.generated.txt` artifact from the same branch, then from `main`. Only the CG portion is cached; local scanning and overrides still run fresh against the current commit.
|
||||
|
||||
Final merge output is uploaded as `ThirdPartyNotices.new.txt` in the `notice_output` artifact.
|
||||
|
||||
## Pipeline flow in CI
|
||||
|
||||
In `product-quality-checks.yml`:
|
||||
|
||||
1. Component Detection runs with `ComponentGovernanceComponentDetection@0`.
|
||||
2. `notice@0` writes `$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt` and is `continueOnError: true`.
|
||||
3. The cache fallback checks whether the CG file exists and is larger than 1 KB. If not, it downloads the latest `notice_output` artifact for the current branch, then `main`, and copies the cached `ThirdPartyNotices.generated.txt` into place.
|
||||
4. TypeScript compiles the OSS scripts into `.oss-build-out`:
|
||||
- `apply-overrides.ts`
|
||||
- `scan-licenses.ts`
|
||||
- `parse-notices.ts`
|
||||
- `merge-notices.ts`
|
||||
5. `scan-licenses.js` runs with `--repo`, `--cg`, and `--output`.
|
||||
6. `merge-notices.js` runs with `--cg`, `--extensions`, `--cglicenses`, and `--output`.
|
||||
7. The generated CG file, scanner file, final merged file, and optional cache metadata are uploaded under `notice_output`.
|
||||
|
||||
## Script reference
|
||||
|
||||
### `scan-licenses.ts`
|
||||
|
||||
Purpose: produces the supplemental NOTICE entries that CG misses.
|
||||
|
||||
Invocation in CI:
|
||||
|
||||
```sh
|
||||
node .oss-build-out/scan-licenses.js \
|
||||
--repo "$(Build.SourcesDirectory)" \
|
||||
--cg "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" \
|
||||
--output "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt"
|
||||
```
|
||||
|
||||
Key inputs:
|
||||
|
||||
- `--repo`: VS Code repo root.
|
||||
- `--cg`: optional CG NOTICE. Used to build `cgCovered` and `cgBodies`, so scanner network work is bounded and Cargo stub bodies can be detected.
|
||||
- `node_modules` trees under root, built-in extensions, `remote`, and `build`.
|
||||
- `cgmanifest.json` files.
|
||||
- `Cargo.lock` files.
|
||||
|
||||
Key outputs:
|
||||
|
||||
- Supplemental NOTICE text at `--output`.
|
||||
- Presence index at `<output>.presence.json` unless `--presence` is supplied.
|
||||
- Cargo stub override index at `<output>.stuboverride.json` unless `--stuboverride` is supplied.
|
||||
|
||||
Scanner sections:
|
||||
|
||||
- Section 1 scans built-in extension dependencies. CG skips npm packages with `engines.vscode`, but VS Code ships built-in extensions, so their bundled dependencies are scanned from extension `node_modules` folders.
|
||||
- Section 2 scans root `node_modules` for packages whose LICENSE files exist on disk but whose ClearlyDefined coverage did not produce CG NOTICE text.
|
||||
- Section 3 reads `licenseDetail` from `cgmanifest.json`, and for uncovered git components without inline text it tries `fetchLicenseFromGitRepo()` at the pinned `commitHash`.
|
||||
- Section 4 harvests Rust crate licenses from `Cargo.lock`; details below.
|
||||
- Section 5 enumerates platform-specific npm binary packages; details below.
|
||||
|
||||
### `merge-notices.ts`
|
||||
|
||||
Purpose: combines CG NOTICE entries, scanner entries, Cargo stub replacements, and `cglicenses.json` overrides into one NOTICE file.
|
||||
|
||||
Invocation in CI:
|
||||
|
||||
```sh
|
||||
node .oss-build-out/merge-notices.js \
|
||||
--cg "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" \
|
||||
--extensions "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt" \
|
||||
--cglicenses "$(Build.SourcesDirectory)/cglicenses.json" \
|
||||
--output "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.new.txt"
|
||||
```
|
||||
|
||||
Key behavior:
|
||||
|
||||
- Parses both input NOTICE files.
|
||||
- Deduplicates by lowercased `<name>@<version>`, preserving multiple shipped versions of the same package.
|
||||
- CG normally wins collisions.
|
||||
- Exception: keys listed in `<extensions>.stuboverride.json` let scanner Cargo entries replace CG entries whose body is only an SPDX-expression stub.
|
||||
- Reads `<extensions>.presence.json` so override injection can distinguish present-but-unlicensed packages from stale overrides.
|
||||
- Applies `cglicenses.json` with `applyOverrides()`.
|
||||
- Writes sorted NOTICE output with provenance markers for scanner, Cargo stub override, and manual override entries.
|
||||
|
||||
### `apply-overrides.ts`
|
||||
|
||||
Purpose: reads and applies human-authored `cglicenses.json` entries.
|
||||
|
||||
Key inputs:
|
||||
|
||||
- `cglicenses.json` parsed by `readCglicenses()` after `stripJsonComments()` removes JSONC line comments.
|
||||
- The merged `Map<string, MergedEntry>` from `merge-notices.ts`.
|
||||
- Optional `presentNames` from the scanner presence index.
|
||||
- Optional URI fetching for `fullLicenseTextUri` entries.
|
||||
|
||||
Supported override shapes:
|
||||
|
||||
- `{ name, prependLicenseText }`
|
||||
- `{ name, fullLicenseText }`
|
||||
- `{ name, fullLicenseTextUri, prependLicenseText? }`
|
||||
- Optional `version` narrows matching to one package version.
|
||||
- Legacy `licenseDetail` is treated as an alias for `fullLicenseText`.
|
||||
|
||||
Key behavior:
|
||||
|
||||
- Matching is case-insensitive on `name` and exact on `version`.
|
||||
- Matching overrides edit existing entries.
|
||||
- Overrides with usable text and a present package inject a new `cglicenses-override` entry.
|
||||
- Overrides for names absent from the presence index are stale: warn and skip.
|
||||
- Overrides with no target and no usable text are unmatched; `--strict` in `merge-notices.ts` fails on unmatched entries or URI errors, but stale entries are warn-only.
|
||||
|
||||
### `parse-notices.ts`
|
||||
|
||||
Purpose: parses NOTICE text files for diagnostics and for scanner logic.
|
||||
|
||||
CLI modes:
|
||||
|
||||
```sh
|
||||
node parse-notices.js --file <path>
|
||||
node parse-notices.js --diff <pathA> <pathB>
|
||||
```
|
||||
|
||||
Key behavior:
|
||||
|
||||
- `parseNoticeFile()` returns package name, version, license, line number, license text length, and optional `licenseText`.
|
||||
- The parser recognizes separator-delimited NOTICE entries and uses `isPackageHeader()` heuristics to avoid treating license prose as package headers.
|
||||
- Section 4 in `scan-licenses.ts` imports this parser to detect CG entries whose body is just an SPDX expression.
|
||||
|
||||
### `check-pr-dependencies.ts`
|
||||
|
||||
Purpose: PR-time license coverage gate for dependency changes.
|
||||
|
||||
Invocation in `pr-oss-check.yml`:
|
||||
|
||||
```sh
|
||||
node .oss-build-out/check-pr-dependencies.js \
|
||||
--repo "$(Build.SourcesDirectory)" \
|
||||
--base "origin/main" \
|
||||
--head "HEAD" \
|
||||
--cglicenses "$(Build.SourcesDirectory)/cglicenses.json"
|
||||
```
|
||||
|
||||
Key behavior:
|
||||
|
||||
- Diffs changed `package.json` files between `--base` and `--head`.
|
||||
- For added dependencies, passes only if one of these is true:
|
||||
- ClearlyDefined has usable license metadata and a non-zero score.
|
||||
- A LICENSE file exists in local or hoisted `node_modules`.
|
||||
- `cglicenses.json` has an override for the package.
|
||||
- For removed dependencies, fails if `cglicenses.json` still has an override for that package.
|
||||
- `--no-clearlydefined` disables the HTTP lookup for offline runs.
|
||||
|
||||
### `cglicenses.json`
|
||||
|
||||
Purpose: last-resort manual override file at repo root.
|
||||
|
||||
Use this only when CG and direct source scanning genuinely cannot provide usable text. Entries are read by `readCglicenses()` and applied by `applyOverrides()` during merge.
|
||||
|
||||
Important: the contents must be human-authored. Do not machine-manufacture license or copyright text. If CG's raw output has wrong text, file an upstream bug and ship CG's output as-is rather than hand-authoring a replacement. Add an override only when CG has no usable text or when a package lacks text in a way CELA has approved for manual override.
|
||||
|
||||
## Notable scanner details
|
||||
|
||||
### Cargo Section 4
|
||||
|
||||
Section 4 closes two Rust gaps:
|
||||
|
||||
1. Crates in `Cargo.lock` that are absent from CG.
|
||||
2. Crates where CG emitted the SPDX expression as the NOTICE body, such as `Zlib OR Apache-2.0 OR MIT`, instead of real license text.
|
||||
|
||||
Important functions and constants:
|
||||
|
||||
- `parseCargoLock()` extracts `name`, `version`, and `source` from `[[package]]` blocks.
|
||||
- `fetchCratesIoJson()` calls `https://crates.io/api/v1/crates/<name>` with `CRATES_IO_USER_AGENT`.
|
||||
- `getCrateRepository()` applies legacy repository URL overrides for crates such as `isatty`, `redox_syscall`, `redox_termios`, and `termion`.
|
||||
- `crateLicenseRefs()` tries `v<version>`, `<version>`, `<name>-v<version>`, `<name>-<version>`, then `main` and `master`.
|
||||
- `isSpdxStub()` detects CG bodies that are only SPDX expressions.
|
||||
- `fetchCargoLicense()` uses SPDX IDs to fetch real per-license files such as `LICENSE-MIT` or `LICENSES/Apache-2.0.txt`, with `fetchLicenseFromGitRepo()` as a generic fallback.
|
||||
- `hasSpdxAnd()` warns when the expression contains `AND`.
|
||||
|
||||
Maintenance rules:
|
||||
|
||||
- OR-license selection is first-in-expression wins: `spdxLicenseIds()` preserves order and `fetchCargoLicense()` returns the first SPDX ID that yields a real body. For `OR`, that is the licensee choice.
|
||||
- Conjunctive `AND` expressions are not reduced to one safe body. The scanner warns with `AND-LICENSE INCOMPLETE`; add a human-authored `cglicenses.json` override with the combined required text if this appears.
|
||||
- Cargo license fetch failures should warn and continue, not crash the build.
|
||||
|
||||
Tests:
|
||||
|
||||
- `cargo-section4.test.ts` covers `isSpdxStub()`, `parseCargoLock()`, `getCrateRepository()`, and `crateLicenseRefs()` without network.
|
||||
- `cargo-section4.live.ts` is a bounded live check that hits crates.io and GitHub for representative crates.
|
||||
|
||||
### Platform-binary Section 5
|
||||
|
||||
Section 5 closes a parity gap with the legacy OSS tool for arch-specific npm packages. Packages such as `@img/sharp-win32-x64` or `@esbuild/linux-x64` are optional dependencies of an arch-independent parent. On a single-platform build agent, only the host arch may be installed, but VS Code ships multiple platform and arch combinations.
|
||||
|
||||
Important functions and constants:
|
||||
|
||||
- `ARCH_SUFFIX_RE` and `isArchPackageName()` identify arch-bearing npm package names.
|
||||
- `VSCODE_SHIPPED_PLATFORMS` is currently `darwin`, `linux`, `linuxmusl`, and `win32`.
|
||||
- `VSCODE_SHIPPED_ARCHS` is currently `x64`, `arm64`, and `arm`.
|
||||
- `isShippedArch()` strips ABI suffixes like `-gnu`, `-musl`, `-msvc`, `-glibc`, `-gnueabihf`, `-eabihf`, and `-androideabi`, then checks the static shipped sets.
|
||||
- `npmLicenseId()` reads npm license shapes: string, `{ type }`, `licenses: [{ type }]`, and `licenses: [string]`.
|
||||
- `fetchNpmRegistryJson()` fetches a package packument when an arch child is not installed on disk.
|
||||
- `familyText()` caches license text by repository URL and reuses it across arch siblings.
|
||||
- `parentTextIfCompatible()` allows parent text fallback only when parent and child license IDs match exactly, case-insensitively.
|
||||
|
||||
Maintenance rules:
|
||||
|
||||
- The shipped-arch filter intentionally uses the static constants `VSCODE_SHIPPED_PLATFORMS` and `VSCODE_SHIPPED_ARCHS` in `scan-licenses.ts`.
|
||||
- Do not import these constants from `build/agent-sdk/common.ts`. Agent SDK excludes `armhf` and uses `alpine` where npm package names use `linuxmusl`, so it is not a direct fit for this scanner.
|
||||
- If VS Code adds or removes a shipped platform or arch, update those constants. The source of truth is noted in the code comment: `build/azure-pipelines/product-build.yml` plus `build/agent-sdk/common.ts`. Expect this to change about once a year.
|
||||
- Each arch child's own license ID is authoritative. Do not blindly reuse parent license text across a different child license ID; that would repeat the legacy sharp/libvips defect.
|
||||
|
||||
Tests:
|
||||
|
||||
- `platform-binary.test.ts` covers arch suffix recognition, shipped-arch filtering, npm license shape parsing, family text reuse, and parent fallback compatibility.
|
||||
|
||||
## Running locally
|
||||
|
||||
A helper exists at `build/azure-pipelines/oss/run-local.ps1`.
|
||||
|
||||
From the oss directory:
|
||||
|
||||
```powershell
|
||||
.\run-local.ps1
|
||||
```
|
||||
|
||||
Useful options:
|
||||
|
||||
```powershell
|
||||
.\run-local.ps1 -CgNotice C:\path\to\ThirdPartyNotices.generated.txt
|
||||
.\run-local.ps1 -SkipScan
|
||||
.\run-local.ps1 -RepoRoot C:\src\vscode.worktrees\oss-cg-validation -OutDir C:\path\to\oss-local
|
||||
```
|
||||
|
||||
What it does:
|
||||
|
||||
1. Runs `scan-licenses.ts` through `npx --yes tsx`.
|
||||
2. Writes `ext-notices.txt` and `ext-notices.txt.presence.json` under `-OutDir`.
|
||||
3. Runs `merge-notices.ts` with `cglicenses.json`.
|
||||
4. Writes `ThirdPartyNotices.new.txt` and a focused summary.
|
||||
|
||||
Notes:
|
||||
|
||||
- Pass `-CgNotice` with a cached CG NOTICE for realistic local validation. Without it, the merge still exercises override injection and stale detection, but it does not represent the full CG base.
|
||||
- If root `node_modules` is missing, run `npm i` first or the presence index will be incomplete.
|
||||
- Full validation is parity against the legacy OSS tool's `ThirdPartyNotices.txt`: the new output should be at least as good as legacy. Use `parse-notices.ts --diff` to compare NOTICE files and inspect package/version differences.
|
||||
|
||||
## Unit checks
|
||||
|
||||
Run from `build/azure-pipelines/oss`:
|
||||
|
||||
```powershell
|
||||
npx tsx cargo-section4.test.ts
|
||||
npx tsx platform-binary.test.ts
|
||||
```
|
||||
|
||||
Optional live Cargo check:
|
||||
|
||||
```powershell
|
||||
npx tsx cargo-section4.live.ts
|
||||
```
|
||||
|
||||
The live check performs bounded network calls to crates.io and GitHub. It is useful for validating the real Section 4 fetch path without running a full scanner pass over every crate.
|
||||
|
||||
## Maintenance rules
|
||||
|
||||
- Keep `cglicenses.json` human-authored. Never synthesize license or copyright text by script, model, or other automation.
|
||||
- Add `cglicenses.json` entries only for genuine no-usable-text gaps. If CG returns wrong raw text, file an upstream bug and ship CG's result as-is rather than replacing it with a hand-authored body.
|
||||
- Preserve the `name@version` merge key behavior. Multiple shipped versions of a package must remain separate NOTICE entries.
|
||||
- Treat stale overrides as warn-only in the release pipeline. The PR-time gate is the enforcement point for dependency additions/removals.
|
||||
- When updating shipped platform or arch support, update `VSCODE_SHIPPED_PLATFORMS` and `VSCODE_SHIPPED_ARCHS` in `scan-licenses.ts`, then update `platform-binary.test.ts` expectations.
|
||||
- When changing Cargo license logic, update `cargo-section4.test.ts`, and run `cargo-section4.live.ts` if the network fetch path changed.
|
||||
- Scanner network failures should log and continue. A service outage should not crash the build unless a later explicit validation step chooses to fail.
|
||||
|
||||
## Pending
|
||||
|
||||
The scanner does not yet consume the bundled `ThirdPartyNotices` files from externally-built, pre-built built-in extensions downloaded into the product. The planned addition under investigation is to read those notices for `js-debug`, `js-debug-companion`, and `js-profile-table`, so their own bundled dependencies are covered even though those extensions are built outside this repo.
|
||||
|
||||
### `audit-notices.ts`
|
||||
|
||||
Purpose: pre-ship sanity check that validates a generated NOTICE file against the repo's declared dependencies.
|
||||
|
||||
Invocation:
|
||||
|
||||
```sh
|
||||
npx tsx build/azure-pipelines/oss/audit-notices.ts \
|
||||
--notice <path-to-ThirdPartyNotices.new.txt> \
|
||||
--repo .
|
||||
```
|
||||
|
||||
What it checks:
|
||||
|
||||
1. **NOTICE stats** — total entries, duplicate detection (same name@version = bug), multi-version packages (expected), license type breakdown.
|
||||
2. **Repo manifest cross-reference** — walks all `package.json` (direct deps), `package-lock.json` (full transitive tree, dev filtered out), `Cargo.lock`, and `cgmanifest.json`. Reports overlap, NOTICE-only entries (CG transitive deps), and manifest-only gaps (packages missing from the NOTICE).
|
||||
3. **Summary** — coverage percentage and actionable gap list.
|
||||
4. **Package source breakdown** — per-lockfile package counts sorted by size. Shows which extensions/directories contribute the most dependencies (e.g., `extensions/copilot: 329 packages`).
|
||||
|
||||
Key behavior:
|
||||
|
||||
- Lockfile parsing filters out `dev: true` and `devOptional: true` entries — only non-dev (shipping) packages are counted.
|
||||
- Cross-reference uses lowercase package names for matching.
|
||||
- The "manifest-only" gap list is the primary ship-readiness signal: each gap should be explainable (binary component, known TODO, platform-specific absence).
|
||||
|
||||
When to use:
|
||||
|
||||
- Before opening a PR: download `notice_output` from the latest CI build and run locally.
|
||||
- After a build: verify the .new.txt artifact covers all declared dependencies.
|
||||
- The script does NOT require `node_modules` on disk — it reads lockfiles as the ground truth for the full transitive dependency tree.
|
||||
@@ -5,6 +5,7 @@ Thumbs.db
|
||||
node_modules/
|
||||
.tmp/
|
||||
.build/
|
||||
/extensionsCG/
|
||||
.vscode/extensions/**/out/
|
||||
extensions/**/dist/
|
||||
src/vs/base/browser/ui/codicons/codicon/codicon.ttf
|
||||
|
||||
Vendored
+12
@@ -13,6 +13,10 @@
|
||||
"type": "string",
|
||||
"description": "The name of the dependency"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Optional exact version. When present, the override applies only to this version. When absent, it applies to every version of the named dependency."
|
||||
},
|
||||
"fullLicenseText": {
|
||||
"type": "array",
|
||||
"description": "The complete license text of the dependency",
|
||||
@@ -40,6 +44,10 @@
|
||||
"type": "string",
|
||||
"description": "The name of the dependency"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Optional exact version. When present, the override applies only to this version. When absent, it applies to every version of the named dependency."
|
||||
},
|
||||
"fullLicenseText": {
|
||||
"type": "array",
|
||||
"description": "The complete license text of the dependency",
|
||||
@@ -67,6 +75,10 @@
|
||||
"type": "string",
|
||||
"description": "The name of the dependency"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Optional exact version. When present, the override applies only to this version. When absent, it applies to every version of the named dependency."
|
||||
},
|
||||
"fullLicenseTextUri": {
|
||||
"type": "string",
|
||||
"description": "The URI to the license text of this repository",
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Apply cglicenses.json overrides on top of a merged NOTICE entry set.
|
||||
*
|
||||
* The override file supports three shapes (see .vscode/cglicenses.schema.json):
|
||||
* - { name, prependLicenseText }
|
||||
* - { name, fullLicenseText }
|
||||
* - { name, fullLicenseTextUri, prependLicenseText? }
|
||||
*
|
||||
* Any shape may include an optional `version` field. When present, the override
|
||||
* applies only to the matching name+version. When absent, the override applies
|
||||
* to every version of the named package.
|
||||
*
|
||||
* CELA guidance (vscode-engineering#2142 comment 4624222337):
|
||||
* Do NOT manufacture copyright statements automatically. Any prepend/full text
|
||||
* must come from a human-authored override entry, not from the tool.
|
||||
*
|
||||
* Override matching is case-insensitive on `name` and exact on `version`.
|
||||
*
|
||||
* When an override matches an existing merged entry it EDITS it. When no entry
|
||||
* exists, the override is the only source of truth for that package, so:
|
||||
* - if the package is present in the shipped-components presence index, the
|
||||
* override is INJECTED as a brand-new entry (tagged `cglicenses-override`);
|
||||
* - if the package is NOT present, the override is almost certainly STALE —
|
||||
* it is skipped (never injected) and reported in `staleNames` so the caller
|
||||
* can warn. A stale override never fails the build; the PR-time check is
|
||||
* the real gate. When no presence index is supplied, staleness cannot be
|
||||
* proven, so the override is injected (conservative, warn-only design).
|
||||
* - if the override carries no usable text at all, it is reported in
|
||||
* `unmatchedNames` so the caller can warn or hard-fail.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import fs from 'fs';
|
||||
import https from 'https';
|
||||
import http from 'http';
|
||||
import { URL } from 'url';
|
||||
|
||||
export interface CglicenseEntry {
|
||||
name: string;
|
||||
/**
|
||||
* Optional version filter. When present, the override applies only to merged
|
||||
* entries whose version matches exactly. When absent, the override applies to
|
||||
* every version of the named package.
|
||||
*/
|
||||
version?: string;
|
||||
fullLicenseText?: string[];
|
||||
/**
|
||||
* Some legacy cglicenses.json entries supply the full license text under
|
||||
* `licenseDetail` (an array of lines) instead of `fullLicenseText`. Treated
|
||||
* as an alias for `fullLicenseText`.
|
||||
*/
|
||||
licenseDetail?: string[];
|
||||
fullLicenseTextUri?: string;
|
||||
prependLicenseText?: string[];
|
||||
}
|
||||
|
||||
export interface MergedEntry {
|
||||
name: string;
|
||||
version: string;
|
||||
license: string;
|
||||
url: string;
|
||||
licenseText: string;
|
||||
/** Provenance marker. Entries injected from an override carry 'cglicenses-override'. */
|
||||
source?: string;
|
||||
}
|
||||
|
||||
export interface ApplyResult {
|
||||
/** Overrides that matched and edited at least one existing merged entry. */
|
||||
appliedNames: string[];
|
||||
/** Overrides that injected a brand-new merged entry (no prior entry existed). */
|
||||
injectedNames: string[];
|
||||
/** Count of (override × matching merged entry) edits (does not include injects). */
|
||||
appliedEntryCount: number;
|
||||
/** Overrides with no matching entry and no usable license text — nothing to contribute. */
|
||||
unmatchedNames: string[];
|
||||
/**
|
||||
* Overrides with usable text whose package is not in the presence index —
|
||||
* almost certainly stale. Skipped (never injected), warn-only, never fails the build.
|
||||
*/
|
||||
staleNames: string[];
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip // line comments from a JSONC file. The existing cglicenses.json uses
|
||||
* only `//` line comments (no block comments) and never inside string literals
|
||||
* in the top-level structure. Conservative implementation: remove `//` to end
|
||||
* of line only when the `//` is not inside a double-quoted string.
|
||||
*/
|
||||
export function stripJsonComments(text: string): string {
|
||||
let out = '';
|
||||
let inString = false;
|
||||
let escape = false;
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
const ch = text[i];
|
||||
if (inString) {
|
||||
out += ch;
|
||||
if (escape) {
|
||||
escape = false;
|
||||
} else if (ch === '\\') {
|
||||
escape = true;
|
||||
} else if (ch === '"') {
|
||||
inString = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (ch === '"') {
|
||||
inString = true;
|
||||
out += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === '/' && text[i + 1] === '/') {
|
||||
// Skip until newline
|
||||
while (i < text.length && text[i] !== '\n') {
|
||||
i++;
|
||||
}
|
||||
if (i < text.length) {
|
||||
out += '\n';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
out += ch;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function readCglicenses(filePath: string): CglicenseEntry[] {
|
||||
const raw = fs.readFileSync(filePath, 'utf8');
|
||||
const stripped = stripJsonComments(raw);
|
||||
const parsed: unknown = JSON.parse(stripped);
|
||||
if (!Array.isArray(parsed)) {
|
||||
throw new Error(`cglicenses.json must be an array, got ${typeof parsed}`);
|
||||
}
|
||||
for (const entry of parsed as Record<string, unknown>[]) {
|
||||
if (typeof entry.name !== 'string' || !entry.name) {
|
||||
throw new Error(`cglicenses.json contains an entry without a string "name"`);
|
||||
}
|
||||
}
|
||||
return parsed as CglicenseEntry[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch text content from an http/https URL. Returns undefined on any failure
|
||||
* so the caller can decide whether to warn or hard-fail.
|
||||
*/
|
||||
export function fetchUriText(uri: string, timeoutMs = 10_000, maxRedirects = 5): Promise<string | undefined> {
|
||||
return new Promise(resolve => {
|
||||
if (maxRedirects <= 0) {
|
||||
resolve(undefined);
|
||||
return;
|
||||
}
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(uri);
|
||||
} catch {
|
||||
resolve(undefined);
|
||||
return;
|
||||
}
|
||||
const lib = url.protocol === 'https:' ? https : (url.protocol === 'http:' ? http : undefined);
|
||||
if (!lib) {
|
||||
resolve(undefined);
|
||||
return;
|
||||
}
|
||||
const req = lib.get(uri, { timeout: timeoutMs }, res => {
|
||||
if (res.statusCode === 301 || res.statusCode === 302 || res.statusCode === 307 || res.statusCode === 308) {
|
||||
const loc = res.headers.location;
|
||||
res.resume();
|
||||
if (loc) {
|
||||
const resolved = new URL(loc, uri).href;
|
||||
resolve(fetchUriText(resolved, timeoutMs, maxRedirects - 1));
|
||||
} else {
|
||||
resolve(undefined);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (res.statusCode !== 200) {
|
||||
res.resume();
|
||||
resolve(undefined);
|
||||
return;
|
||||
}
|
||||
const MAX_BODY_SIZE = 10 * 1024 * 1024; // 10MB
|
||||
let data = '';
|
||||
res.setEncoding('utf8');
|
||||
res.on('data', chunk => {
|
||||
data += chunk;
|
||||
if (data.length > MAX_BODY_SIZE) {
|
||||
req.destroy();
|
||||
resolve(undefined);
|
||||
return;
|
||||
}
|
||||
});
|
||||
res.on('end', () => resolve(data));
|
||||
});
|
||||
req.on('error', () => resolve(undefined));
|
||||
req.on('timeout', () => {
|
||||
req.destroy();
|
||||
resolve(undefined);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
export async function applyOverrides(
|
||||
merged: Map<string, MergedEntry>,
|
||||
overrides: CglicenseEntry[],
|
||||
options: { fetchUris?: boolean; presentNames?: Set<string> } = {}
|
||||
): Promise<ApplyResult> {
|
||||
const appliedNames: string[] = [];
|
||||
const injectedNames: string[] = [];
|
||||
const unmatchedNames: string[] = [];
|
||||
const staleNames: string[] = [];
|
||||
const errors: string[] = [];
|
||||
let appliedEntryCount = 0;
|
||||
|
||||
// Merged-map key scheme — must match merge-notices.ts so injected entries
|
||||
// slot in alongside CG/scanner entries instead of colliding.
|
||||
const mergeKey = (name: string, version: string | undefined) =>
|
||||
`${name.toLowerCase()}@${version || ''}`;
|
||||
|
||||
// Build a name -> entries[] index. The merged map is keyed by `name@version`
|
||||
// (per CELA guidance to preserve every shipped version), but overrides in
|
||||
// cglicenses.json are keyed by name only — a manual override applies to all
|
||||
// versions of the named package.
|
||||
const byName = new Map<string, MergedEntry[]>();
|
||||
for (const entry of merged.values()) {
|
||||
const k = entry.name.toLowerCase();
|
||||
const list = byName.get(k);
|
||||
if (list) {
|
||||
list.push(entry);
|
||||
} else {
|
||||
byName.set(k, [entry]);
|
||||
}
|
||||
}
|
||||
|
||||
for (const override of overrides) {
|
||||
const key = override.name.toLowerCase();
|
||||
const label = override.version ? `${override.name}@${override.version}` : override.name;
|
||||
|
||||
// Resolve the override body FIRST, before looking for a target. This is
|
||||
// what lets us inject a brand-new entry when no target exists: the
|
||||
// human-authored text is the only source of truth for these packages.
|
||||
let bodyOverride: string | undefined;
|
||||
if (override.fullLicenseText && override.fullLicenseText.length > 0) {
|
||||
bodyOverride = override.fullLicenseText.join('\n');
|
||||
} else if (override.licenseDetail && override.licenseDetail.length > 0) {
|
||||
// `licenseDetail` is an older alias for the full license text.
|
||||
bodyOverride = override.licenseDetail.join('\n');
|
||||
} else if (override.fullLicenseTextUri) {
|
||||
if (options.fetchUris) {
|
||||
const fetched = await fetchUriText(override.fullLicenseTextUri);
|
||||
if (fetched === undefined) {
|
||||
errors.push(`Failed to fetch fullLicenseTextUri for "${override.name}" (${override.fullLicenseTextUri})`);
|
||||
continue;
|
||||
}
|
||||
bodyOverride = fetched.trim();
|
||||
} else {
|
||||
errors.push(`Override for "${override.name}" uses fullLicenseTextUri but fetching is disabled`);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
const prefix = override.prependLicenseText && override.prependLicenseText.length > 0
|
||||
? override.prependLicenseText.join('\n')
|
||||
: undefined;
|
||||
|
||||
// Find existing targets (case-insensitive name, exact version when given).
|
||||
const allTargets = byName.get(key);
|
||||
const targets = allTargets
|
||||
? (override.version
|
||||
? allTargets.filter(t => t.version === override.version)
|
||||
: allTargets)
|
||||
: undefined;
|
||||
|
||||
if (targets && targets.length > 0) {
|
||||
// EDIT path — at least one merged entry already exists.
|
||||
for (const target of targets) {
|
||||
let body = bodyOverride ?? target.licenseText;
|
||||
if (prefix) {
|
||||
body = prefix + '\n\n' + body;
|
||||
}
|
||||
target.licenseText = body;
|
||||
appliedEntryCount++;
|
||||
}
|
||||
appliedNames.push(label);
|
||||
continue;
|
||||
}
|
||||
|
||||
// No existing entry. Decide between INJECT, STALE, or UNMATCHED.
|
||||
const hasUsableText = bodyOverride !== undefined || prefix !== undefined;
|
||||
if (!hasUsableText) {
|
||||
// Override carries no text we could inject — nothing to do.
|
||||
unmatchedNames.push(label);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Presence gate: only inject if the package is actually shipped. When no
|
||||
// presence index is supplied we cannot prove staleness, so we inject
|
||||
// (the conservative choice for a warn-only, never-fail design).
|
||||
const isPresent = options.presentNames ? options.presentNames.has(key) : true;
|
||||
if (!isPresent) {
|
||||
// Stale override — warn + skip. Never injected, never fails the build.
|
||||
staleNames.push(label);
|
||||
continue;
|
||||
}
|
||||
|
||||
// INJECT path — build a new entry from the human-authored text.
|
||||
let body: string;
|
||||
if (bodyOverride !== undefined && prefix !== undefined) {
|
||||
body = prefix + '\n\n' + bodyOverride;
|
||||
} else if (bodyOverride !== undefined) {
|
||||
body = bodyOverride;
|
||||
} else {
|
||||
// prepend-only injection (may be copyright-only text)
|
||||
body = prefix as string;
|
||||
}
|
||||
|
||||
const injected: MergedEntry = {
|
||||
name: override.name,
|
||||
version: override.version || '',
|
||||
license: '',
|
||||
url: override.fullLicenseTextUri || '',
|
||||
licenseText: body,
|
||||
source: 'cglicenses-override',
|
||||
};
|
||||
merged.set(mergeKey(override.name, override.version), injected);
|
||||
// Keep byName in sync in case a later override targets the same name.
|
||||
const list = byName.get(key);
|
||||
if (list) {
|
||||
list.push(injected);
|
||||
} else {
|
||||
byName.set(key, [injected]);
|
||||
}
|
||||
injectedNames.push(label);
|
||||
}
|
||||
|
||||
return { appliedNames, injectedNames, appliedEntryCount, unmatchedNames, staleNames, errors };
|
||||
}
|
||||
@@ -0,0 +1,533 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Audit a ThirdPartyNotices file against the actual repo contents.
|
||||
*
|
||||
* Parses the NOTICE file, then walks the repo to collect runtime dependencies
|
||||
* from package.json, Cargo.lock, and cgmanifest.json files. Produces a report
|
||||
* showing coverage gaps, extras, duplicates, and license breakdown.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx build/azure-pipelines/oss/audit-notices.ts --notice <path> --repo .
|
||||
*
|
||||
* --notice Path to ThirdPartyNotices.txt (or .new.txt)
|
||||
* --repo Path to the repo root (defaults to cwd)
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { parseNoticeFile, type NoticeEntry } from './parse-notices.js';
|
||||
import { parseArgs } from './utils.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Recursively walk a directory, yielding file paths that match a predicate. */
|
||||
function walkFiles(dir: string, match: (name: string) => boolean): string[] {
|
||||
const results: string[] = [];
|
||||
const skipDirs = new Set(['.git', '.build', 'out', 'out-build', 'out-editor', 'out-vscode']);
|
||||
|
||||
function walk(d: string): void {
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = fs.readdirSync(d, { withFileTypes: true });
|
||||
} catch {
|
||||
return; // permission errors, broken symlinks, etc.
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (entry.isDirectory()) {
|
||||
if (!skipDirs.has(entry.name)) {
|
||||
walk(path.join(d, entry.name));
|
||||
}
|
||||
} else if (entry.isFile() && match(entry.name)) {
|
||||
results.push(path.join(d, entry.name));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
walk(dir);
|
||||
return results;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// NOTICE analysis
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface NoticeStats {
|
||||
entries: NoticeEntry[];
|
||||
uniqueKey: Set<string>;
|
||||
duplicateEntries: { key: string; count: number }[];
|
||||
duplicateNames: { name: string; versions: string[] }[];
|
||||
licenseBreakdown: { license: string; count: number }[];
|
||||
}
|
||||
|
||||
function analyzeNotice(entries: NoticeEntry[]): NoticeStats {
|
||||
// Unique by name@version
|
||||
const keyCounts = new Map<string, number>();
|
||||
for (const e of entries) {
|
||||
const key = `${e.name.toLowerCase()}@${e.version}`;
|
||||
keyCounts.set(key, (keyCounts.get(key) ?? 0) + 1);
|
||||
}
|
||||
const duplicateEntries: { key: string; count: number }[] = [];
|
||||
for (const [key, count] of keyCounts) {
|
||||
if (count > 1) {
|
||||
duplicateEntries.push({ key, count });
|
||||
}
|
||||
}
|
||||
|
||||
// Same name, different versions
|
||||
const nameVersions = new Map<string, Set<string>>();
|
||||
for (const e of entries) {
|
||||
const lower = e.name.toLowerCase();
|
||||
if (!nameVersions.has(lower)) {
|
||||
nameVersions.set(lower, new Set());
|
||||
}
|
||||
if (e.version) {
|
||||
nameVersions.get(lower)!.add(e.version);
|
||||
}
|
||||
}
|
||||
const duplicateNames: { name: string; versions: string[] }[] = [];
|
||||
for (const [name, versions] of nameVersions) {
|
||||
if (versions.size > 1) {
|
||||
duplicateNames.push({ name, versions: [...versions].sort() });
|
||||
}
|
||||
}
|
||||
|
||||
// License breakdown
|
||||
const licenseCounts = new Map<string, number>();
|
||||
for (const e of entries) {
|
||||
const lic = e.license || '(unknown)';
|
||||
licenseCounts.set(lic, (licenseCounts.get(lic) ?? 0) + 1);
|
||||
}
|
||||
const licenseBreakdown = [...licenseCounts.entries()]
|
||||
.map(([license, count]) => ({ license, count }))
|
||||
.sort((a, b) => b.count - a.count);
|
||||
|
||||
return {
|
||||
entries,
|
||||
uniqueKey: new Set(keyCounts.keys()),
|
||||
duplicateEntries,
|
||||
duplicateNames,
|
||||
licenseBreakdown,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Manifest discovery
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface ManifestPackage {
|
||||
name: string;
|
||||
version: string;
|
||||
source: string; // e.g. "package.json", "Cargo.lock", "cgmanifest.json"
|
||||
}
|
||||
|
||||
/** Read runtime deps (dependencies + optionalDependencies) from a package.json. */
|
||||
function collectPackageJsonDeps(filePath: string): ManifestPackage[] {
|
||||
const results: ManifestPackage[] = [];
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(filePath, 'utf8'));
|
||||
for (const section of ['dependencies', 'optionalDependencies'] as const) {
|
||||
const deps = pkg[section];
|
||||
if (deps && typeof deps === 'object') {
|
||||
for (const [name, ver] of Object.entries(deps)) {
|
||||
results.push({
|
||||
name,
|
||||
version: String(ver).replace(/^[\^~>=<]+/, ''),
|
||||
source: 'package.json',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// malformed JSON -- skip
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/** Parse [[package]] blocks from a Cargo.lock file. */
|
||||
function collectCargoLockDeps(filePath: string): ManifestPackage[] {
|
||||
const results: ManifestPackage[] = [];
|
||||
try {
|
||||
const content = fs.readFileSync(filePath, 'utf8');
|
||||
const blockRe = /\[\[package\]\]\s*\n((?:[^\[]*\n)*)/g;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = blockRe.exec(content)) !== null) {
|
||||
const block = m[1];
|
||||
const nameMatch = block.match(/^name\s*=\s*"(.+?)"/m);
|
||||
const verMatch = block.match(/^version\s*=\s*"(.+?)"/m);
|
||||
if (nameMatch) {
|
||||
results.push({
|
||||
name: nameMatch[1],
|
||||
version: verMatch ? verMatch[1] : '',
|
||||
source: 'Cargo.lock',
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// read error -- skip
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/** Read the full transitive dependency tree from a package-lock.json (v2/v3 format). */
|
||||
function collectPackageLockDeps(filePath: string): ManifestPackage[] {
|
||||
const results: ManifestPackage[] = [];
|
||||
try {
|
||||
const lockfile = JSON.parse(fs.readFileSync(filePath, 'utf8'));
|
||||
const packages: Record<string, any> = lockfile.packages;
|
||||
if (!packages || typeof packages !== 'object') {
|
||||
return results;
|
||||
}
|
||||
for (const [key, entry] of Object.entries(packages)) {
|
||||
// Skip root entry
|
||||
if (key === '') {
|
||||
continue;
|
||||
}
|
||||
// Skip dev-only and devOptional dependencies
|
||||
if (entry.dev === true || entry.devOptional === true) {
|
||||
continue;
|
||||
}
|
||||
// Extract package name from the key path
|
||||
// Keys look like "node_modules/@scope/pkg" or "node_modules/foo/node_modules/bar"
|
||||
const segments = key.split('node_modules/');
|
||||
const lastSegment = segments[segments.length - 1];
|
||||
// Remove trailing slash if present
|
||||
const name = lastSegment.replace(/\/$/, '');
|
||||
if (name && entry.version) {
|
||||
results.push({
|
||||
name,
|
||||
version: String(entry.version),
|
||||
source: 'package-lock.json',
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// malformed JSON or read error -- skip
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/** Parse component registrations from a cgmanifest.json file. */
|
||||
function collectCgManifestDeps(filePath: string): ManifestPackage[] {
|
||||
const results: ManifestPackage[] = [];
|
||||
try {
|
||||
const manifest = JSON.parse(fs.readFileSync(filePath, 'utf8'));
|
||||
const registrations: any[] = manifest.registrations ?? manifest.Registrations ?? [];
|
||||
for (const reg of registrations) {
|
||||
const comp = reg.component ?? reg.Component;
|
||||
if (!comp) {
|
||||
continue;
|
||||
}
|
||||
const compType: string = comp.type ?? comp.Type ?? '';
|
||||
let name = '';
|
||||
let version = reg.version ?? '';
|
||||
|
||||
if (compType === 'git') {
|
||||
const git = comp.git ?? comp.Git;
|
||||
name = git?.name ?? git?.Name ?? '';
|
||||
} else if (compType === 'npm') {
|
||||
const npm = comp.npm ?? comp.Npm;
|
||||
name = npm?.name ?? npm?.Name ?? '';
|
||||
version = version || (npm?.version ?? '');
|
||||
} else if (compType === 'nuget') {
|
||||
const nuget = comp.nuget ?? comp.Nuget;
|
||||
name = nuget?.name ?? nuget?.Name ?? '';
|
||||
version = version || (nuget?.version ?? '');
|
||||
} else if (compType === 'other') {
|
||||
const other = comp.other ?? comp.Other;
|
||||
name = other?.name ?? other?.Name ?? '';
|
||||
version = version || (other?.version ?? '');
|
||||
}
|
||||
|
||||
if (name) {
|
||||
results.push({ name, version: String(version), source: 'cgmanifest.json' });
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// malformed JSON -- skip
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cross-reference
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface CrossRefResult {
|
||||
manifestPackages: ManifestPackage[];
|
||||
manifestUniqueNames: Set<string>;
|
||||
manifestUniqueNameVersions: Set<string>;
|
||||
noticeOnlyNames: string[];
|
||||
manifestOnlyNames: string[];
|
||||
overlapCount: number;
|
||||
bySource: Map<string, number>;
|
||||
}
|
||||
|
||||
function crossReference(noticeNames: Set<string>, manifestPackages: ManifestPackage[]): CrossRefResult {
|
||||
const manifestUniqueNames = new Set<string>();
|
||||
const manifestUniqueNameVersions = new Set<string>();
|
||||
|
||||
for (const pkg of manifestPackages) {
|
||||
const lower = pkg.name.toLowerCase();
|
||||
manifestUniqueNames.add(lower);
|
||||
if (pkg.version) {
|
||||
manifestUniqueNameVersions.add(`${lower}@${pkg.version}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Unique package counts per source (de-duped by name)
|
||||
const uniqueBySource = new Map<string, Set<string>>();
|
||||
for (const pkg of manifestPackages) {
|
||||
if (!uniqueBySource.has(pkg.source)) {
|
||||
uniqueBySource.set(pkg.source, new Set());
|
||||
}
|
||||
uniqueBySource.get(pkg.source)!.add(pkg.name.toLowerCase());
|
||||
}
|
||||
const bySource = new Map<string, number>();
|
||||
for (const [src, names] of uniqueBySource) {
|
||||
bySource.set(src, names.size);
|
||||
}
|
||||
|
||||
const noticeOnlyNames = [...noticeNames].filter(n => !manifestUniqueNames.has(n)).sort();
|
||||
const manifestOnlyNames = [...manifestUniqueNames].filter(n => !noticeNames.has(n)).sort();
|
||||
const overlapCount = [...noticeNames].filter(n => manifestUniqueNames.has(n)).length;
|
||||
|
||||
return {
|
||||
manifestPackages,
|
||||
manifestUniqueNames,
|
||||
manifestUniqueNameVersions,
|
||||
noticeOnlyNames,
|
||||
manifestOnlyNames,
|
||||
overlapCount,
|
||||
bySource,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Report
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function printReport(noticePath: string, stats: NoticeStats, xref: CrossRefResult, lockfileBreakdown: Map<string, number>): void {
|
||||
const sizeMB = (fs.statSync(noticePath).size / 1024 / 1024).toFixed(2);
|
||||
|
||||
console.log('+==============================================================+');
|
||||
console.log('| NOTICE FILE AUDIT REPORT |');
|
||||
console.log('+==============================================================+');
|
||||
console.log('');
|
||||
|
||||
// -- Section 1: NOTICE stats --
|
||||
console.log('-- 1. NOTICE FILE STATS ---------------------------------------');
|
||||
console.log(` File: ${noticePath}`);
|
||||
console.log(` Size: ${sizeMB} MB`);
|
||||
console.log(` Total entries: ${stats.entries.length}`);
|
||||
console.log(` Unique (name@v): ${stats.uniqueKey.size}`);
|
||||
console.log('');
|
||||
|
||||
// Duplicates (same name@version -- bugs)
|
||||
if (stats.duplicateEntries.length > 0) {
|
||||
console.log(` WARNING: DUPLICATE ENTRIES (same name@version): ${stats.duplicateEntries.length}`);
|
||||
for (const d of stats.duplicateEntries) {
|
||||
console.log(` ${d.key} (x${d.count})`);
|
||||
}
|
||||
console.log('');
|
||||
} else {
|
||||
console.log(' OK: No duplicate name@version entries');
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// Same name, different versions (expected but noted)
|
||||
if (stats.duplicateNames.length > 0) {
|
||||
console.log(` INFO: Multi-version packages (same name, different versions): ${stats.duplicateNames.length}`);
|
||||
for (const d of stats.duplicateNames.slice(0, 20)) {
|
||||
console.log(` ${d.name}: ${d.versions.join(', ')}`);
|
||||
}
|
||||
if (stats.duplicateNames.length > 20) {
|
||||
console.log(` ... and ${stats.duplicateNames.length - 20} more`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// License breakdown (top 10)
|
||||
console.log(' License breakdown (top 10):');
|
||||
for (const l of stats.licenseBreakdown.slice(0, 10)) {
|
||||
const bar = '#'.repeat(Math.min(Math.round(l.count / stats.entries.length * 40), 40));
|
||||
const pct = ((l.count / stats.entries.length) * 100).toFixed(1);
|
||||
console.log(` ${l.license.padEnd(30)} ${String(l.count).padStart(5)} ${pct.padStart(5)}% ${bar}`);
|
||||
}
|
||||
if (stats.licenseBreakdown.length > 10) {
|
||||
const rest = stats.licenseBreakdown.slice(10).reduce((s, l) => s + l.count, 0);
|
||||
console.log(` ${'(other)'.padEnd(30)} ${String(rest).padStart(5)}`);
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// -- Section 2: Manifest cross-reference --
|
||||
console.log('-- 2. REPO MANIFEST CROSS-REFERENCE --------------------------');
|
||||
console.log('');
|
||||
console.log(' Manifest sources found:');
|
||||
for (const [src, count] of xref.bySource) {
|
||||
console.log(` ${src.padEnd(20)} ${String(count).padStart(5)} unique packages`);
|
||||
}
|
||||
console.log(` ${'TOTAL'.padEnd(20)} ${String(xref.manifestUniqueNames.size).padStart(5)} unique packages`);
|
||||
console.log(` ${''.padEnd(20)} ${String(xref.manifestUniqueNameVersions.size).padStart(5)} unique name@version (comparable to CG/NOTICE counts)`);
|
||||
console.log('');
|
||||
|
||||
console.log(` Overlap (in both NOTICE and manifests): ${xref.overlapCount}`);
|
||||
console.log(` NOTICE-only (CG + scanner coverage): ${xref.noticeOnlyNames.length}`);
|
||||
console.log(` Manifest-only (potential gaps): ${xref.manifestOnlyNames.length}`);
|
||||
console.log('');
|
||||
|
||||
if (xref.manifestOnlyNames.length > 0) {
|
||||
console.log(' WARNING: Packages in manifests but NOT in NOTICE (investigate):');
|
||||
for (const name of xref.manifestOnlyNames.slice(0, 50)) {
|
||||
console.log(` - ${name}`);
|
||||
}
|
||||
if (xref.manifestOnlyNames.length > 50) {
|
||||
console.log(` ... and ${xref.manifestOnlyNames.length - 50} more`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
if (xref.noticeOnlyNames.length > 0) {
|
||||
console.log(` INFO: Packages in NOTICE but not in manifests: ${xref.noticeOnlyNames.length}`);
|
||||
console.log(' (Expected -- CG auto-detects transitive deps not listed in manifests)');
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// -- Section 3: Summary --
|
||||
console.log('-- 3. SUMMARY ------------------------------------------------');
|
||||
console.log('');
|
||||
const noticeUniqueNames = new Set([...stats.uniqueKey].map(k => k.replace(/@[^@]*$/, '')));
|
||||
console.log(` NOTICE has ${stats.entries.length} entries (${noticeUniqueNames.size} unique names).`);
|
||||
console.log(` Repo manifests declare ${xref.manifestUniqueNames.size} unique runtime packages.`);
|
||||
console.log(` Overlap: ${xref.overlapCount}. NOTICE-only: ${xref.noticeOnlyNames.length} (CG + scanner coverage). Manifest-only: ${xref.manifestOnlyNames.length} (potential gaps).`);
|
||||
console.log('');
|
||||
|
||||
if (xref.manifestOnlyNames.length > 0) {
|
||||
console.log(' FAIL: ACTION REQUIRED: Manifest-only packages need investigation.');
|
||||
console.log(' These may be missing from the NOTICE file.');
|
||||
} else {
|
||||
console.log(' PASS: All manifest packages are covered in the NOTICE file.');
|
||||
}
|
||||
console.log('');
|
||||
|
||||
// -- Section 4: Package source breakdown --
|
||||
if (lockfileBreakdown.size > 0) {
|
||||
console.log('-- 4. PACKAGE SOURCE BREAKDOWN -------------------------------');
|
||||
console.log('');
|
||||
|
||||
// Sort by count descending
|
||||
const sorted = [...lockfileBreakdown.entries()].sort((a, b) => b[1] - a[1]);
|
||||
const total = sorted.reduce((sum, [, count]) => sum + count, 0);
|
||||
|
||||
// Find the widest path for alignment
|
||||
const maxPathLen = Math.max(...sorted.map(([p]) => p.length), 'TOTAL across all lockfiles:'.length);
|
||||
|
||||
console.log(' Packages per lockfile (non-dev, sorted by count):');
|
||||
for (const [relPath, count] of sorted) {
|
||||
console.log(` ${relPath.padEnd(maxPathLen)} ${String(count).padStart(5)}`);
|
||||
}
|
||||
console.log(` ${'TOTAL across all lockfiles:'.padEnd(maxPathLen)} ${String(total).padStart(5)}`);
|
||||
console.log('');
|
||||
|
||||
// Top extensions by package count
|
||||
const extensionEntries = sorted.filter(([p]) => /^extensions\/[^/]+\/package-lock\.json$/.test(p));
|
||||
if (extensionEntries.length > 0) {
|
||||
const top10 = extensionEntries.slice(0, 10);
|
||||
const maxExtLen = Math.max(...top10.map(([p]) => p.replace(/\/package-lock\.json$/, '').length));
|
||||
console.log(' Top 10 extensions by package count:');
|
||||
for (const [relPath, count] of top10) {
|
||||
const extName = relPath.replace(/\/package-lock\.json$/, '');
|
||||
console.log(` ${extName.padEnd(maxExtLen)} ${String(count).padStart(5)}`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function main(): void {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
|
||||
if (!args['notice']) {
|
||||
console.error('Usage: npx tsx build/azure-pipelines/oss/audit-notices.ts --notice <path> [--repo <path>]');
|
||||
console.error('');
|
||||
console.error(' --notice Path to ThirdPartyNotices.txt');
|
||||
console.error(' --repo Path to the repo root (defaults to cwd)');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const noticePath = path.resolve(args['notice']);
|
||||
const repoRoot = path.resolve(args['repo'] ?? process.cwd());
|
||||
|
||||
if (!fs.existsSync(noticePath)) {
|
||||
console.error(`Error: NOTICE file not found: ${noticePath}`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!fs.existsSync(repoRoot)) {
|
||||
console.error(`Error: Repo root not found: ${repoRoot}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`Parsing NOTICE file: ${noticePath}`);
|
||||
const entries = parseNoticeFile(noticePath);
|
||||
const stats = analyzeNotice(entries);
|
||||
|
||||
console.log(`Walking repo for manifests: ${repoRoot}`);
|
||||
|
||||
// Collect all manifest packages
|
||||
const allManifestPkgs: ManifestPackage[] = [];
|
||||
|
||||
// package.json files
|
||||
const packageJsonFiles = walkFiles(repoRoot, n => n === 'package.json');
|
||||
console.log(` Found ${packageJsonFiles.length} package.json files`);
|
||||
for (const f of packageJsonFiles) {
|
||||
allManifestPkgs.push(...collectPackageJsonDeps(f));
|
||||
}
|
||||
|
||||
// package-lock.json files (full transitive dependency tree)
|
||||
const packageLockFiles = walkFiles(repoRoot, n => n === 'package-lock.json');
|
||||
console.log(` Found ${packageLockFiles.length} package-lock.json files`);
|
||||
const lockfileBreakdown = new Map<string, number>();
|
||||
for (const f of packageLockFiles) {
|
||||
const deps = collectPackageLockDeps(f);
|
||||
const relPath = path.relative(repoRoot, f).replace(/\\/g, '/');
|
||||
lockfileBreakdown.set(relPath, deps.length);
|
||||
allManifestPkgs.push(...deps);
|
||||
}
|
||||
|
||||
// Cargo.lock files
|
||||
const cargoLockFiles = walkFiles(repoRoot, n => n === 'Cargo.lock');
|
||||
console.log(` Found ${cargoLockFiles.length} Cargo.lock files`);
|
||||
for (const f of cargoLockFiles) {
|
||||
allManifestPkgs.push(...collectCargoLockDeps(f));
|
||||
}
|
||||
|
||||
// cgmanifest.json files
|
||||
const cgManifestFiles = walkFiles(repoRoot, n => n === 'cgmanifest.json');
|
||||
console.log(` Found ${cgManifestFiles.length} cgmanifest.json files`);
|
||||
for (const f of cgManifestFiles) {
|
||||
allManifestPkgs.push(...collectCgManifestDeps(f));
|
||||
}
|
||||
|
||||
console.log(` Total manifest packages collected: ${allManifestPkgs.length}`);
|
||||
console.log('');
|
||||
|
||||
// Build notice name set (lowercase, without version)
|
||||
const noticeNames = new Set<string>();
|
||||
for (const e of entries) {
|
||||
noticeNames.add(e.name.toLowerCase());
|
||||
}
|
||||
|
||||
const xref = crossReference(noticeNames, allManifestPkgs);
|
||||
printReport(noticePath, stats, xref, lockfileBreakdown);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,155 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Focused unit checks for Section 4 (Cargo.lock harvesting) of scan-licenses.ts.
|
||||
*
|
||||
* Run: npx tsx cargo-section4.test.ts
|
||||
*
|
||||
* Covers (no network required):
|
||||
* 1. isSpdxStub truth table — true for all 17 known CG stub bodies, false for
|
||||
* real short license bodies and license prose.
|
||||
* 2. parseCargoLock — extracts packages from cli/Cargo.lock + build/win32,
|
||||
* skips workspace crates with no source, returns correct name/version for
|
||||
* clap, jiff, slog, term.
|
||||
* 3. getCrateRepository — override map returns mirror URLs.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { isSpdxStub, parseCargoLock, getCrateRepository, crateLicenseRefs } from './scan-licenses.js';
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
function check(name: string, cond: boolean): void {
|
||||
if (cond) {
|
||||
passed++;
|
||||
console.log(` ok ${name}`);
|
||||
} else {
|
||||
failed++;
|
||||
console.error(` FAIL ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
// -- 1. isSpdxStub truth table ------------------------------------------------
|
||||
console.log('isSpdxStub — known stub bodies (must be TRUE):');
|
||||
|
||||
// SPDX strings confirmed via crates.io for each of the 17 known stub crates.
|
||||
const stubBodies: Array<[string, string]> = [
|
||||
['objc2', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-app-kit', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-cloud-kit', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-core-data', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-core-foundation', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-core-graphics', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-foundation', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-io-kit', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-metal', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-quartz-core', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['objc2-encode', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['dispatch2', 'Zlib OR Apache-2.0 OR MIT'],
|
||||
['crc-catalog', 'MIT OR Apache-2.0'],
|
||||
['linux-keyutils', 'Apache-2.0 OR MIT'],
|
||||
['winapi-i686-pc-windows-gnu', 'MIT/Apache-2.0'],
|
||||
['winapi-x86_64-pc-windows-gnu', 'MIT/Apache-2.0'],
|
||||
['yasna', 'MIT OR Apache-2.0'],
|
||||
// Nested-paren compound (internal parens). Must still be detected as a stub
|
||||
// so the SPDX-as-body never survives in the notice. (e.g. encoding_rs)
|
||||
['encoding_rs', '(Apache-2.0 OR MIT) AND BSD-3-Clause'],
|
||||
['nested-leading-paren', '(MIT OR Apache-2.0) AND BSD-3-Clause'],
|
||||
];
|
||||
for (const [crate, body] of stubBodies) {
|
||||
check(`stub: ${crate} ("${body}")`, isSpdxStub(body) === true);
|
||||
}
|
||||
|
||||
console.log('isSpdxStub — real / non-stub bodies (must be FALSE):');
|
||||
const isc = `ISC License
|
||||
|
||||
Copyright (c) 2004-2010 by Internet Systems Consortium, Inc. ("ISC")
|
||||
|
||||
Permission to use, copy, modify, and/or distribute this software for any
|
||||
purpose with or without fee is hereby granted, provided that the above
|
||||
copyright notice and this permission notice appear in all copies.`;
|
||||
const bsd2 = `Copyright (c) 2015, Some Author
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:`;
|
||||
const zeroBsd = `Zero-Clause BSD
|
||||
|
||||
Permission to use, copy, modify, and/or distribute this software for any
|
||||
purpose with or without fee is hereby granted.`;
|
||||
const mit = `MIT License
|
||||
|
||||
Copyright (c) 2020 Author
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal`;
|
||||
const prose = 'Permission is hereby granted, free of charge, to any person obtaining a copy';
|
||||
|
||||
check('real: ISC body', isSpdxStub(isc) === false);
|
||||
check('real: BSD-2-Clause body', isSpdxStub(bsd2) === false);
|
||||
check('real: 0BSD body', isSpdxStub(zeroBsd) === false);
|
||||
check('real: MIT body', isSpdxStub(mit) === false);
|
||||
check('real: bare permission prose line', isSpdxStub(prose) === false);
|
||||
check('real: empty body', isSpdxStub('') === false);
|
||||
check('real: whitespace only', isSpdxStub(' \n ') === false);
|
||||
|
||||
// -- 2. parseCargoLock --------------------------------------------------------
|
||||
console.log('parseCargoLock:');
|
||||
// Run from the oss dir (build/azure-pipelines/oss): repo root is three levels up.
|
||||
const repoRoot = path.resolve(process.cwd(), '..', '..', '..');
|
||||
const cliLockPath = path.join(repoRoot, 'cli', 'Cargo.lock');
|
||||
const win32LockPath = path.join(repoRoot, 'build', 'win32', 'Cargo.lock');
|
||||
|
||||
const cliPkgs = parseCargoLock(fs.readFileSync(cliLockPath, 'utf8'));
|
||||
console.log(` cli/Cargo.lock: ${cliPkgs.length} packages parsed`);
|
||||
check('cli: parsed a substantial number of packages (>300)', cliPkgs.length > 300);
|
||||
|
||||
const cliNoSource = cliPkgs.filter(p => !p.source);
|
||||
const cliWithSource = cliPkgs.filter(p => p.source);
|
||||
console.log(` cli/Cargo.lock: ${cliNoSource.length} workspace crates (no source), ${cliWithSource.length} with source`);
|
||||
check('cli: at least one workspace crate (no source) present', cliNoSource.length >= 1);
|
||||
check('cli: most crates have a source', cliWithSource.length > 300);
|
||||
|
||||
const clap = cliPkgs.find(p => p.name === 'clap');
|
||||
check('cli: clap present with version + registry source', !!clap && /^\d/.test(clap!.version) && !!clap!.source);
|
||||
const jiff = cliPkgs.find(p => p.name === 'jiff');
|
||||
check('cli: jiff present with version + registry source', !!jiff && /^\d/.test(jiff!.version) && !!jiff!.source);
|
||||
|
||||
const win32Pkgs = parseCargoLock(fs.readFileSync(win32LockPath, 'utf8'));
|
||||
console.log(` build/win32/Cargo.lock: ${win32Pkgs.length} packages parsed`);
|
||||
const slog = win32Pkgs.find(p => p.name === 'slog');
|
||||
check('win32: slog present with version', !!slog && /^\d/.test(slog!.version));
|
||||
const term = win32Pkgs.find(p => p.name === 'term');
|
||||
check('win32: term present with version', !!term && /^\d/.test(term!.version));
|
||||
|
||||
// Sanity: no parsed package has a quote/bracket leaking into its fields.
|
||||
check('parse: no malformed names', cliPkgs.every(p => /^[A-Za-z0-9_.-]+$/.test(p.name)));
|
||||
check('parse: no malformed versions', cliPkgs.every(p => /^[0-9]/.test(p.version)));
|
||||
|
||||
// -- 3. getCrateRepository override map ---------------------------------------
|
||||
console.log('getCrateRepository override map:');
|
||||
const mk = (id: string) => ({ crate: { id, repository: 'https://example.com/wrong' }, versions: [] });
|
||||
check('isatty -> dtolnay/isatty', getCrateRepository(mk('isatty')) === 'https://github.com/dtolnay/isatty');
|
||||
check('redox_syscall -> redox-os/syscall', getCrateRepository(mk('redox_syscall')) === 'https://github.com/redox-os/syscall');
|
||||
check('redox_termios -> redox-os/termios', getCrateRepository(mk('redox_termios')) === 'https://github.com/redox-os/termios');
|
||||
check('termion -> redox-os/termion', getCrateRepository(mk('termion')) === 'https://github.com/redox-os/termion');
|
||||
check('fallback -> crate.repository', getCrateRepository({ crate: { id: 'clap', repository: 'https://github.com/clap-rs/clap' }, versions: [] }) === 'https://github.com/clap-rs/clap');
|
||||
|
||||
// -- 4. crateLicenseRefs ordering ---------------------------------------------
|
||||
console.log('crateLicenseRefs:');
|
||||
const refs = crateLicenseRefs('clap', '4.5.20');
|
||||
check('refs: v<version> first', refs[0] === 'v4.5.20');
|
||||
check('refs: <version> second', refs[1] === '4.5.20');
|
||||
check('refs: <name>-v<version>', refs[2] === 'clap-v4.5.20');
|
||||
check('refs: <name>-<version>', refs[3] === 'clap-4.5.20');
|
||||
check('refs: default-branch fallbacks last', refs.includes('main') && refs.includes('master'));
|
||||
|
||||
// -- summary ------------------------------------------------------------------
|
||||
console.log('');
|
||||
console.log(`=== cargo-section4 unit checks: ${passed} passed, ${failed} failed ===`);
|
||||
process.exit(failed === 0 ? 0 : 1);
|
||||
@@ -0,0 +1,427 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Merge CG-generated and extension-scanned NOTICE files into a single output.
|
||||
*
|
||||
* - Keys by lowercased name@version so multiple shipped versions are preserved
|
||||
* - CG entries win on collision; scanner cargo entries override CG on stub
|
||||
* collisions (SPDX-as-body replaced with real fetched license text)
|
||||
* - Logs every package with its source
|
||||
* - Produces ThirdPartyNotices.new.txt
|
||||
*
|
||||
* Usage:
|
||||
* node merge-notices.js --cg <path> --extensions <path> --output <path>
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import fs from 'fs';
|
||||
import { applyOverrides, readCglicenses } from './apply-overrides.js';
|
||||
import { isPackageHeader } from './parse-notices.js';
|
||||
import { parseArgs } from './utils.js';
|
||||
|
||||
interface NoticeEntry {
|
||||
name: string;
|
||||
version: string;
|
||||
license: string;
|
||||
url: string;
|
||||
licenseText: string;
|
||||
}
|
||||
|
||||
const SEPARATOR = '---------------------------------------------------------';
|
||||
const SEPARATOR_REGEX = /^-{50,}$/m;
|
||||
|
||||
function parseNoticeFile(filePath: string): NoticeEntry[] {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return [];
|
||||
}
|
||||
const content = fs.readFileSync(filePath, 'utf8');
|
||||
const entries: NoticeEntry[] = [];
|
||||
const blocks = content.split(SEPARATOR_REGEX);
|
||||
|
||||
for (let i = 1; i < blocks.length; i++) {
|
||||
const block = blocks[i].trim();
|
||||
if (!block) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const lines = block.split('\n');
|
||||
let headerLine = '';
|
||||
let headerIdx = 0;
|
||||
for (let j = 0; j < lines.length; j++) {
|
||||
if (lines[j].trim()) {
|
||||
headerLine = lines[j].trim();
|
||||
headerIdx = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!headerLine || SEPARATOR_REGEX.test(headerLine)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip file header lines
|
||||
if (headerLine.startsWith('NOTICES AND INFORMATION') || headerLine.startsWith('Do Not Translate')) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Validate this looks like a real package header, not license prose
|
||||
// following a decorative dash line inside a LICENSE body.
|
||||
if (!isPackageHeader(headerLine)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const match = headerLine.match(/^(.+?)\s+([\d][^\s]*)\s+-\s+(.+)$/);
|
||||
let name: string, version: string, license: string;
|
||||
let url = '';
|
||||
|
||||
if (match) {
|
||||
name = match[1];
|
||||
version = match[2];
|
||||
license = match[3];
|
||||
} else {
|
||||
const match2 = headerLine.match(/^(.+?)\s+-\s+(.+)$/);
|
||||
if (match2) {
|
||||
name = match2[1];
|
||||
version = '';
|
||||
license = match2[2];
|
||||
} else {
|
||||
name = headerLine;
|
||||
version = '';
|
||||
license = '';
|
||||
}
|
||||
}
|
||||
|
||||
const remainingLines = lines.slice(headerIdx + 1);
|
||||
let licenseStartIdx = 0;
|
||||
for (let j = 0; j < remainingLines.length; j++) {
|
||||
const line = remainingLines[j].trim();
|
||||
if (!line) {
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith('http://') || line.startsWith('https://')) {
|
||||
url = line;
|
||||
licenseStartIdx = j + 1;
|
||||
} else {
|
||||
licenseStartIdx = j;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
const licenseText = remainingLines.slice(licenseStartIdx).join('\n').trim();
|
||||
entries.push({ name, version, license, url, licenseText });
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
async function mainAsync(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const cgPath = args['cg'] || '';
|
||||
const extPath = args['extensions'] || '';
|
||||
const outputPath = args['output'];
|
||||
const cglicensesPath = args['cglicenses'] || '';
|
||||
const strict = process.argv.includes('--strict');
|
||||
const provenance = args['provenance'] !== undefined;
|
||||
|
||||
if (!outputPath) {
|
||||
console.error('Usage: merge-notices.js --cg <path> --extensions <path> --output <path> [--cglicenses <path>] [--strict]');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Parse both inputs
|
||||
const cgEntries = cgPath ? parseNoticeFile(cgPath) : [];
|
||||
const extEntries = extPath ? parseNoticeFile(extPath) : [];
|
||||
|
||||
console.log('=== Merge Step: CG + Extension Licenses ===');
|
||||
console.log('');
|
||||
console.log(`CG input: ${cgPath || '(none)'}`);
|
||||
console.log(` Entries parsed: ${cgEntries.length}`);
|
||||
console.log(`Extension input: ${extPath || '(none)'}`);
|
||||
console.log(` Entries parsed: ${extEntries.length}`);
|
||||
console.log('');
|
||||
|
||||
// Build merged map -- CG entries go in first (they win on conflicts).
|
||||
// Key is `<name>@<version>` so multiple versions of the same package are
|
||||
// preserved as separate entries (per CELA guidance: keep every shipped
|
||||
// version's license text, even if license text is identical between versions).
|
||||
// Entries with no version fall back to a name-only key to keep them distinct
|
||||
// from versioned siblings.
|
||||
const merged = new Map<string, NoticeEntry & { source: string }>();
|
||||
const mergeKey = (e: NoticeEntry) => `${e.name.toLowerCase()}@${e.version || ''}`;
|
||||
|
||||
for (const entry of cgEntries) {
|
||||
const key = mergeKey(entry);
|
||||
if (!merged.has(key)) {
|
||||
merged.set(key, { ...entry, source: 'component-governance' });
|
||||
}
|
||||
}
|
||||
|
||||
const cgCount = merged.size;
|
||||
|
||||
// Load the scanner's stub-override index (sibling of the --extensions file).
|
||||
// These `<name>@<version>` keys identify CG entries whose body is a stub (the
|
||||
// SPDX expression instead of real license text); for them the scanner entry
|
||||
// must WIN the collision, overriding CG's stub with the real fetched text.
|
||||
const stubOverrideKeys = new Set<string>();
|
||||
const stubOverridePath = args['stuboverride'] || (extPath ? extPath + '.stuboverride.json' : '');
|
||||
if (stubOverridePath && fs.existsSync(stubOverridePath)) {
|
||||
try {
|
||||
const raw: unknown = JSON.parse(fs.readFileSync(stubOverridePath, 'utf8'));
|
||||
if (Array.isArray(raw)) {
|
||||
for (const k of raw) {
|
||||
if (typeof k === 'string') {
|
||||
stubOverrideKeys.add(k.toLowerCase());
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log(`Stub-override index: ${stubOverrideKeys.size} cargo entries override CG (${stubOverridePath})`);
|
||||
console.log('');
|
||||
} catch (err) {
|
||||
console.warn(` WARN: could not read stub-override index ${stubOverridePath}: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Add extension entries (only if same name+version not already covered by CG)
|
||||
let extAdded = 0;
|
||||
let extSkipped = 0;
|
||||
let extStubOverridden = 0;
|
||||
const extAddedNames: string[] = [];
|
||||
const extSkippedNames: string[] = [];
|
||||
// Override keys that actually collided with a CG entry (so the override took
|
||||
// effect). Used after the loop to surface keys that never collided.
|
||||
const collidedOverrideKeys = new Set<string>();
|
||||
|
||||
for (const entry of extEntries) {
|
||||
const key = mergeKey(entry);
|
||||
if (merged.has(key)) {
|
||||
// Normally CG wins the collision. Exception: cargo stub-override keys --
|
||||
// the scanner replaces CG's SPDX-as-body stub with real fetched text.
|
||||
if (stubOverrideKeys.has(key)) {
|
||||
merged.set(key, { ...entry, source: 'cargo-stub-override' });
|
||||
extStubOverridden++;
|
||||
collidedOverrideKeys.add(key);
|
||||
extAddedNames.push(`${entry.name} ${entry.version} - ${entry.license} (stub-override)`);
|
||||
} else {
|
||||
extSkipped++;
|
||||
extSkippedNames.push(`${entry.name}@${entry.version || '(no version)'}`);
|
||||
}
|
||||
} else {
|
||||
merged.set(key, { ...entry, source: 'extension-scanner' });
|
||||
extAdded++;
|
||||
extAddedNames.push(`${entry.name} ${entry.version} - ${entry.license}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Stub-override only flips priority on a `name@version` collision. If CG
|
||||
// recorded a crate at a DIFFERENT version than Cargo.lock resolved, there is
|
||||
// no collision: the scanner entry is added as a separate record and CG's stub
|
||||
// SURVIVES -- a silent divergence (the scanner's cargoStubOverride counter
|
||||
// still claimed success). Warn loudly for any override key that never
|
||||
// collided so the mismatch is visible to the operator.
|
||||
const uncollidedOverrideKeys = [...stubOverrideKeys].filter(k => !collidedOverrideKeys.has(k)).sort();
|
||||
if (uncollidedOverrideKeys.length > 0) {
|
||||
console.warn(` WARN: ${uncollidedOverrideKeys.length} stub-override key(s) never collided with a CG entry -- CG's stub may survive at a different version. Verify these crates' notice text:`);
|
||||
for (const k of uncollidedOverrideKeys) {
|
||||
console.warn(` - ${k}`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// Log what happened
|
||||
console.log(`--- Component Governance provided ${cgCount} packages ---`);
|
||||
console.log(' (Source: ClearlyDefined license database via notice@0 task)');
|
||||
console.log('');
|
||||
console.log(`--- Extension scanner added ${extAdded} packages ---`);
|
||||
console.log(' (Source: LICENSE files read from extensions/*/node_modules/)');
|
||||
console.log(' (Why: CG skips packages with engines.vscode, which includes all built-in extensions)');
|
||||
console.log('');
|
||||
|
||||
if (extAddedNames.length > 0) {
|
||||
console.log(' Packages added from extension scanner:');
|
||||
for (const n of extAddedNames.sort()) {
|
||||
console.log(` + ${n}`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
if (extSkippedNames.length > 0) {
|
||||
console.log(` ${extSkipped} extension packages already covered by CG (skipped):`);
|
||||
for (const n of extSkippedNames.sort()) {
|
||||
console.log(` = ${n}`);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// Apply cglicenses.json overrides (the manual "last ~4%" gap-fill file).
|
||||
// CELA-approved: overrides come from human-authored entries, not the tool.
|
||||
let overridesApplied = 0;
|
||||
let overridesInjected = 0;
|
||||
let overrideEntryCount = 0;
|
||||
const unmatchedOverrides: string[] = [];
|
||||
const staleOverrides: string[] = [];
|
||||
if (cglicensesPath) {
|
||||
if (!fs.existsSync(cglicensesPath)) {
|
||||
console.warn(`WARN: --cglicenses path not found: ${cglicensesPath}`);
|
||||
} else {
|
||||
try {
|
||||
const overrides = readCglicenses(cglicensesPath);
|
||||
console.log(`--- Applying ${overrides.length} cglicenses.json overrides ---`);
|
||||
|
||||
// Load the scanner's presence index (packages on disk with no license
|
||||
// file). This lets applyOverrides tell "present but unlicensed" (inject)
|
||||
// apart from "not shipped" (stale -> warn + skip).
|
||||
const presencePath = args['presence'] || (extPath ? extPath + '.presence.json' : '');
|
||||
const presentNames = new Set<string>();
|
||||
if (presencePath && fs.existsSync(presencePath)) {
|
||||
try {
|
||||
const raw: unknown = JSON.parse(fs.readFileSync(presencePath, 'utf8'));
|
||||
if (Array.isArray(raw)) {
|
||||
for (const item of raw as { name?: string }[]) {
|
||||
if (item && typeof item.name === 'string') {
|
||||
presentNames.add(item.name.toLowerCase());
|
||||
}
|
||||
}
|
||||
}
|
||||
console.log(` Presence index: ${presentNames.size} present-but-unlicensed packages (${presencePath})`);
|
||||
} catch (err) {
|
||||
console.warn(` WARN: could not read presence index ${presencePath}: ${(err as Error).message}`);
|
||||
}
|
||||
} else {
|
||||
console.warn(` WARN: no presence index found (${presencePath || 'none'}); stale overrides cannot be detected, so unmatched overrides with usable text will be injected.`);
|
||||
}
|
||||
|
||||
const result = await applyOverrides(merged, overrides, { fetchUris: true, presentNames });
|
||||
overridesApplied = result.appliedNames.length;
|
||||
overridesInjected = result.injectedNames.length;
|
||||
overrideEntryCount = result.appliedEntryCount;
|
||||
unmatchedOverrides.push(...result.unmatchedNames);
|
||||
staleOverrides.push(...result.staleNames);
|
||||
for (const name of result.appliedNames.sort()) {
|
||||
console.log(` * edited: ${name}`);
|
||||
}
|
||||
for (const name of result.injectedNames.sort()) {
|
||||
console.log(` + injected: ${name}`);
|
||||
}
|
||||
for (const err of result.errors) {
|
||||
console.error(` ! ${err}`);
|
||||
}
|
||||
if (result.staleNames.length > 0) {
|
||||
console.warn(` ${result.staleNames.length} override(s) reference a package not present in THIS build. A package can be absent because it is platform-specific (it only ships on other OS/arch builds) OR because the override is genuinely stale. Verify across platforms before removing anything from cglicenses.json:`);
|
||||
// Heuristic: flag names that contain a platform/arch token so a
|
||||
// single-platform build doesn't lead someone to delete an override
|
||||
// that other-platform builds still rely on.
|
||||
const platformToken = /(darwin|win32|linux|linuxmusl|alpine|freebsd|android|arm64|x64|ia32|armhf|ppc64|s390x|musl)/i;
|
||||
for (const name of result.staleNames.sort()) {
|
||||
const hint = platformToken.test(name)
|
||||
? ' (likely platform-specific -- do NOT delete based on a single-platform build)'
|
||||
: '';
|
||||
console.warn(` ? stale: ${name}${hint}`);
|
||||
}
|
||||
}
|
||||
if (result.unmatchedNames.length > 0) {
|
||||
console.warn(` ${result.unmatchedNames.length} override(s) have no matching package and no usable license text:`);
|
||||
for (const name of result.unmatchedNames.sort()) {
|
||||
console.warn(` ? unmatched: ${name}`);
|
||||
}
|
||||
}
|
||||
// Stale overrides are warn-only and NEVER fail the build (the PR-time
|
||||
// check is the real gate). --strict still fails on genuine errors and on
|
||||
// unmatched (no-usable-text) overrides.
|
||||
if (strict && (result.errors.length > 0 || result.unmatchedNames.length > 0)) {
|
||||
console.error('--strict: failing build due to override errors or unmatched entries');
|
||||
process.exit(2);
|
||||
}
|
||||
console.log('');
|
||||
} catch (err) {
|
||||
console.error(`ERROR reading cglicenses overrides from ${cglicensesPath}: ${(err as Error).message}`);
|
||||
if (strict) {
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sort and render
|
||||
const sorted = [...merged.values()].sort((a, b) =>
|
||||
a.name.toLowerCase().localeCompare(b.name.toLowerCase())
|
||||
);
|
||||
|
||||
const header = `NOTICES AND INFORMATION
|
||||
Do Not Translate or Localize
|
||||
|
||||
This software incorporates material from third parties.
|
||||
Microsoft makes certain open source code available at https://3rdpartysource.microsoft.com,
|
||||
or you may send a check or money order for US $5.00, including the product name,
|
||||
the open source component name, platform, and version number, to:
|
||||
|
||||
Source Code Compliance Team
|
||||
Microsoft Corporation
|
||||
One Microsoft Way
|
||||
Redmond, WA 98052
|
||||
USA
|
||||
|
||||
Notwithstanding any other terms, you may reverse engineer this software to the extent
|
||||
required to debug changes to any libraries licensed under the GNU Lesser General Public License.`;
|
||||
|
||||
let output = header + '\n\n';
|
||||
|
||||
for (const entry of sorted) {
|
||||
output += '\n' + SEPARATOR + '\n\n';
|
||||
output += entry.name;
|
||||
if (entry.version) {
|
||||
output += ' ' + entry.version;
|
||||
}
|
||||
if (entry.license) {
|
||||
output += ' - ' + entry.license;
|
||||
}
|
||||
output += '\n';
|
||||
if (entry.url) {
|
||||
output += entry.url + '\n';
|
||||
}
|
||||
// Provenance annotations help reviewers trace where each license text
|
||||
// came from. Gated behind --provenance so they can be stripped from the
|
||||
// final shipped NOTICE when we cut over from the shadow copy.
|
||||
if (provenance) {
|
||||
if (entry.source === 'extension-scanner') {
|
||||
output += '(license text obtained directly from package LICENSE file)\n';
|
||||
} else if (entry.source === 'cargo-stub-override') {
|
||||
output += '(license text fetched from the crate repository -- replaces CG SPDX-expression stub)\n';
|
||||
} else if (entry.source === 'cglicenses-override') {
|
||||
output += '(license text supplied by cglicenses.json manual override)\n';
|
||||
}
|
||||
}
|
||||
output += '\n';
|
||||
if (entry.licenseText) {
|
||||
output += entry.licenseText + '\n';
|
||||
}
|
||||
}
|
||||
|
||||
output += '\n' + SEPARATOR + '\n';
|
||||
|
||||
try {
|
||||
fs.writeFileSync(outputPath, output, 'utf8');
|
||||
} catch (err) {
|
||||
console.error(`ERROR: Failed to write output file ${outputPath}: ${(err as Error).message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const sizeMB = (output.length / 1024 / 1024).toFixed(2);
|
||||
|
||||
console.log('=== Merge Summary ===');
|
||||
console.log(` Total packages: ${sorted.length}`);
|
||||
console.log(` From CG: ${cgCount}`);
|
||||
console.log(` From extension scanner: ${extAdded}`);
|
||||
console.log(` Cargo stub-overrides (beat CG): ${extStubOverridden}`);
|
||||
console.log(` Extension duplicates skipped: ${extSkipped}`);
|
||||
if (cglicensesPath) {
|
||||
console.log(` cglicenses overrides applied: ${overridesApplied} (${overrideEntryCount} merged entries updated)`);
|
||||
console.log(` cglicenses overrides injected: ${overridesInjected}`);
|
||||
console.log(` cglicenses overrides stale (skipped, warn-only): ${staleOverrides.length}`);
|
||||
console.log(` cglicenses overrides unmatched (no usable text): ${unmatchedOverrides.length}`);
|
||||
}
|
||||
console.log(` Output: ${outputPath} (${sizeMB} MB)`);
|
||||
}
|
||||
|
||||
mainAsync().catch(err => { console.error(err); process.exit(1); });
|
||||
@@ -0,0 +1,316 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Parse ThirdPartyNotices.txt files and compare them.
|
||||
*
|
||||
* Usage:
|
||||
* node parse-notices.js --file <path> List all packages
|
||||
* node parse-notices.js --diff <pathA> <pathB> Compare two files
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import fs from 'fs';
|
||||
|
||||
export interface NoticeEntry {
|
||||
name: string;
|
||||
version: string;
|
||||
license: string;
|
||||
licenseTextLength: number;
|
||||
/** 1-indexed line number in the source file where this entry's header appears */
|
||||
lineNumber: number;
|
||||
/**
|
||||
* The license body text, captured between this entry's header and the next
|
||||
* separator, with a single leading URL line stripped (matching how the merge
|
||||
* step splits url vs. body). Used by Section 4 of scan-licenses.ts to detect
|
||||
* CG "stub" bodies (where CG emitted the SPDX expression as the body instead
|
||||
* of the real license text). Optional so existing callers that only need
|
||||
* names/lengths are unaffected.
|
||||
*/
|
||||
licenseText?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristic: does this line look like a package header rather than license prose?
|
||||
*
|
||||
* Real headers: "@scope/pkg 1.2.3 - MIT", "pkg - Apache-2.0", "@scope/pkg"
|
||||
* "org/repo abc123...def - Apache-2.0" (git SHA)
|
||||
* False positives after decorative dashes inside license text:
|
||||
* "END OF TERMS AND CONDITIONS", "Apache License", "The MIT License (MIT)",
|
||||
* "Copyright (c) 2020 Author", "PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2"
|
||||
*
|
||||
* Key signals: real package names are lowercase (or git org/repo), never start
|
||||
* with common license-prose words, and never contain parentheses.
|
||||
*/
|
||||
const LICENSE_PROSE_STARTS = /^(copyright|license|licen[cs]e|permission|the |this |all |end |terms|conditions|disclaimer|granted|redistribution|neither|no |software|unless|provided|except|in |of |for |under |above|note|#|\*|\(|\d+[\.\)]\s)/i;
|
||||
|
||||
export function isPackageHeader(line: string): boolean {
|
||||
// License prose almost always starts with a common English/legal word
|
||||
if (LICENSE_PROSE_STARTS.test(line)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Real package names: @scope/name, org/repo, plain-name
|
||||
// They start with @, a lowercase letter, or a digit (for rare cases)
|
||||
// and consist of [a-z0-9@/._-]
|
||||
// CG git entries use Org/Repo (mixed case) — handle separately
|
||||
const firstWord = line.split(/\s/)[0];
|
||||
if (!firstWord) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Standard npm-style package name (lowercase, may start with @)
|
||||
const npmNameRe = /^@?[a-z0-9][a-z0-9\-\./@_]*$/;
|
||||
// CG git-style: Org/Repo (e.g., "nickel-org/rust-mustache")
|
||||
const gitRepoRe = /^[a-zA-Z0-9_\-]+\/[a-zA-Z0-9_\-\.]+$/;
|
||||
|
||||
if (npmNameRe.test(firstWord) || gitRepoRe.test(firstWord)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Some entries have names with mixed case (rare but legitimate)
|
||||
// Accept if the line matches "name version - license" pattern strictly
|
||||
if (/^.+?\s+[\d][^\s]*\s+-\s+.+$/.test(line)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
export function parseNoticeFile(filePath: string): NoticeEntry[] {
|
||||
const content = fs.readFileSync(filePath, 'utf8');
|
||||
const lines = content.split('\n');
|
||||
const entries: NoticeEntry[] = [];
|
||||
const sepRe = /^-{50,}$/;
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (!sepRe.test(lines[i].trim())) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Find first non-empty line after separator — that's the header candidate
|
||||
let headerLine = '';
|
||||
let headerIdx = -1;
|
||||
for (let j = i + 1; j < lines.length; j++) {
|
||||
if (lines[j].trim()) {
|
||||
headerLine = lines[j].trim();
|
||||
headerIdx = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!headerLine || headerIdx === -1) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Don't re-parse the file header block
|
||||
if (headerLine.startsWith('NOTICES AND INFORMATION') || headerLine.startsWith('Do Not Translate')) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip if the "header" is actually another separator (double-separator pattern)
|
||||
if (sepRe.test(headerLine)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Validate that this looks like a real package header, not license
|
||||
// prose that happens to follow a decorative dash line.
|
||||
// Real headers: "name version - license", "name - license", or "name"
|
||||
// False positives: "END OF TERMS AND CONDITIONS", "Apache License",
|
||||
// "The MIT License (MIT)", "Copyright (c) ...", etc.
|
||||
if (!isPackageHeader(headerLine)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Parse: "name version - license" or just "name"
|
||||
const match = headerLine.match(/^(.+?)\s+([\d][^\s]*)\s+-\s+(.+)$/);
|
||||
let name: string, version: string, license: string;
|
||||
if (match) {
|
||||
name = match[1];
|
||||
version = match[2];
|
||||
license = match[3];
|
||||
} else {
|
||||
// Try: "name - license" (no version)
|
||||
const match2 = headerLine.match(/^(.+?)\s+-\s+(.+)$/);
|
||||
if (match2) {
|
||||
name = match2[1];
|
||||
version = '';
|
||||
license = match2[2];
|
||||
} else {
|
||||
name = headerLine;
|
||||
version = '';
|
||||
license = '';
|
||||
}
|
||||
}
|
||||
|
||||
// Measure license text length (everything until next separator) and
|
||||
// capture the body slice. The body strips a single leading URL line so
|
||||
// stub detection sees the license text, not the repo link (mirrors how
|
||||
// merge-notices.ts splits url vs. licenseText).
|
||||
let textLength = 0;
|
||||
const bodyLines: string[] = [];
|
||||
for (let j = headerIdx + 1; j < lines.length; j++) {
|
||||
if (sepRe.test(lines[j].trim())) {
|
||||
break;
|
||||
}
|
||||
textLength += lines[j].length + 1;
|
||||
bodyLines.push(lines[j]);
|
||||
}
|
||||
|
||||
// Drop leading blank lines, then a single leading URL line if present.
|
||||
while (bodyLines.length > 0 && !bodyLines[0].trim()) {
|
||||
bodyLines.shift();
|
||||
}
|
||||
if (bodyLines.length > 0) {
|
||||
const first = bodyLines[0].trim();
|
||||
if (first.startsWith('http://') || first.startsWith('https://')) {
|
||||
bodyLines.shift();
|
||||
}
|
||||
}
|
||||
const licenseText = bodyLines.join('\n').trim();
|
||||
|
||||
entries.push({
|
||||
name,
|
||||
version,
|
||||
license,
|
||||
licenseTextLength: textLength,
|
||||
lineNumber: headerIdx + 1, // 1-indexed
|
||||
licenseText,
|
||||
});
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
function listMode(filePath: string): void {
|
||||
const entries = parseNoticeFile(filePath);
|
||||
const sizeMB = (fs.statSync(filePath).size / 1024 / 1024).toFixed(2);
|
||||
|
||||
console.log(`File: ${filePath}`);
|
||||
console.log(`Size: ${sizeMB} MB`);
|
||||
console.log(`Total entries: ${entries.length}`);
|
||||
console.log(`Unique names: ${new Set(entries.map(e => e.name.toLowerCase())).size}`);
|
||||
console.log('');
|
||||
console.log('Line | Name | Version | License | Text Length');
|
||||
console.log('---- | ---- | ------- | ------- | -----------');
|
||||
|
||||
for (const e of entries) {
|
||||
console.log(`${e.lineNumber} | ${e.name} | ${e.version} | ${e.license} | ${e.licenseTextLength}`);
|
||||
}
|
||||
}
|
||||
|
||||
function diffMode(pathA: string, pathB: string): void {
|
||||
const entriesA = parseNoticeFile(pathA);
|
||||
const entriesB = parseNoticeFile(pathB);
|
||||
|
||||
const namesA = new Map<string, NoticeEntry[]>();
|
||||
for (const e of entriesA) {
|
||||
const key = e.name.toLowerCase();
|
||||
if (!namesA.has(key)) {
|
||||
namesA.set(key, []);
|
||||
}
|
||||
namesA.get(key)!.push(e);
|
||||
}
|
||||
|
||||
const namesB = new Map<string, NoticeEntry[]>();
|
||||
for (const e of entriesB) {
|
||||
const key = e.name.toLowerCase();
|
||||
if (!namesB.has(key)) {
|
||||
namesB.set(key, []);
|
||||
}
|
||||
namesB.get(key)!.push(e);
|
||||
}
|
||||
|
||||
const onlyA: NoticeEntry[] = [];
|
||||
const onlyB: NoticeEntry[] = [];
|
||||
const inBoth: string[] = [];
|
||||
|
||||
for (const [name, entries] of namesA) {
|
||||
if (namesB.has(name)) {
|
||||
inBoth.push(name);
|
||||
} else {
|
||||
onlyA.push(...entries);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [name, entries] of namesB) {
|
||||
if (!namesA.has(name)) {
|
||||
onlyB.push(...entries);
|
||||
}
|
||||
}
|
||||
|
||||
const sizeA = (fs.statSync(pathA).size / 1024 / 1024).toFixed(2);
|
||||
const sizeB = (fs.statSync(pathB).size / 1024 / 1024).toFixed(2);
|
||||
|
||||
console.log('=== NOTICE File Comparison ===');
|
||||
console.log('');
|
||||
console.log(`File A: ${pathA}`);
|
||||
console.log(` Size: ${sizeA} MB, Entries: ${entriesA.length}, Unique names: ${namesA.size}`);
|
||||
console.log('');
|
||||
console.log(`File B: ${pathB}`);
|
||||
console.log(` Size: ${sizeB} MB, Entries: ${entriesB.length}, Unique names: ${namesB.size}`);
|
||||
console.log('');
|
||||
console.log(`In both: ${inBoth.length}`);
|
||||
console.log(`Only in A: ${onlyA.length} (${namesA.size - inBoth.length} unique names)`);
|
||||
console.log(`Only in B: ${onlyB.length} (${namesB.size - inBoth.length} unique names)`);
|
||||
|
||||
if (onlyA.length > 0) {
|
||||
console.log('');
|
||||
console.log(`=== Only in A (${onlyA.length} entries) ===`);
|
||||
console.log('Line | Name | Version | License');
|
||||
console.log('---- | ---- | ------- | -------');
|
||||
for (const e of onlyA.sort((a, b) => a.name.toLowerCase().localeCompare(b.name.toLowerCase()))) {
|
||||
console.log(`${e.lineNumber} | ${e.name} | ${e.version} | ${e.license}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (onlyB.length > 0) {
|
||||
console.log('');
|
||||
console.log(`=== Only in B (${onlyB.length} entries) ===`);
|
||||
console.log('Line | Name | Version | License');
|
||||
console.log('---- | ---- | ------- | -------');
|
||||
for (const e of onlyB.sort((a, b) => a.name.toLowerCase().localeCompare(b.name.toLowerCase()))) {
|
||||
console.log(`${e.lineNumber} | ${e.name} | ${e.version} | ${e.license}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Version differences for packages in both
|
||||
const versionDiffs: string[] = [];
|
||||
for (const name of inBoth) {
|
||||
const aVersions = new Set(namesA.get(name)!.map(e => e.version).filter(v => v));
|
||||
const bVersions = new Set(namesB.get(name)!.map(e => e.version).filter(v => v));
|
||||
const aOnly = [...aVersions].filter(v => !bVersions.has(v));
|
||||
const bOnly = [...bVersions].filter(v => !aVersions.has(v));
|
||||
if (aOnly.length > 0 || bOnly.length > 0) {
|
||||
versionDiffs.push(`${name}: A has [${aOnly.join(', ')}], B has [${bOnly.join(', ')}]`);
|
||||
}
|
||||
}
|
||||
|
||||
if (versionDiffs.length > 0) {
|
||||
console.log('');
|
||||
console.log(`=== Version differences (${versionDiffs.length} packages) ===`);
|
||||
for (const d of versionDiffs) {
|
||||
console.log(` ${d}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -- CLI ----------------------------------------------------------------------
|
||||
|
||||
// Only run the CLI when parse-notices is the entry point — not when another
|
||||
// module (e.g. scan-licenses) imports parseNoticeFile.
|
||||
if (/parse-notices(\.[jt]s)?$/.test(process.argv[1] || '')) {
|
||||
const args = process.argv.slice(2);
|
||||
|
||||
if (args[0] === '--file' && args[1]) {
|
||||
listMode(args[1]);
|
||||
} else if (args[0] === '--diff' && args[1] && args[2]) {
|
||||
diffMode(args[1], args[2]);
|
||||
} else {
|
||||
console.log('Usage:');
|
||||
console.log(' node parse-notices.js --file <path> List all packages in a NOTICE file');
|
||||
console.log(' node parse-notices.js --diff <pathA> <pathB> Compare two NOTICE files');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Focused unit checks for Section 5 (platform-binary enumeration) of scan-licenses.ts.
|
||||
*
|
||||
* Run: npx tsx platform-binary.test.ts
|
||||
*
|
||||
* Covers (no network required):
|
||||
* 1. ARCH_SUFFIX_RE / isArchPackageName - matches arch package names, rejects
|
||||
* the arch-independent parents.
|
||||
* 2. npmLicenseId - handles string, {type}, and licenses:[{type}] shapes.
|
||||
* 3. familyText - family-level text reuse: one arch has text, a sibling with
|
||||
* none reuses the cached text; parent fallback and empty last-resort.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import { ARCH_SUFFIX_RE, isArchPackageName, isShippedArch, npmLicenseId, familyText, parentTextIfCompatible } from './scan-licenses.js';
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
function check(name: string, cond: boolean): void {
|
||||
if (cond) {
|
||||
passed++;
|
||||
console.log(` ok ${name}`);
|
||||
} else {
|
||||
failed++;
|
||||
console.error(` FAIL ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
// -- 1. arch-suffix recognition -----------------------------------------------
|
||||
console.log('ARCH_SUFFIX_RE / isArchPackageName - arch packages (must MATCH):');
|
||||
const archYes = [
|
||||
'@img/sharp-win32-x64',
|
||||
'@img/sharp-libvips-darwin-arm64',
|
||||
'@napi-rs/canvas-win32-x64-msvc',
|
||||
'@napi-rs/canvas-linux-arm-gnueabihf',
|
||||
'@parcel/watcher-linux-x64-glibc',
|
||||
'@parcel/watcher-win32-ia32',
|
||||
'@github/copilot-linuxmusl-arm64',
|
||||
'@esbuild/linux-x64',
|
||||
'@esbuild/darwin-arm64',
|
||||
'@esbuild/win32-x64',
|
||||
'@esbuild/android-arm',
|
||||
];
|
||||
for (const name of archYes) {
|
||||
check(`match: ${name}`, isArchPackageName(name) === true && ARCH_SUFFIX_RE.test(name) === true);
|
||||
}
|
||||
|
||||
console.log('ARCH_SUFFIX_RE / isArchPackageName - parents / non-arch (must NOT match):');
|
||||
const archNo = [
|
||||
'sharp',
|
||||
'@napi-rs/canvas',
|
||||
'@parcel/watcher',
|
||||
'@github/copilot',
|
||||
'esbuild',
|
||||
'react',
|
||||
];
|
||||
for (const name of archNo) {
|
||||
check(`no-match: ${name}`, isArchPackageName(name) === false);
|
||||
}
|
||||
|
||||
// -- 1b. shipped-arch filter --------------------------------------------------
|
||||
console.log('isShippedArch - shipped platforms (must MATCH):');
|
||||
const shippedYes = [
|
||||
'@rollup/rollup-darwin-arm64',
|
||||
'@rollup/rollup-linux-x64-gnu',
|
||||
'@rollup/rollup-win32-x64-msvc',
|
||||
'@rollup/rollup-linux-arm-gnueabihf',
|
||||
'@esbuild/linux-x64',
|
||||
'@esbuild/darwin-arm64',
|
||||
'@esbuild/win32-arm64',
|
||||
'@img/sharp-linuxmusl-x64',
|
||||
'@parcel/watcher-linux-arm64-glibc',
|
||||
];
|
||||
for (const name of shippedYes) {
|
||||
check(`shipped: ${name}`, isShippedArch(name) === true);
|
||||
}
|
||||
|
||||
console.log('isShippedArch - non-shipped platforms (must NOT match):');
|
||||
const shippedNo = [
|
||||
'@esbuild/android-arm64',
|
||||
'@esbuild/freebsd-x64',
|
||||
'@esbuild/aix-ppc64',
|
||||
'@esbuild/sunos-x64',
|
||||
'@esbuild/linux-ia32',
|
||||
'@esbuild/linux-riscv64',
|
||||
'@esbuild/linux-s390x',
|
||||
'@esbuild/linux-ppc64',
|
||||
'@esbuild/openbsd-x64',
|
||||
'@rollup/rollup-android-arm64',
|
||||
'@napi-rs/canvas-linux-riscv64-gnu',
|
||||
'esbuild',
|
||||
'react',
|
||||
];
|
||||
for (const name of shippedNo) {
|
||||
check(`not-shipped: ${name}`, isShippedArch(name) === false);
|
||||
}
|
||||
|
||||
// -- 2. npm license shape parser ----------------------------------------------
|
||||
console.log('npmLicenseId - license field shapes:');
|
||||
check('string license', npmLicenseId({ license: 'MIT' }) === 'MIT');
|
||||
check('object {type}', npmLicenseId({ license: { type: 'MIT' } }) === 'MIT');
|
||||
check('legacy licenses:[{type}]', npmLicenseId({ licenses: [{ type: 'MIT' }] }) === 'MIT');
|
||||
check('legacy licenses:[string]', npmLicenseId({ licenses: ['MIT'] }) === 'MIT');
|
||||
check('compound string preserved', npmLicenseId({ license: 'Apache-2.0 AND LGPL-3.0-or-later' }) === 'Apache-2.0 AND LGPL-3.0-or-later');
|
||||
check('missing -> empty', npmLicenseId({}) === '');
|
||||
check('undefined -> empty', npmLicenseId(undefined) === '');
|
||||
check('object without type -> empty', npmLicenseId({ license: {} }) === '');
|
||||
|
||||
// -- 3. familyText reuse ------------------------------------------------------
|
||||
console.log('familyText - family-level text reuse:');
|
||||
const LICENSE_BODY = 'Apache License\nVersion 2.0\nCopyright (c) test';
|
||||
|
||||
// First arch of the family resolves real text from disk and caches it by repo.
|
||||
const cache = new Map<string, { text: string; source: string }>();
|
||||
const first = familyText(cache, 'https://github.com/lovell/sharp', LICENSE_BODY, 'disk', undefined);
|
||||
check('first arch: text from disk', first.text === LICENSE_BODY && first.source === 'disk');
|
||||
|
||||
// Sibling arch with NO own text reuses the cached family text (same repo URL).
|
||||
const second = familyText(cache, 'https://github.com/lovell/sharp', undefined, undefined, undefined);
|
||||
check('sibling arch: reuses cached text', second.text === LICENSE_BODY && second.source === 'disk');
|
||||
check('sibling arch: identical body to first', second.text === first.text);
|
||||
|
||||
// A repo-sourced text is also cached and reused.
|
||||
const cache2 = new Map<string, { text: string; source: string }>();
|
||||
const repoFirst = familyText(cache2, 'https://github.com/foo/bar', LICENSE_BODY, 'repo', 'PARENT TEXT');
|
||||
check('repo source wins over parent', repoFirst.source === 'repo' && repoFirst.text === LICENSE_BODY);
|
||||
const repoSibling = familyText(cache2, 'https://github.com/foo/bar', undefined, undefined, 'PARENT TEXT');
|
||||
check('sibling reuses repo text (not parent)', repoSibling.source === 'repo' && repoSibling.text === LICENSE_BODY);
|
||||
|
||||
// Parent fallback when this arch has no own text and nothing is cached yet.
|
||||
const cache3 = new Map<string, { text: string; source: string }>();
|
||||
const parentFallback = familyText(cache3, 'https://github.com/baz/qux', undefined, undefined, 'PARENT TEXT');
|
||||
check('parent fallback used', parentFallback.source === 'parent' && parentFallback.text === 'PARENT TEXT');
|
||||
|
||||
// Empty last resort when there is no own text and no parent text.
|
||||
const cache4 = new Map<string, { text: string; source: string }>();
|
||||
const none = familyText(cache4, 'https://github.com/empty/empty', undefined, undefined, undefined);
|
||||
check('empty last resort', none.source === 'none' && none.text === '');
|
||||
check('empty result not cached', cache4.has('https://github.com/empty/empty') === false);
|
||||
|
||||
// parentTextIfCompatible: parent text is reused only across a matching license id.
|
||||
// Same id (MIT parent -> MIT child) -> reuse.
|
||||
check('compatible: same id reuses parent text',
|
||||
parentTextIfCompatible('MIT', 'MIT', 'PARENT TEXT') === 'PARENT TEXT');
|
||||
// Case-insensitive id match still reuses.
|
||||
check('compatible: id match is case-insensitive',
|
||||
parentTextIfCompatible('Apache-2.0', 'apache-2.0', 'PARENT TEXT') === 'PARENT TEXT');
|
||||
// THE legacy bug guard: Apache parent text must NOT flow under an LGPL child id.
|
||||
check('incompatible: Apache parent text dropped under LGPL child',
|
||||
parentTextIfCompatible('Apache-2.0', 'LGPL-3.0-or-later', 'APACHE TEXT') === undefined);
|
||||
// Compound child id that is not an exact match is also rejected (sharp-win32).
|
||||
check('incompatible: compound child id rejects plain parent id',
|
||||
parentTextIfCompatible('Apache-2.0', 'Apache-2.0 AND LGPL-3.0-or-later', 'APACHE TEXT') === undefined);
|
||||
// Missing ids on either side -> cannot prove compatibility -> drop.
|
||||
check('incompatible: empty parent id dropped', parentTextIfCompatible('', 'MIT', 'PARENT TEXT') === undefined);
|
||||
check('incompatible: empty child id dropped', parentTextIfCompatible('MIT', '', 'PARENT TEXT') === undefined);
|
||||
// No parent text -> undefined regardless of ids.
|
||||
check('no parent text -> undefined', parentTextIfCompatible('MIT', 'MIT', '') === undefined);
|
||||
|
||||
// -- summary ------------------------------------------------------------------
|
||||
console.log('');
|
||||
console.log(`=== platform-binary unit checks: ${passed} passed, ${failed} failed ===`);
|
||||
process.exit(failed === 0 ? 0 : 1);
|
||||
@@ -0,0 +1,115 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Run the OSS NOTICE pipeline locally against this worktree to validate the
|
||||
cglicenses override-inject + presence-index behavior without a CI build.
|
||||
|
||||
.DESCRIPTION
|
||||
Reproduces the CI steps we can run locally:
|
||||
1. scan-licenses.ts -> ext-notices.txt + ext-notices.txt.presence.json
|
||||
2. merge-notices.ts -> ThirdPartyNotices.new.txt (loads the presence
|
||||
index automatically as the --extensions sibling)
|
||||
|
||||
The one CI input we cannot produce locally is the Component Governance
|
||||
NOTICE (notice@0). Pass a cached copy via -CgNotice to get a realistic
|
||||
base set; omit it to validate inject/stale logic on its own (the inject
|
||||
targets are absent from CG by definition, so they still exercise the path).
|
||||
|
||||
Outputs go to -OutDir (default C:\temp\oss-local) so both repos stay clean.
|
||||
|
||||
.EXAMPLE
|
||||
.\run-local.ps1
|
||||
Full run against the worktree, no CG base.
|
||||
|
||||
.EXAMPLE
|
||||
.\run-local.ps1 -CgNotice C:\temp\oss-local\generated.txt
|
||||
Full run with the cached CG NOTICE as the base set.
|
||||
|
||||
.EXAMPLE
|
||||
.\run-local.ps1 -SkipScan
|
||||
Re-run only the merge step (fast iteration on override/inject logic).
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$RepoRoot = (Resolve-Path "$PSScriptRoot\..\..\..").Path,
|
||||
[string]$OutDir = 'C:\temp\oss-local',
|
||||
[string]$CgNotice = '',
|
||||
[switch]$SkipScan
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$ossDir = Join-Path $RepoRoot 'build\azure-pipelines\oss'
|
||||
$cglicenses = Join-Path $RepoRoot 'cglicenses.json'
|
||||
$extNotices = Join-Path $OutDir 'ext-notices.txt'
|
||||
$presence = "$extNotices.presence.json"
|
||||
$newNotice = Join-Path $OutDir 'ThirdPartyNotices.new.txt'
|
||||
$runLog = Join-Path $OutDir 'run.log'
|
||||
|
||||
# --- sanity checks ---------------------------------------------------------
|
||||
if (-not (Test-Path $ossDir)) { throw "OSS scripts not found at $ossDir" }
|
||||
if (-not (Test-Path $cglicenses)) { throw "cglicenses.json not found at $cglicenses" }
|
||||
if (-not (Test-Path (Join-Path $RepoRoot 'node_modules'))) {
|
||||
Write-Warning "No node_modules at $RepoRoot - run 'npm i' there first or the presence index will be empty."
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path $OutDir | Out-Null
|
||||
|
||||
# Run a .ts pipeline script via tsx (handles ESM + .js import specifiers).
|
||||
function Invoke-OssScript {
|
||||
param([string]$Script, [string[]]$ScriptArgs)
|
||||
Push-Location $ossDir
|
||||
try {
|
||||
Write-Host "==> tsx $Script $($ScriptArgs -join ' ')" -ForegroundColor Cyan
|
||||
# npm/tsx write warnings to stderr; with ErrorActionPreference=Stop that
|
||||
# would become a terminating error. Scope it to Continue and rely on the
|
||||
# real exit code instead.
|
||||
$prevEAP = $ErrorActionPreference
|
||||
$ErrorActionPreference = 'Continue'
|
||||
& npx --yes tsx $Script @ScriptArgs 2>&1 | Tee-Object -FilePath $runLog -Append
|
||||
$code = $LASTEXITCODE
|
||||
$ErrorActionPreference = $prevEAP
|
||||
if ($code -ne 0) { throw "$Script exited with code $code" }
|
||||
}
|
||||
finally { Pop-Location }
|
||||
}
|
||||
|
||||
Remove-Item $runLog -ErrorAction SilentlyContinue
|
||||
|
||||
# --- 1. scanner ------------------------------------------------------------
|
||||
if (-not $SkipScan) {
|
||||
$scanArgs = @('--repo', $RepoRoot, '--output', $extNotices)
|
||||
# Pass CG's notice so Section 3 only fetches license text for cgmanifest
|
||||
# git components CG did NOT already cover (avoids redundant network calls).
|
||||
if ($CgNotice -and (Test-Path $CgNotice)) {
|
||||
$scanArgs += @('--cg', $CgNotice)
|
||||
}
|
||||
Invoke-OssScript -Script 'scan-licenses.ts' -ScriptArgs $scanArgs
|
||||
}
|
||||
else {
|
||||
Write-Host "Skipping scan (reusing $extNotices)" -ForegroundColor Yellow
|
||||
if (-not (Test-Path $presence)) { Write-Warning "No presence file at $presence - merge cannot detect stale overrides." }
|
||||
}
|
||||
|
||||
# --- 2. merge --------------------------------------------------------------
|
||||
$mergeArgs = @('--extensions', $extNotices, '--cglicenses', $cglicenses, '--output', $newNotice)
|
||||
if ($CgNotice -and (Test-Path $CgNotice)) {
|
||||
$mergeArgs += @('--cg', $CgNotice)
|
||||
Write-Host "Using CG base: $CgNotice" -ForegroundColor Green
|
||||
}
|
||||
else {
|
||||
Write-Warning "No CG cache (-CgNotice) - merging without a CG base. Inject/stale logic still validated."
|
||||
}
|
||||
$mergeArgs += '--provenance'
|
||||
Invoke-OssScript -Script 'merge-notices.ts' -ScriptArgs $mergeArgs
|
||||
|
||||
# --- 3. focused summary ----------------------------------------------------
|
||||
Write-Host "`n===== LOCAL RUN SUMMARY =====" -ForegroundColor Magenta
|
||||
$patterns = 'Presence index:|\+ injected:|\? stale:|\? unmatched:|overrides injected:|overrides stale|overrides unmatched|Total packages:'
|
||||
Select-String -Path $runLog -Pattern $patterns | ForEach-Object { Write-Host " $($_.Line.Trim())" }
|
||||
|
||||
if (Test-Path $presence) {
|
||||
$presenceCount = (Get-Content $presence -Raw | ConvertFrom-Json).Count
|
||||
Write-Host "`n Presence file: $presence ($presenceCount packages)"
|
||||
}
|
||||
Write-Host " NOTICE output: $newNotice"
|
||||
Write-Host " Full log: $runLog"
|
||||
Write-Host "=============================" -ForegroundColor Magenta
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,90 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Focused unit checks for Section 6 (os-gated whole-package enumeration) of scan-licenses.ts.
|
||||
*
|
||||
* Run: npx tsx section6.test.ts
|
||||
*
|
||||
* Covers (no network required):
|
||||
* 1. osAllows - npm `os` constraint semantics (positive, negation, mixed, empty).
|
||||
* 2. isOsGatedShippedElsewhere - the predicate that decides whether a package
|
||||
* is invisible on the build host yet ships on a platform we target (the
|
||||
* @vscode/windows-ca-certs case), keyed on the host platform.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import { osAllows, isOsGatedShippedElsewhere, NPM_SHIPPED_OS } from './scan-licenses.js';
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
function check(name: string, cond: boolean): void {
|
||||
if (cond) {
|
||||
passed++;
|
||||
console.log(` ok ${name}`);
|
||||
} else {
|
||||
failed++;
|
||||
console.error(` FAIL ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
// -- 1. osAllows --------------------------------------------------------------
|
||||
console.log('osAllows - positive lists:');
|
||||
check('["win32"] allows win32', osAllows(['win32'], 'win32') === true);
|
||||
check('["win32"] denies linux', osAllows(['win32'], 'linux') === false);
|
||||
check('["darwin","linux"] allows linux', osAllows(['darwin', 'linux'], 'linux') === true);
|
||||
check('["darwin","linux"] denies win32', osAllows(['darwin', 'linux'], 'win32') === false);
|
||||
|
||||
console.log('osAllows - negation lists:');
|
||||
check('["!win32"] denies win32', osAllows(['!win32'], 'win32') === false);
|
||||
check('["!win32"] allows linux', osAllows(['!win32'], 'linux') === true);
|
||||
check('["!linux"] denies linux', osAllows(['!linux'], 'linux') === false);
|
||||
check('["!linux"] allows win32', osAllows(['!linux'], 'win32') === true);
|
||||
|
||||
console.log('osAllows - edge cases:');
|
||||
check('[] allows anything (no constraint)', osAllows([], 'linux') === true);
|
||||
check('negation wins over positive', osAllows(['linux', '!linux'], 'linux') === false);
|
||||
check('ignores empty/non-string entries', osAllows(['', 'win32'], 'win32') === true);
|
||||
|
||||
// -- 2. isOsGatedShippedElsewhere (host = linux, the NOTICE agent) ------------
|
||||
console.log('isOsGatedShippedElsewhere - host = linux (the NOTICE agent):');
|
||||
// The motivating case: @vscode/windows-ca-certs is os:["win32"].
|
||||
check('os:["win32"] on linux -> TRUE (ships win32, absent on linux)',
|
||||
isOsGatedShippedElsewhere(['win32'], 'linux') === true);
|
||||
check('os:["darwin"] on linux -> TRUE (ships darwin, absent on linux)',
|
||||
isOsGatedShippedElsewhere(['darwin'], 'linux') === true);
|
||||
check('os:["win32","darwin"] on linux -> TRUE',
|
||||
isOsGatedShippedElsewhere(['win32', 'darwin'], 'linux') === true);
|
||||
check('os:["linux"] on linux -> FALSE (installs on host)',
|
||||
isOsGatedShippedElsewhere(['linux'], 'linux') === false);
|
||||
check('os:["win32","linux"] on linux -> FALSE (installs on host)',
|
||||
isOsGatedShippedElsewhere(['win32', 'linux'], 'linux') === false);
|
||||
check('os:["!win32"] on linux -> FALSE (installs on host)',
|
||||
isOsGatedShippedElsewhere(['!win32'], 'linux') === false);
|
||||
check('os:["!linux"] on linux -> TRUE (absent on linux, ships win32/darwin)',
|
||||
isOsGatedShippedElsewhere(['!linux'], 'linux') === true);
|
||||
check('os:["android"] on linux -> FALSE (we do not ship android)',
|
||||
isOsGatedShippedElsewhere(['android'], 'linux') === false);
|
||||
check('os:["freebsd"] on linux -> FALSE (we do not ship freebsd)',
|
||||
isOsGatedShippedElsewhere(['freebsd'], 'linux') === false);
|
||||
check('empty os -> FALSE', isOsGatedShippedElsewhere([], 'linux') === false);
|
||||
check('non-array os -> FALSE', isOsGatedShippedElsewhere(undefined as unknown as string[], 'linux') === false);
|
||||
|
||||
console.log('isOsGatedShippedElsewhere - host = win32 (sanity, other agent):');
|
||||
check('os:["win32"] on win32 -> FALSE (installs on host)',
|
||||
isOsGatedShippedElsewhere(['win32'], 'win32') === false);
|
||||
check('os:["linux"] on win32 -> TRUE (absent on win32, ships linux)',
|
||||
isOsGatedShippedElsewhere(['linux'], 'win32') === true);
|
||||
|
||||
console.log('NPM_SHIPPED_OS sanity:');
|
||||
check('contains darwin/linux/win32', NPM_SHIPPED_OS.has('darwin') && NPM_SHIPPED_OS.has('linux') && NPM_SHIPPED_OS.has('win32'));
|
||||
check('does not contain android', NPM_SHIPPED_OS.has('android') === false);
|
||||
|
||||
// -- summary ------------------------------------------------------------------
|
||||
console.log('');
|
||||
console.log(`Section 6 unit checks: ${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Focused unit checks for Section 7 (pre-built built-in extension dependency
|
||||
* enumeration) of scan-licenses.ts.
|
||||
*
|
||||
* Run: npx tsx section7.test.ts
|
||||
*
|
||||
* Covers (no network required):
|
||||
* 1. readBuiltInExtensionNames - extracting extension names from product.json
|
||||
* (builtInExtensions + webBuiltInExtensions, missing keys, non-string names).
|
||||
* 2. enumerateLockfileProdDeps - walking a lockfileVersion 2/3 `packages` map
|
||||
* (skips root/dev/link/arch entries, dedups, last-node_modules name, and
|
||||
* returns [] for lockfileVersion 1).
|
||||
* 3. readBuiltInExtensionManifest - extracting name+version+repo (for self-fetch).
|
||||
* 4. builtInExtensionLockfileUrl - building the public raw.githubusercontent.com
|
||||
* package-lock.json URL (no token), rejecting non-GitHub/missing inputs.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
import { readBuiltInExtensionNames, enumerateLockfileProdDeps, readBuiltInExtensionManifest, builtInExtensionLockfileUrl } from './scan-licenses.js';
|
||||
|
||||
let passed = 0;
|
||||
let failed = 0;
|
||||
|
||||
function check(name: string, cond: boolean): void {
|
||||
if (cond) {
|
||||
passed++;
|
||||
console.log(` ok ${name}`);
|
||||
} else {
|
||||
failed++;
|
||||
console.error(` FAIL ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
// -- 1. readBuiltInExtensionNames ---------------------------------------------
|
||||
console.log('readBuiltInExtensionNames - product.json shapes:');
|
||||
|
||||
const realShape = {
|
||||
builtInExtensions: [
|
||||
{ name: 'ms-vscode.js-debug', version: '1.117.0' },
|
||||
{ name: 'ms-vscode.js-debug-companion', version: '1.1.3' },
|
||||
{ name: 'ms-vscode.vscode-js-profile-table', version: '1.0.10' },
|
||||
],
|
||||
};
|
||||
const realNames = readBuiltInExtensionNames(realShape);
|
||||
check('reads the 3 product.json built-in extensions', realNames.length === 3);
|
||||
check('includes js-debug', realNames.includes('ms-vscode.js-debug'));
|
||||
check('includes js-debug-companion', realNames.includes('ms-vscode.js-debug-companion'));
|
||||
check('includes js-profile-table', realNames.includes('ms-vscode.vscode-js-profile-table'));
|
||||
|
||||
check('merges webBuiltInExtensions', readBuiltInExtensionNames({
|
||||
builtInExtensions: [{ name: 'a' }],
|
||||
webBuiltInExtensions: [{ name: 'b' }, { name: 'c' }],
|
||||
}).length === 3);
|
||||
|
||||
check('missing builtInExtensions key -> []', readBuiltInExtensionNames({}).length === 0);
|
||||
check('undefined product json -> []', readBuiltInExtensionNames(undefined).length === 0);
|
||||
check('null product json -> []', readBuiltInExtensionNames(null).length === 0);
|
||||
check('non-array builtInExtensions -> []', readBuiltInExtensionNames({ builtInExtensions: 'nope' }).length === 0);
|
||||
check('skips entries without a string name', readBuiltInExtensionNames({
|
||||
builtInExtensions: [{ name: 'good' }, { version: '1.0.0' }, { name: 42 }, { name: '' }, null],
|
||||
}).join(',') === 'good');
|
||||
|
||||
// -- 2. enumerateLockfileProdDeps ---------------------------------------------
|
||||
console.log('enumerateLockfileProdDeps - lockfileVersion 2/3 packages map:');
|
||||
|
||||
const lockV3 = {
|
||||
lockfileVersion: 3,
|
||||
packages: {
|
||||
'': { name: 'js-debug', version: '1.117.0' }, // root - skip
|
||||
'node_modules/acorn-loose': { version: '8.4.0' },
|
||||
'node_modules/astring': { version: '1.8.6' },
|
||||
'node_modules/preact': { version: '10.20.0' },
|
||||
'node_modules/typescript': { version: '5.4.0', dev: true }, // dev - skip
|
||||
'node_modules/@vscode/l10n': { version: '0.0.18' },
|
||||
'node_modules/@c4312/chromehash': { version: '0.3.0' },
|
||||
// nested: name is taken after the LAST node_modules/
|
||||
'node_modules/execa/node_modules/signal-exit': { version: '3.0.7' },
|
||||
// workspace link - skip
|
||||
'node_modules/local-pkg': { version: '1.0.0', link: true },
|
||||
},
|
||||
};
|
||||
const deps = enumerateLockfileProdDeps(lockV3);
|
||||
const depNames = deps.map(d => d.name).sort();
|
||||
|
||||
check('skips the root entry', !depNames.includes('js-debug'));
|
||||
check('skips dev deps', !depNames.includes('typescript'));
|
||||
check('skips link entries', !depNames.includes('local-pkg'));
|
||||
check('keeps acorn-loose', depNames.includes('acorn-loose'));
|
||||
check('keeps astring', depNames.includes('astring'));
|
||||
check('keeps preact', depNames.includes('preact'));
|
||||
check('keeps scoped @vscode/l10n', depNames.includes('@vscode/l10n'));
|
||||
check('keeps scoped @c4312/chromehash', depNames.includes('@c4312/chromehash'));
|
||||
check('takes name after LAST node_modules/ (nested -> signal-exit)', depNames.includes('signal-exit'));
|
||||
check('does not mistake nested key for execa', !depNames.includes('execa'));
|
||||
check('captures versions', deps.find(d => d.name === 'preact')?.version === '10.20.0');
|
||||
|
||||
console.log('enumerateLockfileProdDeps - dedup + edge cases:');
|
||||
const lockDupe = {
|
||||
lockfileVersion: 2,
|
||||
packages: {
|
||||
'node_modules/color': { version: '4.2.3' },
|
||||
'node_modules/foo/node_modules/color': { version: '3.2.1' }, // same name, different tree
|
||||
},
|
||||
};
|
||||
check('dedups same dependency name', enumerateLockfileProdDeps(lockDupe).filter(d => d.name === 'color').length === 1);
|
||||
|
||||
check('lockfileVersion 1 (no packages map) -> []', enumerateLockfileProdDeps({
|
||||
lockfileVersion: 1,
|
||||
dependencies: { 'acorn-loose': { version: '8.4.0' } },
|
||||
}).length === 0);
|
||||
check('empty packages map -> []', enumerateLockfileProdDeps({ packages: {} }).length === 0);
|
||||
check('undefined lock -> []', enumerateLockfileProdDeps(undefined).length === 0);
|
||||
check('null lock -> []', enumerateLockfileProdDeps(null).length === 0);
|
||||
|
||||
console.log('enumerateLockfileProdDeps - arch packages owned by Section 5:');
|
||||
const lockArch = {
|
||||
lockfileVersion: 3,
|
||||
packages: {
|
||||
'node_modules/esbuild': { version: '0.21.0' }, // parent - keep
|
||||
'node_modules/@esbuild/linux-x64': { version: '0.21.0' }, // arch - skip
|
||||
'node_modules/@esbuild/darwin-arm64': { version: '0.21.0' },// arch - skip
|
||||
},
|
||||
};
|
||||
const archDeps = enumerateLockfileProdDeps(lockArch).map(d => d.name);
|
||||
check('keeps arch-independent parent esbuild', archDeps.includes('esbuild'));
|
||||
check('skips @esbuild/linux-x64 (Section 5 owns it)', !archDeps.includes('@esbuild/linux-x64'));
|
||||
check('skips @esbuild/darwin-arm64 (Section 5 owns it)', !archDeps.includes('@esbuild/darwin-arm64'));
|
||||
|
||||
// -- 3. readBuiltInExtensionManifest ------------------------------------------
|
||||
console.log('readBuiltInExtensionManifest - name + version + repo:');
|
||||
|
||||
const manifestShape = {
|
||||
builtInExtensions: [
|
||||
{ name: 'ms-vscode.js-debug', version: '1.117.0', repo: 'https://github.com/microsoft/vscode-js-debug' },
|
||||
{ name: 'ms-vscode.js-debug-companion', version: '1.1.3', repo: 'https://github.com/microsoft/vscode-js-debug-companion' },
|
||||
],
|
||||
webBuiltInExtensions: [
|
||||
{ name: 'web-ext', version: '2.0.0', repo: 'https://github.com/microsoft/web-ext' },
|
||||
],
|
||||
};
|
||||
const manifest = readBuiltInExtensionManifest(manifestShape);
|
||||
check('manifest merges builtIn + web (3 entries)', manifest.length === 3);
|
||||
check('manifest captures version', manifest.find(e => e.name === 'ms-vscode.js-debug')?.version === '1.117.0');
|
||||
check('manifest captures repo', manifest.find(e => e.name === 'ms-vscode.js-debug')?.repo === 'https://github.com/microsoft/vscode-js-debug');
|
||||
check('manifest undefined product json -> []', readBuiltInExtensionManifest(undefined).length === 0);
|
||||
check('manifest null product json -> []', readBuiltInExtensionManifest(null).length === 0);
|
||||
check('manifest missing key -> []', readBuiltInExtensionManifest({}).length === 0);
|
||||
check('manifest skips nameless entries', readBuiltInExtensionManifest({
|
||||
builtInExtensions: [{ name: 'ok', version: '1.0.0', repo: 'r' }, { version: '2.0.0' }, { name: 42 }, null],
|
||||
}).length === 1);
|
||||
check('manifest defaults missing version/repo to empty string', (() => {
|
||||
const m = readBuiltInExtensionManifest({ builtInExtensions: [{ name: 'x' }] });
|
||||
return m.length === 1 && m[0].version === '' && m[0].repo === '';
|
||||
})());
|
||||
|
||||
// -- 4. builtInExtensionLockfileUrl -------------------------------------------
|
||||
console.log('builtInExtensionLockfileUrl - public raw URL (no token):');
|
||||
|
||||
check('builds the js-debug raw URL at the version tag',
|
||||
builtInExtensionLockfileUrl('https://github.com/microsoft/vscode-js-debug', '1.117.0') ===
|
||||
'https://raw.githubusercontent.com/microsoft/vscode-js-debug/v1.117.0/package-lock.json');
|
||||
check('handles a trailing .git suffix',
|
||||
builtInExtensionLockfileUrl('https://github.com/microsoft/vscode-js-debug.git', '1.0.0') ===
|
||||
'https://raw.githubusercontent.com/microsoft/vscode-js-debug/v1.0.0/package-lock.json');
|
||||
check('never embeds a token in the URL', (() => {
|
||||
const u = builtInExtensionLockfileUrl('https://github.com/microsoft/vscode-js-debug', '1.117.0') || '';
|
||||
return !u.includes('@');
|
||||
})());
|
||||
check('missing version -> undefined', builtInExtensionLockfileUrl('https://github.com/microsoft/vscode-js-debug', '') === undefined);
|
||||
check('missing repo -> undefined', builtInExtensionLockfileUrl('', '1.0.0') === undefined);
|
||||
check('non-GitHub host -> undefined', builtInExtensionLockfileUrl('https://gitlab.com/foo/bar', '1.0.0') === undefined);
|
||||
|
||||
// -- summary ------------------------------------------------------------------
|
||||
console.log('');
|
||||
console.log(`Section 7 unit checks: ${passed} passed, ${failed} failed`);
|
||||
if (failed > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
/**
|
||||
* Parse CLI arguments of the form `--key value` into a Record.
|
||||
* If the next token after a `--key` is another `--`-prefixed flag (or there
|
||||
* is no next token), the key is treated as a boolean flag with value `'true'`.
|
||||
*/
|
||||
export function parseArgs(argv: string[]): Record<string, string> {
|
||||
const args: Record<string, string> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
if (argv[i].startsWith('--')) {
|
||||
const key = argv[i].substring(2);
|
||||
if (i + 1 < argv.length && !argv[i + 1].startsWith('--')) {
|
||||
args[key] = argv[i + 1];
|
||||
i++;
|
||||
} else {
|
||||
args[key] = 'true';
|
||||
}
|
||||
}
|
||||
}
|
||||
return args;
|
||||
}
|
||||
@@ -164,7 +164,7 @@ variables:
|
||||
- name: VSCODE_MIXIN_REPO
|
||||
value: microsoft/vscode-distro
|
||||
- name: skipComponentGovernanceDetection
|
||||
value: true
|
||||
value: false
|
||||
- name: ComponentDetection.Timeout
|
||||
value: 600
|
||||
- name: Codeql.SkipTaskAutoInjection
|
||||
|
||||
@@ -135,10 +135,8 @@ jobs:
|
||||
displayName: Compile & Hygiene
|
||||
|
||||
- script: npm run download-builtin-extensions-cg
|
||||
env:
|
||||
GITHUB_TOKEN: "$(github-distro-mixin-password)"
|
||||
displayName: Download component details of built-in extensions
|
||||
condition: and(succeeded(), eq(lower(variables['VSCODE_PUBLISH']), 'true'))
|
||||
continueOnError: true
|
||||
|
||||
- task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
|
||||
displayName: "Component Detection"
|
||||
@@ -146,7 +144,170 @@ jobs:
|
||||
sourceScanPath: $(Build.SourcesDirectory)
|
||||
alertWarningLevel: Medium
|
||||
continueOnError: true
|
||||
condition: and(succeeded(), eq(lower(variables['VSCODE_PUBLISH']), 'true'))
|
||||
|
||||
- task: notice@0
|
||||
displayName: "Generate ThirdPartyNotices (CG)"
|
||||
inputs:
|
||||
outputfile: $(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt
|
||||
continueOnError: true
|
||||
|
||||
# =========================================================================
|
||||
# Cached-NOTICE fallback
|
||||
#
|
||||
# Goal: never ship without *some* CG NOTICE. If `notice@0` above fails or
|
||||
# produces an empty file (CG service down, timeout, etc.), substitute the
|
||||
# most recent successfully-generated `ThirdPartyNotices.generated.txt`
|
||||
# from a prior build.
|
||||
#
|
||||
# Fallback order:
|
||||
# 1. Most recent successful run on the CURRENT branch
|
||||
# 2. Most recent successful run on `main`
|
||||
#
|
||||
# TODO: consider enforcing a max-age check (~30 days) on cached NOTICEs.
|
||||
# Today the only signal is the warning emitted below.
|
||||
#
|
||||
# Architecture note: the fallback ONLY substitutes the CG-supplied portion.
|
||||
# The local steps (scan-licenses, merge-notices, apply-overrides) ALWAYS
|
||||
# run fresh against current HEAD, so extension licenses and
|
||||
# cglicenses.json overrides are always current-PR-accurate.
|
||||
# =========================================================================
|
||||
- task: DownloadPipelineArtifact@2
|
||||
displayName: "Cache: download last good NOTICE (same branch)"
|
||||
inputs:
|
||||
buildType: specific
|
||||
project: $(System.TeamProject)
|
||||
definition: $(System.DefinitionId)
|
||||
buildVersionToDownload: latestFromBranch
|
||||
branchName: $(Build.SourceBranch)
|
||||
allowPartiallySucceededBuilds: true
|
||||
artifactName: notice_output
|
||||
itemPattern: "**/{ThirdPartyNotices.generated.txt,notice-meta.txt}"
|
||||
targetPath: $(Pipeline.Workspace)/cached-notice-branch
|
||||
continueOnError: true
|
||||
|
||||
- task: DownloadPipelineArtifact@2
|
||||
displayName: "Cache: download last good NOTICE (main fallback)"
|
||||
inputs:
|
||||
buildType: specific
|
||||
project: $(System.TeamProject)
|
||||
definition: $(System.DefinitionId)
|
||||
buildVersionToDownload: latestFromBranch
|
||||
branchName: refs/heads/main
|
||||
allowPartiallySucceededBuilds: true
|
||||
artifactName: notice_output
|
||||
itemPattern: "**/{ThirdPartyNotices.generated.txt,notice-meta.txt}"
|
||||
targetPath: $(Pipeline.Workspace)/cached-notice-main
|
||||
continueOnError: true
|
||||
condition: ne(variables['Build.SourceBranch'], 'refs/heads/main')
|
||||
|
||||
- script: |
|
||||
set -u
|
||||
GEN="$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt"
|
||||
BRANCH_CACHE="$(Pipeline.Workspace)/cached-notice-branch/ThirdPartyNotices.generated.txt"
|
||||
BRANCH_META="$(Pipeline.Workspace)/cached-notice-branch/notice-meta.txt"
|
||||
MAIN_CACHE="$(Pipeline.Workspace)/cached-notice-main/ThirdPartyNotices.generated.txt"
|
||||
MAIN_META="$(Pipeline.Workspace)/cached-notice-main/notice-meta.txt"
|
||||
|
||||
# Emit what a cached NOTICE was built from, if we have the sidecar.
|
||||
# The sidecar is written by the upload step on every successful run.
|
||||
describe_cache() {
|
||||
local meta="$1"
|
||||
local label="$2"
|
||||
if [ -f "$meta" ]; then
|
||||
echo "##[warning]Cache source ($label):"
|
||||
while IFS= read -r line; do echo "##[warning] $line"; done < "$meta"
|
||||
else
|
||||
echo "##[warning]Cache source ($label): no notice-meta.txt sidecar (built before this metadata was added)"
|
||||
fi
|
||||
}
|
||||
|
||||
# `notice@0` succeeded if the file exists and is non-trivial (>1KB).
|
||||
if [ -f "$GEN" ] && [ "$(wc -c < "$GEN")" -gt 1024 ]; then
|
||||
echo "##[section]CG NOTICE generation succeeded; no fallback needed."
|
||||
echo "##vso[task.setvariable variable=NOTICE_FROM_CACHE]false"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "##[warning]CG NOTICE generation failed or produced empty output."
|
||||
|
||||
if [ -f "$BRANCH_CACHE" ] && [ "$(wc -c < "$BRANCH_CACHE")" -gt 1024 ]; then
|
||||
echo "##[warning]Falling back to cached NOTICE from same branch ($(Build.SourceBranch))."
|
||||
describe_cache "$BRANCH_META" "same-branch"
|
||||
cp "$BRANCH_CACHE" "$GEN"
|
||||
echo "##vso[task.setvariable variable=NOTICE_FROM_CACHE]branch"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -f "$MAIN_CACHE" ] && [ "$(wc -c < "$MAIN_CACHE")" -gt 1024 ]; then
|
||||
echo "##[warning]Falling back to cached NOTICE from main."
|
||||
echo "##[warning]Branch $(Build.SourceBranch) may have dependencies that differ from main; review the audit log."
|
||||
describe_cache "$MAIN_META" "main"
|
||||
cp "$MAIN_CACHE" "$GEN"
|
||||
echo "##vso[task.setvariable variable=NOTICE_FROM_CACHE]main"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "##[warning]No cached NOTICE available from either source. Merge step will proceed with empty CG input; final NOTICE will rely entirely on the local scanner + cglicenses.json overrides."
|
||||
echo "##vso[task.setvariable variable=NOTICE_FROM_CACHE]none"
|
||||
displayName: "Cache: apply NOTICE fallback if CG failed"
|
||||
continueOnError: true
|
||||
|
||||
- script: |
|
||||
echo "=== CG NOTICE artifact ==="
|
||||
if [ -f "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" ]; then
|
||||
SIZE=$(wc -c < "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt")
|
||||
echo "Generated file size: $SIZE bytes (NOTICE_FROM_CACHE=$NOTICE_FROM_CACHE)"
|
||||
cp "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.generated.txt"
|
||||
echo "##vso[artifact.upload containerfolder=notice_output;artifactname=notice_output]$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.generated.txt"
|
||||
|
||||
# Write a sidecar so future runs (and humans) know what this NOTICE
|
||||
# was built from. Consumed by the "Cache: apply NOTICE fallback" step
|
||||
# of subsequent builds when they fall back to this artifact.
|
||||
# Only meaningful when this run generated the NOTICE fresh; if we
|
||||
# fell back to a cache, skip writing the sidecar so we don't
|
||||
# mis-attribute someone else's NOTICE to this commit.
|
||||
if [ "${NOTICE_FROM_CACHE:-false}" = "false" ]; then
|
||||
{
|
||||
echo "built_at: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "source_branch: $(Build.SourceBranch)"
|
||||
echo "source_commit: $(Build.SourceVersion)"
|
||||
echo "build_id: $(Build.BuildId)"
|
||||
echo "build_number: $(Build.BuildNumber)"
|
||||
echo "size_bytes: $SIZE"
|
||||
} > "$(Build.ArtifactStagingDirectory)/notice-meta.txt"
|
||||
echo "##vso[artifact.upload containerfolder=notice_output;artifactname=notice_output]$(Build.ArtifactStagingDirectory)/notice-meta.txt"
|
||||
else
|
||||
echo "Skipping notice-meta.txt sidecar: NOTICE_FROM_CACHE=$NOTICE_FROM_CACHE (don't re-publish someone else's metadata as our own)"
|
||||
fi
|
||||
else
|
||||
echo "ERROR: ThirdPartyNotices.generated.txt not found (CG failed AND no cache available)"
|
||||
fi
|
||||
displayName: "Upload CG NOTICE artifact"
|
||||
continueOnError: true
|
||||
|
||||
- script: |
|
||||
set -e
|
||||
npx tsc --skipLibCheck --module nodenext --moduleResolution nodenext --esModuleInterop --target es2022 --types node --outDir .oss-build-out build/azure-pipelines/oss/apply-overrides.ts build/azure-pipelines/oss/scan-licenses.ts build/azure-pipelines/oss/parse-notices.ts build/azure-pipelines/oss/merge-notices.ts
|
||||
node .oss-build-out/scan-licenses.js \
|
||||
--repo "$(Build.SourcesDirectory)" \
|
||||
--cg "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" \
|
||||
--output "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt"
|
||||
displayName: "Scan extension-bundled licenses"
|
||||
continueOnError: true
|
||||
|
||||
- script: |
|
||||
node .oss-build-out/merge-notices.js \
|
||||
--cg "$(Build.SourcesDirectory)/ThirdPartyNotices.generated.txt" \
|
||||
--extensions "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt" \
|
||||
--cglicenses "$(Build.SourcesDirectory)/cglicenses.json" \
|
||||
--output "$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.new.txt" \
|
||||
--provenance
|
||||
|
||||
# Upload all artifacts
|
||||
echo "##vso[artifact.upload containerfolder=notice_output;artifactname=notice_output]$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.extensions.txt"
|
||||
echo "##vso[artifact.upload containerfolder=notice_output;artifactname=notice_output]$(Build.ArtifactStagingDirectory)/ThirdPartyNotices.new.txt"
|
||||
displayName: "Merge CG + extension notices into .new.txt"
|
||||
continueOnError: true
|
||||
|
||||
- task: AzureCLI@2
|
||||
displayName: Fetch secrets
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import url from 'url';
|
||||
import ansiColors from 'ansi-colors';
|
||||
import type { IExtensionDefinition } from './builtInExtensions.ts';
|
||||
|
||||
@@ -14,7 +13,6 @@ const rootCG = path.join(root, 'extensionsCG');
|
||||
const productjson = JSON.parse(fs.readFileSync(path.join(import.meta.dirname, '../../product.json'), 'utf8'));
|
||||
const builtInExtensions = productjson.builtInExtensions as IExtensionDefinition[] || [];
|
||||
const webBuiltInExtensions = productjson.webBuiltInExtensions as IExtensionDefinition[] || [];
|
||||
const token = process.env['GITHUB_TOKEN'];
|
||||
|
||||
const contentBasePath = 'raw.githubusercontent.com';
|
||||
const contentFileNames = ['package.json', 'package-lock.json'];
|
||||
@@ -27,8 +25,15 @@ async function downloadExtensionDetails(extension: IExtensionDefinition): Promis
|
||||
return;
|
||||
}
|
||||
|
||||
const repository = url.parse(extension.repo).path!.substr(1);
|
||||
const repositoryContentBaseUrl = `https://${token ? `${token}@` : ''}${contentBasePath}/${repository}/v${extension.version}`;
|
||||
const repository = new URL(extension.repo).pathname.slice(1);
|
||||
// Do NOT embed a token in the URL userinfo (`https://<token>@host/...`):
|
||||
// Node's native fetch() throws "Request cannot be constructed from a URL that
|
||||
// includes credentials" on such URLs, which silently emptied extensionsCG/ in
|
||||
// CI for ~2y (the throw was swallowed by getContent's try/catch). The built-in
|
||||
// extension repos are public, so the request is unauthenticated — that also
|
||||
// avoids the failure mode where an invalid token makes raw.githubusercontent
|
||||
// return 404 for content that is otherwise reachable.
|
||||
const repositoryContentBaseUrl = `https://${contentBasePath}/${repository}/v${extension.version}`;
|
||||
|
||||
async function getContent(fileName: string): Promise<{ fileName: string; body: Buffer | undefined | null }> {
|
||||
try {
|
||||
@@ -64,10 +69,10 @@ async function downloadExtensionDetails(extension: IExtensionDefinition): Promis
|
||||
|
||||
// Validation
|
||||
if (!results.find(r => r.fileName === 'package.json')?.body) {
|
||||
// throw new Error(`The "package.json" file could not be found for the built-in extension - ${extensionLabel}`);
|
||||
console.warn(`WARN: The "package.json" file could not be found for the built-in extension - ${extensionLabel}`);
|
||||
}
|
||||
if (!results.find(r => r.fileName === 'package-lock.json')?.body) {
|
||||
// throw new Error(`The "package-lock.json" could not be found for the built-in extension - ${extensionLabel}`);
|
||||
console.warn(`WARN: The "package-lock.json" could not be found for the built-in extension - ${extensionLabel}`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+102
-93
@@ -281,26 +281,6 @@
|
||||
"CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
|
||||
]
|
||||
},
|
||||
{
|
||||
// Reason: repo URI is wrong on crate, pending https://github.com/warp-tech/russh/pull/53
|
||||
"name": "russh-cryptovec",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/warp-tech/russh/1da80d0d599b6ee2d257c544c0d6af4f649c9029/LICENSE-2.0.txt"
|
||||
},
|
||||
{
|
||||
// Reason: repo URI is wrong on crate, pending https://github.com/warp-tech/russh/pull/53
|
||||
"name": "russh-keys",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/warp-tech/russh/1da80d0d599b6ee2d257c544c0d6af4f649c9029/LICENSE-2.0.txt"
|
||||
},
|
||||
{
|
||||
// Reason: license is in a subdirectory in repo
|
||||
"name": "openssl",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/sfackler/rust-openssl/e43eb58540b27a17f8029c397e3edc12bbc9011f/openssl/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: license is in a subdirectory in repo
|
||||
"name": "openssl-sys",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/sfackler/rust-openssl/e43eb58540b27a17f8029c397e3edc12bbc9011f/openssl-sys/LICENSE-MIT"
|
||||
},
|
||||
{
|
||||
// Reason: Missing license file
|
||||
"name": "openssl-macros",
|
||||
@@ -325,26 +305,6 @@
|
||||
"CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
|
||||
]
|
||||
},
|
||||
{ // Reason: Missing license file
|
||||
"name": "const_format_proc_macros",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/rodrimati1992/const_format_crates/b2207af46bfbd9f1a6bd12dbffd10feeea3d9fd7/LICENSE-ZLIB.md"
|
||||
},
|
||||
{ // Reason: Missing license file
|
||||
"name": "const_format",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/rodrimati1992/const_format_crates/b2207af46bfbd9f1a6bd12dbffd10feeea3d9fd7/LICENSE-ZLIB.md"
|
||||
},
|
||||
{ // License is MIT/Apache and tool doesn't look in subfolders
|
||||
"name": "toml_edit",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/toml-rs/toml/main/crates/toml_edit/LICENSE-MIT"
|
||||
},
|
||||
{ // License is MIT/Apache and tool doesn't look in subfolders
|
||||
"name": "toml_datetime",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/toml-rs/toml/main/crates/toml_datetime/LICENSE-MIT"
|
||||
},
|
||||
{ // License is MIT/Apache and gitlab API doesn't find the project
|
||||
"name": "libredox",
|
||||
"fullLicenseTextUri": "https://gitlab.redox-os.org/redox-os/libredox/-/raw/master/LICENSE"
|
||||
},
|
||||
{
|
||||
"name": "https-proxy-agent",
|
||||
"fullLicenseText": [
|
||||
@@ -591,16 +551,6 @@
|
||||
"SOFTWARE"
|
||||
]
|
||||
},
|
||||
{
|
||||
// Reason: mono-repo where the individual packages are also dual-licensed under MIT and Apache-2.0
|
||||
"name": "system-configuration",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/mullvad/system-configuration-rs/v0.6.0/system-configuration/LICENSE-MIT"
|
||||
},
|
||||
{
|
||||
// Reason: mono-repo where the individual packages are also dual-licensed under MIT and Apache-2.0
|
||||
"name": "system-configuration-sys",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/mullvad/system-configuration-rs/v0.6.0/system-configuration-sys/LICENSE-MIT"
|
||||
},
|
||||
{
|
||||
// Reason: License missing from the repository https://github.com/isaacs/chownr/issues/35
|
||||
"name": "chownr",
|
||||
@@ -645,10 +595,6 @@
|
||||
"SOFTWARE"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "gethostname",
|
||||
"fullLicenseTextUri": "https://codeberg.org/swsnr/gethostname.rs/raw/commit/d1a7e1162c20106a1df2cdbba9eb2d5174037b3c/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: Unlicense. Does not include a clear Copyright statement
|
||||
"name": "robust-predicates",
|
||||
@@ -721,21 +667,6 @@
|
||||
"name": "wasm-encoder",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/bytecodealliance/wasm-tools/76927bf4bdbddf4b15f835c5eddfffbdfe3bdbd5/LICENSE-APACHE"
|
||||
},
|
||||
{
|
||||
// Reason: mono-repo
|
||||
"name": "@jridgewell/gen-mapping",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/jridgewell/sourcemaps/refs/heads/main/packages/gen-mapping/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: mono-repo
|
||||
"name": "@jridgewell/sourcemap-codec",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/jridgewell/sourcemaps/refs/heads/main/packages/sourcemap-codec/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: mono-repo
|
||||
"name": "@jridgewell/trace-mapping",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/jridgewell/sourcemaps/refs/heads/main/packages/trace-mapping/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: License text from https://github.com/github/copilot-cli/blob/master/LICENSE.md
|
||||
// does not include a copyright statement.
|
||||
@@ -952,10 +883,6 @@
|
||||
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "gcp-metadata",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/googleapis/google-cloud-node-core/76ba85a7f55c6e82943008b4eceb07a0f58b39e1/LICENSE"
|
||||
},
|
||||
{
|
||||
"name": "randombytes",
|
||||
"licenseDetail": [
|
||||
@@ -1074,26 +1001,6 @@
|
||||
"Copyright (c) Pictogrammers (https://pictogrammers.com/)"
|
||||
]
|
||||
},
|
||||
{
|
||||
// Reason: npm repositoryUrl points to a subdirectory; license is at the root of adobe/react-spectrum.
|
||||
"name": "@internationalized/date",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/adobe/react-spectrum/9508b15bf8c0e968c56220548207cc57c7e4f57c/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: npm repositoryUrl points to a subdirectory; license is at the root of nodelib/nodelib.
|
||||
"name": "@nodelib/fs.stat",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/nodelib/nodelib/ba122bd81aadd0a4fa7340ef2cfaabf62984bf1d/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: npm repositoryUrl points to a subdirectory; license is at the root of nodelib/nodelib.
|
||||
"name": "@nodelib/fs.walk",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/nodelib/nodelib/ba122bd81aadd0a4fa7340ef2cfaabf62984bf1d/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: npm repositoryUrl points to a subdirectory; license is at the root of nodelib/nodelib.
|
||||
"name": "@nodelib/fs.scandir",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/nodelib/nodelib/ba122bd81aadd0a4fa7340ef2cfaabf62984bf1d/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: LICENSE.md at github/copilot-cli does not contain a copyright statement.
|
||||
// Copyright holder sourced from the GitHub organization (github = GitHub, Inc.).
|
||||
@@ -1183,5 +1090,107 @@
|
||||
"",
|
||||
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
|
||||
]
|
||||
},
|
||||
{
|
||||
// Reason: @vscode/l10n does not include a LICENSE file in its npm tarball.
|
||||
// Source: https://github.com/microsoft/vscode-l10n/blob/main/LICENSE
|
||||
"name": "@vscode/l10n",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/microsoft/vscode-l10n/main/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @microsoft/dev-tunnels-ssh does not include a LICENSE file in its npm tarball.
|
||||
// Source: https://github.com/microsoft/dev-tunnels-ssh/blob/main/LICENSE
|
||||
"name": "@microsoft/dev-tunnels-ssh",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/microsoft/dev-tunnels-ssh/main/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @microsoft/dev-tunnels-ssh-tcp does not include a LICENSE file in its npm tarball.
|
||||
// Source: https://github.com/microsoft/dev-tunnels-ssh/blob/main/LICENSE (shared repo)
|
||||
"name": "@microsoft/dev-tunnels-ssh-tcp",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/microsoft/dev-tunnels-ssh/main/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @xterm/addon-serialize does not include a LICENSE file in its npm tarball.
|
||||
// Source: https://github.com/xtermjs/xterm.js/blob/master/LICENSE
|
||||
"name": "@xterm/addon-serialize",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/xtermjs/xterm.js/master/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @github/copilot-sdk does not ship a LICENSE file in its npm tarball,
|
||||
// and ClearlyDefined has not harvested it, so CG emits no license text and the
|
||||
// on-disk scanner finds nothing. The source repo is public MIT with a LICENSE file.
|
||||
// Source: https://github.com/github/copilot-sdk/blob/main/LICENSE
|
||||
"name": "@github/copilot-sdk",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/github/copilot-sdk/main/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @vscode/windows-ca-certs is os:["win32"], so npm never installs it on
|
||||
// the single Linux NOTICE agent and the on-disk scanner cannot see it. It is not
|
||||
// harvested by CG either. Section 6 of scan-licenses.ts seeds it into the presence
|
||||
// index from package-lock.json so this override can inject the BSD text below.
|
||||
// Source: https://github.com/microsoft/vscode-windows-ca-certs/blob/main/LICENSE
|
||||
"name": "@vscode/windows-ca-certs",
|
||||
"fullLicenseTextUri": "https://raw.githubusercontent.com/microsoft/vscode-windows-ca-certs/main/LICENSE"
|
||||
},
|
||||
{
|
||||
// Reason: @vscode/markdown-editor comes from a private repo (microsoft/vscode-packages).
|
||||
// License text provided verbatim by the repo owner.
|
||||
"name": "@vscode/markdown-editor",
|
||||
"fullLicenseText": [
|
||||
"MIT License",
|
||||
"",
|
||||
"Copyright (c) Microsoft Corporation.",
|
||||
"",
|
||||
"Permission is hereby granted, free of charge, to any person obtaining a copy",
|
||||
"of this software and associated documentation files (the \"Software\"), to deal",
|
||||
"in the Software without restriction, including without limitation the rights",
|
||||
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell",
|
||||
"copies of the Software, and to permit persons to whom the Software is",
|
||||
"furnished to do so, subject to the following conditions:",
|
||||
"",
|
||||
"The above copyright notice and this permission notice shall be included in all",
|
||||
"copies or substantial portions of the Software.",
|
||||
"",
|
||||
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR",
|
||||
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,",
|
||||
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE",
|
||||
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER",
|
||||
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,",
|
||||
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE",
|
||||
"SOFTWARE"
|
||||
]
|
||||
},
|
||||
{
|
||||
// Reason: linux-keyutils@0.2.4 declares "Apache-2.0 OR MIT" but its repo
|
||||
// (github.com/landhb/linux-keyutils) ships NO license file — only Cargo.toml
|
||||
// declares the SPDX expression. CG emits the SPDX string as the body (a stub)
|
||||
// and the Cargo scanner can't fetch real text (no LICENSE at any ref). OR =
|
||||
// our choice; we supply the canonical MIT text (the shortest option). Author
|
||||
// is "landhb" (no real name documented); no explicit copyright holder exists
|
||||
// in any upstream license file, so we credit the documented crate author.
|
||||
"name": "linux-keyutils",
|
||||
"fullLicenseText": [
|
||||
"MIT License",
|
||||
"",
|
||||
"Copyright (c) landhb",
|
||||
"",
|
||||
"Permission is hereby granted, free of charge, to any person obtaining a copy",
|
||||
"of this software and associated documentation files (the \"Software\"), to deal",
|
||||
"in the Software without restriction, including without limitation the rights",
|
||||
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell",
|
||||
"copies of the Software, and to permit persons to whom the Software is",
|
||||
"furnished to do so, subject to the following conditions:",
|
||||
"",
|
||||
"The above copyright notice and this permission notice shall be included in all",
|
||||
"copies or substantial portions of the Software.",
|
||||
"",
|
||||
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR",
|
||||
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,",
|
||||
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE",
|
||||
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER",
|
||||
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,",
|
||||
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE",
|
||||
"SOFTWARE"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -10,34 +10,13 @@
|
||||
}
|
||||
},
|
||||
"licenseDetail": [
|
||||
"Oniguruma LICENSE",
|
||||
"-----------------",
|
||||
"Copyright (c) 2010-2018 Keith Dahlby, Keith Hill, and contributors",
|
||||
"",
|
||||
"Copyright (c) 2002-2020 K.Kosako <kkosako0@gmail.com>",
|
||||
"All rights reserved.",
|
||||
"Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:",
|
||||
"",
|
||||
"The BSD License",
|
||||
"The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.",
|
||||
"",
|
||||
"Redistribution and use in source and binary forms, with or without",
|
||||
"modification, are permitted provided that the following conditions",
|
||||
"are met:",
|
||||
"1. Redistributions of source code must retain the above copyright",
|
||||
" notice, this list of conditions and the following disclaimer.",
|
||||
"2. Redistributions in binary form must reproduce the above copyright",
|
||||
" notice, this list of conditions and the following disclaimer in the",
|
||||
" documentation and/or other materials provided with the distribution.",
|
||||
"",
|
||||
"THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND",
|
||||
"ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE",
|
||||
"IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE",
|
||||
"ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE",
|
||||
"FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL",
|
||||
"DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS",
|
||||
"OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)",
|
||||
"HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT",
|
||||
"LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY",
|
||||
"OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF",
|
||||
"SUCH DAMAGE."
|
||||
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
|
||||
],
|
||||
"isOnlyProductionDependency": true,
|
||||
"license": "MIT",
|
||||
|
||||
Reference in new issue
Block a user