benchmark: implement node:bench version of bench tools

Signed-off-by: James M Snel <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: https://github.com/nodejs/node/pull/65606
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day>

benchmark/README.md

@@ -10,6 +10,7 @@ directory, see [the guide on benchmarks](../doc/contributing/writing-and-running
 ## Table of Contents
 
 * [File tree structure](#file-tree-structure)
+* [`node:bench` evaluation tools](#nodebench-evaluation-tools)
 * [Common API](#common-api)
 
 ## File tree structure
@@ -44,6 +45,8 @@ directories.
 * `common.js`: see [Common API](#common-api).
 * `compare.js`: command line tool for comparing performance between different
   Node.js binaries.
+* `compare-node-bench.js`: parallel comparison tool for explicit `node:bench`
+  files. It does not change or invoke `compare.js`.
 * `compare.R`: R script for statistically analyzing the output of
   `compare.js`
 * `run.js`: command line tool for running individual benchmark suite(s).
@@ -51,9 +54,83 @@ directories.
   between different parameters in benchmark configurations,
   for example to analyze the time complexity. Pass `--analyze` to
   summarize the results without R.
+* `scatter-node-bench.js`: parallel scatter-data tool for an explicit
+  `node:bench` file. It does not change or invoke `scatter.js`.
 * `scatter.R`: R script for visualizing the output of `scatter.js` with
   scatter plots.
 
+## `node:bench` evaluation tools
+
+The `compare-node-bench.js` and `scatter-node-bench.js` tools run explicit
+`node:bench` files without changing the existing benchmark framework or its
+tools. Each repeated observation for a benchmark identity is collected by a
+separate process invocation with one measured sample. Benchmarks declared in
+the same file still execute serially in that process and can share JIT, garbage
+collector, heap, and cache state. This differs from legacy configuration-level
+process isolation and must be considered when comparing the frameworks.
+
+Compare two binaries and analyze the compatible CSV using `compare.R`:
+
+```console
+./node benchmark/compare-node-bench.js \
+  --old ./node-main --new ./node-pr --runs 30 -- \
+  benchmark/crypto/_create-hash.node-bench.js > compare-node-bench.csv
+Rscript benchmark/compare.R < compare-node-bench.csv
+```
+
+Pass `--analyze` to run the same Welch analysis inline. `--max-regression N`
+implies `--analyze` and makes the command fail only when the Holm-Bonferroni
+adjusted p-value is below 0.05 and the full 95% confidence interval is worse
+than `-N%`. Requiring both conditions prevents a noisy point estimate from
+failing a regression gate.
+
+```console
+./node benchmark/compare-node-bench.js \
+  --old ./node-main --new ./node-pr --runs 30 \
+  --max-regression 5 -- benchmark/crypto/_create-hash.node-bench.js
+```
+
+Collect parameter data for the parallel buffer benchmark and plot it using
+`scatter.R`:
+
+```console
+./node benchmark/scatter-node-bench.js --node ./node --runs 30 -- \
+  benchmark/buffers/_buffer-compare-offset.node-bench.js \
+  > scatter-node-bench.csv
+Rscript benchmark/scatter.R --xaxis size --category method \
+  --plot scatter-node-bench.png < scatter-node-bench.csv
+```
+
+Pass `--analyze` with an x-axis parameter to summarize the samples without R.
+The output includes mean and median confidence intervals, skew warnings, an
+optional bar chart, and Mann-Whitney U and Cliff's delta comparisons between
+consecutive x-axis values. Use `--category` for a second grouping parameter and
+`--no-chart` to omit the chart.
+
+Because configurations in one file share a process, inline analysis averages
+aggregated configurations into one value per outer process. Consecutive
+x-axis comparisons use alternating, disjoint process sets so the unpaired
+Mann-Whitney test does not treat correlated values as independent samples.
+
+```console
+./node benchmark/scatter-node-bench.js --runs 30 --analyze \
+  --xaxis size --category method -- \
+  benchmark/buffers/_buffer-compare-offset.node-bench.js
+```
+
+A file passed to `scatter-node-bench.js` must use one logical benchmark name.
+Parameter values distinguish its configurations. The tool rejects unstable
+identities and names or parameters that would merge unrelated CSV groups.
+
+The underscore-prefixed benchmark files are parallel ports used to compare the
+measurement frameworks. Legacy discovery ignores them, so the original files
+remain the source benchmarks for `run.js`, `compare.js`, and `scatter.js`. The
+ports use the platform-specific original relative filename as their benchmark
+name and preserve parameter column names to keep CSV grouping compatible. For
+a direct framework comparison, collect the same number of runs from an
+original benchmark with `scatter.js` and from its port with
+`scatter-node-bench.js`, then compare their rate distributions.
+
 ## Common API
 
 The common.js module is used by benchmarks for consistency across repeated

benchmark/_node-bench-analysis.js

@@ -0,0 +1,736 @@
+'use strict';
+
+const { createHistogram } = require('node:perf_hooks');
+const { inspect } = require('node:util');
+
+function createRateHistogram(rates, scale, figures) {
+  const histogram = createHistogram({ figures });
+  for (const rate of rates) {
+    const value = Math.max(1, Math.round(rate * scale));
+    if (!Number.isSafeInteger(value)) {
+      throw new RangeError('Benchmark rate is too large for the histogram scale');
+    }
+    histogram.record(value);
+  }
+  return histogram;
+}
+
+function holmAdjust(pValues) {
+  const order = pValues
+    .map((p, index) => ({ index, p }))
+    .sort((a, b) => a.p - b.p);
+  const adjusted = new Array(order.length);
+  let running = 0;
+  for (let index = 0; index < order.length; index++) {
+    running = Math.max(
+      running,
+      Math.min(1, (order.length - index) * order[index].p),
+    );
+    adjusted[order[index].index] = running;
+  }
+  return adjusted;
+}
+
+function isRegressionFailure(row, maxRegression) {
+  return row.pAdjusted < 0.05 &&
+         row.improvement + row.ci95 < -maxRegression;
+}
+
+function analyzeCompare(samples, scale, maxRegression) {
+  const groups = new Map();
+  for (const sample of samples) {
+    let group = groups.get(sample.identity);
+    if (group === undefined) {
+      const suffix = sample.configuration === '' ?
+        '' : ` ${sample.configuration}`;
+      group = {
+        name: `${sample.name}${suffix}`,
+        new: [],
+        old: [],
+      };
+      groups.set(sample.identity, group);
+    }
+    group[sample.binary].push(sample.rate);
+  }
+
+  const rows = [];
+  let skipped = 0;
+  for (const { name, old: oldRates, new: newRates } of groups.values()) {
+    if (oldRates.length < 2 || newRates.length < 2) {
+      skipped++;
+      continue;
+    }
+
+    const oldHistogram = createRateHistogram(oldRates, scale, 3);
+    const newHistogram = createRateHistogram(newRates, scale, 3);
+    const oldMean = oldRates.reduce((sum, rate) => sum + rate, 0) /
+      oldRates.length;
+    const newMean = newRates.reduce((sum, rate) => sum + rate, 0) /
+      newRates.length;
+    const improvement = ((newMean - oldMean) / oldMean) * 100;
+    const w95 = oldHistogram.welchTest(newHistogram, { confidence: 0.95 });
+    const w99 = oldHistogram.welchTest(newHistogram, { confidence: 0.99 });
+    const w999 = oldHistogram.welchTest(newHistogram, { confidence: 0.999 });
+    let stars = '';
+    if (w95.pValue < 0.001) stars = '***';
+    else if (w95.pValue < 0.01) stars = ' **';
+    else if (w95.pValue < 0.05) stars = '  *';
+    const ciPercent = (result) => {
+      const half = (result.confidenceInterval.upper -
+                    result.confidenceInterval.lower) / 2;
+      return (half / (oldMean * scale)) * 100;
+    };
+    rows.push({
+      ci95: ciPercent(w95),
+      ci99: ciPercent(w99),
+      ci999: ciPercent(w999),
+      improvement,
+      name,
+      pValue: Number.isNaN(w95.pValue) ? 1 : w95.pValue,
+      stars,
+    });
+  }
+
+  const adjusted = holmAdjust(rows.map(({ pValue }) => pValue));
+  let underpowered = 0;
+  for (let index = 0; index < rows.length; index++) {
+    const row = rows[index];
+    row.pAdjusted = adjusted[index];
+    row.inconclusive = maxRegression > 0 &&
+                       row.stars.trim() === '' &&
+                       row.ci95 > maxRegression;
+    if (row.inconclusive) underpowered++;
+  }
+
+  const output = [];
+  const maxNameLength = rows.reduce(
+    (maximum, { name }) => Math.max(maximum, name.length), 0);
+  const pad = (value, length) =>
+    value + ' '.repeat(Math.max(0, length - value.length));
+  const padStart = (value, length) =>
+    ' '.repeat(Math.max(0, length - value.length)) + value;
+  output.push(`${pad('', maxNameLength)}  confidence` +
+              '   improvement   accuracy (*)    (**)   (***)');
+  for (const row of rows) {
+    const improvement =
+      `${row.improvement >= 0 ? '+' : ''}${row.improvement.toFixed(2)} %`;
+    output.push(
+      `${pad(row.name, maxNameLength)}  ${pad(row.stars, 10)}` +
+      `  ${padStart(improvement, 11)}` +
+      `   ±${row.ci95.toFixed(2)}%` +
+      `  ±${row.ci99.toFixed(2)}%` +
+      `  ±${row.ci999.toFixed(2)}%` +
+      `${row.inconclusive ? '  (inconclusive)' : ''}`,
+    );
+  }
+
+  if (skipped > 0) {
+    output.push('');
+    output.push(
+      `Note: ${skipped} configuration${skipped === 1 ? ' was' : 's were'}` +
+      ' skipped because Welch\'s t-test requires at least 2 samples per' +
+      ' binary. Use --runs 2 or higher.',
+    );
+  }
+  printCompareChart(output, rows, maxNameLength);
+
+  output.push('');
+  output.push(
+    `Rates were scaled by ${scale}x into HdrHistogram (3 significant figures).`,
+    'Use --scale to adjust precision if needed.',
+    '',
+  );
+  const significant = rows.filter(({ pAdjusted }) => pAdjusted < 0.05).length;
+  output.push(
+    'The confidence markers above are per-benchmark and uncorrected. ' +
+    `After Holm-Bonferroni correction across ${rows.length} comparison` +
+    `${rows.length === 1 ? '' : 's'}, ${significant} remain` +
+    `${significant === 1 ? 's' : ''} significant at 5%.`,
+    '--max-regression uses the corrected values.',
+  );
+
+  if (maxRegression > 0 && underpowered > 0) {
+    output.push('');
+    output.push(
+      `Note: ${underpowered} of ${rows.length} comparison` +
+      `${rows.length === 1 ? '' : 's'} could not resolve an effect as small ` +
+      `as ${maxRegression}% and are marked (inconclusive). Raise --runs to ` +
+      'narrow their confidence intervals.',
+    );
+  }
+
+  const failures = maxRegression > 0 ?
+    rows.filter((row) => isRegressionFailure(row, maxRegression)) : [];
+  if (failures.length > 0) {
+    output.push('');
+    output.push(
+      `FAIL: ${failures.length} benchmark${failures.length === 1 ? '' : 's'}` +
+      ` regressed by more than ${maxRegression}% (the 95% interval excludes ` +
+      `the threshold and significance is family-wise corrected across ` +
+      `${rows.length} comparisons):`,
+    );
+    for (const failure of failures) {
+      output.push(
+        `  ${failure.name}  ${failure.improvement.toFixed(2)}% ` +
+        `(95% CI up to ${(failure.improvement + failure.ci95).toFixed(2)}%, ` +
+        `adjusted p=${failure.pAdjusted.toExponential(2)})`,
+      );
+    }
+  }
+
+  return {
+    failed: failures.length > 0,
+    output: `${output.join('\n')}\n`,
+    rows,
+  };
+}
+
+function printCompareChart(output, rows, maxNameLength) {
+  if (rows.length === 0) return;
+  const width = 40;
+  const halfWidth = width / 2;
+  let maximum = 0;
+  for (const row of rows) {
+    maximum = Math.max(maximum, Math.abs(row.improvement) + row.ci95);
+  }
+  if (maximum === 0) maximum = 1;
+  const left = `-${maximum.toFixed(1)}%`;
+  const right = `+${maximum.toFixed(1)}%`;
+  const centerLabel = '0%';
+  const labelPadding = maxNameLength + 5;
+  output.push('');
+  output.push(
+    ' '.repeat(labelPadding) + left +
+    ' '.repeat(Math.max(
+      0, halfWidth - left.length - Math.floor(centerLabel.length / 2))) +
+    centerLabel +
+    ' '.repeat(Math.max(
+      0, halfWidth - Math.ceil(centerLabel.length / 2) - right.length)) +
+    right,
+  );
+  for (const row of rows) {
+    const center = halfWidth;
+    const result = center + (row.improvement / maximum) * halfWidth;
+    const lower = center +
+      ((row.improvement - row.ci95) / maximum) * halfWidth;
+    const upper = center +
+      ((row.improvement + row.ci95) / maximum) * halfWidth;
+    let bar = '';
+    for (let index = 0; index < width; index++) {
+      const position = index + 0.5;
+      if (index === Math.floor(center)) {
+        bar += '|';
+      } else if ((row.improvement >= 0 &&
+                  position > center && position <= result) ||
+                 (row.improvement < 0 &&
+                  position < center && position >= result)) {
+        bar += row.stars === '' ? '▓' : '█';
+      } else if (position >= lower && position <= upper) {
+        bar += '░';
+      } else {
+        bar += ' ';
+      }
+    }
+    const label = `${row.improvement >= 0 ? '+' : ''}` +
+      `${row.improvement.toFixed(2)}%`;
+    output.push(
+      `${row.name.padEnd(maxNameLength)}  ${bar}  ${label} ${row.stars.trim()}`,
+    );
+  }
+}
+
+function histogramScale(rates) {
+  let minimum = Infinity;
+  let maximum = 0;
+  for (const rate of rates) {
+    if (rate > 0 && rate < minimum) minimum = rate;
+    if (rate > maximum) maximum = rate;
+  }
+  if (!Number.isFinite(minimum) || maximum === 0) return 1;
+  let scale = 1;
+  while (minimum * scale < 1e6 && maximum * scale < 1e15) scale *= 10;
+  return scale;
+}
+
+function validateScatterParameters(samples, xAxis, category) {
+  if (category !== undefined && category === xAxis) {
+    throw new Error('--xaxis and --category must name different parameters');
+  }
+  for (const key of [xAxis, category]) {
+    if (key === undefined) continue;
+    if (samples.some(({ params }) =>
+      !Object.hasOwn(params, key))) {
+      const available = [...new Set(samples.flatMap(
+        ({ params }) => Object.keys(params)))].sort();
+      throw new Error(
+        `The variable '${key}' is not present in every configuration. ` +
+        `Available variables: ${available.join(', ')}`,
+      );
+    }
+  }
+}
+
+function analyzeScatter(samples, xAxis, category, showChart) {
+  validateScatterParameters(samples, xAxis, category);
+
+  const parameterNames = [...new Set(samples.flatMap(
+    ({ params }) => Object.keys(params)))];
+  const aggregated = parameterNames.filter((name) => {
+    if (name === xAxis || name === category) return false;
+    const first = samples[0].params[name];
+    return samples.some(({ params }) => params[name] !== first);
+  });
+  const groups = new Map();
+  for (const sample of samples) {
+    const xValue = sample.params[xAxis];
+    const categoryValue = category === undefined ?
+      undefined : sample.params[category];
+    const key = valueKey([xValue, categoryValue]);
+    let group = groups.get(key);
+    if (group === undefined) {
+      group = {
+        categoryValue,
+        members: [],
+        observations: new Map(),
+        xValue,
+      };
+      groups.set(key, group);
+    }
+    group.members.push(sample);
+    let rates = group.observations.get(sample.observation);
+    if (rates === undefined) {
+      rates = [];
+      group.observations.set(sample.observation, rates);
+    }
+    rates.push(sample.rate);
+  }
+  for (const group of groups.values()) {
+    group.processRates = [...group.observations].map(([observation, rates]) => ({
+      observation,
+      rate: rates.reduce((sum, rate) => sum + rate, 0) / rates.length,
+    }));
+    group.rates = group.processRates.map(({ rate }) => rate);
+  }
+
+  const scale = histogramScale(samples.map(({ rate }) => rate));
+  const compareValues = (a, b) => {
+    if (typeof a === 'number' && typeof b === 'number') return a - b;
+    return String(a).localeCompare(String(b));
+  };
+  const rows = [...groups.values()]
+    .sort((a, b) => compareValues(a.xValue, b.xValue) ||
+                    compareValues(a.categoryValue, b.categoryValue))
+    .map((group) => {
+      const histogram = createRateHistogram(group.rates, scale, 5);
+      const count = group.rates.length;
+      const mean = group.rates.reduce((sum, rate) => sum + rate, 0) / count;
+      const meanInterval = histogram.meanCI();
+      const confidenceInterval = count > 1 ?
+        (meanInterval.upper - meanInterval.lower) / (2 * scale) : NaN;
+      const medianInterval = histogram.percentileCI(50);
+      const median = rawMedian(group.rates);
+      const skewed = count > 1 &&
+        (Math.abs(histogram.skewness) > 1 ||
+         Math.abs(median - mean) > confidenceInterval);
+      return {
+        ...group,
+        confidenceInterval,
+        count,
+        histogram,
+        mean,
+        median,
+        medianLower: medianInterval.lower / scale,
+        medianUpper: medianInterval.upper / scale,
+        skewed,
+      };
+    });
+
+  const legend = assignLabels(rows, 'xValue', 'xLabel');
+  if (category !== undefined) {
+    legend.push(...assignLabels(rows, 'categoryValue', 'categoryLabel'));
+  }
+  const output = [];
+  const contamination = new Map();
+  for (const variable of aggregated) {
+    const share = varianceShare([...groups.values()], variable);
+    contamination.set(variable, share);
+    const percent = share === 1 ?
+      '100' : (share >= 0.995 ? '>99' : (share * 100).toFixed(0));
+    const suffix = Number.isNaN(share) ?
+      '' : ` (explains ${percent}% of within-group variance)`;
+    output.push(`aggregating variable: ${variable}${suffix}`);
+  }
+  const dominant = aggregated.filter(
+    (variable) => contamination.get(variable) > 0.5);
+  if (dominant.length > 0) {
+    output.push('');
+    wrapOutput(
+      output,
+      `${dominant.join(', ')} ${dominant.length === 1 ? 'explains' : 'explain'} ` +
+      'most of the spread within each group. Pin the parameter or use it as ' +
+      '--category; increasing --runs will not remove this source of variance.',
+    );
+  }
+  if (aggregated.length > 0) output.push('');
+  printScatterTable(output, rows, xAxis, category);
+  if (showChart) printScatterChart(output, rows, xAxis, category);
+  printScatterComparisons(output, rows, xAxis, category);
+  if (legend.length > 0) {
+    output.push('', 'Abbreviated values:');
+    for (const { full, label } of legend) {
+      output.push(`  ${label}`, `    = ${full}`);
+    }
+  }
+  const singleSample = rows.filter(({ count }) => count < 2).length;
+  if (singleSample > 0) {
+    output.push('');
+    wrapOutput(
+      output,
+      `Note: ${singleSample} group${singleSample === 1 ? ' has' : 's have'} ` +
+      'only one sample, so no confidence interval could be estimated. Use ' +
+      '--runs 2 or higher.',
+    );
+  }
+  if (rows.some(({ skewed }) => skewed)) {
+    output.push('');
+    wrapOutput(
+      output,
+      '(!) marks groups where the median falls outside the mean confidence ' +
+      'interval or the sample is strongly skewed. The median and its interval ' +
+      'describe the typical run better for those groups.',
+    );
+  }
+  return `${output.join('\n')}\n`;
+}
+
+function rawMedian(rates) {
+  const sorted = [...rates].sort((a, b) => a - b);
+  const middle = Math.floor(sorted.length / 2);
+  return sorted.length % 2 === 0 ?
+    (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle];
+}
+
+function valueKey(value) {
+  return JSON.stringify(value, (_, item) => {
+    if (typeof item === 'bigint') return { bigint: String(item) };
+    return item;
+  });
+}
+
+function varianceShare(groups, variable) {
+  let between = 0;
+  let total = 0;
+  for (const group of groups) {
+    if (group.members.length < 2) continue;
+    const mean = group.members.reduce(
+      (sum, sample) => sum + sample.rate, 0) / group.members.length;
+    const levels = new Map();
+    for (const sample of group.members) {
+      const key = valueKey(sample.params[variable]);
+      let level = levels.get(key);
+      if (level === undefined) {
+        level = { count: 0, sum: 0 };
+        levels.set(key, level);
+      }
+      level.count++;
+      level.sum += sample.rate;
+    }
+    for (const level of levels.values()) {
+      between += level.count * ((level.sum / level.count) - mean) ** 2;
+    }
+    for (const sample of group.members) total += (sample.rate - mean) ** 2;
+  }
+  return total === 0 ? NaN : Math.min(1, between / total);
+}
+
+function effectSizeLabel(delta) {
+  const absolute = Math.abs(delta);
+  if (absolute < 0.147) return 'negligible';
+  if (absolute < 0.33) return 'small';
+  if (absolute < 0.474) return 'medium';
+  return 'large';
+}
+
+const mannWhitneyFloors = new Map();
+function mannWhitneyFloor(firstCount, secondCount) {
+  const key = `${firstCount},${secondCount}`;
+  let floor = mannWhitneyFloors.get(key);
+  if (floor !== undefined) return floor;
+  const low = createHistogram({ figures: 5 });
+  const high = createHistogram({ figures: 5 });
+  for (let index = 0; index < firstCount; index++) low.record(10000 + index);
+  for (let index = 0; index < secondCount; index++) {
+    high.record(10000 + firstCount + index);
+  }
+  floor = high.mannWhitneyTest(low).pValue;
+  mannWhitneyFloors.set(key, floor);
+  return floor;
+}
+
+function printScatterComparisons(output, rows, xAxis, category) {
+  const usable = rows.filter(({ count }) => count > 1);
+  if (usable.length < 2) return;
+  const series = new Map();
+  for (const row of usable) {
+    const key = valueKey(row.categoryValue);
+    if (!series.has(key)) series.set(key, []);
+    series.get(key).push(row);
+  }
+  const sections = [];
+  let floor = 0;
+  for (const group of series.values()) {
+    if (group.length < 2) continue;
+    const entries = [];
+    for (let index = 1; index < group.length; index++) {
+      const previous = group[index - 1];
+      const current = group[index];
+      // Configurations in one file share a process. Split consecutive groups
+      // across disjoint outer-process sets so the unpaired test does not treat
+      // correlated observations as independent.
+      const parity = index % 2;
+      const previousRates = previous.processRates
+        .filter(({ observation }) => observation % 2 === parity)
+        .map(({ rate }) => rate);
+      const currentRates = current.processRates
+        .filter(({ observation }) => observation % 2 !== parity)
+        .map(({ rate }) => rate);
+      if (previousRates.length === 0 || currentRates.length === 0) continue;
+      const comparisonScale = histogramScale([
+        ...previousRates,
+        ...currentRates,
+      ]);
+      const previousHistogram =
+        createRateHistogram(previousRates, comparisonScale, 5);
+      const currentHistogram =
+        createRateHistogram(currentRates, comparisonScale, 5);
+      const { pValue } = currentHistogram.mannWhitneyTest(previousHistogram);
+      const delta = currentHistogram.cliffsD(previousHistogram);
+      const previousMean = previousRates.reduce(
+        (sum, rate) => sum + rate, 0) / previousRates.length;
+      const currentMean = currentRates.reduce(
+        (sum, rate) => sum + rate, 0) / currentRates.length;
+      const change = ((currentMean - previousMean) / previousMean) * 100;
+      floor = Math.max(
+        floor, mannWhitneyFloor(previousRates.length, currentRates.length));
+      let ratio = '';
+      let exponent = '';
+      if (typeof previous.xValue === 'number' &&
+          typeof current.xValue === 'number' &&
+          previous.xValue > 0 && current.xValue > 0 &&
+          previous.xValue !== current.xValue &&
+          previousMean > 0 && currentMean > 0) {
+        const xRatio = current.xValue / previous.xValue;
+        const value = Math.log(currentMean / previousMean) / Math.log(xRatio);
+        ratio = `${xRatio.toFixed(1)}x`;
+        exponent = `${value >= 0 ? '+' : ''}${value.toFixed(2)}`;
+      }
+      entries.push(
+        `    ${previous.xLabel} -> ${current.xLabel}` +
+        `  ${change >= 0 ? '+' : ''}${change.toFixed(2)}%` +
+        (exponent === '' ? '' : `  ${ratio}  exponent=${exponent}`) +
+        `  p=${pValue < 1e-4 ? pValue.toExponential(1) : pValue.toFixed(4)}` +
+        `  delta=${delta >= 0 ? '+' : ''}${delta.toFixed(3)}` +
+        ` (${effectSizeLabel(delta)})`,
+      );
+    }
+    if (entries.length > 0) {
+      sections.push({
+        entries,
+        heading: category === undefined ?
+          undefined : `${category}=${group[0].categoryLabel}`,
+      });
+    }
+  }
+  if (sections.length === 0) return;
+  output.push('', `Change between consecutive ${xAxis} values ` +
+                  `(Mann-Whitney U on disjoint process sets, Cliff's delta):`);
+  for (const section of sections) {
+    output.push('');
+    if (section.heading !== undefined) output.push(`  ${section.heading}`);
+    output.push(...section.entries);
+  }
+  if (floor >= 0.05) {
+    output.push('');
+    wrapOutput(
+      output,
+      `Warning: at this sample size the smallest p-value this test can ` +
+      `produce is ${floor.toFixed(4)}, so no comparison above can reach ` +
+      'significance. Raise --runs.',
+    );
+  } else if (floor >= 0.005) {
+    output.push('');
+    wrapOutput(
+      output,
+      `Note: at this sample size the smallest p-value this test can produce ` +
+      `is ${floor.toFixed(4)}. Raise --runs to strengthen non-significant ` +
+      'results.',
+    );
+  }
+}
+
+function formatRate(rate) {
+  return rate.toLocaleString('en-US', {
+    maximumFractionDigits: 1,
+    minimumFractionDigits: 1,
+  });
+}
+
+function displayWidth(value) {
+  return [...value].length;
+}
+
+function pad(value, width, right) {
+  const padding = ' '.repeat(Math.max(0, width - displayWidth(value)));
+  return right ? padding + value : value + padding;
+}
+
+function truncateMiddle(value, maximum = 24) {
+  const characters = [...value];
+  if (characters.length <= maximum) return value;
+  const retained = maximum - 3;
+  const head = Math.ceil(retained / 2);
+  const tail = Math.floor(retained / 2);
+  return `${characters.slice(0, head).join('')}...` +
+         characters.slice(-tail).join('');
+}
+
+function assignLabels(rows, valueName, labelName) {
+  const assigned = new Map();
+  const used = new Map();
+  const legend = [];
+  for (const row of rows) {
+    const key = valueKey(row[valueName]);
+    let label = assigned.get(key);
+    if (label === undefined) {
+      const full = typeof row[valueName] === 'string' ?
+        inspect(row[valueName]) : String(row[valueName]);
+      const abbreviated = truncateMiddle(full);
+      const collisions = used.get(abbreviated) ?? 0;
+      used.set(abbreviated, collisions + 1);
+      label = collisions === 0 ?
+        abbreviated : `${abbreviated}~${collisions + 1}`;
+      assigned.set(key, label);
+      if (label !== full) legend.push({ full, label });
+    }
+    row[labelName] = label;
+  }
+  return legend;
+}
+
+function printScatterTable(output, rows, xAxis, category) {
+  const header = [xAxis];
+  if (category !== undefined) header.push(category);
+  header.push(
+    'samples', 'rate', 'confidence.interval', 'median', 'median.interval', '');
+  const body = rows.map((row) => {
+    const values = [row.xLabel];
+    if (category !== undefined) values.push(row.categoryLabel);
+    const medianInterval = row.count > 1 ?
+      `[${(((row.medianLower - row.median) / row.median) * 100).toFixed(2)}%, ` +
+      `+${(((row.medianUpper - row.median) / row.median) * 100).toFixed(2)}%]` :
+      'NA';
+    values.push(
+      String(row.count),
+      formatRate(row.mean),
+      Number.isNaN(row.confidenceInterval) ?
+        'NA' :
+        `${formatRate(row.confidenceInterval)} ` +
+        `(±${((row.confidenceInterval / row.mean) * 100).toFixed(2)}%)`,
+      formatRate(row.median),
+      medianInterval,
+      row.skewed ? '(!)' : '',
+    );
+    return values;
+  });
+  const widths = header.map((value, index) => Math.max(
+    displayWidth(value),
+    ...body.map((values) => displayWidth(values[index])),
+  ));
+  const right = [typeof rows[0].xValue === 'number'];
+  if (category !== undefined) {
+    right.push(typeof rows[0].categoryValue === 'number');
+  }
+  right.push(true, true, true, true, true, false);
+  const format = (values) => values.map(
+    (value, index) => pad(value, widths[index], right[index])).join('  ').trimEnd();
+  output.push(format(header));
+  for (const values of body) output.push(format(values));
+}
+
+function printScatterChart(output, rows, xAxis, category) {
+  if (rows.length === 0) return;
+  const width = 40;
+  let maximum = 0;
+  for (const row of rows) {
+    maximum = Math.max(
+      maximum,
+      row.mean + (Number.isNaN(row.confidenceInterval) ?
+        0 : row.confidenceInterval),
+    );
+  }
+  if (maximum === 0) return;
+  const labels = rows.map((row) => {
+    let label = `${xAxis}=${row.xLabel}`;
+    if (category !== undefined) label += ` ${category}=${row.categoryLabel}`;
+    return truncateMiddle(label, 44);
+  });
+  const labelWidth = Math.max(...labels.map(displayWidth));
+  const rateWidth = Math.max(...rows.map(({ mean }) =>
+    displayWidth(formatRate(mean))));
+  const axis = formatRate(maximum);
+  const indent = ' '.repeat(labelWidth + 2);
+  output.push(
+    '',
+    'Rate in operations/second; longer is faster. │ marks the mean and the',
+    'shaded band (░) is its 95% confidence interval.',
+    '',
+    `${indent}0${' '.repeat(Math.max(1, width - 1 - displayWidth(axis)))}${axis}`,
+    `${indent}+${'-'.repeat(width - 2)}+`,
+  );
+  let previous;
+  for (let index = 0; index < rows.length; index++) {
+    const row = rows[index];
+    if (previous !== undefined && previous !== row.xValue) output.push('');
+    previous = row.xValue;
+    const interval = Number.isNaN(row.confidenceInterval) ?
+      0 : row.confidenceInterval;
+    const end = (row.mean / maximum) * width;
+    const lower = ((row.mean - interval) / maximum) * width;
+    const upper = ((row.mean + interval) / maximum) * width;
+    const meanCell = Math.min(width - 1, Math.floor(end));
+    let bar = '';
+    for (let cell = 0; cell < width; cell++) {
+      const position = cell + 0.5;
+      if (cell === meanCell) bar += '│';
+      else if (position >= lower && position <= upper) bar += '░';
+      else if (position <= end) bar += '█';
+      else bar += ' ';
+    }
+    output.push(
+      `${pad(labels[index], labelWidth, false)}  ${bar}  ` +
+      pad(formatRate(row.mean), rateWidth, true),
+    );
+  }
+}
+
+function wrapOutput(output, text, width = 76) {
+  let line = '';
+  for (const word of text.split(/\s+/)) {
+    if (line === '') line = word;
+    else if (displayWidth(line) + displayWidth(word) + 1 <= width) {
+      line += ` ${word}`;
+    } else {
+      output.push(line);
+      line = word;
+    }
+  }
+  if (line !== '') output.push(line);
+}
+
+module.exports = {
+  analyzeCompare,
+  analyzeScatter,
+  holmAdjust,
+  isRegressionFailure,
+  validateScatterParameters,
+};

benchmark/_node-bench.js

@@ -0,0 +1,159 @@
+'use strict';
+
+const { spawn } = require('node:child_process');
+const path = require('node:path');
+const { inspect } = require('node:util');
+
+function parseInteger(value, defaultValue, name, minimum) {
+  if (value === undefined) return defaultValue;
+  if (!/^(?:0|[1-9]\d*)$/.test(value)) {
+    throw new TypeError(`${name} must be an integer`);
+  }
+  const number = Number(value);
+  if (!Number.isSafeInteger(number) || number < minimum) {
+    throw new RangeError(`${name} must be at least ${minimum}`);
+  }
+  return number;
+}
+
+function parseNumber(value, defaultValue, name, minimum) {
+  if (value === undefined) return defaultValue;
+  if (value.trim() === '') throw new TypeError(`${name} must be a number`);
+  const number = Number(value);
+  if (!Number.isFinite(number)) throw new TypeError(`${name} must be a number`);
+  if (number < minimum) {
+    throw new RangeError(`${name} must be at least ${minimum}`);
+  }
+  return number;
+}
+
+function csvEncode(value) {
+  if (typeof value === 'number' || typeof value === 'boolean') {
+    return String(value);
+  }
+  const string = String(value);
+  return `"${string.replace(/"/g, '""')}"`;
+}
+
+function formatConfiguration(params) {
+  return Object.keys(params)
+    .map((key) => `${key}=${inspect(params[key])}`)
+    .join(' ');
+}
+
+function durationToSeconds(duration) {
+  if (!/^\d+$/.test(duration)) {
+    throw new TypeError(`Invalid benchmark duration '${duration}'`);
+  }
+  const padded = duration.padStart(10, '0');
+  return `${padded.slice(0, -9)}.${padded.slice(-9)}`;
+}
+
+function runBenchmark(binary, file, options) {
+  const args = [
+    ...options.nodeArgs,
+    '--no-warnings',
+    '--bench',
+    '--bench-reporter=json',
+    '--bench-samples=1',
+    `--bench-warmup=${options.warmup}`,
+  ];
+  if (options.namePattern !== undefined) {
+    args.push(`--bench-name-pattern=${options.namePattern}`);
+  }
+  args.push('--', path.resolve(file));
+
+  return new Promise((resolve, reject) => {
+    const child = spawn(binary, args, {
+      env: process.env,
+      stdio: ['ignore', 'pipe', 'pipe'],
+    });
+    child.stdout.setEncoding('utf8');
+    child.stderr.setEncoding('utf8');
+
+    let stdout = '';
+    let stderr = '';
+    child.stdout.on('data', (data) => { stdout += data; });
+    child.stderr.on('data', (data) => { stderr += data; });
+    child.once('error', reject);
+    child.once('close', (code, signal) => {
+      let records;
+      try {
+        records = stdout.trim().split('\n')
+          .filter((line) => line.length > 0)
+          .map((line) => JSON.parse(line));
+      } catch (error) {
+        reject(new Error(
+          `Could not parse benchmark output from '${binary}': ${error.message}`,
+          { cause: error },
+        ));
+        return;
+      }
+
+      const summary = records.findLast(
+        ({ type }) => type === 'bench:summary')?.data;
+      if (code !== 0 || signal !== null || summary?.success !== true) {
+        const diagnostics = records
+          .filter(({ type }) => type === 'bench:diagnostic')
+          .map(({ data }) => data.message)
+          .join('\n');
+        const status = signal === null ? `exit code ${code}` : `signal ${signal}`;
+        const details = stderr || diagnostics;
+        reject(new Error(
+          `Benchmark '${file}' failed with ${status}` +
+          (details ? `:\n${details}` : ''),
+        ));
+        return;
+      }
+
+      const samples = [];
+      for (const record of records) {
+        if (record.type !== 'bench:complete' ||
+            record.data.skip !== undefined) {
+          continue;
+        }
+        if (record.data.error !== undefined) {
+          reject(new Error(
+            `Benchmark '${record.data.name}' failed: ` +
+            record.data.error.message,
+          ));
+          return;
+        }
+        if (record.data.samples.length !== 1) {
+          reject(new Error(
+            `Benchmark '${record.data.name}' did not produce exactly one sample`,
+          ));
+          return;
+        }
+        const sample = record.data.samples[0];
+        if (!Number.isFinite(sample.rate)) {
+          reject(new Error(
+            `Benchmark '${record.data.name}' produced a non-finite rate`,
+          ));
+          return;
+        }
+        samples.push({
+          configuration: formatConfiguration(record.data.params),
+          duration: durationToSeconds(sample.duration_ns),
+          identity: record.data.benchId,
+          logicalIdentity: JSON.stringify([
+            record.data.file,
+            record.data.parentId,
+            record.data.name,
+          ]),
+          name: record.data.name,
+          params: record.data.params,
+          rate: sample.rate,
+        });
+      }
+      resolve(samples);
+    });
+  });
+}
+
+module.exports = {
+  csvEncode,
+  parseInteger,
+  parseNumber,
+  runBenchmark,
+};

benchmark/buffers/_buffer-compare-offset.node-bench.js

@@ -0,0 +1,37 @@
+'use strict';
+
+const { bench } = require('node:bench');
+const path = require('node:path');
+
+const methods = ['offset', 'slice'];
+const sizes = [16, 512, 4096, 16386];
+const n = 1e6;
+const name = path.join('buffers', 'buffer-compare-offset.js');
+
+function compareUsingSlice(b0, b1, len, iterations) {
+  for (let i = 0; i < iterations; i++)
+    Buffer.compare(b0.slice(1, len), b1.slice(1, len));
+}
+
+function compareUsingOffset(b0, b1, len, iterations) {
+  for (let i = 0; i < iterations; i++)
+    b0.compare(b1, 1, len, 1, len);
+}
+
+for (const method of methods) {
+  for (const size of sizes) {
+    const compare = method === 'slice' ?
+      compareUsingSlice : compareUsingOffset;
+
+    bench(name, {
+      params: { method, n, size },
+    }, (b) => {
+      b.start();
+      compare(Buffer.alloc(size, 'a'),
+              Buffer.alloc(size, 'b'),
+              size >> 1,
+              n);
+      b.end(n);
+    });
+  }
+}

benchmark/compare-node-bench.js

@@ -0,0 +1,116 @@
+'use strict';
+
+const path = require('node:path');
+const CLI = require('./_cli.js');
+const { analyzeCompare } = require('./_node-bench-analysis.js');
+const {
+  csvEncode,
+  parseInteger,
+  parseNumber,
+  runBenchmark,
+} = require('./_node-bench.js');
+
+const cli = new CLI(`usage: ./node compare-node-bench.js [options] [--] <file> ...
+  Run explicit node:bench files repeatedly with two Node.js binaries. Each
+  observation runs in a fresh process. Output is compatible with compare.R,
+  or --analyze can summarize it directly.
+
+  --new          binary     new Node.js binary (required)
+  --old          binary     old Node.js binary (required)
+  --runs         30         observations per binary
+  --warmup       0          warmup samples before each observation
+  --name-pattern pattern    only run matching benchmarks
+  --node-arg     argument   pass an argument to both binaries (repeatable)
+  --analyze                 analyze with Welch's t-test instead of writing CSV
+  --scale        1000       rate multiplier used for histogram precision
+  --max-regression N        fail if a family-wise significant regression's
+                            95% confidence interval is entirely beyond N%
+                            (implies --analyze)
+`, { arrayArgs: ['node-arg'], boolArgs: ['analyze'] });
+
+if (!cli.optional.new || !cli.optional.old || cli.items.length === 0) {
+  cli.abort(cli.usage);
+}
+
+async function main() {
+  const runs = parseInteger(cli.optional.runs, 30, '--runs', 1);
+  const warmup = parseInteger(cli.optional.warmup, 0, '--warmup', 0);
+  const scale = parseInteger(cli.optional.scale, 1000, '--scale', 1);
+  const maxRegression = parseNumber(
+    cli.optional['max-regression'], 0, '--max-regression', 0);
+  const analyze = !!cli.optional.analyze || maxRegression > 0;
+  const options = {
+    namePattern: cli.optional['name-pattern'],
+    nodeArgs: cli.optional['node-arg'],
+    warmup,
+  };
+  const binaries = [
+    { label: 'old', path: cli.optional.old },
+    { label: 'new', path: cli.optional.new },
+  ];
+  const rows = [];
+  const counts = new Map();
+  const csvGroups = new Map();
+
+  for (const file of cli.items) {
+    const resolved = path.resolve(file);
+    for (let run = 0; run < runs; run++) {
+      const order = run % 2 === 0 ? binaries : [binaries[1], binaries[0]];
+      for (const binary of order) {
+        const samples = await runBenchmark(binary.path, resolved, options);
+        for (const sample of samples) {
+          const identity = JSON.stringify([resolved, sample.identity]);
+          const csvGroup = JSON.stringify([
+            sample.name,
+            sample.configuration,
+          ]);
+          const groupedIdentity = csvGroups.get(csvGroup);
+          if (groupedIdentity !== undefined && groupedIdentity !== identity) {
+            throw new Error(
+              `Distinct benchmarks would share the CSV group '${sample.name} ` +
+              `${sample.configuration}'`,
+            );
+          }
+          csvGroups.set(csvGroup, identity);
+          let count = counts.get(identity);
+          if (count === undefined) {
+            count = { name: sample.name, new: 0, old: 0 };
+            counts.set(identity, count);
+          }
+          count[binary.label]++;
+          rows.push({ binary: binary.label, ...sample });
+        }
+      }
+    }
+  }
+
+  if (rows.length === 0) {
+    throw new Error('No benchmark samples were produced');
+  }
+  for (const count of counts.values()) {
+    if (count.old !== runs || count.new !== runs) {
+      throw new Error(
+        `Benchmark '${count.name}' was not reported by both binaries in every run`,
+      );
+    }
+  }
+
+  if (analyze) {
+    const result = analyzeCompare(rows, scale, maxRegression);
+    process.stdout.write(result.output);
+    if (result.failed) process.exitCode = 1;
+    return;
+  }
+
+  const output = ['"binary","filename","configuration","rate","time"'];
+  for (const row of rows) {
+    output.push(`${csvEncode(row.binary)},${csvEncode(row.name)},` +
+                `${csvEncode(row.configuration)},${row.rate},${row.duration}`);
+  }
+  process.stdout.write(`${output.join('\n')}\n`);
+}
+
+main().catch((error) => {
+  console.error(error.stack);
+  process.exitCode = 1;
+});

benchmark/crypto/_create-hash.node-bench.js

@@ -0,0 +1,19 @@
+'use strict';
+
+const assert = require('node:assert');
+const { bench } = require('node:bench');
+const { createHash } = require('node:crypto');
+const path = require('node:path');
+
+const n = 1e5;
+const name = path.join('crypto', 'create-hash.js');
+
+bench(name, { params: { n } }, (b) => {
+  const array = [];
+  for (let i = 0; i < n; ++i) array.push(null);
+  b.start();
+  for (let i = 0; i < n; ++i)
+    array[i] = createHash('sha1');
+  b.end(n);
+  assert.strictEqual(typeof array[n - 1], 'object');
+});

benchmark/scatter-node-bench.js

@@ -0,0 +1,131 @@
+'use strict';
+
+const CLI = require('./_cli.js');
+const {
+  analyzeScatter,
+  validateScatterParameters,
+} = require('./_node-bench-analysis.js');
+const {
+  csvEncode,
+  parseInteger,
+  runBenchmark,
+} = require('./_node-bench.js');
+
+const cli = new CLI(`usage: ./node scatter-node-bench.js [options] [--] <file>
+  Run an explicit node:bench file repeatedly and output each independent
+  observation with its benchmark parameters as CSV, or summarize the results
+  directly with --analyze.
+
+  --node         ./node     Node.js binary
+  --runs         30         number of observations
+  --warmup       0          warmup samples before each observation
+  --name-pattern pattern    only run matching benchmarks
+  --node-arg     argument   pass an argument to the binary (repeatable)
+  --analyze                 print a statistical summary instead of CSV
+  --xaxis        parameter  parameter to group by with --analyze (required)
+  --category     parameter  optional second grouping parameter
+  --no-chart                omit the analysis bar chart
+`, {
+  arrayArgs: ['node-arg'],
+  boolArgs: ['analyze', 'no-chart'],
+});
+
+if (cli.items.length !== 1) cli.abort(cli.usage);
+if (cli.optional.analyze && cli.optional.xaxis === undefined) {
+  cli.abort('--analyze requires --xaxis <parameter>');
+}
+
+async function main() {
+  const runs = parseInteger(cli.optional.runs, 30, '--runs', 1);
+  const warmup = parseInteger(cli.optional.warmup, 0, '--warmup', 0);
+  const options = {
+    namePattern: cli.optional['name-pattern'],
+    nodeArgs: cli.optional['node-arg'],
+    warmup,
+  };
+  const binary = cli.optional.node || process.execPath;
+  const rows = [];
+  const paramNames = new Set();
+  const csvGroups = new Map();
+  let expectedIdentities;
+  let logicalIdentity;
+
+  for (let run = 0; run < runs; run++) {
+    const samples = await runBenchmark(binary, cli.items[0], options);
+    if (run === 0 && cli.optional.analyze) {
+      validateScatterParameters(
+        samples, cli.optional.xaxis, cli.optional.category);
+    }
+    const identities = new Set();
+    for (const sample of samples) {
+      if (logicalIdentity !== undefined &&
+          logicalIdentity !== sample.logicalIdentity) {
+        throw new Error(
+          'scatter-node-bench.js requires one logical benchmark name per file',
+        );
+      }
+      logicalIdentity = sample.logicalIdentity;
+      if (identities.has(sample.identity)) {
+        throw new Error(`Benchmark '${sample.name}' was reported more than once`);
+      }
+      identities.add(sample.identity);
+      const csvGroup = JSON.stringify([sample.name, sample.params]);
+      const groupedIdentity = csvGroups.get(csvGroup);
+      if (groupedIdentity !== undefined &&
+          groupedIdentity !== sample.identity) {
+        throw new Error(
+          `Distinct benchmarks would share the CSV group '${sample.name}'`,
+        );
+      }
+      csvGroups.set(csvGroup, sample.identity);
+      rows.push({ ...sample, observation: run });
+      for (const name of Object.keys(sample.params)) paramNames.add(name);
+    }
+    if (expectedIdentities === undefined) {
+      expectedIdentities = identities;
+    } else if (identities.size !== expectedIdentities.size ||
+               ![...identities].every((id) => expectedIdentities.has(id))) {
+      throw new Error('The set of reported benchmarks changed between runs');
+    }
+  }
+  if (rows.length === 0) {
+    throw new Error('No benchmark samples were produced');
+  }
+
+  const params = [...paramNames].sort();
+  if (cli.optional.analyze) {
+    const output = analyzeScatter(
+      rows,
+      cli.optional.xaxis,
+      cli.optional.category,
+      !cli.optional['no-chart'],
+    );
+    process.stdout.write(output);
+    return;
+  }
+
+  for (const name of params) {
+    if (name === 'filename' || name === 'rate' || name === 'time') {
+      throw new Error(`Benchmark parameter '${name}' is reserved in scatter CSV`);
+    }
+  }
+  const header = ['filename', ...params, 'rate', 'time']
+    .map(csvEncode)
+    .join(',');
+  const output = [header];
+  for (const row of rows) {
+    const values = [
+      csvEncode(row.name),
+      ...params.map((name) => csvEncode(row.params[name] ?? '')),
+      row.rate,
+      row.duration,
+    ];
+    output.push(values.join(','));
+  }
+  process.stdout.write(`${output.join('\n')}\n`);
+}
+
+main().catch((error) => {
+  console.error(error.stack);
+  process.exitCode = 1;
+});

doc/api/bench.md

@@ -6,7 +6,7 @@
 added: REPLACEME
 -->
 
-> Stability: 1 - Experimental
+> Stability: 1.0 - Early Development
 
 <!-- source_link=lib/bench.js -->
 

doc/contributing/writing-and-running-benchmarks.md

@@ -17,6 +17,7 @@
     * [Using `--analyze` (no external tools needed)](#using---analyze-no-external-tools-needed)
     * [Using R scripts or node-benchmark-compare](#using-r-scripts-or-node-benchmark-compare)
   * [Comparing parameters](#comparing-parameters)
+  * [Evaluating `node:bench` ports](#evaluating-nodebench-ports)
   * [Running benchmarks on the CI](#running-benchmarks-on-the-ci)
 * [Creating a benchmark](#creating-a-benchmark)
   * [Basics of a benchmark](#basics-of-a-benchmark)
@@ -706,6 +707,41 @@ chunkLen     encoding      rate confidence.interval
 
 ![compare tool boxplot](doc_img/scatter-plot.png)
 
+### Evaluating `node:bench` ports
+
+The experimental `compare-node-bench.js` and `scatter-node-bench.js` tools are
+parallel versions of the existing tools for explicit `node:bench` files. They
+do not modify or replace the legacy benchmark framework. Each repeated
+observation for a benchmark identity uses one measured sample from a separate
+process invocation. Configurations declared in the same file still execute
+serially in that process, unlike the legacy framework's configuration-level
+process isolation, and can share runtime state.
+
+Both parallel tools support inline analysis. `scatter-node-bench.js --analyze`
+uses the same `--xaxis`, `--category`, and `--no-chart` interface described for
+`scatter.js`. `compare-node-bench.js --analyze` performs Welch's t-test, while
+`--max-regression N` adds a corrected regression gate. The gate requires both a
+Holm-Bonferroni-adjusted p-value below 0.05 and a 95% confidence interval lying
+entirely beyond `-N%`; the point estimate alone cannot fail the command.
+Scatter analysis reduces aggregated configurations to one value per outer
+process and uses disjoint process sets for consecutive Mann-Whitney comparisons
+so configurations sharing a process are not treated as independent samples.
+
+Underscore-prefixed ports are kept beside selected legacy benchmarks and are
+excluded from legacy discovery. For example:
+
+```console
+./node benchmark/scatter.js --runs 30 \
+  benchmark/crypto/create-hash.js > legacy.csv
+./node benchmark/scatter-node-bench.js --runs 30 -- \
+  benchmark/crypto/_create-hash.node-bench.js > node-bench.csv
+```
+
+The port uses the legacy relative filename as its benchmark name and preserves
+the same parameter names. The two CSV files can therefore be analyzed with the
+same scripts to check whether their rate distributions and measurement units
+agree. See [`benchmark/README.md`][] for compare and scatter examples.
+
 ### Running benchmarks on the CI
 
 To see the performance impact of a pull request by running benchmarks on
@@ -889,6 +925,7 @@ Supported options keys are:
 
 [Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Effect_size_for_ordinal_data
 [Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
+[`benchmark/README.md`]: ../../benchmark/README.md#nodebench-evaluation-tools
 [autocannon]: https://github.com/mcollina/autocannon
 [benchmark-ci]: https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
 [git-for-windows]: https://git-scm.com/download/win

test/fixtures/bench-runner/tools-collision.cjs

@@ -0,0 +1,14 @@
+'use strict';
+
+const { bench, suite } = require('node:bench');
+
+function register(name) {
+  bench(name, { params: { size: 1 } }, (b) => {
+    b.start();
+    process.hrtime.bigint();
+    b.end(1);
+  });
+}
+
+suite('first', () => register('same'));
+suite('second', () => register('same'));

test/fixtures/bench-runner/tools-no-params.cjs

@@ -0,0 +1,9 @@
+'use strict';
+
+const { bench } = require('node:bench');
+
+bench('tools/no-params.js', (b) => {
+  b.start();
+  process.hrtime.bigint();
+  b.end(1);
+});

test/fixtures/bench-runner/tools-reserved-param.cjs

@@ -0,0 +1,9 @@
+'use strict';
+
+const { bench } = require('node:bench');
+
+bench('tools/reserved.js', { params: { rate: 'parameter' } }, (b) => {
+  b.start();
+  process.hrtime.bigint();
+  b.end(1);
+});

test/fixtures/bench-runner/tools.cjs

@@ -0,0 +1,20 @@
+'use strict';
+
+const { bench } = require('node:bench');
+
+if (process.env.NODE_BENCH_PID_LOG !== undefined) {
+  require('fs').appendFileSync(
+    process.env.NODE_BENCH_PID_LOG, `${process.pid}\n`);
+}
+
+for (const size of [1, 2]) {
+  bench('tools/simple.js', {
+    params: { method: 'loop', size },
+  }, (b) => {
+    let value = 0;
+    b.start();
+    for (let i = 0; i < 1_000; i++) value += size;
+    b.end(1_000);
+    if (value === 0) throw new Error('unreachable');
+  });
+}

test/parallel/test-benchmark-node-bench-tools.js

@@ -0,0 +1,261 @@
+// Flags: --no-warnings
+'use strict';
+
+require('../common');
+const assert = require('assert');
+const { spawnSync } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+const fixtures = require('../common/fixtures');
+const tmpdir = require('../common/tmpdir');
+const {
+  analyzeScatter,
+  holmAdjust,
+  isRegressionFailure,
+} = require('../../benchmark/_node-bench-analysis.js');
+const { csvEncode } = require('../../benchmark/_node-bench.js');
+
+const compare = path.resolve(__dirname, '../../benchmark/compare-node-bench.js');
+const legacyScatter = path.resolve(__dirname, '../../benchmark/scatter.js');
+const scatter = path.resolve(__dirname, '../../benchmark/scatter-node-bench.js');
+const benchmark = fixtures.path('bench-runner/tools.cjs');
+
+tmpdir.refresh();
+
+assert.strictEqual(csvEncode(true), 'true');
+assert.deepStrictEqual(holmAdjust([0.01, 0.03, 0.04]), [0.03, 0.06, 0.06]);
+assert.strictEqual(isRegressionFailure({
+  ci95: 3,
+  improvement: -12,
+  pAdjusted: 0.01,
+}, 10), false);
+assert.strictEqual(isRegressionFailure({
+  ci95: 1,
+  improvement: -12,
+  pAdjusted: 0.06,
+}, 10), false);
+assert.strictEqual(isRegressionFailure({
+  ci95: 1,
+  improvement: -12,
+  pAdjusted: 0.01,
+}, 10), true);
+assert.throws(
+  () => analyzeScatter([{
+    observation: 0,
+    params: { size: 1 },
+    rate: 1,
+  }], 'size', 'size', false),
+  /must name different parameters/,
+);
+assert.doesNotMatch(analyzeScatter([0, 1].map((observation) => ({
+  observation,
+  params: { size: 1 },
+  rate: 1_234_567.89,
+})), 'size', undefined, false), /\(!\)/);
+assert.match(analyzeScatter([
+  { observation: 0, params: { method: 'a', size: 1 }, rate: 10 },
+  { observation: 0, params: { method: 'b', size: 1 }, rate: 20 },
+  { observation: 1, params: { method: 'a', size: 1 }, rate: 30 },
+  { observation: 1, params: { method: 'b', size: 1 }, rate: 50 },
+], 'size', undefined, false), /\n\s*1\s+2\s+/);
+
+function run(script, args, options = undefined) {
+  return spawnSync(process.execPath, [script, ...args], {
+    encoding: 'utf8',
+    timeout: 30_000,
+    ...options,
+  });
+}
+
+{
+  const pidLog = tmpdir.resolve('pids');
+  const result = run(compare, [
+    '--old', process.execPath,
+    '--new', process.execPath,
+    '--runs', '2',
+    '--', benchmark,
+  ], {
+    env: { __proto__: null, ...process.env, NODE_BENCH_PID_LOG: pidLog },
+  });
+  assert.strictEqual(result.status, 0, result.stderr);
+  assert.strictEqual(result.stderr, '');
+  const lines = result.stdout.trim().split('\n');
+  assert.strictEqual(lines[0],
+                     '"binary","filename","configuration","rate","time"');
+  assert.strictEqual(lines.length, 9);
+  assert.strictEqual(lines.filter((line) => line.startsWith('"old",')).length,
+                     4);
+  assert.strictEqual(lines.filter((line) => line.startsWith('"new",')).length,
+                     4);
+  assert.deepStrictEqual(lines.slice(1).map((line) => line.slice(0, 5)), [
+    '"old"', '"old"', '"new"', '"new"',
+    '"new"', '"new"', '"old"', '"old"',
+  ]);
+  assert(lines.slice(1).every(
+    (line) => line.includes('"tools/simple.js"')));
+  const pids = fs.readFileSync(pidLog, 'utf8').trim().split('\n');
+  assert.strictEqual(new Set(pids).size, 4);
+}
+
+{
+  const result = run(scatter, [
+    '--node', process.execPath,
+    '--runs', '2',
+    '--', benchmark,
+  ]);
+  assert.strictEqual(result.status, 0, result.stderr);
+  assert.strictEqual(result.stderr, '');
+  const lines = result.stdout.trim().split('\n');
+  assert.strictEqual(lines[0],
+                     '"filename","method","size","rate","time"');
+  assert.strictEqual(lines.length, 5);
+  assert(lines.slice(1).every(
+    (line) => line.startsWith('"tools/simple.js","loop",')));
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '1',
+    '--', fixtures.path('bench-runner/tools-no-params.cjs'),
+  ]);
+  assert.strictEqual(result.status, 0, result.stderr);
+  const lines = result.stdout.trim().split('\n');
+  assert.strictEqual(lines[0], '"filename","rate","time"');
+  assert.strictEqual(lines.length, 2);
+  assert.strictEqual(lines[1].split(',').length, 3);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '1',
+    '--', fixtures.path('bench-runner/tools-reserved-param.cjs'),
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /parameter 'rate' is reserved/);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '1',
+    '--', fixtures.path('bench-runner/tools-collision.cjs'),
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /requires one logical benchmark name per file/);
+}
+
+{
+  const result = run(compare, [
+    '--old', process.execPath,
+    '--new', process.execPath,
+    '--runs', '1',
+    '--', fixtures.path('bench-runner/tools-collision.cjs'),
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /Distinct benchmarks would share the CSV group/);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '2',
+    '--', fixtures.path('bench-runner/a.cjs'),
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /set of reported benchmarks changed between runs/);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '1',
+    '--name-pattern', 'missing',
+    '--', benchmark,
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /No benchmark samples were produced/);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', 'invalid',
+    '--', benchmark,
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /--runs must be an integer/);
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '2',
+    '--analyze',
+    '--xaxis', 'size',
+    '--category', 'method',
+    '--no-chart',
+    '--', benchmark,
+  ]);
+  assert.strictEqual(result.status, 0, result.stderr);
+  assert.strictEqual(result.stderr, '');
+  assert.match(result.stdout,
+               /size\s+method\s+samples\s+rate\s+confidence\.interval/);
+  assert.match(result.stdout, /Change between consecutive size values/);
+  assert.match(result.stdout, /Mann-Whitney U.*Cliff's delta/);
+  assert.doesNotMatch(result.stdout, /"filename","method"/);
+}
+
+{
+  const result = run(scatter, [
+    '--analyze',
+    '--', benchmark,
+  ]);
+  assert.strictEqual(result.status, 1);
+  assert.match(result.stderr, /--analyze requires --xaxis/);
+}
+
+{
+  const result = run(compare, [
+    '--old', process.execPath,
+    '--new', process.execPath,
+    '--runs', '2',
+    '--max-regression', '100',
+    '--', fixtures.path('bench-runner/tools-no-params.cjs'),
+  ]);
+  assert.strictEqual(result.status, 0, result.stderr);
+  assert.strictEqual(result.stderr, '');
+  assert.match(result.stdout, /confidence\s+improvement\s+accuracy/);
+  assert.match(result.stdout, /Holm-Bonferroni correction/);
+  assert.match(result.stdout, /--max-regression uses the corrected values/);
+  assert.doesNotMatch(result.stdout, /"binary","filename"/);
+}
+
+{
+  const legacy = run(legacyScatter, [
+    '--runs', '1',
+    path.resolve(__dirname, '../../benchmark/crypto/create-hash.js'),
+  ]);
+  const modern = run(scatter, [
+    '--runs', '1',
+    '--', path.resolve(
+      __dirname, '../../benchmark/crypto/_create-hash.node-bench.js'),
+  ]);
+  assert.strictEqual(legacy.status, 0, legacy.stderr);
+  assert.strictEqual(modern.status, 0, modern.stderr);
+  const legacyLines = legacy.stdout.trim().split('\n');
+  const modernLines = modern.stdout.trim().split('\n');
+  assert.strictEqual(legacyLines[0].replaceAll(' ', ''), modernLines[0]);
+  const name = path.join('crypto', 'create-hash.js');
+  assert(legacyLines[1].startsWith(`"${name}",`));
+  assert(modernLines[1].startsWith(`"${name}",`));
+}
+
+{
+  const result = run(scatter, [
+    '--runs', '1',
+    '--', path.resolve(
+      __dirname,
+      '../../benchmark/buffers/_buffer-compare-offset.node-bench.js',
+    ),
+  ]);
+  assert.strictEqual(result.status, 0, result.stderr);
+  const lines = result.stdout.trim().split('\n');
+  assert.strictEqual(lines[0],
+                     '"filename","method","n","size","rate","time"');
+  assert.strictEqual(lines.length, 9);
+}