Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: https://github.com/nodejs/node/pull/65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This adds a new `test` option. Using it automatically uses a single
minimal option matrix to verify the benchmark works as expected.
Using the new `all` option makes sure all test suites are run.
On top of that the benchmarks will from now on report properly
what category might have a typo, if any.
The http duration was also refactored to use a option instead of
relying on a configuration setting.
The fixture folder is ignored as test suite from now on.
PR-URL: https://github.com/nodejs/node/pull/31396
Fixes: https://github.com/nodejs/node/issues/31083
Reviewed-By: Sam Roberts <vieuxtech@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
* Let users provide more than one pattern by repeating the flag
* Add new flag --exclude to exclude patterns
* Add tests for --filter
* Document --filter
This commit also fixes a bug where things like
`compare.js --new --old binary --new binary` was acceptable (now the
script will exit and print the usage message).
PR-URL: https://github.com/nodejs/node/pull/29987
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Denys Otrishko <shishugi@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Code in benchmark directory sometimes uses `function () {}` for
anonymous callbacks and sometimes uses `() => {}`. Multi-line arrays
sometimes have a trailing comma and sometimes do not. Update to always
use arrow functions for anonymous callbacks and trailing commas for
multiline arrays.
PR-URL: https://github.com/nodejs/node/pull/25944
Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This changes the execution order from "iter, file, binary" to "file,
iter, binary". This means the csv no longer has to buffered completely.
This also has the added effect that stopping compare.js early or
interfering with performance only affects a single benchmark, instead of
all of them.
Refs: https://github.com/nodejs/node/issues/8659
PR-URL: https://github.com/nodejs/node/pull/9064
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
In order to comply with linting rules used in the rest of the code base,
eliminate redeclared variables. A conservative approach is used so as to
avoid unintentional performance issues (for example, as might be seen in
some situations when using `let` instead of `var`).
PR-URL: https://github.com/nodejs/node/pull/5468
Reviewed-By: Brian White <mscdex@mscdex.net>
Before this commit, only benchmark targets defined in Makefile could
be used. This commit allows execution of common.js directly and
passing of filter arguments directly, allowing you to run either a
subset of benchmarks or a single specific benchmark for comparison.
PR-URL: https://github.com/iojs/io.js/pull/711
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
The benchmark compare would drop the last run of the binary pairs. So
when they were only run once an error would arise because no data was
generated for the second binary.
This will run the benchmarks the number of times specified by NODE_BENCH_RUNS,
to attempt to reduce variability.
If the number of runs is high enough, it'll also throw out the top and bottom
quartiles, since that's where the outliers will be.
It's not very fancy statistics-fu, but it's better than nothing.
Also, linted this file. It had tabs in it. TABS!
James M Snell