mirror of
https://github.com/nodejs/node.git
synced 2026-10-28 10:15:07 +08:00
test_runner: support test order randomization
PR-URL: https://github.com/nodejs/node/pull/61747 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
This commit is contained in:
Pietro Marchini authored and GitHub committed
1 parent
25443db104
commit
5ff1eab951
28 files changed
+1557
-21
No files matched your search
@@ -2784,6 +2784,38 @@ changes:
|
||||
Configures the test runner to only execute top level tests that have the `only`
|
||||
option set. This flag is not necessary when test isolation is disabled.
|
||||
|
||||
### `--test-random-seed`
|
||||
|
||||
<!-- YAML
|
||||
added: REPLACEME
|
||||
-->
|
||||
|
||||
Set the seed used to randomize test execution order. This applies to both test
|
||||
file execution order and queued tests within each file. Providing this flag
|
||||
enables randomization implicitly, even without `--test-randomize`.
|
||||
|
||||
The value must be an integer between `0` and `4294967295`.
|
||||
|
||||
This flag cannot be used with `--watch` or `--test-rerun-failures`.
|
||||
|
||||
### `--test-randomize`
|
||||
|
||||
<!-- YAML
|
||||
added: REPLACEME
|
||||
-->
|
||||
|
||||
Randomize test execution order. This applies to both test file execution order
|
||||
and queued tests within each file. This can help detect tests that rely on
|
||||
shared state or execution order.
|
||||
|
||||
The seed used for randomization is printed in the test summary and can be
|
||||
reused with `--test-random-seed`.
|
||||
|
||||
For detailed behavior and examples, see
|
||||
[randomizing tests execution order][].
|
||||
|
||||
This flag cannot be used with `--watch` or `--test-rerun-failures`.
|
||||
|
||||
### `--test-reporter`
|
||||
|
||||
<!-- YAML
|
||||
@@ -3702,6 +3734,8 @@ one is included in the list below.
|
||||
* `--test-isolation`
|
||||
* `--test-name-pattern`
|
||||
* `--test-only`
|
||||
* `--test-random-seed`
|
||||
* `--test-randomize`
|
||||
* `--test-reporter-destination`
|
||||
* `--test-reporter`
|
||||
* `--test-rerun-failures`
|
||||
@@ -4286,6 +4320,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
|
||||
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
|
||||
[module compile cache]: module.md#module-compile-cache
|
||||
[preloading asynchronous module customization hooks]: module.md#registration-of-asynchronous-customization-hooks
|
||||
[randomizing tests execution order]: test.md#randomizing-tests-execution-order
|
||||
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
|
||||
[running tests from the command line]: test.md#running-tests-from-the-command-line
|
||||
[scavenge garbage collector]: https://v8.dev/blog/orinoco-parallel-scavenger
|
||||
|
||||
+100
@@ -627,6 +627,94 @@ prevent shell expansion, which can reduce portability across systems.
|
||||
node --test "**/*.test.js" "**/*.spec.js"
|
||||
```
|
||||
|
||||
### Randomizing tests execution order
|
||||
|
||||
<!-- YAML
|
||||
added: REPLACEME
|
||||
-->
|
||||
|
||||
> Stability: 1.0 - Early development
|
||||
|
||||
The test runner can randomize execution order to help detect
|
||||
order-dependent tests. When enabled, the runner randomizes both discovered
|
||||
test files and queued tests within each file. Use `--test-randomize` to
|
||||
enable this mode.
|
||||
|
||||
```bash
|
||||
node --test --test-randomize
|
||||
```
|
||||
|
||||
When randomization is enabled, the test runner prints the seed used for the run
|
||||
as a diagnostic message:
|
||||
|
||||
```text
|
||||
Randomized test order seed: 12345
|
||||
```
|
||||
|
||||
Use `--test-random-seed=<number>` to replay the same randomized order
|
||||
deterministically. Supplying `--test-random-seed` also enables randomization,
|
||||
so `--test-randomize` is optional when a seed is provided:
|
||||
|
||||
```bash
|
||||
node --test --test-random-seed=12345
|
||||
```
|
||||
|
||||
In most test files, randomization works automatically. One important exception
|
||||
is when subtests are awaited one by one. In that pattern, each subtest starts
|
||||
only after the previous one finishes, so the runner keeps declaration order
|
||||
instead of randomizing it.
|
||||
|
||||
Example: this runs sequentially and is **not** randomized.
|
||||
|
||||
```mjs
|
||||
import test from 'node:test';
|
||||
|
||||
test('math', async (t) => {
|
||||
for (const name of ['adds', 'subtracts', 'multiplies']) {
|
||||
// Sequentially awaiting each subtest preserves declaration order.
|
||||
await t.test(name, async () => {});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
```cjs
|
||||
const test = require('node:test');
|
||||
|
||||
test('math', async (t) => {
|
||||
for (const name of ['adds', 'subtracts', 'multiplies']) {
|
||||
// Sequentially awaiting each subtest preserves declaration order.
|
||||
await t.test(name, async () => {});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
|
||||
still allows randomization, because sibling tests are enqueued together.
|
||||
|
||||
Example: this remains eligible for randomization.
|
||||
|
||||
```mjs
|
||||
import { describe, it } from 'node:test';
|
||||
|
||||
describe('math', () => {
|
||||
it('adds', () => {});
|
||||
it('subtracts', () => {});
|
||||
it('multiplies', () => {});
|
||||
});
|
||||
```
|
||||
|
||||
```cjs
|
||||
const { describe, it } = require('node:test');
|
||||
|
||||
describe('math', () => {
|
||||
it('adds', () => {});
|
||||
it('subtracts', () => {});
|
||||
it('multiplies', () => {});
|
||||
});
|
||||
```
|
||||
|
||||
`--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
|
||||
|
||||
Matching files are executed as test files.
|
||||
More information on the test file execution can be found
|
||||
in the [test runner execution model][] section.
|
||||
@@ -667,6 +755,10 @@ test runner functionality:
|
||||
* `--test-reporter` - Reporting is managed by the parent process
|
||||
* `--test-reporter-destination` - Output destinations are controlled by the parent
|
||||
* `--experimental-config-file` - Config file paths are managed by the parent
|
||||
* `--test-randomize` - Randomization is managed by the parent process and
|
||||
propagated to child processes
|
||||
* `--test-random-seed` - Randomization seed is managed by the parent process and
|
||||
propagated to child processes
|
||||
|
||||
All other Node.js options from command line arguments, environment variables,
|
||||
and configuration files are inherited by the child processes.
|
||||
@@ -1575,6 +1667,14 @@ changes:
|
||||
that specifies the index of the shard to run. This option is _required_.
|
||||
* `total` {number} is a positive integer that specifies the total number
|
||||
of shards to split the test files to. This option is _required_.
|
||||
* `randomize` {boolean} Randomize execution order for test files and queued tests.
|
||||
This option is not supported with `watch: true`.
|
||||
**Default:** `false`.
|
||||
* `randomSeed` {number} Seed used when randomizing execution order. If this
|
||||
option is set, runs can replay the same randomized order deterministically,
|
||||
and setting this option also enables randomization. The value must be an
|
||||
integer between `0` and `4294967295`.
|
||||
**Default:** `undefined`.
|
||||
* `rerunFailuresFilePath` {string} A file path where the test runner will
|
||||
store the state of the tests to allow rerunning only the failed tests on a next run.
|
||||
see \[Rerunning failed tests]\[] for more information.
|
||||
|
||||
@@ -524,6 +524,14 @@
|
||||
"type": "boolean",
|
||||
"description": "run tests with 'only' option set"
|
||||
},
|
||||
"test-random-seed": {
|
||||
"type": "number",
|
||||
"description": "seed used to randomize test execution order"
|
||||
},
|
||||
"test-randomize": {
|
||||
"type": "boolean",
|
||||
"description": "run tests in a random order"
|
||||
},
|
||||
"test-reporter": {
|
||||
"oneOf": [
|
||||
{
|
||||
@@ -906,6 +914,14 @@
|
||||
"type": "boolean",
|
||||
"description": "run tests with 'only' option set"
|
||||
},
|
||||
"test-random-seed": {
|
||||
"type": "number",
|
||||
"description": "seed used to randomize test execution order"
|
||||
},
|
||||
"test-randomize": {
|
||||
"type": "boolean",
|
||||
"description": "run tests in a random order"
|
||||
},
|
||||
"test-reporter": {
|
||||
"oneOf": [
|
||||
{
|
||||
|
||||
+20
@@ -1363,6 +1363,22 @@ tests must satisfy \fBboth\fR requirements in order to be executed.
|
||||
Configures the test runner to only execute top level tests that have the \fBonly\fR
|
||||
option set. This flag is not necessary when test isolation is disabled.
|
||||
.
|
||||
.It Fl -test-random-seed
|
||||
Set the seed used to randomize test execution order.
|
||||
This applies to both test file execution order and queued tests within each file.
|
||||
Providing this flag enables randomization implicitly, even without
|
||||
\fB--test-randomize\fR.
|
||||
The value must be an integer between 0 and 4294967295.
|
||||
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
|
||||
.
|
||||
.It Fl -test-randomize
|
||||
Randomize test execution order.
|
||||
This applies to both test file execution order and queued tests within each file.
|
||||
This can help detect tests that rely on shared state or execution order.
|
||||
The seed used for randomization is printed in the test summary and can be
|
||||
reused with \fB--test-random-seed\fR.
|
||||
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
|
||||
.
|
||||
.It Fl -test-reporter
|
||||
A test reporter to use when running tests. See the documentation on
|
||||
test reporters for more details.
|
||||
@@ -2040,6 +2056,10 @@ one is included in the list below.
|
||||
.It
|
||||
\fB--test-reporter-destination\fR
|
||||
.It
|
||||
\fB--test-randomize\fR
|
||||
.It
|
||||
\fB--test-random-seed\fR
|
||||
.It
|
||||
\fB--test-reporter\fR
|
||||
.It
|
||||
\fB--test-rerun-failures\fR
|
||||
|
||||
@@ -59,6 +59,7 @@ const {
|
||||
validateObject,
|
||||
validateOneOf,
|
||||
validateInteger,
|
||||
validateUint32,
|
||||
validateString,
|
||||
validateStringArray,
|
||||
} = require('internal/validators');
|
||||
@@ -84,6 +85,7 @@ const {
|
||||
const { FastBuffer } = require('internal/buffer');
|
||||
|
||||
const {
|
||||
createRandomSeed,
|
||||
convertStringToRegExp,
|
||||
countCompletedTest,
|
||||
kDefaultPattern,
|
||||
@@ -105,12 +107,14 @@ const kIsolatedProcessName = Symbol('kIsolatedProcessName');
|
||||
const kFilterArgs = [
|
||||
'--test',
|
||||
'--experimental-test-coverage',
|
||||
'--test-randomize',
|
||||
'--watch',
|
||||
'--experimental-default-config-file',
|
||||
];
|
||||
const kFilterArgValues = [
|
||||
'--test-reporter',
|
||||
'--test-reporter-destination',
|
||||
'--test-random-seed',
|
||||
'--experimental-config-file',
|
||||
];
|
||||
const kDiagnosticsFilterArgs = ['tests', 'suites', 'pass', 'fail', 'cancelled', 'skipped', 'todo', 'duration_ms'];
|
||||
@@ -168,6 +172,8 @@ function getRunArgs(path, { forceExit,
|
||||
argv: suppliedArgs,
|
||||
execArgv,
|
||||
rerunFailuresFilePath,
|
||||
randomize,
|
||||
randomSeed,
|
||||
root: { timeout },
|
||||
cwd }) {
|
||||
const processNodeOptions = getOptionsAsFlagsFromBinding();
|
||||
@@ -209,6 +215,12 @@ function getRunArgs(path, { forceExit,
|
||||
if (rerunFailuresFilePath) {
|
||||
ArrayPrototypePush(runArgs, `--test-rerun-failures=${rerunFailuresFilePath}`);
|
||||
}
|
||||
if (randomize) {
|
||||
ArrayPrototypePush(runArgs, '--test-randomize');
|
||||
}
|
||||
if (randomSeed != null) {
|
||||
ArrayPrototypePush(runArgs, `--test-random-seed=${randomSeed}`);
|
||||
}
|
||||
|
||||
ArrayPrototypePushApply(runArgs, execArgv);
|
||||
|
||||
@@ -646,6 +658,8 @@ function run(options = kEmptyObject) {
|
||||
lineCoverage = 0,
|
||||
branchCoverage = 0,
|
||||
functionCoverage = 0,
|
||||
randomize: suppliedRandomize,
|
||||
randomSeed: suppliedRandomSeed,
|
||||
execArgv = [],
|
||||
argv = [],
|
||||
cwd = process.cwd(),
|
||||
@@ -674,6 +688,56 @@ function run(options = kEmptyObject) {
|
||||
if (globPatterns != null) {
|
||||
validateArray(globPatterns, 'options.globPatterns');
|
||||
}
|
||||
if (suppliedRandomize != null) {
|
||||
validateBoolean(suppliedRandomize, 'options.randomize');
|
||||
}
|
||||
if (suppliedRandomSeed != null) {
|
||||
validateUint32(suppliedRandomSeed, 'options.randomSeed');
|
||||
}
|
||||
let randomize = suppliedRandomize;
|
||||
let randomSeed = suppliedRandomSeed;
|
||||
|
||||
if (randomSeed != null) {
|
||||
randomize = true;
|
||||
}
|
||||
if (watch) {
|
||||
if (randomSeed != null) {
|
||||
throw new ERR_INVALID_ARG_VALUE(
|
||||
'options.randomSeed',
|
||||
randomSeed,
|
||||
'is not supported with watch mode',
|
||||
);
|
||||
}
|
||||
if (randomize) {
|
||||
throw new ERR_INVALID_ARG_VALUE(
|
||||
'options.randomize',
|
||||
randomize,
|
||||
'is not supported with watch mode',
|
||||
);
|
||||
}
|
||||
}
|
||||
if (rerunFailuresFilePath) {
|
||||
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
|
||||
// TODO(pmarchini): Support rerun-failures with randomization by
|
||||
// persisting the randomization seed in the rerun state file.
|
||||
if (randomSeed != null) {
|
||||
throw new ERR_INVALID_ARG_VALUE(
|
||||
'options.randomSeed',
|
||||
randomSeed,
|
||||
'is not supported with rerun failures mode',
|
||||
);
|
||||
}
|
||||
if (randomize) {
|
||||
throw new ERR_INVALID_ARG_VALUE(
|
||||
'options.randomize',
|
||||
randomize,
|
||||
'is not supported with rerun failures mode',
|
||||
);
|
||||
}
|
||||
}
|
||||
if (randomize) {
|
||||
randomSeed ??= createRandomSeed();
|
||||
}
|
||||
|
||||
validateString(cwd, 'options.cwd');
|
||||
|
||||
@@ -683,10 +747,6 @@ function run(options = kEmptyObject) {
|
||||
);
|
||||
}
|
||||
|
||||
if (rerunFailuresFilePath) {
|
||||
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
|
||||
}
|
||||
|
||||
if (shard != null) {
|
||||
validateObject(shard, 'options.shard');
|
||||
// Avoid re-evaluating the shard object in case it's a getter
|
||||
@@ -783,11 +843,18 @@ function run(options = kEmptyObject) {
|
||||
functionCoverage: functionCoverage,
|
||||
cwd,
|
||||
globalSetupPath,
|
||||
randomize,
|
||||
randomSeed,
|
||||
};
|
||||
|
||||
const root = createTestTree(rootTestOptions, globalOptions);
|
||||
let testFiles = files ?? createTestFileList(globPatterns, cwd);
|
||||
const { isTestRunner } = globalOptions;
|
||||
|
||||
if (randomize) {
|
||||
root.diagnostic(`Randomized test order seed: ${randomSeed}`);
|
||||
}
|
||||
|
||||
if (shard) {
|
||||
testFiles = ArrayPrototypeFilter(testFiles, (_, index) => index % shard.total === shard.index - 1);
|
||||
}
|
||||
@@ -833,6 +900,8 @@ function run(options = kEmptyObject) {
|
||||
rerunFailuresFilePath,
|
||||
env,
|
||||
workerIdPool: isolation === 'process' ? workerIdPool : null,
|
||||
randomize,
|
||||
randomSeed,
|
||||
};
|
||||
|
||||
if (isolation === 'process') {
|
||||
|
||||
@@ -11,6 +11,7 @@ const {
|
||||
ArrayPrototypeUnshiftApply,
|
||||
Error,
|
||||
FunctionPrototype,
|
||||
MathFloor,
|
||||
MathMax,
|
||||
Number,
|
||||
NumberPrototypeToFixed,
|
||||
@@ -50,6 +51,7 @@ const { MockTracker } = require('internal/test_runner/mock/mock');
|
||||
const { TestsStream } = require('internal/test_runner/tests_stream');
|
||||
const {
|
||||
createDeferredCallback,
|
||||
createSeededGenerator,
|
||||
countCompletedTest,
|
||||
isTestFailureError,
|
||||
reporterScope,
|
||||
@@ -702,11 +704,17 @@ class Test extends AsyncResource {
|
||||
this.message = typeof skip === 'string' ? skip :
|
||||
typeof todo === 'string' ? todo : null;
|
||||
this.activeSubtests = 0;
|
||||
this.subtestQueueRandom = this.config.randomize ?
|
||||
createSeededGenerator(this.config.randomSeed) :
|
||||
null;
|
||||
this.pendingSubtests = [];
|
||||
this.pendingSubtestsScheduled = false;
|
||||
this.readySubtests = new SafeMap();
|
||||
this.unfinishedSubtests = new SafeSet();
|
||||
this.subtestsPromise = null;
|
||||
this.subtests = [];
|
||||
this.nextReportOrder = 1;
|
||||
this.reportOrder = 0;
|
||||
this.waitingOn = 0;
|
||||
this.finished = false;
|
||||
this.hooks = {
|
||||
@@ -832,13 +840,49 @@ class Test extends AsyncResource {
|
||||
ArrayPrototypePush(this.pendingSubtests, deferred);
|
||||
}
|
||||
|
||||
schedulePendingSubtests() {
|
||||
if (this.pendingSubtestsScheduled) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.pendingSubtestsScheduled = true;
|
||||
queueMicrotask(() => {
|
||||
this.pendingSubtestsScheduled = false;
|
||||
this.processPendingSubtests();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure each subtest has a contiguous, per-parent reporting order.
|
||||
* This is assigned at dequeue time for randomized runs, but tests that are
|
||||
* cancelled before dequeue still need an order to be reported.
|
||||
* @param {Test} subtest
|
||||
* @returns {void}
|
||||
*/
|
||||
assignReportOrder(subtest) {
|
||||
if (subtest.reportOrder === 0) {
|
||||
subtest.reportOrder = this.nextReportOrder++;
|
||||
}
|
||||
}
|
||||
|
||||
dequeuePendingSubtest() {
|
||||
if (!this.subtestQueueRandom || this.pendingSubtests.length < 2) {
|
||||
return ArrayPrototypeShift(this.pendingSubtests);
|
||||
}
|
||||
|
||||
// Pick a uniformly random pending sibling when randomization is enabled.
|
||||
const index = MathFloor(this.subtestQueueRandom() * this.pendingSubtests.length);
|
||||
return ArrayPrototypeSplice(this.pendingSubtests, index, 1)[0];
|
||||
}
|
||||
|
||||
/**
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async processPendingSubtests() {
|
||||
while (this.pendingSubtests.length > 0 && this.hasConcurrency()) {
|
||||
const deferred = ArrayPrototypeShift(this.pendingSubtests);
|
||||
const deferred = this.dequeuePendingSubtest();
|
||||
const test = deferred.test;
|
||||
this.assignReportOrder(test);
|
||||
test.reporter.dequeue(test.nesting, test.loc, test.name, this.reportedType);
|
||||
await test.run();
|
||||
deferred.resolve();
|
||||
@@ -850,7 +894,8 @@ class Test extends AsyncResource {
|
||||
* @returns {void}
|
||||
*/
|
||||
addReadySubtest(subtest) {
|
||||
this.readySubtests.set(subtest.childNumber, subtest);
|
||||
this.assignReportOrder(subtest);
|
||||
this.readySubtests.set(subtest.reportOrder, subtest);
|
||||
|
||||
if (this.unfinishedSubtests.delete(subtest) &&
|
||||
this.unfinishedSubtests.size === 0) {
|
||||
@@ -1104,6 +1149,20 @@ class Test extends AsyncResource {
|
||||
return deferred.promise;
|
||||
}
|
||||
|
||||
// When this parent dequeues subtests through subtestQueueRandom, defer the
|
||||
// first start to the next microtask so siblings created in the same
|
||||
// synchronous turn are all eligible for the first randomized pick, rather
|
||||
// than letting declaration order claim the initial concurrency slot(s).
|
||||
if (this.parent.subtestQueueRandom) {
|
||||
const deferred = PromiseWithResolvers();
|
||||
|
||||
setOwnProperty(deferred, 'test', this);
|
||||
this.parent.addPendingSubtest(deferred);
|
||||
this.parent.schedulePendingSubtests();
|
||||
return deferred.promise;
|
||||
}
|
||||
|
||||
this.parent.assignReportOrder(this);
|
||||
this.reporter.dequeue(this.nesting, this.loc, this.name, this.reportedType);
|
||||
return this.run();
|
||||
}
|
||||
@@ -1410,7 +1469,7 @@ class Test extends AsyncResource {
|
||||
isClearToSend() {
|
||||
return this.parent === null ||
|
||||
(
|
||||
this.parent.waitingOn === this.childNumber && this.parent.isClearToSend()
|
||||
this.parent.waitingOn === this.reportOrder && this.parent.isClearToSend()
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -7,11 +7,14 @@ const {
|
||||
ArrayPrototypePop,
|
||||
ArrayPrototypePush,
|
||||
ArrayPrototypeReduce,
|
||||
ArrayPrototypeSlice,
|
||||
ArrayPrototypeSome,
|
||||
JSONParse,
|
||||
MathFloor,
|
||||
MathImul,
|
||||
MathMax,
|
||||
MathMin,
|
||||
MathRandom,
|
||||
NumberParseInt,
|
||||
NumberPrototypeToFixed,
|
||||
ObjectGetOwnPropertyDescriptor,
|
||||
@@ -45,6 +48,7 @@ const { compose } = require('stream');
|
||||
const {
|
||||
validateInteger,
|
||||
validateFunction,
|
||||
validateUint32,
|
||||
} = require('internal/validators');
|
||||
const { validatePath } = require('internal/fs/utils');
|
||||
const { kEmptyObject } = require('internal/util');
|
||||
@@ -58,6 +62,7 @@ const coverageColors = {
|
||||
|
||||
const kMultipleCallbackInvocations = 'multipleCallbackInvocations';
|
||||
const kRegExpPattern = /^\/(.*)\/([a-z]*)$/;
|
||||
const kMaxRandomSeed = 0xFFFF_FFFF;
|
||||
|
||||
const kPatterns = ['test', 'test/**/*', 'test-*', '*[._-]test'];
|
||||
const kFileExtensions = ['js', 'mjs', 'cjs'];
|
||||
@@ -133,6 +138,54 @@ const kBuiltinReporters = new SafeMap([
|
||||
const kDefaultReporter = 'spec';
|
||||
const kDefaultDestination = 'stdout';
|
||||
|
||||
/**
|
||||
* Create a random uint32 seed.
|
||||
* @returns {number}
|
||||
*/
|
||||
function createRandomSeed() {
|
||||
return MathFloor(MathRandom() * (kMaxRandomSeed + 1));
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Mulberry32 pseudo-random number generator from a uint32 seed.
|
||||
* @param {number} seed
|
||||
* @returns {() => number}
|
||||
*/
|
||||
function createSeededGenerator(seed) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (state + 0x6D2B79F5) | 0;
|
||||
let value = MathImul(state ^ state >>> 15, 1 | state);
|
||||
value ^= value + MathImul(value ^ value >>> 7, 61 | value);
|
||||
return ((value ^ value >>> 14) >>> 0) / 4_294_967_296;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Return a deterministically shuffled copy of an array.
|
||||
* @template T
|
||||
* @param {T[]} values
|
||||
* @param {number} seed
|
||||
* @returns {T[]}
|
||||
*/
|
||||
function shuffleArrayWithSeed(values, seed) {
|
||||
if (values.length < 2) {
|
||||
return values;
|
||||
}
|
||||
|
||||
const randomized = ArrayPrototypeSlice(values);
|
||||
const random = createSeededGenerator(seed);
|
||||
|
||||
for (let i = randomized.length - 1; i > 0; i--) {
|
||||
const j = MathFloor(random() * (i + 1));
|
||||
const tmp = randomized[i];
|
||||
randomized[i] = randomized[j];
|
||||
randomized[j] = tmp;
|
||||
}
|
||||
|
||||
return randomized;
|
||||
}
|
||||
|
||||
function tryBuiltinReporter(name) {
|
||||
const builtinPath = kBuiltinReporters.get(name);
|
||||
|
||||
@@ -217,6 +270,10 @@ function parseCommandLine() {
|
||||
const updateSnapshots = getOptionValue('--test-update-snapshots');
|
||||
const watch = getOptionValue('--watch');
|
||||
const timeout = getOptionValue('--test-timeout') || Infinity;
|
||||
let randomize = getOptionValue('--test-randomize');
|
||||
const hasRandomSeedOption = getOptionValue('[has_test_random_seed]');
|
||||
const randomSeedOption = getOptionValue('--test-random-seed');
|
||||
let randomSeed;
|
||||
const rerunFailuresFilePath = getOptionValue('--test-rerun-failures');
|
||||
const isChildProcess = process.env.NODE_TEST_CONTEXT === 'child';
|
||||
const isChildProcessV8 = process.env.NODE_TEST_CONTEXT === 'child-v8';
|
||||
@@ -328,6 +385,12 @@ function parseCommandLine() {
|
||||
validatePath(rerunFailuresFilePath, '--test-rerun-failures');
|
||||
}
|
||||
|
||||
if (hasRandomSeedOption) {
|
||||
validateUint32(randomSeedOption, '--test-random-seed');
|
||||
randomSeed = randomSeedOption;
|
||||
randomize = true;
|
||||
}
|
||||
|
||||
const setup = reporterScope.bind(async (rootReporter) => {
|
||||
const reportersMap = await getReportersMap(reporters, destinations);
|
||||
for (let i = 0; i < reportersMap.length; i++) {
|
||||
@@ -362,6 +425,8 @@ function parseCommandLine() {
|
||||
timeout,
|
||||
updateSnapshots,
|
||||
watch,
|
||||
randomize,
|
||||
randomSeed,
|
||||
rerunFailuresFilePath,
|
||||
};
|
||||
|
||||
@@ -652,10 +717,14 @@ async function setupGlobalSetupTeardownFunctions(globalSetupPath, cwd) {
|
||||
module.exports = {
|
||||
convertStringToRegExp,
|
||||
countCompletedTest,
|
||||
createRandomSeed,
|
||||
createSeededGenerator,
|
||||
createDeferredCallback,
|
||||
isTestFailureError,
|
||||
kDefaultPattern,
|
||||
kMaxRandomSeed,
|
||||
parseCommandLine,
|
||||
shuffleArrayWithSeed,
|
||||
reporterScope,
|
||||
shouldColorizeTestFiles,
|
||||
getCoverageReport,
|
||||
|
||||
@@ -1003,6 +1003,20 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
|
||||
&EnvironmentOptions::test_global_setup_path,
|
||||
kAllowedInEnvvar,
|
||||
OptionNamespaces::kTestRunnerNamespace);
|
||||
AddOption("--test-randomize",
|
||||
"run tests in a random order",
|
||||
&EnvironmentOptions::test_randomize,
|
||||
kAllowedInEnvvar,
|
||||
false,
|
||||
OptionNamespaces::kTestRunnerNamespace);
|
||||
AddOption(
|
||||
"[has_test_random_seed]", "", &EnvironmentOptions::has_test_random_seed);
|
||||
AddOption("--test-random-seed",
|
||||
"seed used to randomize test execution order",
|
||||
&EnvironmentOptions::test_random_seed,
|
||||
kAllowedInEnvvar,
|
||||
OptionNamespaces::kTestRunnerNamespace);
|
||||
Implies("--test-random-seed", "[has_test_random_seed]");
|
||||
AddOption("--test-rerun-failures",
|
||||
"specifies the path to the rerun state file",
|
||||
&EnvironmentOptions::test_rerun_failures_path,
|
||||
|
||||
@@ -204,6 +204,9 @@ class EnvironmentOptions : public Options {
|
||||
std::string test_rerun_failures_path;
|
||||
std::vector<std::string> test_reporter_destination;
|
||||
std::string test_global_setup_path;
|
||||
bool test_randomize = false;
|
||||
bool has_test_random_seed = false;
|
||||
uint64_t test_random_seed = 0;
|
||||
bool test_only = false;
|
||||
bool test_udp_no_try_send = false;
|
||||
std::string test_isolation = "process";
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
import test from 'node:test';
|
||||
import { setTimeout as sleep } from 'node:timers/promises';
|
||||
|
||||
test('parent', { timeout: 50, concurrency: 1 }, async (t) => {
|
||||
t.test('first', async () => {
|
||||
await sleep(1000);
|
||||
});
|
||||
|
||||
t.test('second', async () => {});
|
||||
t.test('third', async () => {});
|
||||
});
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
'use strict';
|
||||
require('../../../common');
|
||||
const fixtures = require('../../../common/fixtures');
|
||||
const spawn = require('node:child_process').spawn;
|
||||
|
||||
spawn(
|
||||
process.execPath,
|
||||
[
|
||||
'--no-warnings',
|
||||
'--test-reporter', 'spec',
|
||||
'--test-random-seed=1',
|
||||
'--test-isolation=none',
|
||||
'--test',
|
||||
fixtures.path('test-runner/randomize/internal-order-nested-scenarios.cjs'),
|
||||
],
|
||||
{ stdio: 'inherit' },
|
||||
);
|
||||
+213
@@ -0,0 +1,213 @@
|
||||
▶ nested-scenarios
|
||||
▶ scenario describe-it
|
||||
▶ describe-it block
|
||||
▶ describe-it 2
|
||||
▶ describe-it 2-2
|
||||
✔ describe-it 2-2-2 (*ms)
|
||||
✔ describe-it 2-2-1 (*ms)
|
||||
✔ describe-it 2-2-3 (*ms)
|
||||
✔ describe-it 2-2 (*ms)
|
||||
▶ describe-it 2-1
|
||||
✔ describe-it 2-1-2 (*ms)
|
||||
✔ describe-it 2-1-1 (*ms)
|
||||
✔ describe-it 2-1-3 (*ms)
|
||||
✔ describe-it 2-1 (*ms)
|
||||
▶ describe-it 2-3
|
||||
✔ describe-it 2-3-2 (*ms)
|
||||
✔ describe-it 2-3-1 (*ms)
|
||||
✔ describe-it 2-3-3 (*ms)
|
||||
✔ describe-it 2-3 (*ms)
|
||||
✔ describe-it 2 (*ms)
|
||||
▶ describe-it 1
|
||||
▶ describe-it 1-2
|
||||
✔ describe-it 1-2-2 (*ms)
|
||||
✔ describe-it 1-2-1 (*ms)
|
||||
✔ describe-it 1-2-3 (*ms)
|
||||
✔ describe-it 1-2 (*ms)
|
||||
▶ describe-it 1-1
|
||||
✔ describe-it 1-1-2 (*ms)
|
||||
✔ describe-it 1-1-1 (*ms)
|
||||
✔ describe-it 1-1-3 (*ms)
|
||||
✔ describe-it 1-1 (*ms)
|
||||
▶ describe-it 1-3
|
||||
✔ describe-it 1-3-2 (*ms)
|
||||
✔ describe-it 1-3-1 (*ms)
|
||||
✔ describe-it 1-3-3 (*ms)
|
||||
✔ describe-it 1-3 (*ms)
|
||||
✔ describe-it 1 (*ms)
|
||||
▶ describe-it 3
|
||||
▶ describe-it 3-2
|
||||
✔ describe-it 3-2-2 (*ms)
|
||||
✔ describe-it 3-2-1 (*ms)
|
||||
✔ describe-it 3-2-3 (*ms)
|
||||
✔ describe-it 3-2 (*ms)
|
||||
▶ describe-it 3-1
|
||||
✔ describe-it 3-1-2 (*ms)
|
||||
✔ describe-it 3-1-1 (*ms)
|
||||
✔ describe-it 3-1-3 (*ms)
|
||||
✔ describe-it 3-1 (*ms)
|
||||
▶ describe-it 3-3
|
||||
✔ describe-it 3-3-2 (*ms)
|
||||
✔ describe-it 3-3-1 (*ms)
|
||||
✔ describe-it 3-3-3 (*ms)
|
||||
✔ describe-it 3-3 (*ms)
|
||||
✔ describe-it 3 (*ms)
|
||||
✔ describe-it block (*ms)
|
||||
✔ scenario describe-it (*ms)
|
||||
▶ scenario static-no-await
|
||||
▶ static-no-await block
|
||||
▶ static-no-await 2
|
||||
▶ static-no-await 2-2
|
||||
✔ static-no-await 2-2-2 (*ms)
|
||||
✔ static-no-await 2-2-1 (*ms)
|
||||
✔ static-no-await 2-2-3 (*ms)
|
||||
✔ static-no-await 2-2 (*ms)
|
||||
▶ static-no-await 2-1
|
||||
✔ static-no-await 2-1-2 (*ms)
|
||||
✔ static-no-await 2-1-1 (*ms)
|
||||
✔ static-no-await 2-1-3 (*ms)
|
||||
✔ static-no-await 2-1 (*ms)
|
||||
▶ static-no-await 2-3
|
||||
✔ static-no-await 2-3-2 (*ms)
|
||||
✔ static-no-await 2-3-1 (*ms)
|
||||
✔ static-no-await 2-3-3 (*ms)
|
||||
✔ static-no-await 2-3 (*ms)
|
||||
✔ static-no-await 2 (*ms)
|
||||
▶ static-no-await 1
|
||||
▶ static-no-await 1-2
|
||||
✔ static-no-await 1-2-2 (*ms)
|
||||
✔ static-no-await 1-2-1 (*ms)
|
||||
✔ static-no-await 1-2-3 (*ms)
|
||||
✔ static-no-await 1-2 (*ms)
|
||||
▶ static-no-await 1-1
|
||||
✔ static-no-await 1-1-2 (*ms)
|
||||
✔ static-no-await 1-1-1 (*ms)
|
||||
✔ static-no-await 1-1-3 (*ms)
|
||||
✔ static-no-await 1-1 (*ms)
|
||||
▶ static-no-await 1-3
|
||||
✔ static-no-await 1-3-2 (*ms)
|
||||
✔ static-no-await 1-3-1 (*ms)
|
||||
✔ static-no-await 1-3-3 (*ms)
|
||||
✔ static-no-await 1-3 (*ms)
|
||||
✔ static-no-await 1 (*ms)
|
||||
▶ static-no-await 3
|
||||
▶ static-no-await 3-2
|
||||
✔ static-no-await 3-2-2 (*ms)
|
||||
✔ static-no-await 3-2-1 (*ms)
|
||||
✔ static-no-await 3-2-3 (*ms)
|
||||
✔ static-no-await 3-2 (*ms)
|
||||
▶ static-no-await 3-1
|
||||
✔ static-no-await 3-1-2 (*ms)
|
||||
✔ static-no-await 3-1-1 (*ms)
|
||||
✔ static-no-await 3-1-3 (*ms)
|
||||
✔ static-no-await 3-1 (*ms)
|
||||
▶ static-no-await 3-3
|
||||
✔ static-no-await 3-3-2 (*ms)
|
||||
✔ static-no-await 3-3-1 (*ms)
|
||||
✔ static-no-await 3-3-3 (*ms)
|
||||
✔ static-no-await 3-3 (*ms)
|
||||
✔ static-no-await 3 (*ms)
|
||||
✔ static-no-await block (*ms)
|
||||
✔ scenario static-no-await (*ms)
|
||||
▶ scenario static-await
|
||||
▶ static-await block
|
||||
▶ static-await 1
|
||||
▶ static-await 1-1
|
||||
✔ static-await 1-1-1 (*ms)
|
||||
✔ static-await 1-1-2 (*ms)
|
||||
✔ static-await 1-1-3 (*ms)
|
||||
✔ static-await 1-1 (*ms)
|
||||
▶ static-await 1-2
|
||||
✔ static-await 1-2-1 (*ms)
|
||||
✔ static-await 1-2-2 (*ms)
|
||||
✔ static-await 1-2-3 (*ms)
|
||||
✔ static-await 1-2 (*ms)
|
||||
▶ static-await 1-3
|
||||
✔ static-await 1-3-1 (*ms)
|
||||
✔ static-await 1-3-2 (*ms)
|
||||
✔ static-await 1-3-3 (*ms)
|
||||
✔ static-await 1-3 (*ms)
|
||||
✔ static-await 1 (*ms)
|
||||
▶ static-await 2
|
||||
▶ static-await 2-1
|
||||
✔ static-await 2-1-1 (*ms)
|
||||
✔ static-await 2-1-2 (*ms)
|
||||
✔ static-await 2-1-3 (*ms)
|
||||
✔ static-await 2-1 (*ms)
|
||||
▶ static-await 2-2
|
||||
✔ static-await 2-2-1 (*ms)
|
||||
✔ static-await 2-2-2 (*ms)
|
||||
✔ static-await 2-2-3 (*ms)
|
||||
✔ static-await 2-2 (*ms)
|
||||
▶ static-await 2-3
|
||||
✔ static-await 2-3-1 (*ms)
|
||||
✔ static-await 2-3-2 (*ms)
|
||||
✔ static-await 2-3-3 (*ms)
|
||||
✔ static-await 2-3 (*ms)
|
||||
✔ static-await 2 (*ms)
|
||||
▶ static-await 3
|
||||
▶ static-await 3-1
|
||||
✔ static-await 3-1-1 (*ms)
|
||||
✔ static-await 3-1-2 (*ms)
|
||||
✔ static-await 3-1-3 (*ms)
|
||||
✔ static-await 3-1 (*ms)
|
||||
▶ static-await 3-2
|
||||
✔ static-await 3-2-1 (*ms)
|
||||
✔ static-await 3-2-2 (*ms)
|
||||
✔ static-await 3-2-3 (*ms)
|
||||
✔ static-await 3-2 (*ms)
|
||||
▶ static-await 3-3
|
||||
✔ static-await 3-3-1 (*ms)
|
||||
✔ static-await 3-3-2 (*ms)
|
||||
✔ static-await 3-3-3 (*ms)
|
||||
✔ static-await 3-3 (*ms)
|
||||
✔ static-await 3 (*ms)
|
||||
✔ static-await block (*ms)
|
||||
✔ scenario static-await (*ms)
|
||||
▶ scenario mixed-await-normal
|
||||
▶ mixed-await-normal block
|
||||
▶ mixed-await-normal 2
|
||||
✔ mixed-await-normal 2-1-awaited (*ms)
|
||||
✔ mixed-await-normal 2-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 2-2-awaited (*ms)
|
||||
✔ mixed-await-normal 2-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-3-awaited (*ms)
|
||||
✔ mixed-await-normal 2-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 2-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 2 (*ms)
|
||||
▶ mixed-await-normal 1
|
||||
✔ mixed-await-normal 1-1-awaited (*ms)
|
||||
✔ mixed-await-normal 1-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 1-2-awaited (*ms)
|
||||
✔ mixed-await-normal 1-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-3-awaited (*ms)
|
||||
✔ mixed-await-normal 1-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 1-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 1 (*ms)
|
||||
▶ mixed-await-normal 3
|
||||
✔ mixed-await-normal 3-1-awaited (*ms)
|
||||
✔ mixed-await-normal 3-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 3-2-awaited (*ms)
|
||||
✔ mixed-await-normal 3-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-3-awaited (*ms)
|
||||
✔ mixed-await-normal 3-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 3-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 3 (*ms)
|
||||
✔ mixed-await-normal block (*ms)
|
||||
✔ scenario mixed-await-normal (*ms)
|
||||
✔ nested-scenarios (*ms)
|
||||
ℹ Randomized test order seed: 1
|
||||
ℹ tests 144
|
||||
ℹ suites 12
|
||||
ℹ pass 144
|
||||
ℹ fail 0
|
||||
ℹ cancelled 0
|
||||
ℹ skipped 0
|
||||
ℹ todo 0
|
||||
ℹ duration_ms *
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
'use strict';
|
||||
require('../../../common');
|
||||
const fixtures = require('../../../common/fixtures');
|
||||
const spawn = require('node:child_process').spawn;
|
||||
|
||||
spawn(
|
||||
process.execPath,
|
||||
[
|
||||
'--no-warnings',
|
||||
'--test-reporter', 'spec',
|
||||
'--test-random-seed=1',
|
||||
'--test-isolation=process',
|
||||
'--test',
|
||||
fixtures.path('test-runner/randomize/internal-order-nested-scenarios.cjs'),
|
||||
],
|
||||
{ stdio: 'inherit' },
|
||||
);
|
||||
+213
@@ -0,0 +1,213 @@
|
||||
▶ nested-scenarios
|
||||
▶ scenario describe-it
|
||||
▶ describe-it block
|
||||
▶ describe-it 2
|
||||
▶ describe-it 2-2
|
||||
✔ describe-it 2-2-2 (*ms)
|
||||
✔ describe-it 2-2-1 (*ms)
|
||||
✔ describe-it 2-2-3 (*ms)
|
||||
✔ describe-it 2-2 (*ms)
|
||||
▶ describe-it 2-1
|
||||
✔ describe-it 2-1-2 (*ms)
|
||||
✔ describe-it 2-1-1 (*ms)
|
||||
✔ describe-it 2-1-3 (*ms)
|
||||
✔ describe-it 2-1 (*ms)
|
||||
▶ describe-it 2-3
|
||||
✔ describe-it 2-3-2 (*ms)
|
||||
✔ describe-it 2-3-1 (*ms)
|
||||
✔ describe-it 2-3-3 (*ms)
|
||||
✔ describe-it 2-3 (*ms)
|
||||
✔ describe-it 2 (*ms)
|
||||
▶ describe-it 1
|
||||
▶ describe-it 1-2
|
||||
✔ describe-it 1-2-2 (*ms)
|
||||
✔ describe-it 1-2-1 (*ms)
|
||||
✔ describe-it 1-2-3 (*ms)
|
||||
✔ describe-it 1-2 (*ms)
|
||||
▶ describe-it 1-1
|
||||
✔ describe-it 1-1-2 (*ms)
|
||||
✔ describe-it 1-1-1 (*ms)
|
||||
✔ describe-it 1-1-3 (*ms)
|
||||
✔ describe-it 1-1 (*ms)
|
||||
▶ describe-it 1-3
|
||||
✔ describe-it 1-3-2 (*ms)
|
||||
✔ describe-it 1-3-1 (*ms)
|
||||
✔ describe-it 1-3-3 (*ms)
|
||||
✔ describe-it 1-3 (*ms)
|
||||
✔ describe-it 1 (*ms)
|
||||
▶ describe-it 3
|
||||
▶ describe-it 3-2
|
||||
✔ describe-it 3-2-2 (*ms)
|
||||
✔ describe-it 3-2-1 (*ms)
|
||||
✔ describe-it 3-2-3 (*ms)
|
||||
✔ describe-it 3-2 (*ms)
|
||||
▶ describe-it 3-1
|
||||
✔ describe-it 3-1-2 (*ms)
|
||||
✔ describe-it 3-1-1 (*ms)
|
||||
✔ describe-it 3-1-3 (*ms)
|
||||
✔ describe-it 3-1 (*ms)
|
||||
▶ describe-it 3-3
|
||||
✔ describe-it 3-3-2 (*ms)
|
||||
✔ describe-it 3-3-1 (*ms)
|
||||
✔ describe-it 3-3-3 (*ms)
|
||||
✔ describe-it 3-3 (*ms)
|
||||
✔ describe-it 3 (*ms)
|
||||
✔ describe-it block (*ms)
|
||||
✔ scenario describe-it (*ms)
|
||||
▶ scenario static-no-await
|
||||
▶ static-no-await block
|
||||
▶ static-no-await 2
|
||||
▶ static-no-await 2-2
|
||||
✔ static-no-await 2-2-2 (*ms)
|
||||
✔ static-no-await 2-2-1 (*ms)
|
||||
✔ static-no-await 2-2-3 (*ms)
|
||||
✔ static-no-await 2-2 (*ms)
|
||||
▶ static-no-await 2-1
|
||||
✔ static-no-await 2-1-2 (*ms)
|
||||
✔ static-no-await 2-1-1 (*ms)
|
||||
✔ static-no-await 2-1-3 (*ms)
|
||||
✔ static-no-await 2-1 (*ms)
|
||||
▶ static-no-await 2-3
|
||||
✔ static-no-await 2-3-2 (*ms)
|
||||
✔ static-no-await 2-3-1 (*ms)
|
||||
✔ static-no-await 2-3-3 (*ms)
|
||||
✔ static-no-await 2-3 (*ms)
|
||||
✔ static-no-await 2 (*ms)
|
||||
▶ static-no-await 1
|
||||
▶ static-no-await 1-2
|
||||
✔ static-no-await 1-2-2 (*ms)
|
||||
✔ static-no-await 1-2-1 (*ms)
|
||||
✔ static-no-await 1-2-3 (*ms)
|
||||
✔ static-no-await 1-2 (*ms)
|
||||
▶ static-no-await 1-1
|
||||
✔ static-no-await 1-1-2 (*ms)
|
||||
✔ static-no-await 1-1-1 (*ms)
|
||||
✔ static-no-await 1-1-3 (*ms)
|
||||
✔ static-no-await 1-1 (*ms)
|
||||
▶ static-no-await 1-3
|
||||
✔ static-no-await 1-3-2 (*ms)
|
||||
✔ static-no-await 1-3-1 (*ms)
|
||||
✔ static-no-await 1-3-3 (*ms)
|
||||
✔ static-no-await 1-3 (*ms)
|
||||
✔ static-no-await 1 (*ms)
|
||||
▶ static-no-await 3
|
||||
▶ static-no-await 3-2
|
||||
✔ static-no-await 3-2-2 (*ms)
|
||||
✔ static-no-await 3-2-1 (*ms)
|
||||
✔ static-no-await 3-2-3 (*ms)
|
||||
✔ static-no-await 3-2 (*ms)
|
||||
▶ static-no-await 3-1
|
||||
✔ static-no-await 3-1-2 (*ms)
|
||||
✔ static-no-await 3-1-1 (*ms)
|
||||
✔ static-no-await 3-1-3 (*ms)
|
||||
✔ static-no-await 3-1 (*ms)
|
||||
▶ static-no-await 3-3
|
||||
✔ static-no-await 3-3-2 (*ms)
|
||||
✔ static-no-await 3-3-1 (*ms)
|
||||
✔ static-no-await 3-3-3 (*ms)
|
||||
✔ static-no-await 3-3 (*ms)
|
||||
✔ static-no-await 3 (*ms)
|
||||
✔ static-no-await block (*ms)
|
||||
✔ scenario static-no-await (*ms)
|
||||
▶ scenario static-await
|
||||
▶ static-await block
|
||||
▶ static-await 1
|
||||
▶ static-await 1-1
|
||||
✔ static-await 1-1-1 (*ms)
|
||||
✔ static-await 1-1-2 (*ms)
|
||||
✔ static-await 1-1-3 (*ms)
|
||||
✔ static-await 1-1 (*ms)
|
||||
▶ static-await 1-2
|
||||
✔ static-await 1-2-1 (*ms)
|
||||
✔ static-await 1-2-2 (*ms)
|
||||
✔ static-await 1-2-3 (*ms)
|
||||
✔ static-await 1-2 (*ms)
|
||||
▶ static-await 1-3
|
||||
✔ static-await 1-3-1 (*ms)
|
||||
✔ static-await 1-3-2 (*ms)
|
||||
✔ static-await 1-3-3 (*ms)
|
||||
✔ static-await 1-3 (*ms)
|
||||
✔ static-await 1 (*ms)
|
||||
▶ static-await 2
|
||||
▶ static-await 2-1
|
||||
✔ static-await 2-1-1 (*ms)
|
||||
✔ static-await 2-1-2 (*ms)
|
||||
✔ static-await 2-1-3 (*ms)
|
||||
✔ static-await 2-1 (*ms)
|
||||
▶ static-await 2-2
|
||||
✔ static-await 2-2-1 (*ms)
|
||||
✔ static-await 2-2-2 (*ms)
|
||||
✔ static-await 2-2-3 (*ms)
|
||||
✔ static-await 2-2 (*ms)
|
||||
▶ static-await 2-3
|
||||
✔ static-await 2-3-1 (*ms)
|
||||
✔ static-await 2-3-2 (*ms)
|
||||
✔ static-await 2-3-3 (*ms)
|
||||
✔ static-await 2-3 (*ms)
|
||||
✔ static-await 2 (*ms)
|
||||
▶ static-await 3
|
||||
▶ static-await 3-1
|
||||
✔ static-await 3-1-1 (*ms)
|
||||
✔ static-await 3-1-2 (*ms)
|
||||
✔ static-await 3-1-3 (*ms)
|
||||
✔ static-await 3-1 (*ms)
|
||||
▶ static-await 3-2
|
||||
✔ static-await 3-2-1 (*ms)
|
||||
✔ static-await 3-2-2 (*ms)
|
||||
✔ static-await 3-2-3 (*ms)
|
||||
✔ static-await 3-2 (*ms)
|
||||
▶ static-await 3-3
|
||||
✔ static-await 3-3-1 (*ms)
|
||||
✔ static-await 3-3-2 (*ms)
|
||||
✔ static-await 3-3-3 (*ms)
|
||||
✔ static-await 3-3 (*ms)
|
||||
✔ static-await 3 (*ms)
|
||||
✔ static-await block (*ms)
|
||||
✔ scenario static-await (*ms)
|
||||
▶ scenario mixed-await-normal
|
||||
▶ mixed-await-normal block
|
||||
▶ mixed-await-normal 2
|
||||
✔ mixed-await-normal 2-1-awaited (*ms)
|
||||
✔ mixed-await-normal 2-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 2-2-awaited (*ms)
|
||||
✔ mixed-await-normal 2-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-3-awaited (*ms)
|
||||
✔ mixed-await-normal 2-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 2-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 2-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 2 (*ms)
|
||||
▶ mixed-await-normal 1
|
||||
✔ mixed-await-normal 1-1-awaited (*ms)
|
||||
✔ mixed-await-normal 1-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 1-2-awaited (*ms)
|
||||
✔ mixed-await-normal 1-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-3-awaited (*ms)
|
||||
✔ mixed-await-normal 1-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 1-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 1-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 1 (*ms)
|
||||
▶ mixed-await-normal 3
|
||||
✔ mixed-await-normal 3-1-awaited (*ms)
|
||||
✔ mixed-await-normal 3-1-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-1-sync-after (*ms)
|
||||
✔ mixed-await-normal 3-2-awaited (*ms)
|
||||
✔ mixed-await-normal 3-2-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-3-awaited (*ms)
|
||||
✔ mixed-await-normal 3-3-sync-before (*ms)
|
||||
✔ mixed-await-normal 3-2-sync-after (*ms)
|
||||
✔ mixed-await-normal 3-3-sync-after (*ms)
|
||||
✔ mixed-await-normal 3 (*ms)
|
||||
✔ mixed-await-normal block (*ms)
|
||||
✔ scenario mixed-await-normal (*ms)
|
||||
✔ nested-scenarios (*ms)
|
||||
ℹ Randomized test order seed: 1
|
||||
ℹ tests 144
|
||||
ℹ suites 12
|
||||
ℹ pass 144
|
||||
ℹ fail 0
|
||||
ℹ cancelled 0
|
||||
ℹ skipped 0
|
||||
ℹ todo 0
|
||||
ℹ duration_ms *
|
||||
@@ -0,0 +1,26 @@
|
||||
'use strict';
|
||||
require('../../../common');
|
||||
const fixtures = require('../../../common/fixtures');
|
||||
const spawn = require('node:child_process').spawn;
|
||||
|
||||
spawn(
|
||||
process.execPath,
|
||||
[
|
||||
'--no-warnings',
|
||||
'--test-reporter', 'spec',
|
||||
'--test-concurrency', '1',
|
||||
'--test-random-seed=12345',
|
||||
'--test',
|
||||
fixtures.path('test-runner/shards/a.cjs'),
|
||||
fixtures.path('test-runner/shards/b.cjs'),
|
||||
fixtures.path('test-runner/shards/c.cjs'),
|
||||
fixtures.path('test-runner/shards/d.cjs'),
|
||||
fixtures.path('test-runner/shards/e.cjs'),
|
||||
fixtures.path('test-runner/shards/f.cjs'),
|
||||
fixtures.path('test-runner/shards/g.cjs'),
|
||||
fixtures.path('test-runner/shards/h.cjs'),
|
||||
fixtures.path('test-runner/shards/i.cjs'),
|
||||
fixtures.path('test-runner/shards/j.cjs'),
|
||||
],
|
||||
{ stdio: 'inherit' },
|
||||
);
|
||||
@@ -0,0 +1,19 @@
|
||||
✔ j.cjs this should pass (*ms)
|
||||
✔ c.cjs this should pass (*ms)
|
||||
✔ e.cjs this should pass (*ms)
|
||||
✔ h.cjs this should pass (*ms)
|
||||
✔ f.cjs this should pass (*ms)
|
||||
✔ b.cjs this should pass (*ms)
|
||||
✔ a.cjs this should pass (*ms)
|
||||
✔ i.cjs this should pass (*ms)
|
||||
✔ g.cjs this should pass (*ms)
|
||||
✔ d.cjs this should pass (*ms)
|
||||
ℹ Randomized test order seed: 12345
|
||||
ℹ tests 10
|
||||
ℹ suites 0
|
||||
ℹ pass 10
|
||||
ℹ fail 0
|
||||
ℹ cancelled 0
|
||||
ℹ skipped 0
|
||||
ℹ todo 0
|
||||
ℹ duration_ms *
|
||||
@@ -0,0 +1,27 @@
|
||||
'use strict';
|
||||
require('../../../common');
|
||||
const fixtures = require('../../../common/fixtures');
|
||||
const spawn = require('node:child_process').spawn;
|
||||
|
||||
spawn(
|
||||
process.execPath,
|
||||
[
|
||||
'--no-warnings',
|
||||
'--test-reporter', 'spec',
|
||||
'--test-concurrency', '1',
|
||||
'--test-random-seed=12345',
|
||||
'--test-isolation=none',
|
||||
'--test',
|
||||
fixtures.path('test-runner/shards/a.cjs'),
|
||||
fixtures.path('test-runner/shards/b.cjs'),
|
||||
fixtures.path('test-runner/shards/c.cjs'),
|
||||
fixtures.path('test-runner/shards/d.cjs'),
|
||||
fixtures.path('test-runner/shards/e.cjs'),
|
||||
fixtures.path('test-runner/shards/f.cjs'),
|
||||
fixtures.path('test-runner/shards/g.cjs'),
|
||||
fixtures.path('test-runner/shards/h.cjs'),
|
||||
fixtures.path('test-runner/shards/i.cjs'),
|
||||
fixtures.path('test-runner/shards/j.cjs'),
|
||||
],
|
||||
{ stdio: 'inherit' },
|
||||
);
|
||||
@@ -0,0 +1,19 @@
|
||||
✔ j.cjs this should pass (*ms)
|
||||
✔ c.cjs this should pass (*ms)
|
||||
✔ e.cjs this should pass (*ms)
|
||||
✔ h.cjs this should pass (*ms)
|
||||
✔ f.cjs this should pass (*ms)
|
||||
✔ b.cjs this should pass (*ms)
|
||||
✔ a.cjs this should pass (*ms)
|
||||
✔ i.cjs this should pass (*ms)
|
||||
✔ g.cjs this should pass (*ms)
|
||||
✔ d.cjs this should pass (*ms)
|
||||
ℹ Randomized test order seed: 12345
|
||||
ℹ tests 10
|
||||
ℹ suites 0
|
||||
ℹ pass 10
|
||||
ℹ fail 0
|
||||
ℹ cancelled 0
|
||||
ℹ skipped 0
|
||||
ℹ todo 0
|
||||
ℹ duration_ms *
|
||||
@@ -0,0 +1,9 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
|
||||
test('parent', (t) => {
|
||||
t.test('a', () => {});
|
||||
t.test('b', () => {});
|
||||
t.test('c', () => {});
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
'use strict';
|
||||
|
||||
const { describe, it, test } = require('node:test');
|
||||
const { setImmediate: setImmediatePromise } = require('node:timers/promises');
|
||||
|
||||
const levelOne = ['1', '2', '3'];
|
||||
const levelTwo = ['1', '2', '3'];
|
||||
const levelThree = ['1', '2', '3'];
|
||||
|
||||
test('nested-scenarios', async (t) => {
|
||||
await t.test('scenario describe-it', async (scenario) => {
|
||||
await scenario.test('describe-it block', () => {
|
||||
for (const outer of levelOne) {
|
||||
describe(`describe-it ${outer}`, () => {
|
||||
for (const middle of levelTwo) {
|
||||
describe(`describe-it ${outer}-${middle}`, () => {
|
||||
for (const inner of levelThree) {
|
||||
it(`describe-it ${outer}-${middle}-${inner}`, () => {});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
await t.test('scenario static-no-await', async (scenario) => {
|
||||
await scenario.test('static-no-await block', (block) => {
|
||||
for (const outer of levelOne) {
|
||||
block.test(`static-no-await ${outer}`, (outerBlock) => {
|
||||
for (const middle of levelTwo) {
|
||||
outerBlock.test(`static-no-await ${outer}-${middle}`, (middleBlock) => {
|
||||
for (const inner of levelThree) {
|
||||
middleBlock.test(`static-no-await ${outer}-${middle}-${inner}`, () => {});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
await t.test('scenario static-await', async (scenario) => {
|
||||
await scenario.test('static-await block', async (block) => {
|
||||
for (const outer of levelOne) {
|
||||
await block.test(`static-await ${outer}`, async (outerBlock) => {
|
||||
await setImmediatePromise();
|
||||
|
||||
for (const middle of levelTwo) {
|
||||
await outerBlock.test(`static-await ${outer}-${middle}`, async (middleBlock) => {
|
||||
await setImmediatePromise();
|
||||
|
||||
for (const inner of levelThree) {
|
||||
await middleBlock.test(`static-await ${outer}-${middle}-${inner}`, async () => {
|
||||
await setImmediatePromise();
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
await t.test('scenario mixed-await-normal', async (scenario) => {
|
||||
await scenario.test('mixed-await-normal block', (block) => {
|
||||
for (const outer of levelOne) {
|
||||
block.test(`mixed-await-normal ${outer}`, async (outerBlock) => {
|
||||
for (const middle of levelTwo) {
|
||||
outerBlock.test(`mixed-await-normal ${outer}-${middle}-sync-before`, () => {});
|
||||
|
||||
await outerBlock.test(`mixed-await-normal ${outer}-${middle}-awaited`, async () => {
|
||||
await setImmediatePromise();
|
||||
});
|
||||
|
||||
outerBlock.test(`mixed-await-normal ${outer}-${middle}-sync-after`, () => {});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,16 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const { setImmediate: setImmediatePromise } = require('node:timers/promises');
|
||||
const executionOrder = [];
|
||||
|
||||
for (const name of ['a', 'b', 'c', 'd', 'e']) {
|
||||
test(`internal ${name}`, async () => {
|
||||
executionOrder.push(name);
|
||||
await setImmediatePromise();
|
||||
});
|
||||
}
|
||||
|
||||
process.on('exit', () => {
|
||||
process.stdout.write(`EXECUTION_ORDER:${executionOrder.join(',')}\n`);
|
||||
});
|
||||
@@ -0,0 +1,228 @@
|
||||
'use strict';
|
||||
|
||||
require('../common');
|
||||
const assert = require('node:assert');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const { join } = require('node:path');
|
||||
const { describe, it } = require('node:test');
|
||||
const fixtures = require('../common/fixtures');
|
||||
|
||||
const testFixtures = fixtures.path('test-runner');
|
||||
const internalOrderFixture = fixtures.path('test-runner', 'randomize', 'internal-order.cjs');
|
||||
const rerunStateFile = fixtures.path('test-runner', 'rerun-state.json');
|
||||
const kShardFiles = ['a.cjs', 'b.cjs', 'c.cjs', 'd.cjs', 'e.cjs', 'f.cjs', 'g.cjs', 'h.cjs', 'i.cjs', 'j.cjs'];
|
||||
const kInternalTests = ['a', 'b', 'c', 'd', 'e'];
|
||||
|
||||
function getShardOrder(stdout) {
|
||||
return Array.from(stdout.matchAll(/ok \d+ - ([a-j]\.cjs) this should pass/g), ({ 1: name }) => name);
|
||||
}
|
||||
|
||||
function getInternalExecutionOrder(stdout) {
|
||||
const match = stdout.match(/EXECUTION_ORDER:([a-e](?:,[a-e])*)/);
|
||||
assert(match, `Missing EXECUTION_ORDER marker in output: ${stdout}`);
|
||||
return match[1].split(',');
|
||||
}
|
||||
|
||||
describe('test runner randomize flags via command line', () => {
|
||||
it('should be deterministic with --test-random-seed', () => {
|
||||
const args = [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-concurrency=1',
|
||||
'--test-randomize',
|
||||
'--test-random-seed=12345',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
];
|
||||
const first = spawnSync(process.execPath, args);
|
||||
const second = spawnSync(process.execPath, args);
|
||||
|
||||
assert.strictEqual(first.stderr.toString(), '');
|
||||
assert.strictEqual(second.stderr.toString(), '');
|
||||
assert.strictEqual(first.status, 0);
|
||||
assert.strictEqual(second.status, 0);
|
||||
|
||||
const firstOrder = getShardOrder(first.stdout.toString());
|
||||
const secondOrder = getShardOrder(second.stdout.toString());
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), kShardFiles);
|
||||
});
|
||||
|
||||
it('should use different orders for different seeds', () => {
|
||||
const first = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-concurrency=1',
|
||||
'--test-randomize',
|
||||
'--test-random-seed=11111',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
const second = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-concurrency=1',
|
||||
'--test-randomize',
|
||||
'--test-random-seed=22222',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(first.stderr.toString(), '');
|
||||
assert.strictEqual(second.stderr.toString(), '');
|
||||
assert.strictEqual(first.status, 0);
|
||||
assert.strictEqual(second.status, 0);
|
||||
|
||||
const firstOrder = getShardOrder(first.stdout.toString());
|
||||
const secondOrder = getShardOrder(second.stdout.toString());
|
||||
assert.notDeepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), kShardFiles);
|
||||
assert.deepStrictEqual([...secondOrder].sort(), kShardFiles);
|
||||
});
|
||||
|
||||
it('should randomize deterministically with --test-random-seed alone', () => {
|
||||
const args = [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-concurrency=1',
|
||||
'--test-random-seed=24680',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
];
|
||||
const first = spawnSync(process.execPath, args);
|
||||
const second = spawnSync(process.execPath, args);
|
||||
|
||||
assert.strictEqual(first.stderr.toString(), '');
|
||||
assert.strictEqual(second.stderr.toString(), '');
|
||||
assert.strictEqual(first.status, 0);
|
||||
assert.strictEqual(second.status, 0);
|
||||
|
||||
const firstOrder = getShardOrder(first.stdout.toString());
|
||||
const secondOrder = getShardOrder(second.stdout.toString());
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), kShardFiles);
|
||||
assert.match(first.stdout.toString(), /# Randomized test order seed: 24680/);
|
||||
});
|
||||
|
||||
it('should print the randomization seed when --test-randomize is used', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-concurrency=1',
|
||||
'--test-randomize',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stderr.toString(), '');
|
||||
assert.strictEqual(child.status, 0);
|
||||
assert.match(child.stdout.toString(), /# Randomized test order seed: \d+/);
|
||||
});
|
||||
|
||||
it('should randomize internal test order deterministically with --test-random-seed', () => {
|
||||
const args = [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
'--test-random-seed=12345',
|
||||
internalOrderFixture,
|
||||
];
|
||||
const first = spawnSync(process.execPath, args);
|
||||
const second = spawnSync(process.execPath, args);
|
||||
|
||||
assert.strictEqual(first.stderr.toString(), '');
|
||||
assert.strictEqual(second.stderr.toString(), '');
|
||||
assert.strictEqual(first.status, 0);
|
||||
assert.strictEqual(second.status, 0);
|
||||
|
||||
const firstOrder = getInternalExecutionOrder(first.stdout.toString());
|
||||
const secondOrder = getInternalExecutionOrder(second.stdout.toString());
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), kInternalTests);
|
||||
});
|
||||
|
||||
it('should randomize internal test order differently across seeds', () => {
|
||||
const orders = [];
|
||||
for (const seed of [11111, 22222, 33333, 44444]) {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-reporter=tap',
|
||||
`--test-random-seed=${seed}`,
|
||||
internalOrderFixture,
|
||||
]);
|
||||
assert.strictEqual(child.stderr.toString(), '');
|
||||
assert.strictEqual(child.status, 0);
|
||||
|
||||
const order = getInternalExecutionOrder(child.stdout.toString());
|
||||
assert.deepStrictEqual([...order].sort(), kInternalTests);
|
||||
orders.push(order.join(','));
|
||||
}
|
||||
|
||||
assert.notStrictEqual(new Set(orders).size, 1);
|
||||
});
|
||||
|
||||
it('should reject --test-randomize with --watch', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--watch',
|
||||
'--test-randomize',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stdout.toString(), '');
|
||||
assert.match(child.stderr.toString(), /The property 'options\.randomize' is not supported with watch mode\./);
|
||||
assert.strictEqual(child.status, 1);
|
||||
assert.strictEqual(child.signal, null);
|
||||
});
|
||||
|
||||
it('should reject --test-random-seed with --watch', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--watch',
|
||||
'--test-random-seed=12345',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stdout.toString(), '');
|
||||
assert.match(child.stderr.toString(), /The property 'options\.randomSeed' is not supported with watch mode\./);
|
||||
assert.strictEqual(child.status, 1);
|
||||
assert.strictEqual(child.signal, null);
|
||||
});
|
||||
|
||||
it('should reject --test-randomize with --test-rerun-failures', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-randomize',
|
||||
'--test-rerun-failures',
|
||||
rerunStateFile,
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stdout.toString(), '');
|
||||
assert.match(child.stderr.toString(), /The property 'options\.randomize' is not supported with rerun failures mode\./);
|
||||
assert.strictEqual(child.status, 1);
|
||||
assert.strictEqual(child.signal, null);
|
||||
});
|
||||
|
||||
it('should reject --test-random-seed with --test-rerun-failures', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-random-seed=12345',
|
||||
'--test-rerun-failures',
|
||||
rerunStateFile,
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stdout.toString(), '');
|
||||
assert.match(child.stderr.toString(), /The property 'options\.randomSeed' is not supported with rerun failures mode\./);
|
||||
assert.strictEqual(child.status, 1);
|
||||
assert.strictEqual(child.signal, null);
|
||||
});
|
||||
|
||||
it('should reject out of range --test-random-seed values', () => {
|
||||
const child = spawnSync(process.execPath, [
|
||||
'--test',
|
||||
'--test-random-seed=4294967296',
|
||||
join(testFixtures, 'shards/*.cjs'),
|
||||
]);
|
||||
|
||||
assert.strictEqual(child.stdout.toString(), '');
|
||||
assert.match(child.stderr.toString(), /The value of "--test-random-seed" is out of range\. It must be >= 0 && <= 4294967295\. Received 4294967296/);
|
||||
assert.strictEqual(child.status, 1);
|
||||
assert.strictEqual(child.signal, null);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,37 @@
|
||||
import '../common/index.mjs';
|
||||
import assert from 'node:assert';
|
||||
import { test, run } from 'node:test';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
|
||||
const fixture = fixtures.path('test-runner', 'randomize', 'first-draw.cjs');
|
||||
const cases = [
|
||||
{ seed: 0, expected: ['a', 'b', 'c'] },
|
||||
{ seed: 1, expected: ['b', 'a', 'c'] },
|
||||
{ seed: 2, expected: ['c', 'a', 'b'] },
|
||||
];
|
||||
|
||||
test('randomizes the first subtest draw deterministically', async () => {
|
||||
for (const { seed, expected } of cases) {
|
||||
const actual = [];
|
||||
const stream = run({
|
||||
files: [fixture],
|
||||
isolation: 'process',
|
||||
randomSeed: seed,
|
||||
});
|
||||
|
||||
stream.on('test:fail', ({ details }) => {
|
||||
throw details.error;
|
||||
});
|
||||
|
||||
stream.on('test:pass', ({ name }) => {
|
||||
if (name === 'a' || name === 'b' || name === 'c') {
|
||||
actual.push(name);
|
||||
}
|
||||
});
|
||||
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
for await (const _ of stream);
|
||||
|
||||
assert.deepStrictEqual(actual, expected, `seed ${seed}`);
|
||||
}
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
import * as common from '../common/index.mjs';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
import { join } from 'node:path';
|
||||
import { basename, join } from 'node:path';
|
||||
import { describe, it, run } from 'node:test';
|
||||
import { dot, spec, tap } from 'node:test/reporters';
|
||||
import consumers from 'node:stream/consumers';
|
||||
@@ -8,6 +8,7 @@ import assert from 'node:assert';
|
||||
import util from 'node:util';
|
||||
|
||||
const testFixtures = fixtures.path('test-runner');
|
||||
const rerunStateFile = join(testFixtures, 'rerun-state.json');
|
||||
|
||||
describe('require(\'node:test\').run', { concurrency: true }, () => {
|
||||
it('should run with no tests', async () => {
|
||||
@@ -99,6 +100,33 @@ describe('require(\'node:test\').run', { concurrency: true }, () => {
|
||||
for await (const _ of stream);
|
||||
});
|
||||
|
||||
for (const isolation of ['process', 'none']) {
|
||||
it(`should preserve cancelled queued subtest order without randomization (${isolation})`, async () => {
|
||||
const stream = run({
|
||||
files: [fixtures.path('test-runner', 'cancelled-report-order.mjs')],
|
||||
isolation,
|
||||
});
|
||||
const cancelledFailures = [];
|
||||
|
||||
stream.on('test:fail', ({ name, details }) => {
|
||||
if (name === 'second' || name === 'third') {
|
||||
cancelledFailures.push({
|
||||
name,
|
||||
failureType: details.error.failureType,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
for await (const _ of stream);
|
||||
|
||||
assert.deepStrictEqual(cancelledFailures, [
|
||||
{ name: 'second', failureType: 'cancelledByParent' },
|
||||
{ name: 'third', failureType: 'cancelledByParent' },
|
||||
]);
|
||||
});
|
||||
}
|
||||
|
||||
it('should be piped with dot', async () => {
|
||||
const result = await run({
|
||||
files: [join(testFixtures, 'default-behavior/test/random.cjs')]
|
||||
@@ -343,20 +371,64 @@ describe('require(\'node:test\').run', { concurrency: true }, () => {
|
||||
});
|
||||
});
|
||||
|
||||
const shardsTestsFixtures = fixtures.path('test-runner', 'shards');
|
||||
const internalOrderTestFile = join(testFixtures, 'randomize', 'internal-order.cjs');
|
||||
const shardFileNames = [
|
||||
'a.cjs',
|
||||
'b.cjs',
|
||||
'c.cjs',
|
||||
'd.cjs',
|
||||
'e.cjs',
|
||||
'f.cjs',
|
||||
'g.cjs',
|
||||
'h.cjs',
|
||||
'i.cjs',
|
||||
'j.cjs',
|
||||
];
|
||||
const internalTestNames = ['a', 'b', 'c', 'd', 'e'];
|
||||
const shardsTestsFiles = shardFileNames.map((file) => join(shardsTestsFixtures, file));
|
||||
|
||||
async function getExecutedShardOrder(options = {}) {
|
||||
const stream = run({
|
||||
files: shardsTestsFiles,
|
||||
concurrency: false,
|
||||
...options,
|
||||
});
|
||||
const executedTestFiles = [];
|
||||
|
||||
stream.on('test:fail', common.mustNotCall());
|
||||
stream.on('test:pass', (passedTest) => {
|
||||
if (passedTest.nesting === 0) {
|
||||
executedTestFiles.push(basename(passedTest.file));
|
||||
}
|
||||
});
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
for await (const _ of stream) ;
|
||||
|
||||
return executedTestFiles;
|
||||
}
|
||||
|
||||
async function getExecutedInternalOrder(options = {}) {
|
||||
const stream = run({
|
||||
files: [internalOrderTestFile],
|
||||
concurrency: false,
|
||||
...options,
|
||||
});
|
||||
const executionOrder = [];
|
||||
|
||||
stream.on('test:fail', common.mustNotCall());
|
||||
stream.on('test:pass', (passedTest) => {
|
||||
if (passedTest.file === internalOrderTestFile && passedTest.name.startsWith('internal ')) {
|
||||
executionOrder.push(passedTest.name.slice('internal '.length));
|
||||
}
|
||||
});
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
for await (const _ of stream) ;
|
||||
|
||||
return executionOrder;
|
||||
}
|
||||
|
||||
describe('sharding', () => {
|
||||
const shardsTestsFixtures = fixtures.path('test-runner', 'shards');
|
||||
const shardsTestsFiles = [
|
||||
'a.cjs',
|
||||
'b.cjs',
|
||||
'c.cjs',
|
||||
'd.cjs',
|
||||
'e.cjs',
|
||||
'f.cjs',
|
||||
'g.cjs',
|
||||
'h.cjs',
|
||||
'i.cjs',
|
||||
'j.cjs',
|
||||
].map((file) => join(shardsTestsFixtures, file));
|
||||
|
||||
describe('validation', () => {
|
||||
it('should require shard.total when having shard option', () => {
|
||||
@@ -517,6 +589,73 @@ describe('require(\'node:test\').run', { concurrency: true }, () => {
|
||||
|
||||
assert.deepStrictEqual(executedTestFiles.sort(), [...shardsTestsFiles].sort());
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe('randomization', () => {
|
||||
it('should randomize file order deterministically when using the same seed', async () => {
|
||||
const firstOrder = await getExecutedShardOrder({ randomize: true, randomSeed: 12345 });
|
||||
const secondOrder = await getExecutedShardOrder({ randomize: true, randomSeed: 12345 });
|
||||
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), [...shardFileNames].sort());
|
||||
});
|
||||
|
||||
it('should randomize file order differently with different seeds', async () => {
|
||||
const firstOrder = await getExecutedShardOrder({ randomize: true, randomSeed: 11111 });
|
||||
const secondOrder = await getExecutedShardOrder({ randomize: true, randomSeed: 22222 });
|
||||
|
||||
assert.notDeepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), [...shardFileNames].sort());
|
||||
assert.deepStrictEqual([...secondOrder].sort(), [...shardFileNames].sort());
|
||||
});
|
||||
|
||||
it('should randomize file order when only randomSeed is provided', async () => {
|
||||
const firstOrder = await getExecutedShardOrder({ randomSeed: 24680 });
|
||||
const secondOrder = await getExecutedShardOrder({ randomSeed: 24680 });
|
||||
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), [...shardFileNames].sort());
|
||||
});
|
||||
|
||||
it('should randomize internal test order deterministically when using the same seed', async () => {
|
||||
const firstOrder = await getExecutedInternalOrder({ randomSeed: 12345 });
|
||||
const secondOrder = await getExecutedInternalOrder({ randomSeed: 12345 });
|
||||
|
||||
assert.deepStrictEqual(firstOrder, secondOrder);
|
||||
assert.deepStrictEqual([...firstOrder].sort(), [...internalTestNames].sort());
|
||||
});
|
||||
|
||||
it('should randomize internal test order differently across seeds', async () => {
|
||||
const orders = [];
|
||||
for (const seed of [11111, 22222, 33333, 44444]) {
|
||||
const order = await getExecutedInternalOrder({ randomSeed: seed });
|
||||
assert.deepStrictEqual([...order].sort(), [...internalTestNames].sort());
|
||||
orders.push(order.join(','));
|
||||
}
|
||||
|
||||
assert.notStrictEqual(new Set(orders).size, 1);
|
||||
});
|
||||
|
||||
it('should emit the randomization seed as a diagnostic message', async () => {
|
||||
const stream = run({
|
||||
files: shardsTestsFiles,
|
||||
concurrency: false,
|
||||
randomize: true,
|
||||
});
|
||||
const diagnostics = [];
|
||||
|
||||
stream.on('test:diagnostic', ({ message }) => {
|
||||
diagnostics.push(message);
|
||||
});
|
||||
// eslint-disable-next-line no-unused-vars
|
||||
for await (const _ of stream) ;
|
||||
|
||||
assert(
|
||||
diagnostics.some((message) => /Randomized test order seed: \d+/.test(message)),
|
||||
`Missing randomization seed diagnostic. Received diagnostics: ${diagnostics.join(', ')}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validation', () => {
|
||||
@@ -540,6 +679,40 @@ describe('require(\'node:test\').run', { concurrency: true }, () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('should not allow randomize with watch mode', () => {
|
||||
assert.throws(() => run({ watch: true, randomize: true }), {
|
||||
code: 'ERR_INVALID_ARG_VALUE',
|
||||
message: /The property 'options\.randomize' is not supported with watch mode\./,
|
||||
});
|
||||
});
|
||||
|
||||
it('should not allow randomSeed with watch mode', () => {
|
||||
assert.throws(() => run({ watch: true, randomSeed: 12345 }), {
|
||||
code: 'ERR_INVALID_ARG_VALUE',
|
||||
message: /The property 'options\.randomSeed' is not supported with watch mode\./,
|
||||
});
|
||||
});
|
||||
|
||||
it('should not allow randomize with rerunFailuresFilePath', () => {
|
||||
assert.throws(() => run({ randomize: true, rerunFailuresFilePath: rerunStateFile }), {
|
||||
code: 'ERR_INVALID_ARG_VALUE',
|
||||
message: /The property 'options\.randomize' is not supported with rerun failures mode\./,
|
||||
});
|
||||
});
|
||||
|
||||
it('should not allow randomSeed with rerunFailuresFilePath', () => {
|
||||
assert.throws(() => run({ randomSeed: 12345, rerunFailuresFilePath: rerunStateFile }), {
|
||||
code: 'ERR_INVALID_ARG_VALUE',
|
||||
message: /The property 'options\.randomSeed' is not supported with rerun failures mode\./,
|
||||
});
|
||||
});
|
||||
|
||||
it('should not allow decimal randomSeed values', () => {
|
||||
assert.throws(() => run({ randomSeed: 1.5 }), {
|
||||
code: 'ERR_OUT_OF_RANGE',
|
||||
});
|
||||
});
|
||||
|
||||
it('should only allow a string in options.cwd', async () => {
|
||||
[Symbol(), {}, [], () => {}, 0, 1, 0n, 1n, true, false]
|
||||
.forEach((cwd) => assert.throws(() => run({ cwd }), {
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
// Test that the output of test-runner/output/randomize_nested_scenarios_output_cli_none.js
|
||||
// matches test-runner/output/randomize_nested_scenarios_output_cli_none.snapshot
|
||||
import '../common/index.mjs';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
import { spawnAndAssert, specTransform, ensureCwdIsProjectRoot } from '../common/assertSnapshot.js';
|
||||
|
||||
ensureCwdIsProjectRoot();
|
||||
await spawnAndAssert(
|
||||
fixtures.path('test-runner/output/randomize_nested_scenarios_output_cli_none.js'),
|
||||
specTransform,
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
// Test that the output of test-runner/output/randomize_nested_scenarios_output_cli_process.js
|
||||
// matches test-runner/output/randomize_nested_scenarios_output_cli_process.snapshot
|
||||
import '../common/index.mjs';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
import { spawnAndAssert, specTransform, ensureCwdIsProjectRoot } from '../common/assertSnapshot.js';
|
||||
|
||||
ensureCwdIsProjectRoot();
|
||||
await spawnAndAssert(
|
||||
fixtures.path('test-runner/output/randomize_nested_scenarios_output_cli_process.js'),
|
||||
specTransform,
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
// Test that the output of test-runner/output/randomize_output_cli_none.js matches
|
||||
// test-runner/output/randomize_output_cli_none.snapshot
|
||||
import '../common/index.mjs';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
import { spawnAndAssert, specTransform, ensureCwdIsProjectRoot } from '../common/assertSnapshot.js';
|
||||
|
||||
ensureCwdIsProjectRoot();
|
||||
await spawnAndAssert(
|
||||
fixtures.path('test-runner/output/randomize_output_cli_none.js'),
|
||||
specTransform,
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
// Test that the output of test-runner/output/randomize_output_cli.js matches
|
||||
// test-runner/output/randomize_output_cli.snapshot
|
||||
import '../common/index.mjs';
|
||||
import * as fixtures from '../common/fixtures.mjs';
|
||||
import { spawnAndAssert, specTransform, ensureCwdIsProjectRoot } from '../common/assertSnapshot.js';
|
||||
|
||||
ensureCwdIsProjectRoot();
|
||||
await spawnAndAssert(
|
||||
fixtures.path('test-runner/output/randomize_output_cli.js'),
|
||||
specTransform,
|
||||
);
|
||||
Reference in new issue
Block a user