Fix test statistics reporting issue in pool mode - consolidate results properly to prevent duplicate counting #5089

Original file line number	Diff line number	Diff line change
Expand Up		@@ -64,6 +64,7 @@ You don't need to worry about asynchronous nature of NodeJS or about various API
		- Also plays nice with TypeScript.
		- </> Smart locators: use names, labels, matching text, CSS or XPath to locate elements.
		- 🌐 Interactive debugging shell: pause test at any point and try different commands in a browser.
	- ⚡ Parallel testing with dynamic test pooling for optimal load balancing and performance.
		- Easily create tests, pageobjects, stepobjects with CLI generators.

		## Installation
Expand Down

1 change: 1 addition & 0 deletions bin/codecept.js

Show comments View file Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -196,6 +196,7 @@ program
		.option('-i, --invert', 'inverts --grep matches')
		.option('-o, --override [value]', 'override current config options')
		.option('--suites', 'parallel execution of suites not single tests')
	.option('--by <strategy>', 'test distribution strategy: "test" (pre-assign individual tests), "suite" (pre-assign test suites), or "pool" (dynamic distribution for optimal load balancing, recommended)')
		.option(commandFlags.debug.flag, commandFlags.debug.description)
		.option(commandFlags.verbose.flag, commandFlags.verbose.description)
		.option('--features', 'run only *.feature files and skip tests')
Expand Down

24 changes: 22 additions & 2 deletions docs/commands.md

Show comments View file Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -102,12 +102,32 @@ DEBUG=codeceptjs:* npx codeceptjs run

		## Run Workers

	Run tests in parallel threads.
	Run tests in parallel threads. CodeceptJS supports different distribution strategies for optimal performance.

	```
	```bash
	# Run with 3 workers using default strategy (pre-assign tests)
		npx codeceptjs run-workers 3

	# Run with pool mode for dynamic test distribution (recommended)
	npx codeceptjs run-workers 3 --by pool

	# Run with suite distribution
	npx codeceptjs run-workers 3 --by suite

	# Pool mode with filtering
	npx codeceptjs run-workers 4 --by pool --grep "@smoke"
		```

	Test Distribution Strategies:

	- `--by test` (default): Pre-assigns individual tests to workers
	- `--by suite`: Pre-assigns entire test suites to workers
	- `--by pool`: Dynamic distribution for optimal load balancing (recommended for best performance)

	The pool mode provides the best load balancing by maintaining tests in a shared pool and distributing them dynamically as workers become available. This prevents workers from sitting idle and ensures optimal CPU utilization, especially when tests have varying execution times.

	See [Parallel Execution](/parallel) documentation for more details.

		## Run Rerun <Badge text="Since 3.3.6" type="warning"/>

		Run tests multiple times to detect and fix flaky tests.
Expand Down

82 changes: 82 additions & 0 deletions docs/parallel.md

Show comments View file Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -32,6 +32,88 @@ By default, the tests are assigned one by one to the available workers this may
		npx codeceptjs run-workers --suites 2
		```

	### Test Distribution Strategies

	CodeceptJS supports three different strategies for distributing tests across workers:

	#### Default Strategy (`--by test`)
	Tests are pre-assigned to workers at startup, distributing them evenly across all workers. Each worker gets a predetermined set of tests to run.

	```sh
	npx codeceptjs run-workers 3 --by test
	```

	#### Suite Strategy (`--by suite`)
	Test suites are pre-assigned to workers, with all tests in a suite running on the same worker. This ensures better test isolation but may lead to uneven load distribution.

	```sh
	npx codeceptjs run-workers 3 --by suite
	```

	#### Pool Strategy (`--by pool`) - Recommended for optimal performance
	Tests are maintained in a shared pool and distributed dynamically to workers as they become available. This provides the best load balancing and resource utilization.

	```sh
	npx codeceptjs run-workers 3 --by pool
	```

	## Dynamic Test Pooling Mode

	The pool mode enables dynamic test distribution for improved worker load balancing. Instead of pre-assigning tests to workers at startup, tests are stored in a shared pool and distributed on-demand as workers become available.

	### Benefits of Pool Mode

	* Better load balancing: Workers never sit idle while others are still running long tests
	* Improved performance: Especially beneficial when tests have varying execution times
	* Optimal resource utilization: All CPU cores stay busy until the entire test suite is complete
	* Automatic scaling: Workers continuously process tests until the pool is empty

	### When to Use Pool Mode

	Pool mode is particularly effective in these scenarios:

	* Uneven test execution times: When some tests take significantly longer than others
	* Large test suites: With hundreds or thousands of tests where load balancing matters
	* Mixed test types: When combining unit tests, integration tests, and end-to-end tests
	* CI/CD pipelines: For consistent and predictable test execution times

	### Usage Examples

	```bash
	# Basic pool mode with 4 workers
	npx codeceptjs run-workers 4 --by pool

	# Pool mode with grep filtering
	npx codeceptjs run-workers 3 --by pool --grep "@smoke"

	# Pool mode in debug mode
	npx codeceptjs run-workers 2 --by pool --debug

	# Pool mode with specific configuration
	npx codeceptjs run-workers 3 --by pool -c codecept.conf.js
	```

	### How Pool Mode Works

	1. Pool Creation: All tests are collected into a shared pool of test identifiers
	2. Worker Initialization: The specified number of workers are spawned
	3. Dynamic Assignment: Workers request tests from the pool when they're ready
	4. Continuous Processing: Each worker runs one test, then immediately requests the next
	5. Automatic Completion: Workers exit when the pool is empty and no more tests remain

	### Performance Comparison

	```bash
	# Traditional mode - tests pre-assigned, some workers may finish early
	npx codeceptjs run-workers 3 --by test # ✓ Good for uniform test times

	# Suite mode - entire suites assigned to workers
	npx codeceptjs run-workers 3 --by suite # ✓ Good for test isolation

	# Pool mode - tests distributed dynamically
	npx codeceptjs run-workers 3 --by pool # ✓ Best for mixed test execution times
	```

		## Test stats with Parallel Execution by Workers

		```js
Expand Down

17 changes: 16 additions & 1 deletion lib/command/run-workers.js

Show comments View file Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -10,7 +10,22 @@ module.exports = async function (workerCount, selectedRuns, options) {

		const { config: testConfig, override = '' } = options
		const overrideConfigs = tryOrDefault(() => JSON.parse(override), {})
	const by = options.suites ? 'suite' : 'test'

	// Determine test split strategy
	let by = 'test' // default
	if (options.by) {
	// Explicit --by option takes precedence
	by = options.by
	} else if (options.suites) {
	// Legacy --suites option
	by = 'suite'
	}

	// Validate the by option
	const validStrategies = ['test', 'suite', 'pool']
	if (!validStrategies.includes(by)) {
	throw new Error(`Invalid --by strategy: ${by}. Valid options are: ${validStrategies.join(', ')}`)
	}
		delete options.parent
		const config = {
		by,
Expand Down

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix test statistics reporting issue in pool mode - consolidate results properly to prevent duplicate counting #5089

Uh oh!

Fix test statistics reporting issue in pool mode - consolidate results properly to prevent duplicate counting #5089

Filter by extension

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!