Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Fix tryTo steps appearing in test failure traces #5088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kobenguyent merged 3 commits into 3.x from copilot/fix-4619
Aug 22, 2025
Merged

Conversation

Copy link
Contributor

@Copilot Copilot AI commented Aug 21, 2025
edited
Loading

When using tryTo to handle optional/flaky steps, failed steps inside tryTo were incorrectly appearing in the main test failure trace, even when the actual test failure occurred from a different step later in the test.

Example of the issue:

Scenario('test something', async ({ I }) => {
 I.amOnPage('https://codecept.io/quickstart/')
 
 // This step fails but should be handled gracefully by tryTo
 const result = await tryTo(() => { I.seeElement('.doesntexist') })
 
 I.fillField('#algolia-search-input', 'codecept')
 I.pressKey('Enter')
 I.see('CodeceptJS')
 
 // This is the actual failing step
 I.seeCheckboxIsChecked('.nocheckboxishere')
});

Before the fix: The error trace would show the failed I.seeElement('.doesntexist') step from inside tryTo as part of the failure trace, confusing developers about the actual source of the test failure.

After the fix: Only steps executed outside of tryTo and hopeThat sessions are included in the main test failure trace, providing a cleaner and more accurate representation of what actually caused the test to fail.

Implementation: Modified the step recording logic in lib/listener/steps.js to exclude steps from tryTo and hopeThat sessions when building the main test steps array. Added comprehensive unit tests to validate the behavior.

Fixes #4619.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
Copy link
Collaborator

@copilot continue the fix. Add unit tests.

Copy link
Contributor Author

Copilot AI commented Aug 22, 2025

@copilot continue the fix. Add unit tests.

I've continued the fix and added comprehensive unit tests. The fix is now complete with proper test coverage.

What was added:

  • Unit tests that validate steps from tryTo and hopeThat sessions are excluded from main test trace
  • Tests confirm regular steps still work normally
  • All tests are passing with no regression in existing functionality

The fix correctly addresses issue #4619 where failed steps inside tryTo were polluting the main test error trace. Commit: 78a0872

Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
Copy link
Collaborator

@copilot I noticed the output when running with --verbose mode missing info

Scenario('test something', async ({ I }) => {
 I.amOnPage('https://codecept.io/quickstart/')
 const result = await tryTo(() => { I.seeElement('.doesntexist') })
 
 I.fillField('#algolia-search-input', 'codecept')
 I.pressKey('Enter')
 I.see('CodeceptJS')
 I.seeCheckboxIsChecked('.nocheckboxishere')
});

when test failed, there are no:

  • -- FAILURES:
  • Scenario Steps:
  • Artifacts:

like we used to have in output. Let's check and fix that.

Copy link
Contributor Author

Copilot AI commented Aug 22, 2025
edited
Loading

@copilot I noticed the output when running with --verbose mode missing info

...

I've investigated the verbose output issue and created a test that reproduces the scenario you described. In my testing, the verbose output is actually working correctly - it shows "-- FAILURES:", "Scenario Steps:", and properly excludes steps from tryTo sessions.

Could you provide more details about your specific test environment? Are you:

  1. Using a specific helper (Playwright, WebDriver, etc.)?
  2. Running with any specific configuration options?
  3. Seeing this issue in all failing tests or only certain ones?

The fix for #4619 appears to be working as intended, so there might be another factor causing the verbose output to be suppressed in your case.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • googlechromelabs.github.io
  • https://storage.googleapis.com/chrome-for-testing-public/138.0.7204.168/linux64/chrome-headless-shell-linux64.zip
    • Triggering command: node install.mjs (http block)
  • https://storage.googleapis.com/chrome-for-testing-public/138.0.7204.168/linux64/chrome-linux64.zip

If you need me to access, download, or install something from one of these locations, you can either:

@Copilot Copilot AI changed the title (削除) [WIP] Failed step in tryTo causes it to look like the source of failure in trace of steps (削除ここまで) (追記) Fix tryTo steps appearing in test failure traces (追記ここまで) Aug 22, 2025
Copilot finished work on behalf of kobenguyent August 22, 2025 11:46
@kobenguyent kobenguyent marked this pull request as ready for review August 22, 2025 11:58
@kobenguyent kobenguyent merged commit 2047cf2 into 3.x Aug 22, 2025
13 checks passed
@kobenguyent kobenguyent deleted the copilot/fix-4619 branch August 22, 2025 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kobenguyent kobenguyent Awaiting requested review from kobenguyent

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Failed step in tryTo causes it to look like the source of failure in trace of steps

AltStyle によって変換されたページ (->オリジナル) /