Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Playright tests to integration_tests.yml (Issue #14755) #14775

Merged
merged 2 commits into from
Jul 2, 2024

Conversation

alexgibson
Copy link
Member

One-line summary

This doesn't yet remove the Firefox / Chrome Selenium tests, but for now adds Playwright as an additional testing step to our integration tests for main/stage/prod etc.

Significant changes and points to review

Rather than run both the existing integration tests and the playwright tests in parallel, I've gone for a more serial approach for now to try and avoid extra load. The ideas is that the pipeline should look like:

notify-of-test-run-start -> integration-tests -> playwright-tests -> notify-of-test-run-completion

This will add a few extra minutes to our deployment process, but should hopefully be temporary and then we can look to retire the Firefox. / Chrome Selenium tests 🤞

Issue / Bugzilla link

#14755

- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
Copy link
Member Author

@alexgibson alexgibson Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question: do I need to add if: always() to these steps?

Note we use if: always() below to keep things going, rather than
continue-on-error, because that approach falsely marks the overall
test suite as green/passed even if it has some failures.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think don't it's worth adding them to here.

What's not clear from my original comment that you quoted is that the need to keep going under failure is something specific to using a matrix strategy. Our preference (influenced by what we had in GitLab before, as well as practicaltiy) is that we don't stop running through the various test options set up in the matrix if something fails. By using if: always(), even if the tests fail under certain parameters, we get to see wholesale which do and which do not pass, right across the matrix, rather than failing fast and maybe having to re-run a few times to be shown all the bugs to be sorted.

Because your playwright tests don't use the matrix strategy, there's no need to worry about failing "too" fast.

Copy link
Collaborator

@stevejalim stevejalim Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't suggest an inline change to the comment in this pr, but if you could change it to the following that'd be welcome

OLD

     # Note we use if: always() below to keep things going, rather than
    # continue-on-error, because that approach falsely marks the overall
    # test suite as green/passed even if it has some failures.

NEW

    # Note we use `if: always()` below to keep the entire matrix strategy going if one
    # or more permutations fail. This gives us the whole picture, not just some of it.
    # We can't use `continue-on-error`, because that approach falsely marks 
    # the overall test suite as green/passed even if it has some failures.
@alexgibson alexgibson merged commit 2656cbd into mozilla:main Jul 2, 2024
3 checks passed
@alexgibson alexgibson deleted the playwright-ci branch July 2, 2024 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants