Re-run Only Failed Tests
A guide on rerunning only failed Playwright tests in CI
Rerunning only failed tests in CI is one of the most popular way to unblock CI pipelines - indeed, often the tests pass on a rerun, unblocking pending Pull Requests and marking all the checkmarks green.
Speaking strictly, the tests that randomly fail and then pass without any change to the tests, the application under test or the environment, are considered flaky (see Flaky Tests). Even if tests pass after a rerun, it is recommended to explore the failures and eliminate the flakiness.
However there are plenty of pragmatic reasons why teams defer the investigation and just want to unblock the CI pipeline, for example:
skipping a known flaky tests that is currently under investigation
fixing an infrastructure component or CI environment
temporary outage of the CI environment
urgent deployment of a hot fix
Starting from version 1.44 Playwright supports running only the failed test from the last run using --last-failed
CLI flag, for example:
While this feature works well for local environments, there are a few challenges related to rerunning the failed only tests in CI that are related to how Playwright implements parallel execution with shards:
Solving those issues requires maintaining cumbersome configuration and custom scripts, that's why we have created a set of tools that unlock rerunning only the failed Playwright tests in CI, including sharded parallel CI runs and runs created by Currents Orchestration.
📖 Good to know
Playwright allows setting output directory on a project level. If you have multiple projects and they have different output directories, Playwright will generate .last-run.json
in the directory of the first project (according to execution level).
Re-run Failed Playwright Tests - Playwright Shards in CI
The suggested templates do not require maintaining complex CI configurations and scripts - they are compatible with popular CI providers and can be used even without Currents reporter.
Re-run Failed Playwright Tests - Currents Orchestration
When rerunning failed orchestrated CI runs, you'd select "Rerun All Jobs" instead of "Rerun Failed Only"
Currents orchestration for Playwright (see Playwright Orchestration) improves CI execution time by up to 40%, compared to the native Playwright Sharding. It eliminates the need to maintain and update CI configuration and automatically balances the tests between available CI machines in the most optimal way.
Orchestrated runs are conceptually different from the native playwright shards. Currents assigns the tests dynamically to all the available machines - i.e. the more machines are available, the better (also, there's no need to provide the overall number of CI machines in advance).
That's why when rerunning failed orchestrated CI runs, you'd select "Rerun All Jobs" instead of "Rerun Failed Only":
The CI provider will spin up all the containers
Currents will dynamically assign the failed tests to all the available containers
More available containers will run the tests in parallel faster
Last updated