Playwright - GitLab CI/CD

Running Playwright test in parallel using GitLab CI/CD

TL;DR take a look at the example repository:

https://gitlab.com/currents.dev/gitlab-playwright-currents

The example .gitlab-ci.yml file creates a GitLab CI pipeline with 3 workers using 3 Playwright Shards:

# .gitlab-ci.yml
default:
  image: mcr.microsoft.com/playwright:latest

stages:
  - test

test:
  image: mcr.microsoft.com/playwright:latest
  stage: test
  parallel: 3

  script:
    - npm ci
    - npx playwright install
    - cd ./basic
    - npx pwc --key $CURRENTS_RECORD_KEY --project-id bnsqNa --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Running 3 parallel jobs Playwright jobs in GitLab CI

Setup

• Create an account at https://app.currents.dev and grab the Project ID and Record Key.
• Add CURRENTS_RECORD_KEY as GitLab CI/CD Secrets (make sure it's not protected)

Additional resources:

• Playwright Features on Currents: https://currents.dev/playwright
• Integration Documentation: https://currents.dev/readme/integration-with-playwright/currents-playwright
• CI Build ID Guide: https://currents.dev/readme/guides/cypress-ci-build-id

GitLab CI Playwright Results

The results are being reported to Currents for more efficient troubleshooting, and monitoring test suite flakiness and performance.

Currents will collect the following information:

• console output

• screenshots

• videos

• trace files

• timing

• outcomes

• flaky tests

• error details

• tags for more convenient management of the tests

GitLab Merge Request Notes

Take a look at the example merge request: https://gitlab.com/currents.dev/gitlab-playwright-currents/-/merge_requests/3

When GitLab integration is enabled, Currents will post a Merge Commit note with run results:

Currents Merge Request Notes

In addition, Currents will post an External Status check for every Playwright project configured. For example:

Example of GitLab CI External Status Check

Re-run only failed Playwright tests in GitLab CI/CD

When a workflow fails in GitLab CI/CD you have the option to re-run the failed jobs. However, an additional setup is required for properly configuring Playwright for rerunning only the failed tests. See Re-run Only Failed Tests guide for details.

Rerunning only the failed jobs on GitLab CI/CD

Playwright Sharding

If you're using Playwright Sharding for running your tests in parallel, use currents cache command to store the last run results and simplify re-run workflows.

Example workflows are available in our GitLab repositories:

• reruns-pwc.yml - re-run only failed tests on GitLab CI using 3 parallel runners and Playwright Shards + Currents pwc command
• reruns-reporter.yml - re-run only failed tests on GitLab CI using 3 parallel runners and Playwright Shards + Currents reporter in playwright.config.ts

Install the @currents/cmd package

npm i -D @currents/cmd

Add an after_script to upload the cache

Add an after_script to the end of your job that uploads run information to the cache

after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

See the configuration for details on the flags.

Add a download cache step to you script

Update your job script to download the cache prior to running tests

script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PW_FLAGS will contain the correct --shard flag, as well as --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    - npx playwright install
    # Run playwright, provide a build id with the run-attempt included and also pass $EXTRA_PW_FLAGS 
    - export CURRENTS_CI_BUILD_ID="reporter-$CI_PIPELINE_ID-$RUN_ATTEMPT"
    - npx playwright test -c playwright.config.reporter.ts $EXTRA_PW_FLAGS

See the configuration for details on the flags.

A full example

test-rerun-reporter:
  image: mcr.microsoft.com/playwright:v1.47.0
  stage: test
  parallel: 3
  variables:
    CURRENTS_PROJECT_ID: bnsqNa
    # CURRENTS_RECORD_KEY: <set in your CI/CD variables>
  script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PW_FLAGS will contain the correct --shard flag, as well as --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    - npx playwright install
    - cd ./basic
    # Run playwright, provide a build id with the run-attempt included and also pass $EXTRA_PW_FLAGS 
    - export CURRENTS_CI_BUILD_ID="reporter-$CI_PIPELINE_ID-$RUN_ATTEMPT"
    - npx playwright test -c playwright.config.reporter.ts $EXTRA_PW_FLAGS
  after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

Currents Orchestration

In case you're using Orchestration Setup for running your Playwright tests in parallel, use currents api command to fetch the results of the last run from the API.

An example workflow is available in our GitLab demo repository

• rerun-or8n.yml - rerun only failed tests on GitLab CI/CD with Currents Orchestration

Install the @currents/cmd package

npm i -D @currents/cmd

Set CURRENTS_API_KEY environment variable

Obtain an API key (see API Keys) and Record Key from Currents Dashboard and set GitLab CI/CD variable accordingly

variables:
    CURRENTS_PROJECT_ID: bnsqNa
    CURRENTS_RECORD_KEY: # set CI/CD variable
    CURRENTS_API_KEY: # set  CI/CD variable

Add an after_script to upload the cache

Add an after_script to the end of your job that uploads GitLab retry information to the cache

after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

See currents cache documentation for all the available options

Add an api get-run step to you script

Update your job script to download the cache prior to running tests, and then also grab the previous run.

script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PWCP_FLAGS will contain the the --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    # Grab the complete last run from the API so orchestration can find all the failures
    - npx currents api get-run --branch=$CI_COMMIT_REF_NAME --pw-last-run --output=basic/test-results/.last-run.json || true
    - npx playwright install
    # Run pwc-p, provide a build id with the run-attempt included, and also pass EXTRA_PWCP_FLAGS
    - npx pwc-p --ci-build-id=or8n-$CI_PIPELINE_ID-$RUN_ATTEMPT $EXTRA_PWCP_FLAGS

See currents api documentation to explore all the available options

A full example

test-rerun-pwcp:
  image: mcr.microsoft.com/playwright:v1.47.0
  stage: test
  parallel: 3
  variables:
    CURRENTS_PROJECT_ID: bnsqNa
    # CURRENTS_RECORD_KEY: <set in your CI/CD variables>
    # CURRENTS_API_KEY: <set in your CI/CD variables>
  script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PWCP_FLAGS will contain the the --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    # Grab the complete last run from the API so orchestration can find all the failures
    - npx currents api get-run --branch=$CI_COMMIT_REF_NAME --pw-last-run --output=basic/test-results/.last-run.json || true
    - npx playwright install
    # Run pwc-p, provide a build id with the run-attempt included, and also pass EXTRA_PWCP_FLAGS
    - npx pwc-p --ci-build-id=or8n-$CI_PIPELINE_ID-$RUN_ATTEMPT $EXTRA_PWCP_FLAGS
  after_script:
    # Save the last-run to cache in order to track the run attempts
    - npx currents cache set --preset last-run --pw-output-dir basic/test-results