Currents Documentation
Currents.devGitHubChangelog
  • Getting Started
    • What is Currents?
    • Playwright
      • Playwright: Quick Start
      • Troubleshooting Playwright
    • Cypress
      • Your First Cypress Run
      • Integrating with Cypress
        • Compatibility
        • Alternative Cypress Binaries
      • Troubleshooting Cypress
    • Jest
      • Your First Jest Run
      • Detox + Jest
      • Troubleshooting Jest
    • Others
    • CI Setup
      • GitHub Actions
        • Cypress - GitHub Actions
        • Playwright - GitHub Actions
        • Jest - GitHub Actions
        • Node.js - GitHub Actions
        • Commit data for GitHub Actions
        • Custom Docker runners
        • Named Runners
      • GitLab
        • Cypress - GitLab CI/CD
        • Playwright - GitLab CI/CD
        • Custom Docker runners
      • Jenkins
        • Cypress - Jenkins
        • Playwright - Jenkins
      • CircleCI
        • Cypress - CircleCI
        • Playwright - CircleCI
      • Bitbucket
        • Cypress - Bitbucket Pipelines
      • Azure DevOps
        • Cypress - Azure DevOps
        • Playwright - Azure DevOps
      • AWS Code Build
        • Cypress - AWS Code Build
        • Playwright - AWS Code Build
      • NX
        • Playwright - NX
        • Cypress - NX
  • Guides
    • Record Key
    • CI Build ID
    • Reporting
      • Reporting Strategy
      • Reporting in CI
      • Step-Level Reporting
    • CI Optimization
      • Playwright Parallelization
      • Orchestration Setup
      • Fully Parallel Mode
      • Re-run Only Failed Tests
      • Cloud Spot Instances
      • Failing Fast
      • Load Balancing
    • Code Coverage
      • Code Coverage for Playwright
      • Code Coverage for Cypress
    • Currents Actions
      • Setup Currents Actions
      • Using Currents Actions
      • Reference
        • Conditions
        • Actions
    • Playwright Component Testing
    • Playwright Visual Testing
    • Playwright Annotations
    • Playwright Tags
    • MCP Server
  • Dashboard
    • Projects
      • Projects Summary view
      • Project Settings
      • Archive and Unarchive Projects
    • Runs
      • Run Status
      • Run Details
      • Commit Information
      • Tags
      • Run Timeouts
      • Cancelling Runs
      • Deleting Runs
      • Run Progress
    • Tests
      • Spec File Status
      • Test Status
      • Flaky Tests
      • Test History
    • Test Suite Explorer
      • Test Explorer
        • Tests Performance
      • Spec Files Explorer
        • Spec Files Performance
      • Errors Explorer
  • Automated Reports
  • Insights and Analytics
  • Administration
    • Email Domain Based Access
    • SSO SAML2.0
      • SAML2.0 Configuration
      • SCIM User Provisioning
      • IdP-initiated Sessions
      • JumpCloud
        • JumpCloud User provisioning
      • Okta
        • Okta User provisioning
      • Troubleshooting SSO
    • Billing & Usage
  • Billing and Pricing
  • Resources
    • Reporters
      • cypress-cloud
        • Batched Orchestration
        • Migration to Cypress@13
      • @currents/cli
      • @currents/playwright
        • Configuration
        • pwc
        • pwc-p (orchestration)
        • Playwright Fixtures
      • @currents/jest
      • @currents/node-test-reporter
      • @currents/cmd
        • currents api
        • currents upload
        • currents cache
        • currents convert
      • Data Format Reference
    • Integrations
      • GitHub
        • GitHub App
        • GitHub OAuth
      • GitLab
      • Slack
      • Microsoft Teams
      • HTTP Webhooks
      • Bitbucket
    • API
      • Introduction
      • Authentication
      • API Keys
      • Errors
      • Pagination
      • API Resources
        • Instances
        • Runs
        • Projects
        • Spec Files
        • Test Signature
        • Test Results
    • Data Privacy
      • Access to Customer Data
      • Data Retention
      • Cloud Endpoints
    • Support
Powered by GitBook
On this page
  • Playwright Test Status
  • Cypress Tests Status

Was this helpful?

  1. Dashboard
  2. Tests

Test Status

Playwright and Cypress Test Status - detailed guide and explanation

PreviousSpec File StatusNextFlaky Tests

Last updated 10 months ago

Was this helpful?

Playwright and Cypress test status is mostly straightforward to interpret, however, there are a few confusing concepts that can be tricky to figure out.

Playwright Test Status

Playwright test's status is a composition of:

  • the status of its

  • its

The final status is determined after a test has finished all its retries, hook and fixtures and depends on the expected status and the status of all the retries.

See Playwright Test Status - Summary Table for reference of how Playwright statuses appear in Currents.

Playwright Test Retry Status

Each test retry can be in one of the following statuses:

  • passed - means that the test retry didn't encounter any error or exception while executing the test body or the associated test.beforeEach or test.afterEach hooks

  • failed - means that the test retry failed due to an assertion error, exception in the test's body or the associated test.beforeEach or test.afterEach hooks. An exception or failure of test.beforeAll hooks will cause a failed status in the first test attempt according to the execution order.

  • interrupted - means that the execution of a test retry was halted before it could be completed. This interruption could occur due to a variety of reasons, such as:

    • Uncaught Exception.

    • Unhandled Rejection.

    • Interrupted by a system signal like SIGINT (CTRL / CMD + C).

    • limit reached (the rest of the tests get stats interrupted).

  • timedOut - indicates that the test retry did not finish within a specified time limit, leading to its automatic termination. Each test's timeout depends on , , , and .

  • skipped - indicates that the test retry was not executed. This can happen for several reasons:

    • If you programmatically skip tests based on certain conditions.

    • If you explicitly skip a test with or .

    • If you're using a feature like .only to focus on specific test.

    • If a previous test in a test group failed or timed out (see below).

Playwright Expected Status

  • Other tests are expected to be passed.

Playwright Test Outcome

  • expected - means all the attempts' status matches the expectedStatus

    • a test has expectedStatus=failed all attempts were failed (but not timedOut )

    • a test has expectedStatus=passed and all attempts have a status passed

  • unexpected - all attempts' status does not match the expectedStatus

  • flaky - some attempts' status matches the expected status, and some do not. See below.

Playwright Flaky Tests

Playwright marks a test as flaky if some attempts match the expected status and some don't. For example:

  • there are multiple test attempts and

  • there is at least one passed attempt and

  • there's at least one failed or timedOut attempt

It can be tricky to interpret the result depending on the expected status. For example, all the tests below will be considered flaky:


import { expect, test } from "@playwright/test";

const wait = (ms: number) => new Promise((r) => setTimeout(r, ms));
test.describe.configure({ timeout: 500 });

test("expectedStatus=passed; failed, passed", async ({ page }, { retry }) => {
  if (retry === 0) {
    throw new Error("oh!");
  }

  expect(true).toBe(true);
});

test("expectedStatus=failed; timedOut, passed", async ({ page }, { retry }) => {
  if (retry === 0) {
    await wait(1000);
  }
  expect(true).toBe(true);
});

test("expectedStatus=failed; passed, failed", async ({ page }, { retry }) => {
  test.fail();
  if (retry === 0) {
    expect(true).toBe(true);
  }
  if (retry === 1) {
    expect(true).toBe(false);
  }
});

test("expectedStatus=failed; timedOut, failed", async ({ page }, { retry }) => {
  test.fail();
  if (retry === 0) {
    await wait(1000);
  }
  if (retry === 1) {
    expect(true).toBe(false);
  }
});

See Flaky Testsfor more details about learning how to use Currents Dashboard when dealing with flaky tests.

Serial Mode and Playwright Test Status

It is recommended to avoid dependencies between tests and ensure that every test is isolated for more efficient retries and parallelization

Determining test status for serial tests follows the same principles as for other tests, however, you should be aware of that behaviour and interpret the results accordingly.

Console Output of the Playwrigth Reporter

The Playwright Console Output has a precise and accurate presentation of the execution workflow.

  • Each attempt has its start and end timestamps, status details, detected errors, stdout and stderr output.

Moreover, inline links to individual test results is part of console output:

Playwright Test Status - Summary Table

Refer to the table below to determine how a certain combination of expectedStatus and outcome affects the status reported to Currents Dashboard.

PW Expected Status
PW Actual Status
PW Outcome
Currents Status

failed

passed

unexpected

failed

failed

timedOut

unexpected

failed

failed

failed

expected

passed

passed

passed

expected

passed

passed

timedOut

unexpected

failed

passed

failed

unexpected

failed

skipped

skipped

expected

ignored

Cypress Tests Status

Cypress test can be in one of the following states:

  • Passed - a test completed all attempts without any exceptions or errors during its execution, including all before and beforeEach expressions. During its execution, the test runner didn't encounter any timeouts, exceptions or failed assertions. Please note, for tests with multiple attempts, if the last attempt was successful, the whole test will be marked as "Passed" but "Flaky". See Flaky Tests for details.

  • Failed - a test that triggered an exception or one of its assertions failed during the execution of before, beforeEach or the test body for each of its attempts. A failed test activates capturing a stack trace, a screenshot and a video recording. All of those will be shown in Currents dashboard and are available for exploration of failed test details. In some cases, cypress runner isn't able to capture a screenshot and useful error information - for example, when the spec file associated with the test cannot be parsed, or there's a crash in the runtime environment.

  • Ignored / Pending - a test was excluded by a developer, e.g. it.skip() and was excluded by a runner. Ignored/pending tests are tests that were:

    • explicitly marked by a developer to be excluded, e.g. it.skip(), describe.skip() or xit()

    • a test that is not implemented - i.e. a test that has no "body". For example, it('nothing is here')

    • describe('Skip in Chrome', { browser: '!chrome' }, () => {/* ... */})

  • Skipped - a test that was supposed to run but has been skipped because of a runtime error. This state is exclusive to Cypress executions. One of the most common examples is a situation where there's a crash in beforeEach. After unsuccessfully running the first test and recognizing an error in beforeEach, Cypress runner "skips" the rest of the tests because they would fail due to the same error. Currents marks spec files and runs with skipped tests as "failed".

In addition to the statuses above, a test can be marked as Flaky if the test passes after a few failing attempts. See Flaky Tests.

Playwright tests have , you can set the expected status as follows:

Tests marked as or are expected to be skipped.

Tests marked as are expected to be failed.

Combining the expected status with test retries status defines :

skipped - means all the attempts were skipped (because the test was marked as or )

Enabling for Playwright Tests changes the default behaviour: a group of tests designated to run serially always run together, one after another. A failure or a timeout in a test causes all the subsequent tests in the group to become skipped. Depending on the number of retries left, all the tests will run again until completion.

Use to group dependent tests to ensure they will always run together and in order. If one of the tests fails, all subsequent tests are skipped. All tests in the group are retried together.

Clear separation between tests flakiness and outcome status - a test's "status" and flakiness requires considering the outcome of all the attempts, the expected status and the execution mode (e.g. can generate extra attempts).

a test suite that was filtered out in , for example:

attempts
expected status
test.afterEach()
maxFailures
testConfig.timeout
testProject.timeout
test.setTimeout()
test.slow()
testInfo.setTimeout()
test.skip()
test.fixme()
expected status
test.skip()
test.fixme()
test. fail()
Playwright test outcome
test.skip()
test.fixme()
Serial Mode
test.describe.serial()
determining
"serial" mode
test/suite configuration
Example - Console output of the Playwright reporter
Example - Console output of the Playwright reporter