Insights and Analytics

Use Insights to explore the aggregated metrics related to your test suite performance:

• changes and trends in duration and success rate of your builds and tests over time,

• test suite flakiness and stability.

Customization

Customize the reports by applying various filters for a more focused and meaningful insights.

All charts support filtering by tag, git author, and git branch. In addition:

• Suite Size, Test Results, and Test Flakiness charts also support the group filter.

• Test Results and Test Flakiness charts also support the Playwright Annotationsfilter.

The aggregated metrics are based in the samples collected during the selected Date Range.

Insight Charts

Run Status

Represents the distribution of the outcomes of your builds/runs over time.

• Overall runs: overall runs created during the selected period, regardless of their completion or the end state.

• Failed runs: runs that have one or more failed tests. Failed runs count includes cancelled and timed-out tests.

• Passed runs: runs that have zero failed tests.

• Pass Rate: passed runs / overall runs.

Run Duration

Represents the daily/weekly average duration of fully reported runs. A fully reported run is a run that wasn't cancelled and didn't time out.

Please note that cancelled or timed-out runs are excluded from the report.

Run Completion

Represents the distribution of runs by their completion over time.

• Overall runs: overall runs created during the selected period, regardless of their completion or the end state.

• Fully reported runs: runs that weren't cancelled and didn't time out.

• Cancelled runs: see Cancelling Runs

• Timed out runs: see Run Timeouts

Test Suite Size

Represents the maximum daily/weekly amount of specs/tests for fully completed runs. A fully reported run is a run that wasn't cancelled and didn't time out.

• Spec files: the maximum number of spec files detected in a run for the selected period

• Tests: the maximum number of tests detected in a run for the selected period

Please note that cancelled or timed-out runs are excluded from the report.

Test Results

Shows the distribution of test outcomes over time.

• Total tests: overall tests recorded for the selected period, regardless of their outcome.

• Passed tests: tests that were successfully completed on all attempts without any exceptions or errors during their execution.

• Failed tests: tests that either failed or were skipped because of an error in beforeEach/beforeAll

• Ignored test: tests that weren't run, e.g. it.skip()

• Success Rate: passed / (total - pending)

Test Flakiness

Shows the distribution of flaky tests over time.

• Flaky tests: overall flaky tests detected for the period. See Flaky Tests

• Flakiness rate: flaky tests count / passed tests count