Flaky Tests
1 min read
The Flaky Tests report highlights tests that flip statuses across runs — bouncing between pass and fail rather than failing consistently. It gives you a stable triage shortlist so you can fix the noisiest tests first and tell genuine bugs apart from flakes.
Filtering
Section titled “Filtering”- Search by title or code: Find a specific test case.
- Min flips: Set a minimum number of pass↔fail transitions a test must have before it appears.
- Classification: Filter to Flaky or Unstable tests.
- Folder: Limit results to a folder.
Key metrics
Section titled “Key metrics”The report summarizes the tests with run data in scope:
- Tests with data
- Flaky
- Unstable
- Stable
Probable breakages and folder instability
Section titled “Probable breakages and folder instability”- Probable breakages: Tests whose recent signals point to a real break rather than a flake — for example, a failing streak or open defects.
- Folder instability: Each folder’s instability percentage, calculated as (Flaky + Unstable) ÷ tests with data, so you can see which areas misbehave the most.
Reading the flaky tests table
Section titled “Reading the flaky tests table”The main table lists each flaky test with:
- Streak: How many consecutive recent runs ended with the same status (for example, “3× Failed”).
- Last failed: The most recent run that ended Failed or Blocked.
- Flips: The number of pass-to-fail or fail-to-pass transitions across the recent runs in scope. Higher means more bouncing.
- Fail %: Failed runs ÷ total runs in the flakiness window.
- Open defects: Defects linked to the test that are not closed — a value greater than zero often signals a real bug rather than a flake.
- Classification: Whether the test is classified as Flaky or Unstable by the flakiness model.
Exporting
Section titled “Exporting”Download the report as a PDF or as Excel (XLSX). The PDF dialog lets you choose sections — cover page, executive summary, key metrics, probable breakages, folder instability, the flaky tests table, and a legend explaining flips, fail rate, streaks, and the classifications.