How do I tell a real bug from a flaky test?

Run the test several times with retries off and count. A real bug fails the same way on every run, and a flaky test passes some runs and fails others with nothing changed. The example below does exactly that on any machine with Playwright, and then shows what Nexus Studio does with each result.

By the Nexus Studio team, Verdict System LLC. October 27, 2026.

How can I tell them apart in one minute?

Run the test several times in a row with retries off. Playwright does it with two flags: --retries=0, so a retry cannot hide a failure, and --repeat-each=6, which runs every test six times. Then count the results of each test.

• It failed every time: treat it as a bug until you can show otherwise. A failure on every run can still come from the environment, such as a blocked request or an unreachable host, so read the error first.
• It passed some runs and failed others: it is flaky. The code and the test did not change between those runs, so something unstable is involved, in the application or in the test.
• It passed every time: nothing to see yet. A defect that shows up once in fifty runs needs more runs, or a look at the History of the test.

Can I try it on a runnable example?

Save this as bug-or-flaky.spec.js. The first test simulates a flake: it fails on one run in three. The second test is a real bug: the page says "Welcome" and the test expects "Welcome back", so no run can pass.

import { test, expect } from '@playwright/test';

// Fails on its first attempt of every third run, passes otherwise: a simulated flake.
test('flaky: a slow load now and then', async ({ page }, info) => {
    await page.setContent('<h1>Welcome</h1>');
    expect((info.repeatEachIndex + info.retry) % 3, 'simulated slow load').not.toBe(0);
});

// Fails the same way every time: the heading says "Welcome", the test expects "Welcome back".
test('real bug: wrong heading', async ({ page }) => {
    await page.setContent('<h1>Welcome</h1>');
    await expect(page.locator('h1')).toHaveText('Welcome back', { timeout: 300 });
});

Run it six times with retries off:

npx playwright test --retries=0 --repeat-each=6

Playwright ends with 8 failed and 4 passed. The count per test is what answers the question, and it is what the list reporter printed, line by line:

flaky: a slow load now and then    4 passed, 2 failed
real bug: wrong heading            0 passed, 6 failed

Now run it once with two retries:

npx playwright test --retries=2

Playwright ends with one test in each group. The flaky test failed its first attempt and passed on retry #1. The real bug failed all three attempts.

1 failed
  bug-or-flaky.spec.js:10:1 › real bug: wrong heading
1 flaky
  bug-or-flaky.spec.js:4:1 › flaky: a slow load now and then

Does a passing retry mean the application is fine?

No. A retry that passes tells you the attempts disagreed, nothing more. If the run turns green and nobody looks, an intermittent defect in the application, such as a race, shared data or a slow dependency, becomes invisible, and those are among the most expensive defects to find. Read a flaky result as a finding, not as a pass to ignore, and do not assume the fault is in the test.

What does Nexus Studio do with each result?

Nexus Studio records every attempt and gives the result its own verdict, so a retry never turns a defect into a plain green run.

• FLAKY: the test failed, then a retry passed it, and nothing was repaired. It exits 0, so the build does not break, and the report and History show the reservation.
• BREACHED: the test failed on a single attempt, and no environment or harness pattern explains the error. It exits 1.
• BREACHED_FLAKY: the test failed on its last attempt after an earlier retry, and no environment pattern explains it. It exits 1. Compare the attempts before you file a bug.

History keeps one row per run of each test, so a test that goes green, red and green again shows up as a trend instead of disappearing into reruns. The flaky tests guide lists the files each verdict leaves.

What are the limits?

• The flake in the example is simulated with arithmetic so that it repeats. Real flakiness has a cause, and finding it is still work for a person.
• Six runs is a quick screen, not statistics. A failure that happens once in fifty runs can pass six times.
• A failure on every run is not always a bug in your application: a slow network, a block page or a dead browser fail tests too. The false failures guide shows how to tell.
• With retries off, an intermittent failure in Nexus Studio is a BREACHED, because only a retry can tell FLAKY from a plain failure. The classification reads error text, so open metadata.json when a verdict surprises you.

Where can I see it?

The verdicts guide explains in what order the engine decides all 14 verdicts, and the Evidence page follows one failed test through every file it leaves. The 30-day trial runs the same engine on your own tests.