How to Fix Flaky Tests: A Practical Playbook for QA Engineers
Flaky tests destroy trust in your CI. This playbook covers the seven root causes, how to detect them, and the patterns to fix each — with a focus on test data hygiene.
· 10 min read
A flaky test is one that passes and fails on the same code without changes. Three of them is a nuisance. Thirty is a CI you stop trusting. This playbook walks through the seven root causes we see again and again, how to detect each, and the fixes that actually stick.
Why flakiness is more expensive than you think
Every flaky test costs you twice: once when it fails spuriously and someone investigates, and again when a real failure gets dismissed as flake. Google's testing team has published data showing that ~16% of their tests have some level of flakiness — and the biggest hidden cost is engineers learning to ignore red builds.
Cause 1: shared mutable test data
Two tests grab the same user account, one mutates it, the other asserts on the original state. Classic flake. The fix is isolation: each test gets its own freshly-seeded fixtures, or pulls from a pool of disposable accounts that are reset after the run. A shared catalog of immutable reference assets (test cards, sample inputs) is fine — shared mutable accounts are the trap.
Cause 2: timing and async races
Tests that rely on `sleep(500)` or arbitrary waits will fail under CI load. Replace them with explicit waits on observable conditions: 'wait until this element is visible', 'wait until this network request completes'. Playwright, Cypress, and Selenium all expose proper waiters — use them. Bare sleeps are a flake factory.
Cause 3: test order dependency
If your suite passes serially but fails in parallel, you have order-dependent tests. Run with random ordering locally (`--shuffle` in most runners) to surface them. The fix is the same as cause 1: every test must set up and tear down its own state.
Cause 4: third-party dependencies
Hitting a real Stripe sandbox, a real OAuth provider, or a real email API in unit tests guarantees flake. Mock them at the network boundary (MSW, nock, WireMock). Reserve real third-party calls for a small, separate suite of contract tests that you accept might be slower and flakier.
Cause 5: time and timezone
`new Date()` in a test is a future bug. Freeze time with `sinon.useFakeTimers`, `vi.useFakeTimers`, or `jest.useFakeTimers`. Pin the test environment to UTC. Tests that pass in PST and fail at midnight UTC are pure pain.
Cause 6: animations and transitions
An element that fades in over 300ms will sometimes be queried before it is interactive. Disable CSS animations in your test environment with a global stylesheet override, or set `prefers-reduced-motion` in your browser context. Both Playwright and Cypress support this in two lines.
Cause 7: locator brittleness
Selectors like `div > div > .btn-primary:nth-child(2)` will break when anyone touches the markup. Prefer accessible, semantic selectors: roles, labels, test IDs added intentionally for testing. Playwright's strict-mode error catches duplicate matches before they become flake.
How to detect flake at scale
Run your suite ten times in a row in CI on a known-good commit and tag any test that fails at least once as a flake suspect. Tools like Buildkite Test Analytics, Datadog CI Visibility, and Trunk Flaky Tests do this automatically and quarantine offenders. Quarantine first, fix second — never disable.
The test data hygiene checklist
Most flake roots in test data, so audit yours: every test owns its fixtures, no test mutates a shared account, reference assets (cards, IDs, sample files) live in a versioned shared catalog, and no test reads 'now' without freezing it. Get these four right and your flake rate drops by an order of magnitude.