There is a particular kind of silence in a team whose tests are red. Nobody defends the suite, nobody deletes it, and the pipeline has a line somewhere that lets the build through regardless. It is one of the most common states we find, and it is worth understanding how it happens, because the way out is not "try harder".

Flakiness is a design problem, not a discipline problem

A test that fails once in every twenty runs for no discernible reason will destroy a suite faster than a hundred genuinely broken tests. Genuine failures get fixed. Random ones teach the team that red means nothing.

The usual causes are structural:

  • Fixed waits. `sleep(2)` passes on a fast machine and fails on a loaded one. Every wait should be on a condition — an element appearing, a request completing — never on a clock.
  • Shared data. Two tests using the same account, one changing its state, order deciding the outcome. Each test should create what it needs and be runnable alone, in parallel, in any order.
  • Third parties in the loop. A payment provider's sandbox having a slow morning is not your build failing. External services get stubbed at the boundary.
  • Time and timezones. Tests that pass except near midnight, or in the month a country changes its clocks.

None of these are fixed by asking developers to care more.

The second cause: tests that assert nothing

The other half of a red suite is tests that were never worth trusting. They check that a page rendered, that a button exists, that a heading contains a word. They pass while the product is broken and fail whenever a designer renames something. A team learns quickly that these failures are noise, and the lesson generalises to the whole suite.

A test earns its place by asserting an outcome a person would care about: the money moved, the order reached the right state, the wrong role could not open the page.

Getting out

The way back is unpopular because it starts with deletion.

Decide what is worth saving. Usually a minority. Run the suite ten times and list every test that failed at least once without a real cause — those are candidates for deletion, not repair.

Delete the rest. A smaller suite you trust is worth more than a large one you ignore. This is the step teams resist, because deleting tests feels like moving backwards, and it is the step that makes everything after it possible.

Make red mean stop. Once the remaining tests are trustworthy, the pipeline must fail on red and nobody may merge past it. This only works after the first two steps; do it before, and the team will simply revert the rule within a week.

Add back deliberately. New tests only for paths that carry money or would be a public embarrassment. Each one with real assertions and its own data.

The honest caveat

If a suite is large, old and unloved, rescuing it can cost more than writing a small new one. We say so when we see it, because the alternative is billing for weeks of archaeology and delivering a slightly less red version of the same thing. A short suite that is green for a reason changes how a team ships. A long one that is amber forever changes nothing at all. If that is where you are, our test automation starts by naming the fifteen journeys that carry money and leaving everything else red until it earns a fix.

Earlier in this series: what automated testing actually costs, and what the money buys.