Put a flaky test in quarantine

See which tests flip between pass and fail, and quarantine one with an owner so its runs stop counting as evidence until it is fixed.

In short

A flaky test passes and fails with no change in between. The tests list shows "Flaky tests" when a test's verdict flips often. Quarantine it with an owner and a reason. It keeps running, but its runs are not evidence for a design step, a bug or the figures until someone takes it out.

Before you start

Who: quarantining needs Edit on the site. The owner you choose must be able to open it.
Plan: every plan.
First: nothing. A test appears once its verdict flips often enough.

Steps

Open the site's tests. "Flaky tests" lists each flaky test, such as "Changed 7 times in 20 runs (35%)".
Press "Quarantine", write "Why" and choose the "Owner".
Press "Quarantine" to confirm. Once it is fixed, press "Take out of quarantine".

Result: the test is marked "In quarantine" with its owner and reason.

Good to know

A test is flaky when its verdict changes in at least 30% of its last 20 runs, with 6 runs at least. Preview runs never count.
The owner must be able to open the site. Quarantining needs Edit on the site.
A run that passed only on its second try is marked "Flaky" on the runs list. Choose "Try once more after a failure" when you run a test in your browser.

If something goes wrong

"Say why" - a reason is needed so the owner knows where to start.