Find flaky checks and put them in quarantine

See which saved requests and suite cases flip between pass and fail, and quarantine one with an owner so its results stop counting as evidence until it is fixed.

In short

A flaky check passes and fails with no change in between, so nobody trusts it. {{brand}} counts how often each check's result flips. Quarantine one, with an owner and a reason. It keeps running, but its results stop counting anywhere until someone takes it out. That covers Can we ship?, a design step, a bug and the figures.

Before you start

Who: quarantining and taking a check out need Edit on the project. The owner you choose must be able to open it.
Plan: every plan.
First: nothing to set up. A check appears once its results flip often enough.

Steps

Open the project. "Flaky checks" appears under the run summary when some check flips.
Read each line, such as "Changed 7 times in 20 runs (35%)".
Press "Quarantine", write "Why" and choose the "Owner" who fixes it.
Press "Quarantine" to confirm.
Once it is fixed, press "Take out of quarantine".

Result: the check is marked "In quarantine" with its owner and reason. Its owner is told in their inbox.

Good to know

A check is flaky when its result changes in at least 30% of its last 20 runs, with 6 runs at least. Only runs {{brand}} sent count - not the CLI, the browser, an agent or the mock.
Can we ship? lists checks in quarantine as a risk of their own, because each is a behaviour nothing proves.
A card's linked tests cannot pass on a check in quarantine, so a card never closes on it.
Past figures leave out exactly the runs made while the check was held, also after it comes out.
Quarantining needs Edit on the project. A project holds up to 200 checks in quarantine.
UI Tester tracks work the same way. See /help/ui-tester/flaky-tracks.

If something goes wrong

"Say why this check goes into quarantine."
"This check is already in quarantine."
The owner is refused: they must be able to open this project.
"Choose a saved request or a suite case of this project to quarantine."
"A project holds at most 200 checks in quarantine. Take some out first."