Failure threshold

A failure threshold means the number of consecutive failed checks required before a monitor raises an alert or declares an incident. It is also called a confirmation period or retries before alert. At one, the monitor alerts on the first failure and every transient blip becomes a page. At two or three, brief failures are absorbed and only a sustained problem raises anything. It is the simplest and most effective control over false alarms that monitoring offers.

The cost is arithmetic and entirely predictable: each additional required failure adds one check interval to detection. Two failures at a one-minute interval means an outage is announced roughly two minutes in; the same threshold at five minutes means ten. This is why the interval and the threshold are one decision rather than two.

The important design question is what the threshold gates. It should gate ALERTING, not measurement. If a below-threshold failure is discarded rather than recorded, short outages vanish from your uptime figures entirely and the availability number becomes a fiction — the monitor is asserting that a failure it observed did not happen. Recording the failure and merely declining to page for it keeps both facts true.

Recovery should not have a threshold at all, or should have a much smaller one. Waiting for confirmation before believing bad news is prudence; waiting for confirmation before believing good news keeps an incident open after it has ended and delays the all-clear to everyone watching.