Heartbeat monitoring

Heartbeat monitoring reverses the usual direction. Rather than the monitoring system reaching in to check your service, your service reaches out — a job, a script or a device sends a request to a unique URL each time it completes successfully. The monitor's only job is to notice when that request stops arriving.

This is the only way to monitor things that cannot be reached from outside: a nightly backup, a cron job on a private host, a worker behind a firewall, an offline device. There is nothing to poll, so the thing being watched has to volunteer. It is also called a dead man's switch, because the alerting condition is the absence of a signal rather than the presence of a failure.

The setting that matters is the grace period: how long silence is tolerated before an alert. It has to accommodate the job's normal variation — a backup that usually takes twenty minutes and occasionally forty needs a period generous enough to cover the bad night, or it will page every time the job runs slowly. Too generous and a failure sits undetected until the next window.

One caveat: a heartbeat proves the job reached its final line, not that the job did anything useful. A backup script that silently wrote a zero-byte file and exited cleanly will report a perfect heartbeat. Sending the heartbeat only after verifying the result is what closes that gap.