DOCS

When infrastructure fails

What HowlOps does when a prober dies, a region goes dark, on-call resolves to nobody, or the thing being watched goes completely silent — the safety nets that keep an outage from going unnoticed.

Who watches the watchers

The scariest failure for any monitoring tool is the silent one: the outage nobody is paged for. This page describes what happens when the monitoring path itself degrades — a prober node dies, a region stops reporting, an on-call rotation resolves to nobody, or the monitored thing goes completely dark. These are the safety nets that turn "silent" into "you still get paged".

A prober or region stops reporting

Each monitor's checks come from one or more prober regions. If a region goes quiet (the prober crashes, the region is decommissioned, or the network partitions), HowlOps does not let that region keep voting with its last stale sample.

A region whose newest check is older than 3× the check interval (at least 180 seconds) is treated as not reporting and is excluded from evaluation.
An excluded region is dropped from the region-quorum denominator and from the "all regions down" test — so a dead region cannot silently hold a real outage below quorum, cannot make a full outage look like a partial one, and cannot keep an incident from resolving.

The net effect: a broken prober degrades your coverage for that region, but it cannot create a silent no-page.

The monitored thing goes completely silent

If every region stops returning data for a monitor — including the case of a single-region monitor whose only region goes dark — a staleness watchdog opens an incident within roughly 3× the check interval titled along the lines of "No data — monitor has not been checked recently." You are told that the monitor stopped being checked, rather than being left to assume "no news is good news".

On-call resolves to nobody

Escalation is only as good as the people it can reach. HowlOps closes the "paged into the void" gap:

If an escalation step has no target, or its schedule/rotation resolves to nobody available, the step falls back to paging the workspace owner, and the reason is written to the alert's activity log.
Unavailable on-call members are skipped (up to a bounded number of hops) to the next available person, and each skip is recorded in the activity log — so a rotation that "paged the wrong person" is explainable after the fact.

The guarantee is that an alert never resolves to no one without that being recorded and the owner being paged.

Dead-man's switch for things that push to you

For jobs and workers that can't be probed from outside — cron jobs, batch pipelines, background workers — use a heartbeat. The heartbeat is the inverse safety net: instead of us checking your service, your service checks in with us, and silence itself is the alert. If the expected ping doesn't arrive within interval + grace (or a missed cron time), an alert opens.

Was this page helpful?