DOCS

When infrastructure fails

What HowlOps does when a prober dies, a region goes dark, on-call resolves to nobody, the thing being watched goes completely silent, or HowlOps itself cannot see — the safety nets that keep an outage from going unnoticed.

Who watches the watchers

The scariest failure for any monitoring tool is the silent one: the outage nobody is paged for. This page describes what happens when the monitoring path itself degrades — a prober node dies, a region stops reporting, an on-call rotation resolves to nobody, or the monitored thing goes completely dark. These are the safety nets that turn "silent" into "you still get paged".

A prober or region stops reporting

Each monitor's checks come from one or more prober regions. If a region goes quiet (the prober crashes, the region is decommissioned, or the network partitions), HowlOps does not let that region keep voting with its last stale sample.

A region whose newest check is older than 3× the check interval (at least 180 seconds) is treated as not reporting and is excluded from evaluation.
An excluded region is dropped from the region-quorum denominator and from the "all regions down" test — so a dead region cannot silently hold a real outage below quorum, cannot make a full outage look like a partial one, and cannot keep an incident from resolving.

The net effect: a broken prober degrades your coverage for that region, but it cannot create a silent no-page.

When HowlOps itself is the thing that is down

The uncomfortable case is the one where the problem is on our side. If our probe fleet stops collecting results, your services might be perfectly healthy while we simply cannot see them — and a monitoring tool that stays quiet in that situation is worse than useless.

Two things happen.

You are told once, not once per monitor. When a fleet-wide gap is detected, HowlOps opens a single platform-level event and sends you one summary email. It deliberately does not open an incident for every affected monitor, and it never sends SMS or voice for this case. A gap on our side must not arrive as a hundred pages at three in the morning.

Our status page says so. status.howlops.com is served from the edge and renders independently of the application, so it stays up when the app does not. It shows a Monitoring fleet component with the current probe-region coverage, for example 4 of 5 probe regions healthy — and it shows that line all the time, not only during a declared incident. A page that only speaks up once somebody declares an outage still reads as reassuring while coverage quietly erodes.

The same per-component health is on howlops.com/status, alongside the 90-day availability history.

One honest limitation: the status page is served by Cloudflare's edge, so it does not survive a full Cloudflare outage. It survives our own.

The monitored thing goes completely silent

If every region stops returning data for a monitor — including the case of a single-region monitor whose only region goes dark — a staleness watchdog opens an incident within roughly 3× the check interval titled along the lines of "No data — monitor has not been checked recently." You are told that the monitor stopped being checked, rather than being left to assume "no news is good news".

On-call resolves to nobody

Escalation is only as good as the people it can reach. HowlOps closes the "paged into the void" gap:

If an escalation step has no target, or its schedule/rotation resolves to nobody available, the step falls back to paging the workspace owner, and the reason is written to the alert's activity log.
Unavailable on-call members are skipped (up to a bounded number of hops) to the next available person, and each skip is recorded in the activity log — so a rotation that "paged the wrong person" is explainable after the fact.

The guarantee is that an alert never resolves to no one without that being recorded and the owner being paged.

Dead-man's switch for things that push to you

For jobs and workers that can't be probed from outside — cron jobs, batch pipelines, background workers — use a heartbeat. The heartbeat is the inverse safety net: instead of us checking your service, your service checks in with us, and silence itself is the alert. If the expected ping doesn't arrive within interval + grace (or a missed cron time), an alert opens.

Was this page helpful?