When infrastructure fails

Who watches the watchers

The scariest failure for any monitoring tool is the silent one: the outage nobody is paged for. This page describes what happens when the monitoring path itself degrades because a prober node dies, a region stops reporting, an on-call rotation resolves to nobody, or the monitored thing goes completely dark. These are the safety nets that turn "silent" into "you still get paged".

A prober or region stops reporting

Each monitor's checks come from one or more prober regions. If a region goes quiet (the prober crashes, the region is decommissioned, or the network partitions), HowlOps does not let that region keep voting with its last stale sample.

• A region whose newest check is older than the monitor's freshness window is treated as not reporting and is excluded from evaluation. The window is three check intervals, at least 180 seconds, and never shorter than one worst-case check plus an interval, so a slow target being actively measured is not mistaken for a silent region.
• An excluded region is dropped from the region-quorum denominator and from the "all regions down" test. A dead region therefore cannot silently hold a real outage below quorum, cannot make a full outage look like a partial one, and cannot keep an incident from resolving.

The net effect: a broken prober degrades your coverage for that region, but it cannot create a silent no-page.

When HowlOps itself is the thing that is down

The uncomfortable case is the one where the problem is on our side. If our probe fleet stops collecting results, your services might be perfectly healthy while we simply cannot see them. A monitoring tool that stays quiet in that situation is worse than useless.

Two things happen.

You are told once, not once per monitor. When a fleet-wide gap is detected, HowlOps opens a single platform-level event and sends you one summary email. It deliberately does not open an incident for every affected monitor, and it never sends SMS or voice for this case. A gap on our side must not arrive as a hundred pages at three in the morning.

Our status page says so. status.howlops.com is served from the edge and renders independently of the application, so it stays up when the app does not. It shows a Monitoring fleet component with the current probe-region coverage, for example 6 of 7 probe regions healthy. It shows that line all the time, not only during a declared incident. A page that only speaks up once somebody declares an outage still reads as reassuring while coverage quietly erodes.

The same per-component health is on howlops.com/status, alongside the 90-day availability history.

One honest limitation: the status page is served by Cloudflare's edge, so it does not survive a full Cloudflare outage. It survives our own.

The monitored thing goes completely silent

If every region stops returning data for a monitor, including a single-region monitor whose only region goes dark, a staleness watchdog opens an incident titled along the lines of "No data: monitor has not been checked recently." You are told that the monitor stopped being checked, rather than being left to assume "no news is good news".

The watchdog waits for three check intervals, at least three minutes, and always longer than one check can legitimately take. That last part matters for slow targets: a monitor with a 30-second timeout and retries enabled can spend over three minutes on a single round of checks, and waiting only three intervals would report a check still in progress as silence.

The incident names which region went quiet and which ones were still reporting, so a single dead vantage point is not confused with the whole monitor going dark. When there is nothing to go on, because the monitor has no per-region samples to compare, the incident says the silent region could not be determined instead of leaving the field blank.

On-call resolves to nobody

Escalation is only as good as the people it can reach. HowlOps closes the "paged into the void" gap:

• If an escalation step has no target, or its schedule/rotation resolves to nobody available, the step falls back to paging the workspace owner, and the reason is written to the alert's activity log.
• Unavailable on-call members are skipped (up to a bounded number of hops) to the next available person, and each skip is recorded in the activity log. This means a rotation that "paged the wrong person" is explainable after the fact.

The guarantee is that an alert never resolves to no one without that being recorded and the owner being paged.

Dead-man's switch for things that push to you

For jobs and workers that cannot be probed from outside, such as cron jobs, batch pipelines, and background workers, use a heartbeat. The heartbeat is the inverse safety net: instead of us checking your service, your service checks in with us, and silence itself is the alert. If the expected ping doesn't arrive within interval + grace (or a missed cron time), an alert opens.