How we verify outages

Why this page exists

When a check fails, that failure could be three different things: your service is really down, one of our probes hit a network blip, or a firewall decided our prober looked like a bot. Paging you for the last two wastes your on-call. HowlOps runs a verification step that looks at the failure from several independent vantage points before deciding what it means. This page explains what that step does, what evidence you can see, and what data we collect along the way.

What verification does

A single failing vantage point can never, on its own, confirm an outage. Verification gathers the result from independent vantage points (different providers, regions, and network paths) and classifies the picture:

• Confirmed — enough independent vantage points agree the target failed. This is a real outage and pages you.
• Partial — some vantage points failed while others served your target. A genuine but narrower problem.
• Regionally degraded — a single vantage point struggled while the rest were fine. The service is still up, so this does not page.
• Disproved — every independent vantage point saw the target healthy. This is the only outcome allowed to hold a page back.
• Inconclusive — the evidence was mixed or too thin to decide. We err on the side of waking you (fail-open), never on the side of silence.
• Our own infrastructure — our vantage points could not report at all. That points at us, not your target, so it does not page you.

The type of failure matters too. A server error the origin actively returned (an HTTP 5xx) is strong, target-side evidence. A connection that timed out or was refused could be a network path on our side, so it needs corroboration before it counts.

The evidence you can see

Open a monitor or an incident and you will find a How we verified this panel. It shows:

• A plain-language summary of the verdict.
• A timeline of every vantage point that took part, oldest first: which location, what it saw (healthy, slow, a DNS or TLS problem, a connection failure, an HTTP error, or a block), the status code and response time it measured, and when it checked.
• A verification badge. Today this badge reads not yet verified — we are rolling out the signals (an independent control probe and fleet-wide correlation) that let us state with confidence whether a failure was your target or our side. Until those are live, we show you the evidence and stay honest about what we can and cannot yet claim.

The panel never shows response bodies, request or response headers, cookies, or anything that could carry a secret. It is a sanitized view of the same evidence our engine reasoned over.

What each check collects

We are a monitoring tool, so we send requests to your targets and record what comes back. Here is exactly what that means, per check type, and who can see it.

Check type	What we collect	Who can see it	Retention
HTTP / keyword	Status code, response time, resolved IP, and — only on a failure — a curated set of diagnostic response headers, the names (never the values) of any Set-Cookie headers, a short signal snippet if a WAF/block page is detected, the redirect host chain, and TLS certificate details. The raw response body is scanned in memory and discarded, never stored.	You and your team (sanitized). Full failure diagnostics are visible to HowlOps platform operators for debugging.	Failure diagnostics are short-lived; check results follow your plan's history window.
Ping	Round-trip time, packet loss, jitter, resolved IP.	You and your team.	Your plan's history window.
TCP	Whether the port accepted a connection, and the connect time.	You and your team.	Your plan's history window.
DNS	The resolved records for the type you monitor, and resolution time.	You and your team.	Your plan's history window.

Two things we deliberately do not do: we never store your target's raw response body, and the evidence shown to you is redacted of anything secret-shaped (cookie values, authorization headers, tokens, and internal addresses).

Keeping sensitive data out of monitoring

Because a check hits a real endpoint, the safest practice is to point HowlOps at a URL that does not require or return sensitive data:

• Prefer a dedicated health endpoint over an authenticated, data-bearing page.
• If a monitor must send credentials, use dedicated, non-personal test credentials scoped to the smallest possible access — never a real user's login.
• Keyword checks match against text you choose; pick a marker string that is not itself sensitive.

What each state means

A monitored endpoint moves through a small set of clearly named states. These are separate from whether an alert was delivered to you (a page can be held back while the endpoint is genuinely down). The states you may see:

State	Meaning
Pending / Verifying	A failure was seen and we are gathering evidence from independent vantage points before deciding.
Degraded	The endpoint answered but is impaired (slow everywhere, or a narrow regional problem). It is up, just not healthy.
Down	A real outage, confirmed by an independent quorum.
Unknown	Evidence was mixed or too thin to decide. We treat this cautiously and lean toward alerting.
Suppressed	A verification concluded this did not warrant paging, so the page was held. This is an action on the alert, not a claim the endpoint was up and not a claim the incident never existed.
Recovering	The endpoint is answering again and we are waiting for a sustained healthy streak before closing.
Maintenance	Inside a maintenance window you configured, so checks are recorded but do not page.

"Suppressed" and "Down" can be true at the same time: the outage was real and recorded, and the page was held. That distinction matters for your uptime numbers below.

How this affects your uptime and SLO

Your availability numbers are computed from the verified service impact, never from whether an alert happened to be delivered:

• A confirmed outage burns your error budget even if its page was suppressed. Holding a page back is a delivery choice; it never rewrites what actually happened to your service.
• A latency degradation (up everywhere, just slow) does not burn availability. It is tracked as a separate latency signal.
• A partial outage (a region genuinely down or flapping) burns proportionally when you have configured audience or traffic weights, and conservatively (as a full outage) otherwise.
• Verified-up (disproved) is excluded, and states we genuinely cannot classify (our-infrastructure, maintenance, paused, still-verifying) are excluded from the denominator, not silently counted as healthy. When coverage is low we say so rather than showing an inflated 100 percent.

We publish this rule up front and apply it consistently, so a suppressed page never quietly improves your SLO.

Honest delivery status

Our guiding principle for notifications is that a message counts as delivered only when the channel actually accepted it, never merely because we queued it. We are extending this into a fuller delivery ledger, so that for each notification you will be able to see whether it was not requested, held back by policy, queued, attempted, accepted by the provider, retrying, or permanently failed, and so that a recovery notice is only sent when a real page or degrade notice went out first for that same episode. Until that ledger is fully rolled out, delivery outcomes are shown at the level your channels already report.

Related

• Verification edge-case cookbook — scenario-by-scenario reference for the tricky patterns: single-region soft fails, all-region blocks, shared-hosting bans, flapping certificates, suppressed-but-still-down outages, and single-vantage monitors.
• Noise control — every mechanism that keeps alerting quiet.
• Alert & incident flow — the end-to-end path from a failed check to a page.
• Incidents & severities — how a confirmed outage is classified.