How we verify outages

Why this page exists

When a check fails, that failure could be three different things: your service is really down, one of our probes hit a network blip, or a firewall decided our prober looked like a bot. Paging you for the last two wastes your on-call. HowlOps runs a verification step that looks at the failure from several independent vantage points before deciding what it means. This page explains what that step does, what evidence you can see, and what data we collect along the way.

What verification does

A single failing vantage point can never, on its own, confirm an outage. Verification gathers the result from independent vantage points (different providers, regions, and network paths) and classifies the picture:

The same evidence can end in six different places:

The type of failure matters too. A server error the origin actively returned (an HTTP 5xx) is strong, target-side evidence. A connection that timed out or was refused could be a network path on our side, so it needs corroboration before it counts.

The evidence you can see

Open a monitor or an incident and you will find a How we verified this panel. It shows:

• A plain-language summary of the verdict.
• A timeline of every vantage point that took part, oldest first, showing its location and what it saw (healthy, slow, a DNS or TLS problem, a connection failure, an HTTP error, or a block), the status code and response time it measured, and when it checked.
• A verification badge. Today this badge reads not yet verified. We are rolling out the signals (an independent control probe and fleet-wide correlation) that let us state with confidence whether a failure was your target or our side. Until those are live, we show you the evidence and stay honest about what we can and cannot yet claim.

The panel never shows response bodies, request or response headers, cookies, or anything that could carry a secret. It is a sanitized view of the same evidence our engine reasoned over.

What each check collects

We are a monitoring tool, so we send requests to your targets and record what comes back. Here is exactly what that means, per check type, and who can see it.

Check type	What we collect	Who can see it	Retention
HTTP / keyword	Status code, response time, and resolved IP. Failures also include a curated set of diagnostic response headers, the names (never the values) of any Set-Cookie headers, a short signal snippet if a WAF/block page is detected, the redirect host chain, and TLS certificate details. The raw response body is scanned in memory and discarded, never stored.	You and your team (sanitized). Full failure diagnostics are visible to HowlOps platform operators for debugging.	Failure diagnostics are short-lived; check results follow your plan's history window.
Ping	Round-trip time, packet loss, jitter, resolved IP.	You and your team.	Your plan's history window.
TCP	Whether the port accepted a connection, and the connect time.	You and your team.	Your plan's history window.
DNS	The resolved records for the type you monitor, and resolution time.	You and your team.	Your plan's history window.

Two things we deliberately do not do: we never store your target's raw response body, and the evidence shown to you is redacted of anything secret-shaped (cookie values, authorization headers, tokens, and internal addresses).

Keeping sensitive data out of monitoring

Because a check hits a real endpoint, the safest practice is to point HowlOps at a URL that does not require or return sensitive data:

• Prefer a dedicated health endpoint over an authenticated, data-bearing page.
• If a monitor must send credentials, use dedicated, non-personal test credentials scoped to the smallest possible access: never a real user's login.
• Keyword checks match against text you choose; pick a marker string that is not itself sensitive.

What each state means

A monitored endpoint moves through a small set of clearly named states. These are separate from whether an alert was delivered to you (a page can be held back while the endpoint is genuinely down). The states you may see:

State	Meaning
Pending / Verifying	A failure was seen and we are gathering evidence from independent vantage points before deciding.
Degraded	The endpoint answered but is impaired (slow everywhere, or a narrow regional problem). It is up, just not healthy.
Down	A real outage, confirmed by an independent quorum.
Unknown	Evidence was mixed or too thin to decide. We treat this cautiously and lean toward alerting.
Suppressed	A verification concluded this did not warrant paging, so the page was held. This is an action on the alert, not a claim the endpoint was up and not a claim the incident never existed.
Recovering	The endpoint is answering again and we are waiting for a sustained healthy streak before closing.
Maintenance	Inside a maintenance window you configured, so checks are recorded but do not page.

"Suppressed" and "Down" can be true at the same time: the outage was real and recorded, and the page was held. That distinction matters for your uptime numbers below.

The settings that decide

Six settings decide how a failure on your monitor is verified.

Setting	What it controls
Required failing regions	How many independent vantage points must see the failure before it is treated as real. This is the core of the quorum: one region failing alone is a regional problem, not your outage.
Degraded threshold	The response time above which an answering endpoint is called degraded rather than up.
Confirmation threshold	How many consecutive failed checks are needed before verification even starts. It absorbs a single unlucky request.
Recovery period	How long the endpoint must stay healthy before we close the incident. Zero closes on the first good check, which is faster but flaps more.
Never auto-suppress	When on, a verification is never allowed to hold back the page for this monitor. Use it on anything where a missed page is worse than a false one.
Blocked-response codes	Which status codes count as "we were blocked" rather than "the site is down". A WAF answering 403 to our prober says nothing about whether your customers can reach you.

Where a value comes from. Anything you set on the monitor wins. Where you have set nothing, the value is inherited, and the monitor's edit screen labels which of the three it came from:

• monitor: you set it explicitly.
• adaptive baseline: for the degraded threshold only, we derived it from this monitor's own measured response times, so a normally slow endpoint is not called degraded for being itself.
• default: the platform-wide value.

Leaving the degraded threshold empty is not the same as switching it off. Empty means inherit, and the inherited value is the platform default, so an endpoint that becomes much slower can still be marked degraded. If you want slowness never to affect the state, say so explicitly rather than leaving the field blank.

Required failing regions works slightly differently from the other five. It is not a field on the monitor: it comes from a verification contract bound to that monitor, so that the size of the quorum is set deliberately rather than drifting monitor by monitor. Where no contract exists, the platform default applies, and the edit screen labels the value accordingly.

How this affects your uptime and SLO

Your availability numbers are computed from the verified service impact, never from whether an alert happened to be delivered:

• A confirmed outage burns your error budget even if its page was suppressed. Holding a page back is a delivery choice; it never rewrites what actually happened to your service.
• A latency degradation (up everywhere, just slow) does not burn availability. It is tracked as a separate latency signal.
• A partial outage (a region genuinely down or flapping) burns proportionally when you have configured audience or traffic weights, and conservatively (as a full outage) otherwise.
• Verified-up (disproved) is excluded, and states we genuinely cannot classify (our-infrastructure, maintenance, paused, still-verifying) are excluded from the denominator, not silently counted as healthy. When coverage is low we say so rather than showing an inflated 100 percent.

We publish this rule up front and apply it consistently, so a suppressed page never quietly improves your SLO.

Honest delivery status

Our guiding principle for notifications is that a message counts as delivered only when the channel actually accepted it, never merely because we queued it. We are extending this into a fuller delivery ledger, so that for each notification you will be able to see whether it was not requested, held back by policy, queued, attempted, accepted by the provider, retrying, or permanently failed, and so that a recovery notice is only sent when a real page or degrade notice went out first for that same episode. Until that ledger is fully rolled out, delivery outcomes are shown at the level your channels already report.