Verification edge-case cookbook

Why this page exists

How we verify outages explains the verdicts, the states, and the SLO rule in general terms. This page is the cookbook: it walks through the specific, sometimes confusing situations that show up in the real world, and tells you exactly what happens in each one. If you have ever wondered "wait, why didn't that page me" or "why did that page me when the site looked fine to me," the answer is probably below.

Every scenario follows the same shape: what our probes saw, what verification concluded, what that means for paging, and what you will actually see on the monitor or incident.

1. One region times out, the rest are fine

What the probes see: one vantage point cannot establish or complete a connection. Every other vantage point gets a normal, healthy response.

What verification concludes: a connection timeout or refusal is not, by itself, strong evidence that your target is unhealthy. It is exactly the kind of failure a network hiccup on our side can cause. With every other independent vantage point serving the target successfully, this is a regionally degraded picture, not a confirmed outage.

Does it page: no. The service is up everywhere that matters, so this does not page and does not burn your error budget.

What you see: the monitor briefly shows verifying, then settles back to healthy, or (if it is a recurring pattern) a regional degradation you can see in the evidence timeline. No incident is opened for a single soft regional fail.

2. One region returns an HTTP 5xx, the rest are fine

What the probes see: one vantage point gets a server error the origin actively returned. Every other vantage point is healthy.

What verification concludes: an HTTP 5xx is target-side evidence. The origin itself answered and said it was broken. That is a real, if narrower, problem, so this classifies as partial, a genuine outage confined to a region rather than a false alarm.

Does it page: yes, unless you have marked the monitor non-critical. A confirmed partial outage is real impact for whoever is being served by that region.

What you see: an incident is opened at the appropriate severity, the evidence timeline shows the one region's 5xx alongside the healthy vantage points, and your uptime numbers count the partial impact proportionally if you have configured audience or traffic weights, or as a full outage if you have not.

3. Every region gets 403, 429, or 451

What the probes see: every single vantage point gets blocked. Not a timeout, not a 5xx, but a response that looks like a WAF challenge, a rate limit, or a legal/geo block.

What verification concludes: an edge device answering with a block page does not prove your origin is healthy for real users. It proves something in front of your origin is refusing our probes, and we cannot see past that. Because we cannot rule out that the same wall is also turning away legitimate traffic, we treat this as needing a page by default.

Does it page: yes, by default. If you know a given block pattern is expected and harmless for your target (say, a WAF that always 403s automated clients but never blocks real users), you can opt those specific status codes into "degrade instead of page" on that monitor, so the same pattern shows as degraded rather than paging next time.

What you see: an incident (unless you have opted the codes into degrade), with the evidence timeline showing the same block code across every vantage point so you can tell at a glance that this is a wall, not a crash.

4. Every region is up but slow

What the probes see: every vantage point gets a healthy response, but the response time is above the monitor's configured degraded threshold everywhere.

What verification concludes: this is not an outage. The target answered correctly from every independent vantage point. It is a latency problem.

Does it page: no, it degrades and notifies. A slow-everywhere pattern does not page as an outage; it is surfaced as a degrade notice so your team knows performance has slipped without treating it as downtime.

What you see: the monitor shows degraded, you get a notification about the degrade (not a page-style alert), and critically, this does not burn your availability. It is tracked as a separate latency signal, kept apart from your uptime/SLO number.

5. A shared-hosting egress ban makes several of our probes fail while you are actually up

What the probes see: several of our probes, often ones that happen to share an outbound network path or provider, fail in a pattern that lines up with our own infrastructure rather than with your target. For example, a hosting provider's network temporarily blocks probing traffic from an IP range we use.

What verification concludes: the failure pattern points at us, not at you. This is classified as our own infrastructure being the problem.

Does it page: no, not you. We page our own on-call to fix the probing path on our side. Your team is not woken up for a problem that is ours to solve.

What you see: the monitor does not flip to down. Depending on where you are looking, you may see a neutral indicator that verification attributed the failure to our side rather than yours, so you are not left wondering whether something happened silently.

6. Our entire verification mesh cannot report

What the probes see: no vantage point can report a result at all. This is not "every vantage point saw a failure," it is "we have no evidence to reason over."

What verification concludes: with zero independent evidence, we cannot make any claim about your target. This state is explicitly our own infrastructure, not your target.

Does it page: no. We cannot honestly tell you your service is down when we have no evidence either way, so this does not page as an outage.

What you see: the monitor shows an our-infrastructure state rather than down, and this window is excluded from your uptime denominator rather than being silently counted as healthy or silently counted against you. If coverage was low during a period, your reports say so.

7. A TLS certificate problem seen from one vantage vs. a cert that flaps over time

One vantage, one failure: if a single vantage point reports a TLS problem while every other vantage point completes a healthy TLS handshake, the healthy vantage points are not hidden or overridden by the one failure. The same corroboration logic applies as with any other single-region failure: one bad read does not outweigh several good ones.

A certificate that flaps over time: a certificate that is sometimes fine and sometimes failing across checks, rather than failing from a single vantage at a single moment, is a different pattern worth calling out on its own. That pattern is surfaced to you as a certificate warning, separate from a one-off regional TLS blip, so you can catch a cert that is genuinely close to expiring, misconfigured on a subset of servers, or intermittently served incorrectly, before it becomes a real outage.

8. An outage was briefly suppressed but stays down

What happens: verification (or a noise-control rule) held a page back for a period, for example while waiting for corroboration or a short grace window. The target does not recover; it is still down when that window ends.

What verification concludes: the outage was real the entire time. Suppression is a decision about whether to send a page, never a claim about what was actually happening to your service.

Does it page: yes, once the hold expires without recovery, the page fires. It arrives later than an immediately-confirmed outage would have, but it does fire.

What you see: the incident and the monitor state both show down, and your uptime/SLO number counts the outage for its full real duration, including the suppressed window. A suppressed page never improves your numbers. "Suppressed" and "down" can and do show up as true at the same time, and that is by design: it is a record of what happened to the alert, not a record of what happened to your service.

9. A monitor with only one vantage point (free tier)

What the probes see: there is only one prober checking this monitor, so there is no second, independent vantage point to corroborate a failure against.

What verification concludes: without spatial quorum, verification leans on two other signals instead. First, temporal persistence: a single vantage point that keeps failing across several consecutive checks is treated as stronger evidence than one failing check, since a transient network blip usually does not repeat. Second, our own prober-health check: we confirm the one vantage point that is checking your target is itself healthy and not the thing that is actually broken, before trusting what it reports.

Does it page: yes, once persistence and prober health both check out, a single-vantage monitor pages like any other. It simply confirms outages the way one honest, repeated observation would, rather than by cross-referencing several vantage points at once.

What you see: the same evidence panel and states as any other monitor, with the timeline showing the repeated checks from your one vantage point that built the case, plus a note that this monitor is not running with spatial quorum. Adding more vantage points (available on higher plans) shortens how long it takes to confirm a real outage with confidence.

How to make a monitor always page

If you would rather a monitor never wait on verification, even when the evidence is thin, two per-monitor settings exist for that:

• Always page when unsure — an override on the monitor that treats an inconclusive or regionally-degraded reading as a page instead of holding back, for endpoints where you would rather be woken unnecessarily than miss something.
• Degrade instead of page for specific block codes — lets you tell verification that certain status codes (say, a 403 your CDN always returns for automated clients) are expected and should degrade the monitor rather than open an incident, instead of relying on the default page-on-block behavior described in scenario 3.

Both settings live on the monitor's configuration, not globally, so you can dial individual endpoints tighter or looser without changing how verification behaves for everything else.

Related

• How we verify outages — the verdicts, states, evidence panel, and SLO rule this cookbook builds on.
• Noise control — every mechanism that keeps alerting quiet.
• Alert & incident flow — the end-to-end path from a failed check to a page.