Alert & incident flow

The big picture

The source-specific checks differ, but every accepted signal enters the same routing, paging, response, and recovery pipeline. Select a source to compare the paths.

The rest of this page owns the detailed behavior. Setup guides link here instead of copying the rules into several places.

Alerts and incidents are two levels

HowlOps separates a raw alert from a confirmed incident:

An alert becomes an incident in one of two ways:

1. Automatically, when it matches a promotion rule. Out of the box there is one rule: any alert with critical severity is promoted. You can add your own rules that match on severity, source (monitor, heartbeat, integration), or labels.
2. Manually, with the Promote to incident button on any alert.

Step 1: where alerts come from

Monitor goes down

A single failed check does nothing on its own. A detector evaluates each completed check and opens an alert only when the configured confirmation rule is satisfied:

For monitors that run from several regions you also get:

External alert arrives

Alerts can come from Prometheus Alertmanager, Grafana, Datadog, CloudWatch, or a generic webhook. HowlOps normalises the payload to a name, a state, a severity, and a fingerprint (a stable key the source provides). Deduplication is keyed on that fingerprint:

• A new fingerprint opens a new alert.
• A repeat of the same fingerprint updates it and bumps a "fired N times" counter. It does not open a duplicate or re-notify.

External alerts have no monitor of their own, so there is no screenshot, prober log, or recheck for them.

Heartbeat is missed

A heartbeat expects a ping on a schedule. It is marked down when:

• the last ping is older than interval + grace, or
• it was never pinged and is older than twice its interval, or
• it missed its cron-scheduled time plus grace.

The alert is titled with the heartbeat's name and carries its configured priority (default P1). It is idempotent. While one alert is open, missing again does not open another.

One alert per incident (deduplication)

Whatever the source, you are notified once when something breaks and once when it recovers, never on every failed check. Subsequent failures for the same open alert are deduplicated:

14:00  check fails      → alert sent (incident opens)
14:10  check fails again → no alert (same incident)
14:20  check fails again → no alert
14:30  check succeeds    → recovery alert sent (incident resolves)

You can set a per-monitor repeat alert interval (see Step 4). While an alert stays open and unacknowledged, HowlOps sends another notification after that interval and continues until the alert is acknowledged, silenced, resolved, or deleted.

Step 2: screenshots (HTTP monitors)

If a monitor is HTTP, the workspace has screenshot_monitoring, and you enabled Capture screenshots on failure, HowlOps takes a screenshot of the page after the alert opens. The setting is off by default.

Step 3: how the alert is delivered

The first decision is whether an escalation policy governs the alert. HowlOps can pick one for you automatically:

Configure a routing rule

A routing rule is a condition + target pair. When a new alert is created, HowlOps evaluates your rules in ascending priority order (1 → 2 → 3 …). The first matching rule wins and pins its escalation policy; the rest are skipped. If no rule matches, the alert falls back to the monitor's own policy, then the workspace-level default escalation policy.

To create one, go to Alerts → Routing Rules → New Rule, then set a condition, a match value, the escalation policy to apply, and a priority (lower number = evaluated first).

Rules are evaluated from the lowest priority number upward. The first match selects its policy and stops evaluation; when nothing matches, the monitor policy and then the workspace default are used.

A policy governs the alert

The immediate broadcast is skipped (so you are not notified twice). The escalation engine takes over and pages on-call step by step over time (see Step 4).

No policy governs the alert

The alert is broadcast to every enabled channel plus mobile push. Each channel still passes through a set of filters and can be skipped:

Step 4: escalation steps and your personal chain

When a policy governs the alert, two layers of timing apply, and both are yours to configure.

Escalation steps run in order. Each step has its own delay, measured from when the previous step fired, and its own target. The policy can also repeat from step 1 if nobody acknowledges (see Escalation policies). Your personal chain, alongside it, decides how you are reached: each channel has a priority and a fallback delay, so a single page spreads across channels over time.

The chain advances only while the alert is still unacknowledged and not silenced. If you acknowledge at minute 3, the SMS and voice steps never fire.

You can also set a per-monitor renotify interval to re-send an alert every N seconds while it stays unacknowledged.

Step 5: routing labels

Routing rules match on labels. Where those labels come from depends on the source:

• Monitors and heartbeats: HowlOps sets the labels from your configuration (the monitor type, its tags, the severity).
• External integrations: the labels come from the incoming payload. HowlOps reads the labels and tags the source sent and routes on those, so the quality of your routing depends on the source sending good labels.

Step 6: recovery and resolve

When the service recovers, the alert resolves automatically:

• A monitor that returns healthy auto-resolves (multi-region waits for every region).
• An external source that sends a resolved event closes the matching alert by fingerprint.
• A heartbeat that pings again recovers and closes its alert.

The all-clear travels the same path the alert did:

• If a policy paged on-call, only the people who were actually paged get the all-clear.
• If it was a broadcast, the all-clear goes to the channels that received the original alert.

Acknowledge, silence, unacknowledge

What you can configure (summary)