Best practices
Hard-learned lessons for getting paged for real problems, not noise — naming, intervals, thresholds, keyword and SSL checks, and heartbeats.
Good monitoring is not about catching everything — it is about getting paged for the things that matter and staying quiet for the things that don't. These practices help you build a setup your team trusts.
1. Name monitors clearly
Use a consistent naming pattern: [Service]: [What it checks]. For example: API: Health, DB: Backup job, Website: Homepage. This makes it immediately obvious what failed when you get paged at 3am.
2. Set the right check interval
Don't use the shortest interval just because it is available. Match the interval to how quickly you actually need to know about failures. 60 seconds is fine for most web services. Reserve 15–30 seconds for truly critical, latency-sensitive APIs.
3. Tune alert thresholds to avoid fatigue
The default threshold is 2, which requires two consecutive failures before alerting and ignores transient network hiccups. Opt in to threshold 1 (alert on the first failure) only for your most critical production APIs. Alert fatigue — too many false positives — leads to real alerts being ignored.
4. Use keywords for deeper checks
A 200 OK response doesn't mean your app is healthy; it might be returning a cached error page. Add a keyword check for a string that only appears in a valid response, e.g. "status":"ok" or your app version number.
5. Monitor SSL certificates proactively
Enable SSL monitoring — it warns you 30 and 7 days before a certificate expires. Certificate expiry is entirely preventable, so don't let it cause an outage. Most CAs and Let's Encrypt tools support auto-renewal, but still monitor as a safety net.
6. Make heartbeat pings resilient
In cron jobs, add || true after the ping curl so cron doesn't report failure if the ping itself fails. The goal is: your job ran, HowlOps heard about it. A failing ping should not mask a successful job.
7. Keep API keys secret
Use environment variables for API keys, and never hardcode them. Set appropriate scopes (read-only tokens for dashboards, write tokens for automation). Rotate keys immediately if they may have been exposed.
8. Organize monitors in workspaces
Use separate workspaces for different environments (production, staging) or different clients. This prevents accidents (pausing prod instead of staging) and allows clean access control.
The golden rule: your alerting is only as good as your on-call process. Great monitoring with no runbooks means wasted time during incidents. Document your response procedures alongside your monitoring setup.
Related
Was this page helpful?