How we treat robots.txt

The short version

Endpoint discovery respects robots.txt. When you point it at a domain, we follow that domain's sitemaps to find things worth monitoring — we skip anything its robots.txt disallows, and we pace ourselves by its Crawl-delay.

Your monitors are not affected. At all. A monitor checks one address you chose, at the interval you chose, and robots.txt does not change either of those things.

Why the two are different

robots.txt is a message from a site's owner to automated visitors, and it was written for crawlers: programs that wander a site fetching pages nobody specifically asked for.

Endpoint discovery is exactly that, so it obeys.

A monitor is not. It fetches one address that a person typed into a form, on a fixed interval, forever — much closer to somebody keeping a tab open than to a crawl. You told us what to watch and how often, and that is the arrangement.

We used to slow such a monitor down when a site's robots.txt disallowed the path. We stopped, for three reasons worth stating plainly:

• It was a half-measure. Either robots.txt governs a check, in which case we should have refused to run it, or it does not, in which case it should not touch the interval either.
• Nobody else in this industry applies robots.txt to uptime checks, and arriving from another tool to find your checks quietly slowed down is not a pleasant surprise.
• The thing it was guarding against — us putting too much load on a site — is handled properly elsewhere. We cap the total rate of requests any one hostname receives from HowlOps, across every customer and every probing location. That is our own limit, it sees the real total rather than one monitor's share, and it does not depend on a file we found on somebody's server.

If you are monitoring a site you do not run, you are responsible for having the right to do so. That was always true, and it is not something a robots.txt can decide for you.

What you will see

If a site's robots.txt disallows the path a monitor checks, the monitor's page says so.

That is a note, not a restriction. It is there because a site that asks bots to stay away often blocks them at the firewall too — so when a check gets a 403 that a browser never sees, this is usually the first clue.

Setting it

The workspace-wide setting and the per-monitor override control discovery only:

• Respect (the default). Discovery skips disallowed paths.
• Ignore. Discovery scans the whole site. Only takes effect on a domain you have verified you own — on any other domain, robots.txt is a stranger's instruction and we keep honouring it.

Verification is a single DNS TXT record. See verifying a domain.

Which rules apply to us

robots.txt addresses bots by a short product token. Ours is HowlOpsMonitor. If your file has a group for it, that group wins; otherwise we use the User-agent: * group.

If you have set a monitor's User-Agent to look like a browser, that makes no difference to which rules we follow. We match on our own identity, always. A rule you write for HowlOps applies to HowlOps, and we are not going to slip past it by claiming to be Chrome.

We follow the usual matching rules: the most specific pattern wins, Allow beats Disallow when they are the same length, and * and $ work as wildcards. An empty Disallow: means "everything is allowed", as it should.

If we cannot read your robots.txt

Nothing is blocked. A file we cannot fetch is missing information, not a refusal, and we will not stop scanning because a DNS lookup failed. We cache what we last read successfully and keep using it while the site is unreachable.

We fetch the file at most once a day per site, and share that between all of our probing locations, so honouring it costs your server one request a day.