Crawler Activity

Blocking a crawler in robots.txt is a request. This is the record of what actually happened: every fetch by a known AI crawler, logged on your own server as it arrives, with the path, the response code and the time.

It answers the questions a settings screen cannot. Is anything reading this site at all? Did the crawler I blocked stop coming? Are they spending their fetches on my best pages or on my tag archives?

The most useful view is the one nobody expects: published but never fetched — pages that exist and that no crawler has read. That is usually a linking or sitemap problem, and it is invisible everywhere else.

Free. Per crawler, kept up to a year, plus the never-fetched list.

Pro adds a breakdown by page, CSV export, and an email the first time a crawler you have never seen turns up.