AI Crawler Directory
The 19 crawlers PinMeOut knows by name: 5 search engines and 14 AI crawlers. What each one is for, and what blocking it does.
Search engines
Blocking these removes your site from that search engine. Almost never what you want.
| Crawler | Operator | What it does |
|---|---|---|
Googlebot |
Crawls pages for Google Search. | |
Bingbot |
Microsoft | Crawls pages for Bing, which also feeds Copilot. |
DuckDuckBot |
DuckDuckGo | Crawls pages for DuckDuckGo. |
YandexBot |
Yandex | Crawls pages for Yandex Search. |
Applebot |
Apple | Crawls pages for Siri and Spotlight Suggestions. |
AI answer engines
These fetch pages to answer a question and usually cite the source. Allowing them is how you get mentioned.
| Crawler | Operator | What it does |
|---|---|---|
OAI-SearchBot |
OpenAI | Builds the index ChatGPT search draws on. Separate from training. |
PerplexityBot |
Perplexity | Indexes pages so they can be surfaced and cited in answers. |
Claude-SearchBot |
Anthropic | Indexes pages to support search results in Claude. |
AI training crawlers
These collect content used to train models. Blocking them does not affect search or AI answers.
| Crawler | Operator | What it does |
|---|---|---|
GPTBot |
OpenAI | Collects content that may be used to train OpenAI models. |
ClaudeBot |
Anthropic | Collects content that may be used to train Anthropic models. |
Google-Extended |
Controls use of your content for Gemini training. Does not affect Google Search. (robots.txt token only: it never appears in your logs) | |
Applebot-Extended |
Apple | Controls use of your content for Apple model training. Does not affect Applebot. (robots.txt token only: it never appears in your logs) |
meta-externalagent |
Meta | Collects content for Meta AI. |
Amazonbot |
Amazon | Collects content for Amazon services including Alexa. |
CCBot |
Common Crawl | Builds a public web archive widely used as AI training data. |
Bytespider |
ByteDance | Collects content for ByteDance models. |
User-triggered fetchers
These fetch a page because a person asked an assistant about that specific link.
| Crawler | Operator | What it does |
|---|---|---|
ChatGPT-User |
OpenAI | Fetches a page because someone asked ChatGPT about that link. |
Claude-User |
Anthropic | Fetches a page because someone asked Claude about that link. |
Perplexity-User |
Perplexity | Fetches a page in response to a specific user request. |
How do I block one of them?
Add a group for it to robots.txt. This keeps GPTBot out of the whole site:
User-agent: GPTBot
Disallow: /Crawlers that follow robots.txt read it before anything else. A crawler with its own group ignores the rules under User-agent: *, so any general rules you rely on have to be repeated in its group. The free PinMeOut plugin writes these groups for you and repeats the general rules automatically.