AI crawlers
AI crawler
A program an AI company runs to fetch web pages: to build its assistant's search index, to collect training data, or to open a page a person asked about.
Also called AI bot, AI web crawler, LLM crawler · Sources checked 14 November 2026
AI companies send three kinds, and they do different jobs:
- Search crawlers, such as OAI-SearchBot and PerplexityBot, build the index an assistant searches when it answers. Blocking one keeps you out of that assistant's answers.
- Training crawlers, such as GPTBot and ClaudeBot, collect pages that may be used to train models. Blocking one does not change whether you are cited.
- User fetchers, such as ChatGPT-User, open a page because a person asked. OpenAI says robots.txt may not apply to its fetcher, and Perplexity says its own generally ignores it.
You control the first two in robots.txt, by the names each company publishes. When we read the robots.txt of 56 large sites on 14 November 2026, 19 blocked GPTBot, 8 blocked OAI-SearchBot and 31 blocked no AI crawler at all.
When Vercel measured them in December 2024, none of the major AI crawlers ran JavaScript. Text that only appears after scripts run may never reach them.