AI crawlers
AI crawlers explained: which bots to allow and which to block
AI companies run three kinds of crawler, and they answer three different questions. Search crawlers decide whether you can appear in AI answers. Training crawlers decide whether your pages are used to train models. User fetchers open a page when a person asks for it. Allow the first kind if you want to be cited. The second is a choice about your content that does not affect visibility. The third is partly out of your hands.
In short
- Blocking GPTBot does not remove you from ChatGPT's answers. Blocking OAI-SearchBot does.
- Of 56 large sites we checked on 14 November 2026, 19 block GPTBot and 8 block OAI-SearchBot.
- OpenAI and Perplexity both say robots.txt may not apply when a person asks the assistant to open a page.
The 13 crawlers worth knowing
| Crawler | Run by | What it is for | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search | You are left out of ChatGPT's search answers |
| Claude-SearchBot | Anthropic | Improves Claude's search results | Less visibility in Claude's answers, per Anthropic |
| PerplexityBot | Perplexity | Builds Perplexity's search index. Not used for training | You are left out of Perplexity's results |
| Googlebot | Google Search, and with it AI Overviews and AI Mode | You leave Google | |
| Bingbot | Microsoft | Bing, and with it Copilot | You leave Bing and Copilot |
| ChatGPT-User | OpenAI | Opens a page when a person asks ChatGPT to | Uncertain: OpenAI says robots.txt "may not apply" |
| Claude-User | Anthropic | Opens a page when a person asks Claude to | Less visibility for those requests, per Anthropic |
| Perplexity-User | Perplexity | Opens a page when a person asks Perplexity to | Little: Perplexity says it "generally ignores robots.txt" |
| GPTBot | OpenAI | Collects pages that may be used in training | Your pages are not used to train OpenAI's models |
| ClaudeBot | Anthropic | Collects pages that may be used in training | Your pages are not used to train Anthropic's models |
| Google-Extended | A switch, not a crawler: controls Gemini training and grounding | No effect on Google Search or ranking | |
| Applebot-Extended | Apple | A switch, not a crawler: controls training of Apple's models | No effect on Apple search results |
| CCBot | Common Crawl | A public web archive that many AI labs have trained on | You leave future Common Crawl archives |
Each row comes from the company's own page: OpenAI, Anthropic, Perplexity, Google and Apple.
Two details are easy to miss. Google's AI Overviews are part of Search, so they follow your Googlebot rules and your snippet settings; Google's guidance says Google-Extended does not control them. And Google-Extended and Applebot-Extended never visit your site. They are names you put in robots.txt to say how pages already crawled may be used.
What 56 large sites do
Sites blocking each crawler, of 56
- ClaudeBot25
- CCBot23
- Applebot-Extended23
- GPTBot19
- PerplexityBot19
- Claude-SearchBot15
- Google-Extended14
- Claude-User14
- Perplexity-User13
- ChatGPT-User9
- OAI-SearchBot8
- Googlebot1
- Bingbot1
Show as a table
| ClaudeBot | 25 |
| CCBot | 23 |
| Applebot-Extended | 23 |
| GPTBot | 19 |
| PerplexityBot | 19 |
| Claude-SearchBot | 15 |
| Google-Extended | 14 |
| Claude-User | 14 |
| Perplexity-User | 13 |
| ChatGPT-User | 9 |
| OAI-SearchBot | 8 |
| Googlebot | 1 |
| Bingbot | 1 |
On 14 November 2026 we fetched the robots.txt of 60 well-known sites, 10 to 15 each from news, software, shopping, reference and developer tools, and checked each of the 13 crawlers against the home page. Four files could not be read (npr.org, homedepot.com, stackoverflow.com, npmjs.com), which leaves 56.
| Sites blocking it | Share of 56 | |
|---|---|---|
| ClaudeBot (training) | 25 | 45% |
| CCBot (training) | 23 | 41% |
| Applebot-Extended (training) | 23 | 41% |
| GPTBot (training) | 19 | 34% |
| PerplexityBot (search) | 19 | 34% |
| Claude-SearchBot (search) | 15 | 27% |
| Google-Extended (training) | 14 | 25% |
| OAI-SearchBot (search) | 8 | 14% |
| No AI crawler blocked at all | 31 | 55% |
The split by type of site says more than the totals:
- News. All 14 block at least one AI crawler. Only 3 block OAI-SearchBot.
- Software companies. 13 of 15 block nothing. The other two, Figma and Canva, block training crawlers and leave the search crawlers alone.
- Developer tools. None of the 9 block anything.
- Shopping. 7 of 9 block nothing. Amazon and eBay do.
Six sites block training and nothing else: TechCrunch, Figma, Canva, Medium, Tripadvisor and Healthline. That is the "train on me no, cite me yes" position, and it is the one most businesses asking this question want.
One file is a warning against copying the giants. Reddit's robots.txt disallows everything for every crawler, Googlebot included, and Reddit is plainly still in Google. Platforms that size arrange access by agreement. Their robots.txt tells you little about what a business site should do.
How we ran this check
- When and where. 14 November 2026, from a connection in Germany.
- What was fetched. One request for robots.txt per site, trying the www address first and then the bare domain. The request named itself as our own bot. It did not pretend to be any of the 13 crawlers.
- How the file was read. By the rules in RFC 9309: a crawler with a group of its own ignores the group for everyone, the longest matching rule wins, and Allow wins a tie. Each crawler was tested against the home page.
- What was left out. The four sites whose file could not be read.
- The data. Every site and what it blocks, as JSON.
What it does not show. These are 60 large sites we picked, so the percentages describe this list and not the web. Only the home page path was tested. And robots.txt is what a site asks for, which is not always what its firewall enforces.
Frase published a larger check of the same question on 2 November 2026, covering 98 B2B software vendors, with a fuller method than ours. If you want a second measurement, read that one.
The 60 sites: nytimes.com, theguardian.com, bbc.com, cnn.com, washingtonpost.com, reuters.com, apnews.com, wsj.com, bloomberg.com, theverge.com, wired.com, techcrunch.com, forbes.com, theatlantic.com, npr.org, stripe.com, shopify.com, hubspot.com, salesforce.com, notion.so, slack.com, zoom.us, atlassian.com, figma.com, canva.com, mailchimp.com, zendesk.com, asana.com, airtable.com, intercom.com, amazon.com, ebay.com, etsy.com, walmart.com, target.com, bestbuy.com, ikea.com, nike.com, wayfair.com, homedepot.com, wikipedia.org, reddit.com, stackoverflow.com, quora.com, medium.com, substack.com, imdb.com, yelp.com, tripadvisor.com, healthline.com, github.com, developer.mozilla.org, vercel.com, cloudflare.com, docker.com, npmjs.com, python.org, digitalocean.com, netlify.com and supabase.com.
To cite this: Content Autopilot, "robots.txt check of 60 large sites", 14 November 2026, with a link to this page.
What to put in your robots.txt
If you sell something and want to be recommended, this is enough:
User-agent: *
Disallow: /admin/
Sitemap: https://yoursite.com/sitemap.xml
If you also want to opt out of training:
User-agent: *
Disallow: /admin/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://yoursite.com/sitemap.xml
The mistake to avoid: a crawler that has its own group ignores the group for everyone. If you write a separate group for the AI search crawlers, repeat your private paths inside it, or you have just let them into /admin/. The generator below does this for you.
Build the file
One per line. Leave empty to allow everything.
# Everyone: search engines and anything not named below User-agent: * Disallow: /admin/ Disallow: /cart
The training opt-out is changing
Naming every training crawler in robots.txt works today and means editing the file whenever a new one appears. That is starting to be replaced by a single signal. Anthropic's page says that since 30 October 2026 it honours a machine-readable "no AI training" signal, and Google's crawler documentation says it intends to support one in early 2027. Until that settles, the per-crawler rules above are the dependable way.
Questions
- Should I block GPTBot?
- Only if you do not want your pages used to train OpenAI's models. It has no effect on whether ChatGPT cites you. A third of the large sites we checked block it; most software companies do not.
- How long does a robots.txt change take to work?
- OpenAI says about 24 hours for its search results. Perplexity says up to 24 hours. Others do not publish a figure.
- Can a crawler just ignore robots.txt?
- Yes. It is a convention, not a lock. The companies listed here say they follow it for their search and training crawlers. To stop a crawler that does not, you need a firewall rule.
Sources
Each one was opened and read on 14 November 2026. Figures in the text say what was counted, how many and when.
- 1.OpenAI (/docs/bots) · platform.openai.com
- 2.Anthropic (/en/articles/8896518) · support.claude.com
- 3.Perplexity (/guides/bots) · docs.perplexity.ai
- 4.Google (/search/docs/crawling-indexing/google-common-crawlers) · developers.google.com
- 5.Apple (/en-us/119829) · support.apple.com
- 6.guidance (/search/docs/appearance/ai-features) · developers.google.com
- 7.RFC 9309 (/rfc/rfc9309.html) · rfc-editor.org
- 8.98 B2B software vendors · frase.io
Content Autopilot team
We make Content Autopilot, an SEO and AI visibility product
Our guides are drafted with AI from the sources listed above, and every figure is checked against its source before it is published. Where we sell something a guide discusses, the guide says so.