Study · 14 November 2026
Which AI crawlers do large websites block?
Large sites block AI training crawlers far more often than AI search crawlers. Of 56 well-known sites whose robots.txt we could read, 25 block ClaudeBot and 19 block GPTBot, while 8 block the crawler behind ChatGPT's search answers. 31 block no AI crawler at all. The pattern follows the business: every news site blocks something, and almost no software company does.
- block GPTBot (training)
- 19 of 56
- block ChatGPT's search crawler
- 8 of 56
- block no AI crawler
- 31 of 56
- block training only
- 6
By crawler
Sites blocking each crawler, of 56
- ClaudeBot25
- CCBot23
- Applebot-Extended23
- GPTBot19
- PerplexityBot19
- Claude-SearchBot15
- Google-Extended14
- Claude-User14
- Perplexity-User13
- ChatGPT-User9
- OAI-SearchBot8
- Googlebot1
- Bingbot1
Show as a table
| ClaudeBot | 25 |
| CCBot | 23 |
| Applebot-Extended | 23 |
| GPTBot | 19 |
| PerplexityBot | 19 |
| Claude-SearchBot | 15 |
| Google-Extended | 14 |
| Claude-User | 14 |
| Perplexity-User | 13 |
| ChatGPT-User | 9 |
| OAI-SearchBot | 8 |
| Googlebot | 1 |
| Bingbot | 1 |
| Crawler | What it is for | Sites blocking it | Share of 56 |
|---|---|---|---|
| ClaudeBot | Training | 25 | 45% |
| CCBot | Training (Common Crawl) | 23 | 41% |
| Applebot-Extended | Training | 23 | 41% |
| GPTBot | Training | 19 | 34% |
| PerplexityBot | Search | 19 | 34% |
| Claude-SearchBot | Search | 15 | 27% |
| Google-Extended | Training | 14 | 25% |
| Perplexity-User | Opens a page on request | 13 | 23% |
| Claude-User | Opens a page on request | 14 | 25% |
| ChatGPT-User | Opens a page on request | 9 | 16% |
| OAI-SearchBot | Search | 8 | 14% |
| Googlebot | Google Search | 1 | 2% |
| Bingbot | Bing | 1 | 2% |
The one site that blocks Googlebot and Bingbot is Reddit, whose file disallows everything for everyone. Reddit is plainly still in Google, so its access is arranged some other way. It is the clearest sign that a platform's robots.txt is a poor guide for a business site.
By type of site
| Type | Sites read | Block any AI crawler | Block a training crawler | Block an AI search crawler |
|---|---|---|---|---|
| News | 14 | 14 | 14 | 13 |
| Software companies | 15 | 2 | 2 | 0 |
| Shopping | 9 | 2 | 2 | 2 |
| Reference and community | 9 | 7 | 7 | 4 |
| Developer tools | 9 | 0 | 0 | 0 |
News publishers block because their content is the product. Software companies and developer tools want to be found and quoted, and nearly all of them leave the door open. If you sell something, you are in the second group.
6 sites block training crawlers and leave every search crawler and fetcher alone: techcrunch.com, figma.com, canva.com, medium.com, tripadvisor.com, healthline.com. That is the position most businesses asking "should I block GPTBot?" are looking for: do not train on me, do cite me.
Site by site
| Site | Type | Blocks |
|---|---|---|
| nytimes.com | News | Search: ChatGPT search, Claude search, Perplexity. Training: all five |
| theguardian.com | News | Search: Claude search, Perplexity. Training: ClaudeBot, Applebot-Extended, CCBot |
| bbc.com | News | Search: ChatGPT search, Claude search, Perplexity. Training: all five |
| cnn.com | News | Search: ChatGPT search, Claude search, Perplexity. Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| washingtonpost.com | News | Search: Perplexity. Training: ClaudeBot, Applebot-Extended, CCBot |
| reuters.com | News | Search: Claude search, Perplexity. Training: all five |
| apnews.com | News | Search: Claude search, Perplexity. Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| wsj.com | News | Search: Claude search, Perplexity. Training: ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| bloomberg.com | News | Search: Perplexity. Training: all five |
| theverge.com | News | Search: Claude search, Perplexity. Training: ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| wired.com | News | Search: Claude search, Perplexity. Training: ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| techcrunch.com | News | Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| forbes.com | News | Search: Perplexity. Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| theatlantic.com | News | Search: Claude search, Perplexity. Training: ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| stripe.com | Software companies | Nothing |
| shopify.com | Software companies | Nothing |
| hubspot.com | Software companies | Nothing |
| salesforce.com | Software companies | Nothing |
| notion.so | Software companies | Nothing |
| slack.com | Software companies | Nothing |
| zoom.us | Software companies | Nothing |
| atlassian.com | Software companies | Nothing |
| figma.com | Software companies | Training: GPTBot, ClaudeBot, Google-Extended, CCBot |
| canva.com | Software companies | Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| mailchimp.com | Software companies | Nothing |
| zendesk.com | Software companies | Nothing |
| asana.com | Software companies | Nothing |
| airtable.com | Software companies | Nothing |
| intercom.com | Software companies | Nothing |
| amazon.com | Shopping | Search: ChatGPT search, Claude search, Perplexity. Training: GPTBot, ClaudeBot, Google-Extended, CCBot |
| ebay.com | Shopping | Search: Perplexity. Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| etsy.com | Shopping | Nothing |
| walmart.com | Shopping | Nothing |
| target.com | Shopping | Nothing |
| bestbuy.com | Shopping | Nothing |
| ikea.com | Shopping | Nothing |
| nike.com | Shopping | Nothing |
| wayfair.com | Shopping | Nothing |
| wikipedia.org | Reference and community | Nothing |
| reddit.com | Reference and community | Everything, search engines included |
| quora.com | Reference and community | Search: ChatGPT search, Claude search, Perplexity. Training: GPTBot, ClaudeBot, Applebot-Extended |
| medium.com | Reference and community | Training: GPTBot, ClaudeBot, Applebot-Extended |
| substack.com | Reference and community | Nothing |
| imdb.com | Reference and community | Search: ChatGPT search, Claude search, Perplexity. Training: all five |
| yelp.com | Reference and community | Search: ChatGPT search, Claude search, Perplexity. Training: all five |
| tripadvisor.com | Reference and community | Training: all five |
| healthline.com | Reference and community | Training: GPTBot, ClaudeBot, Applebot-Extended, CCBot |
| github.com | Developer tools | Nothing |
| developer.mozilla.org | Developer tools | Nothing |
| vercel.com | Developer tools | Nothing |
| cloudflare.com | Developer tools | Nothing |
| docker.com | Developer tools | Nothing |
| python.org | Developer tools | Nothing |
| digitalocean.com | Developer tools | Nothing |
| netlify.com | Developer tools | Nothing |
| supabase.com | Developer tools | Nothing |
How we ran this
- When and where. 2026-11-14, from a connection in Germany.
- What was fetched. One request for robots.txt per site, the www address first and then the bare domain. The request named itself as our own bot and did not pretend to be any crawler.
- How the file was read. By the rules in RFC 9309: a crawler with its own group ignores the group for everyone, the longest matching rule wins, and Allow wins a tie. Each crawler was tested against the home page.
- What was left out. 4 of the 60 could not be read and are not counted: npr.org (no answer), homedepot.com (403), stackoverflow.com (418), npmjs.com (403).
What this does not show
These are 60 large sites we chose, 10 to 15 from each of five types. The percentages describe this list and not the web. Only the home page path was tested, so a site that blocks a section is counted as open. And robots.txt is what a site asks for: a firewall can refuse a crawler the file welcomes, and we did not test firewalls.
Frase published a larger check of a narrower group on 2 November 2026: 98 B2B software vendors, read on 9 October. It found none of them blocking the AI search crawlers and 2 blocking a training crawler, which agrees with our software rows.
What to do with it
Run the crawler checker on your own site, and read which crawlers to allow for what each one does.
Use this data
The site-by-site results are published as JSON. Quote any figure with a link to this page. To cite it:
Content Autopilot, “Which AI crawlers do large websites block?”, 14 November 2026. https://apexmention.com/studies/ai-crawler-access