AI crawlers
ClaudeBot
Anthropic's training crawler. Anthropic runs two others: Claude-SearchBot for Claude's search results, and Claude-User for pages people ask Claude to open.
Also called Claude-SearchBot, Claude-User · Sources checked 14 November 2026
Anthropic publishes three names, and each can be set separately in robots.txt:
| Name | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects pages that may be used in training | Your pages are not used to train Anthropic's models |
| Claude-SearchBot | Improves Claude's search results | Less visibility in Claude's answers, per Anthropic |
| Claude-User | Opens a page when a person asks Claude to | Less visibility for those requests, per Anthropic |
ClaudeBot was the most blocked AI crawler in our check of 56 large sites on 14 November 2026: 25 block it, against 15 for Claude-SearchBot.
Anthropic also says that since 30 October 2026 it honours a machine-readable "no AI training" signal, which may in time replace naming each training crawler.