AI crawlers
GPTBot
OpenAI's training crawler. Blocking it keeps your pages out of OpenAI's model training and has no effect on whether ChatGPT search shows you.
Sources checked 15 November 2026
GPTBot collects pages that may be used to train OpenAI's models. ChatGPT's search answers come from a different crawler, OAI-SearchBot, and OpenAI's documentation says the two settings are independent.
So blocking GPTBot is a decision about your content, not about visibility. 19 of the 56 large sites we checked on 14 November 2026 block it. All 14 news sites in the list block at least one AI crawler, while 13 of the 15 software companies block none.
To block it and nothing else:
User-agent: GPTBot
Disallow: /