Free tool · No sign-up · Nothing is stored
AI crawler checker: can ChatGPT, Claude and Perplexity read your site?
Enter an address. We read your robots.txt and the page, and show which of 13 crawlers are let in, which are kept out, and whether that stops AI assistants from citing you.
Or try one:
What it reads
Your robots.txt, by the rules crawlers themselves follow (RFC 9309): the longest matching rule wins, and a crawler that has its own group ignores the rules under the star. Most mistakes we see come from that second rule. A file that blocks /admin/ for everyone and then adds a separate group for GPTBot has, without meaning to, let GPTBot into /admin/.
It then checks the address you entered against 13 crawlers in three groups:
- Search crawlers build the indexes assistants answer from: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, Bingbot (which also feeds Copilot) and Googlebot (which also feeds AI Overviews).
- User fetchers open a page when a person asks an assistant to read it: ChatGPT-User, Claude-User, Perplexity-User. Robots.txt has less hold here. OpenAI's documentation says its rules "may not apply" to these visits, and Perplexity's says its fetcher generally ignores them, because a person asked for the page.
- Training crawlers collect text for training models: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
Blocking the search crawlers keeps you out of AI answers. Blocking the training crawlers does not, which is why the result treats that as your choice and not as an error.
It also reads the page once to see whether its text is in the HTML before any JavaScript runs, whether it carries a noindex, and whether the site has a sitemap and an llms.txt.
What it cannot see
Your firewall. Robots.txt is a request, and a bot-protection rule can refuse a crawler that robots.txt welcomes. Cloudflare, for one, has settings that block AI crawlers by category. We do not pretend to be GPTBot to test this: sites that verify crawler IP addresses would refuse our copy and let the real one through, so the result would mislead you. If the check says you are open and assistants still never cite you, look at your firewall's bot settings next.
What we found on 60 well-known sites
On 14 November 2026 we ran this check on 60 large sites across news, software, shopping, reference and developer tools, and could read the robots.txt of 56. Of those, 19 block GPTBot but only 8 block OAI-SearchBot. All 14 news sites block at least one AI crawler. Of 15 software companies, 13 block none. The full list and the script are in our crawler guide.
Questions
- If I block GPTBot, do I disappear from ChatGPT?
- No. GPTBot collects training data. ChatGPT's search results come from OAI-SearchBot, and OpenAI's documentation says the two settings are independent. Pages a person asks ChatGPT to open are fetched by a third agent, ChatGPT-User.
- The check says I'm open but AI assistants never mention me. Why?
- Being readable is the entry ticket, not the reason to be cited. Assistants cite pages that answer the question asked and that other sites refer to. The usual gap is content, not access.
- Do you store the addresses I check?
- No. The check runs when you press the button and the result is not saved.
Read next
Updated 14 November 2026.