Free tool · No sign-up · Runs in your browser
Robots.txt generator with AI crawler rules
Two decisions and a list of private folders. The file updates as you choose, and it handles the rule that trips most hand-written files.
One per line. Leave empty to allow everything.
# Everyone: search engines and anything not named below User-agent: * Disallow: /admin/ Disallow: /cart
The two decisions
May AI assistants read and cite your site? This covers the crawlers that build search indexes for ChatGPT, Claude and Perplexity, and the fetchers that open a page when someone asks an assistant to read it. Say no and you will not appear in their answers.
May AI companies train models on your site? This covers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Saying no does not affect whether you are cited. It is a question about your content, not your visibility, and reasonable site owners answer it both ways.
Search engines are not part of either choice. The file never blocks Googlebot or Bingbot.
The rule the generator handles for you
A crawler that has a group of its own ignores the group for everyone. So if you block /admin/ under the star and then write a separate group to allow the AI assistants, those assistants are no longer bound by the /admin/ rule. The generator repeats your private paths inside every group it writes. If you edit the file by hand later, keep doing that.
What robots.txt does not do
It does not hide a page. A blocked page can still appear in results if other sites link to it, just without a description; to keep a page out of search, use a noindex tag and leave the page crawlable so the tag can be read. It also does not stop a crawler that ignores it. The well-known ones follow it, and for the rest you need a firewall rule.
When the file is live, run the crawler checker on your site to confirm it reads the way you meant.
Questions
- Where do I put robots.txt?
- At the root of the domain, so it loads at yoursite.com/robots.txt. Each subdomain needs its own.
- Do I need a robots.txt at all?
- Not to be crawled. With no file, everything is allowed. You need one to keep crawlers out of something or to point them at your sitemap.
- What about crawlers that are not on the list?
- They fall under the first group, the one for everyone. New AI crawlers appear a few times a year, so check the list again from time to time.
Read next
Updated 14 November 2026.