AI crawlers
robots.txt
A plain text file at the root of a site that tells crawlers which paths they may fetch. It is a request, not a lock.
Also called robots file, robots exclusion protocol · Sources checked 14 November 2026
Crawlers read yoursite.com/robots.txt before fetching anything else. The rules are standardised in RFC 9309:
- Rules sit in groups under one or more
User-agentlines. - A crawler that has a group of its own ignores the group for everyone (
*). This causes the most common mistake: an AI crawler gets its own group, and the private paths are not repeated in it. - The longest matching rule wins, and
Allowwins a tie.
It enforces nothing. The large search and AI crawlers say they follow it; fetchers acting for a person may not. Stopping a crawler that ignores it takes a firewall rule.
Large platforms are a poor model for your own file. Reddit's robots.txt disallows everything for every crawler, Googlebot included, and Reddit is plainly in Google, because platforms that size arrange access by agreement.