AI crawlers
PerplexityBot
Perplexity's search crawler. It builds the index behind Perplexity's answers and, according to Perplexity, is not used to train models.
Also called Perplexity-User · Sources checked 14 November 2026
Blocking PerplexityBot leaves a site out of Perplexity's results. Perplexity says a robots.txt change can take up to 24 hours.
Perplexity-User is different. It fetches a page when a person asks Perplexity about it, and Perplexity says it generally ignores robots.txt, because a person asked for the page.
19 of the 56 large sites we checked on 14 November 2026 block PerplexityBot, more than twice as many as block OpenAI's search crawler (8).