BotsRead demo page
This site blocks AI search in robots.txt
Everything on this page is plain HTML, so any crawler could read it. But this site's robots.txt asks the search bots of ChatGPT and Perplexity to stay out. Bots that follow robots.txt won't read the page, so AI search can't quote it or link to it.
The rules in this site's robots.txt
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: GPTBot User-agent: CCBot Disallow: /
What each rule asks
- OAI-SearchBot reads pages for ChatGPT search. Blocking it keeps a site out of ChatGPT search answers.
- PerplexityBot reads pages for Perplexity search.
- ChatGPT-User opens a page when a person asks ChatGPT to. OpenAI says robots.txt rules may not apply to these visits.
- GPTBot collects pages for training OpenAI's models. Blocking it doesn't remove a site from ChatGPT search.
- CCBot collects pages for Common Crawl, a public web archive that many AI models are trained on.
Check your own site
Open any page in Chrome and click BotsRead. It shows which AI and search bots your robots.txt blocks, quotes the exact line, and checks how many of your words reach crawlers that don't run JavaScript.