Plenty of sites block the AI search crawlers by accident: an old “Disallow: /” for a bot nobody remembers, or a blanket rule copied from a forum in 2019. If ChatGPT or Perplexity can’t read your pages, they can’t cite them.
How to use it
- Open yoursite.com/robots.txt, copy everything and paste it in.
- List the pages you care about: the homepage, a key product page, a blog post, anything you’ve had trouble getting indexed.
- Read the table. Hover a result to see the exact rule and line that decided it.
What to look for
- Search crawlers blocked by mistake. Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot are the ones that put you in answers. These should almost always be allowed.
- Training crawlers. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot collect content for model training. Blocking them is a business decision, not an SEO fix.
- Wildcards that catch too much. A rule like
Disallow: /*?blocks every URL with a query string, including some you might want crawled.
One thing robots.txt can’t do
It controls crawling, not indexing. A blocked page can still show up in Google as a bare URL if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex tag. The indexability checker shows both.