Skip to main content

Free AI crawler checker.

See which AI search, assistant, and training crawlers your robots.txt file allows, limits, or blocks.

Website

Reads the public robots.txt file and maps rules to known AI crawlers.

Search

Check OpenAI, Anthropic, and Perplexity search crawlers.

User requests

See whether assistants may fetch pages for users.

Training

Separate model-training policies from search access.

AI crawler checker FAQ

Read crawler access correctly.

What does an AI crawler checker do?

It reads a website's public robots.txt file and applies its user-agent rules to known AI search, assistant, and model-training crawlers.

Which AI crawlers are checked?

The check covers OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot, and Google-Extended, alongside their published robots.txt names.

Does blocking GPTBot also block ChatGPT search?

Not necessarily. GPTBot is associated with model training, while OAI-SearchBot and ChatGPT-User serve different search and user-requested access. Each user-agent should be reviewed separately.

What happens when a site has no robots.txt file?

Robots.txt does not restrict public crawling when the file is absent. Other controls, authentication, response status, and page-level directives may still affect access.