Syntax
01Find malformed directives, missing user-agent groups, and conflicting path rules.
Free robots.txt generator and checker.
Generate a clean file or check the one already live. See exactly which search and AI-answer crawlers can access your site.
Crawler access
Choose who can crawl, then add the paths you want to keep private.
Search crawlers
Google, Bing, and other general crawlers
OpenAI crawlers
ChatGPT search, user visits, and GPTBot
Anthropic crawlers
Claude search, user visits, and ClaudeBot
Perplexity crawlers
PerplexityBot and user-requested visits
Other training crawlers
Common Crawl, Google-Extended, and Bytespider
One path per line.
Optional, but useful for discovery.
robots.txt
Edit anything before you publish it at /robots.txt.
See whether AI recommends your brand
Enter your domain to track brand mentions, competitor positions, and cited sources across ChatGPT, Gemini, Perplexity, AI Mode, and AI Overviews.
What gets checked
The file, without the guesswork.
Crawler access
02Summarize effective rules for search, AI-answer, and training crawlers.
Sitemap links
03Extract and validate every sitemap URL declared in the file.
Live response
04Confirm the public file exists and is served with a sensible response and content type.
robots.txt generator FAQ
Crawling rules, plainly stated.
What does a robots.txt file do?
A robots.txt file tells compliant crawlers which paths they may request. It is a crawling directive, not a security control, and blocked URLs can still appear in search results.
Can I control AI-answer crawlers separately?
Yes. Many answer engines publish distinct crawler tokens for search, user-requested visits, and model training. The generator creates separate groups so you can choose a policy for each use.
Where should I publish the file?
Publish it at the root of the exact host it covers, such as https://example.com/robots.txt. A file in a subdirectory does not control the rest of the site.
Does the checker crawl my website?
No. It makes one bounded request to the public /robots.txt file and analyzes the returned text without a model or search API.