robots.txt generator
Build a valid robots.txt without memorizing the syntax. Set your allow and disallow paths, decide per AI crawler whether it may read your site, add your sitemap, and copy the result.
How to use
- Enter the paths you want to allow or disallow for all crawlers, one per line.
- Review the 16 known AI crawlers and uncheck any you want to block. Allowed crawlers get an explicit Allow group so your policy is unambiguous.
- Add your sitemap URL so crawlers can discover all of your pages.
- Generate the file, copy it, and place it at the root of your domain as /robots.txt.
- Verify the result with the AI crawler checker before you ship it.
Frequently asked questions
Where does robots.txt go?
At the root of your domain: https://yourdomain.com/robots.txt. It must be served as plain text and be reachable without a login or redirect to another host.
Does robots.txt guarantee a bot stays out?
No. robots.txt is a convention, not access control. Reputable crawlers (Google, OpenAI, Anthropic, Perplexity) honor it. Anything sensitive still needs real authentication.
Why does the generator emit explicit Allow: / groups for AI crawlers?
An explicit group per crawler makes your policy unambiguous to both the crawler and to anyone reading the file later. It also protects the allow decision if you later add a restrictive wildcard rule.
Can I block AI training but stay visible in AI answers?
Yes. Uncheck the Training crawlers (GPTBot, ClaudeBot, CCBot and friends) and keep the Search and Live-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) allowed.
How do I know the generated file is valid?
The generator emits standard robots.txt syntax and the output is verified against the same parser our audit uses. You can double-check any file with the AI crawler checker tool.