About this tool
A robots.txt file tells search engine crawlers which parts of your site they may or may not access. This generator lets you allow or disallow paths, target specific bots, set a crawl delay and add your sitemap URL, then outputs a valid robots.txt ready to upload to your site's root. It is a simple but important SEO file.
How to use
- 1Choose to allow or disallow paths.
- 2Add your sitemap URL.
- 3Copy the file and upload it to your site root.
Benefits
- Valid, standards-compliant output.
- Supports sitemap and crawl-delay.
- No syntax mistakes to worry about.
Generate a crawl policy without hiding indexable content
Practical example
Blocking /admin/ can keep a private interface out of normal crawling, while blocking a shared /assets/ folder can prevent Google from rendering public pages correctly. Each path needs a specific reason.
Interpreting the result
Place the file at the host root, use absolute sitemap URLs and test rules for the intended user-agent. Use authentication or authorization—not robots.txt—to protect confidential resources.
Limits and common mistakes
Robots.txt controls crawling, not guaranteed indexing or access. Crawlers can ignore it, blocked URLs can still appear without snippets and one character can affect an entire directory.
Detailed guides for this tool
Learn the underlying concepts, worked examples and common mistakes before relying on the result.
Frequently asked questions
Where do I put robots.txt?+
In the root of your domain, so it is reachable at yoursite.com/robots.txt.
Does disallow hide a page from Google?+
It asks crawlers not to crawl it, but does not guarantee it stays out of the index. Use noindex for that.