Über dieses Tool
A robots.txt file tells search engine crawlers which parts of your site they may or may not access. This generator lets you allow or disallow paths, target specific bots, set a crawl delay and add your sitemap URL, then outputs a valid robots.txt ready to upload to your site's root. It is a simple but important SEO file.
So funktioniert es
- 1Choose to allow or disallow paths.
- 2Add your sitemap URL.
- 3Copy the file and upload it to your site root.
Vorteile
- Valid, standards-compliant output.
- Supports sitemap and crawl-delay.
- No syntax mistakes to worry about.
Detailed guides for this tool
Learn the underlying concepts, worked examples and common mistakes before relying on the result.
Häufig gestellte Fragen
Where do I put robots.txt?+
In the root of your domain, so it is reachable at yoursite.com/robots.txt.
Does disallow hide a page from Google?+
It asks crawlers not to crawl it, but does not guarantee it stays out of the index. Use noindex for that.