How to Create a robots.txt File (Step by Step)
A robots.txt file tells search engines which pages to crawl. Learn the syntax, see an example, and avoid common mistakes.
Creating a robots.txt file is one of the simplest yet most important steps in technical SEO. This small text file sits at the root of your website and tells search engine crawlers which parts of your site they may or may not visit, giving you control over how your pages are discovered.
What robots.txt does
When a search engine like Google visits your site, the first thing it looks for is a file at yoursite.com/robots.txt. The rules inside guide the crawler — you can welcome it everywhere, block private folders, or point it to your sitemap so it finds your content faster.
The basic syntax
- User-agent — which crawler the rule applies to ("*" means all).
- Allow — a path the crawler may access.
- Disallow — a path the crawler should not visit.
- Sitemap — the full URL of your XML sitemap.
A simple example
Here is a typical robots.txt that allows everything except an admin folder and lists a sitemap:
| Line | Meaning |
|---|---|
| User-agent: * | Applies to every crawler. |
| Disallow: /admin | Blocks the admin folder. |
| Allow: / | Allows the rest of the site. |
| Sitemap: .../sitemap.xml | Points to your sitemap. |
Common mistakes to avoid
Disallow tells crawlers not to crawl a page, but it does not reliably keep it out of Google. To truly hide a page from search results, use a noindex meta tag instead.
Other frequent errors include blocking your CSS and JavaScript (which hurts how Google renders your pages) and placing the file anywhere other than the site root, where crawlers will never find it.
Generate yours in seconds
You do not have to write the syntax by hand. Our robots.txt generator builds a valid file from simple options, and to control indexing at the page level the meta tag generator creates the tags you need. For the official rules straight from Google, see the Google Search Central documentation. Upload the finished file to your root, and search engines will follow your directions from the next crawl.