How to Create a robots.txt File (Step by Step)

Creating a robots.txt file is one of the simplest yet most important steps in technical SEO. This small text file sits at the root of your website and tells search engine crawlers which parts of your site they may or may not visit, giving you control over how your pages are discovered.

What robots.txt does

When a search engine like Google visits your site, the first thing it looks for is a file at yoursite.com/robots.txt. The rules inside guide the crawler — you can welcome it everywhere, block private folders, or point it to your sitemap so it finds your content faster.

The basic syntax

• User-agent — which crawler the rule applies to ("*" means all).
• Allow — a path the crawler may access.
• Disallow — a path the crawler should not visit.
• Sitemap — the full URL of your XML sitemap.

A simple example

Here is a typical robots.txt that allows everything except an admin folder and lists a sitemap:

Line	Meaning
User-agent: *	Applies to every crawler.
Disallow: /admin	Blocks the admin folder.
Allow: /	Allows the rest of the site.
Sitemap: .../sitemap.xml	Points to your sitemap.

Common mistakes to avoid

Disallow tells crawlers not to crawl a page, but it does not reliably keep it out of Google. To truly hide a page from search results, use a noindex meta tag instead.

Other frequent errors include blocking your CSS and JavaScript (which hurts how Google renders your pages) and placing the file anywhere other than the site root, where crawlers will never find it.

Generate yours in seconds

You do not have to write the syntax by hand. Our robots.txt generator builds a valid file from simple options, and to control indexing at the page level the meta tag generator creates the tags you need. For the official rules straight from Google, see the Google Search Central documentation. Upload the finished file to your root, and search engines will follow your directions from the next crawl.

robots.txt, noindex and authentication solve different problems

A robots.txt rule manages crawling; it is not a privacy control. A blocked URL can still be known through links and may appear without a useful snippet. A page-level noindex directive controls search-result eligibility, but the crawler must be allowed to fetch the page to see that directive. Authentication is the correct protection for private content. Never rely on Disallow to hide customer records, staging credentials or confidential files.

How Google chooses between matching rules

Rules are grouped by user-agent. A crawler uses the most specific matching group, then evaluates the most specific path rule. Paths are case-sensitive and begin from the site root. A missing slash, wrong hostname or copied rule from another platform can change the scope completely. Test important patterns with Search Console and inspect the exact production file rather than a local copy.

Safe deployment checklist

1. Publish the file at the root, such as https://example.com/robots.txt.
2. Return a normal text response without a login wall or redirect chain.
3. Keep CSS, JavaScript and public images crawlable so search engines can render pages.
4. List the canonical HTTPS sitemap URL.
5. Check production after every migration, staging launch or CMS change.

Common real-world failure

A staging site may use Disallow: / to prevent crawling. If that file reaches production during deployment, discovery can stop across the whole site. Add robots.txt to the launch checklist, monitor the live response and keep an approved production version in source control.

Validate the outcome

Open the file in a private browser window, test representative URLs and watch Search Console for blocked resources or unexpected crawl changes. A sitemap entry does not override a Disallow rule. After editing, allow time for crawlers to fetch the file again; do not repeatedly submit URLs as a substitute for correcting the rule.