About this tool
Robots.txt Checker reviews pasted crawler directives without fetching a website or sending data to a server. It identifies missing user-agent groups, orphaned rules, malformed sitemap URLs, invalid crawl delays, duplicate directives and unknown fields. It is a syntax and structure assistant—not a promise that a search engine will crawl or index a page.
How to use
- 1Open the robots.txt file from your domain and copy its complete contents.
- 2Paste the contents into the checker and review errors before warnings.
- 3Correct the source file, then test important URLs in the relevant search-engine tools.
Benefits
- No remote URL fetching, which avoids SSRF and privacy risks.
- Line-numbered findings make corrections easier.
- Explains the difference between valid syntax and safe crawl policy.
A syntax pass is only the first safety check
Practical example
“Disallow: /” is valid syntax but blocks the entire matching user-agent group. A green structural result therefore does not mean the policy is suitable for a public production site.
Interpreting the result
Resolve errors first, then inspect each group as the named crawler would read it. Confirm that CSS, JavaScript, images and canonical pages needed for rendering remain crawlable, and keep sitemap URLs absolute.
Limits and common mistakes
Crawler support for non-standard directives varies. This local checker does not fetch paths, apply a specific search engine’s precedence algorithm, test HTTP status codes or verify that sitemap URLs exist.
Frequently asked questions
Does a valid robots.txt guarantee indexing?+
No. Robots.txt controls crawling. Indexing also depends on page quality, status, canonical signals and other factors.
Can robots.txt remove a page from Google?+
Blocking crawling is not a reliable removal method. Use an indexable noindex directive or the search engine's removal process where appropriate.
Does this checker fetch my website?+
No. It checks only the text you paste and runs entirely in your browser.