Free Tool
Robots.txt Checker
Fetch and parse robots.txt to check crawler access rules, validate syntax, flag blocked important paths, and identify misconfigurations that affect crawlability.
What is robots.txt?
Robots.txt is a text file placed at the root of your domain that provides instructions to search engine crawlers about which pages or paths they are allowed to crawl. It is the first file compliant crawlers look for when visiting a site.
Robots.txt provides crawler instructions at the site level. A well-configured robots.txt file helps optimize crawl budget by guiding crawlers toward your important content and away from low-value paths. Learn how it fits into technical SEO.
What does robots.txt control?
It is important to distinguish between three separate processes:
- Crawling — Whether a search engine is allowed to fetch a page. Robots.txt controls this.
- Indexing — Whether a page is included in the search engine's index. Robots.txt does not directly control this.
- Ranking — Where a page appears in search results. Robots.txt does not control this.
Robots.txt is primarily about crawler access. A page blocked in robots.txt may still be indexed if it was previously crawled or if other signals suggest it should be indexed. If you want to remove a page from search results, use noindex, not robots.txt.
Understanding User-agent
The User-agent directive specifies which crawler the following rules apply to:
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /admin/An asterisk (*) means the rules apply to all crawlers. You can also specify rules for individual crawlers, such as Googlebot or Bingbot, to provide crawler-specific instructions. Rules for specific crawlers take precedence over the wildcard rules.
Allow and Disallow
The Disallow directive tells crawlers not to access specific paths:
User-agent: *
Disallow: /private/
Disallow: /admin/
Allow: /This tells all crawlers not to access anything under /private/ or /admin/, but allows access to everything else. An empty Disallow (Disallow:) means nothing is blocked. Note that not all crawlers necessarily obey robots.txt identically — compliance is voluntary, though major search engines generally follow the rules.
Robots.txt vs noindex
This is one of the most commonly confused areas of technical SEO:
- Robots.txt controls whether compliant crawlers are allowed to fetch a resource. It prevents crawling.
- Noindex is an indexing directive that tells search engines not to include a page in search results, when the crawler can access the page and process the directive.
The critical interaction: if you block a page in robots.txt, crawlers cannot access the page to see its noindex directive. This means the page may remain indexed if it was previously crawled. If you want to remove a page from search results, allow crawling and use noindex — do not block it in robots.txt. Check for noindex directives with the Noindex/Nofollow Checker.
Robots.txt vs sitemap
Robots.txt and XML sitemaps serve different purposes:
- Robots.txt — Controls crawler access and provides site-level instructions.
- Sitemap — Helps search engines discover URLs by listing them in a structured file.
They can work together: you can declare your sitemap location in robots.txt using the Sitemap directive, making it easier for search engines to find. Validate your sitemap with the XML Sitemap Validator.
Common robots.txt mistakes
- Accidentally blocking important sections — A Disallow rule that is too broad can block critical pages from being crawled.
- Blocking CSS/JS resources — Search engines need to render pages, which requires CSS and JavaScript. Blocking these resources can prevent proper understanding of your content.
- Disallowing the entire site — A Disallow: / rule blocks all crawling. This is sometimes left on after staging or maintenance.
- Incorrect path patterns — Path matching in robots.txt can be subtle. Test your rules carefully.
- Forgetting sitemap declaration — Adding your sitemap URL to robots.txt helps search engines discover it without manual submission.
- Assuming robots.txt removes indexed URLs — Blocking a page in robots.txt does not remove it from search results if it is already indexed.
- Testing against the wrong user-agent — Rules for specific crawlers differ from wildcard rules. Test with the correct user-agent.
- Relying on robots.txt for security — Robots.txt is not an access-control mechanism. Non-compliant crawlers and direct visitors can still access blocked paths.
Robots.txt and AI search visibility
Crawler accessibility is part of the technical foundation for discoverability. If a page is blocked in robots.txt, compliant crawlers — including those that feed AI search engines — may not be able to access it. However, not all crawlers behave identically, and allowing every crawler does not automatically increase visibility.
The most effective approach is to ensure important pages are crawlable while using meta robots or noindex for pages you want excluded from search results. Learn how SEO audits can help identify crawlability issues across your site.
Why this matters
Robots.txt provides crawler instructions at the site level. A misconfigured robots.txt can accidentally block search engines from crawling important pages — one of the most damaging SEO mistakes. This tool helps you catch access restrictions before they affect your search visibility.
Related glossary terms
Frequently Asked Questions
Robots.txt is a text file placed at the root of your domain that provides instructions to search engine crawlers about which pages or paths they are allowed to crawl. It is the first file compliant crawlers look for when visiting a site.
Robots.txt controls crawler access — whether search engines are allowed to fetch specific pages or paths. It does not control indexing (whether a page appears in search results) or ranking. These are separate processes. A page blocked in robots.txt may still be indexed if it was previously crawled or if other signals suggest it should be indexed.
No. Robots.txt prevents crawling but not indexing. If a page is already indexed and you want it removed, use a noindex meta tag or a URL removal request in Google Search Console. Blocking crawling via robots.txt while the page is indexed can prevent search engines from seeing the noindex directive, causing the page to remain indexed.
The User-agent directive specifies which crawler the following rules apply to. An asterisk (*) means the rules apply to all crawlers. You can also specify rules for individual crawlers, such as User-agent: Googlebot, to provide crawler-specific instructions.
No. Search engines need to render your pages to understand their content and layout, which requires access to CSS and JavaScript files. Blocking these resources can prevent crawlers from understanding your page, which may affect how your content is interpreted and ranked.
Robots.txt controls whether compliant crawlers are allowed to fetch a resource. Noindex is an indexing directive that tells search engines not to include a page in search results when the crawler can access and process the directive. Blocking a page in robots.txt prevents crawlers from seeing any noindex tag on that page.
No. Robots.txt is a set of voluntary instructions for compliant crawlers. It does not prevent access to your pages — non-compliant crawlers and direct visitors can still access blocked paths. Do not use robots.txt as an access-control or security mechanism. Use server-side authentication and authorization for sensitive content.
Yes. You can add a Sitemap directive in robots.txt to help search engines discover your sitemap. This is optional but useful. The directive is independent of Allow/Disallow rules.
Related articles
Related tools
Let's talk
Your next customer might not click a blue link — they'll ask an AI. Be the answer.
Book a free strategy call and I'll audit your visibility across Google, AI Overviews, ChatGPT, and Perplexity — then map out exactly what it takes to be found and cited.