How to Configure robots.txt Correctly

robots.txt is one of the most consequential files on your site. A single misconfigured line can block your entire site from Google — and you may not notice for weeks. Equally, a robots.txt that is too permissive can expose admin paths and staging pages to crawlers. This guide walks through writing a correct, complete robots.txt that gives you control over every crawler that visits your site.

Step 1: Understand what robots.txt can and cannot do

robots.txt is served at https://yourdomain.com/robots.txt and tells well-behaved crawlers which URLs they are allowed to fetch.

What robots.txt can do:

• Prevent crawlers from fetching specific URLs or path patterns
• Reduce crawl budget waste on non-indexable paths
• Point crawlers to your sitemap

What robots.txt cannot do:

• Remove a page from the index (use noindex for that)
• Block malicious bots (they ignore robots.txt)
• Prevent a URL from appearing in search results if it has inbound links

The distinction between blocking crawling (robots.txt) and blocking indexing (noindex) is critical. If you want a page deindexed, you need noindex — not a robots.txt block.

Check your robots.txt now — Use the robots.txt checker to test any URL against your current robots.txt rules and see exactly which bots are blocked or allowed.

Step 2: Write the basic robots.txt structure

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=
Allow: /

Sitemap: https://izajulhaque.com/sitemap.xml

Syntax rules:

• User-agent: * — applies to all crawlers
• Disallow: /path/ — blocks the path and everything under it
• Allow: / — explicitly allows everything else (useful for clarity)
• Blank line between rule groups (different User-agent blocks)
• Case-sensitive paths: /Admin/ and /admin/ are different
• Allow: takes precedence over Disallow: for the same specificity

Step 3: Block non-indexable paths from crawlers

Block paths that should never be crawled or indexed:

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/
Disallow: /logout/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /?s=           # WordPress search results
Disallow: /search?       # Generic search results
Disallow: /tag/          # If tag pages are noindex
Disallow: /staging/      # Staging paths if on same domain
Disallow: /preview/

Do NOT block these:

# Wrong: blocking assets prevents rendering
Disallow: /wp-content/
Disallow: /assets/
Disallow: /static/
Disallow: *.css
Disallow: *.js

Googlebot renders pages using CSS and JavaScript. Blocking these prevents rendering, which hurts page quality assessment, Core Web Vitals measurement, and structured data detection. Never block asset directories.

Step 4: Explicitly handle AI crawlers

AI crawlers use separate user-agent strings and may be caught by wildcard rules. Here is how to handle the main AI crawlers:

# Allow all major AI crawlers explicitly
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

If you want to block a specific AI crawler from accessing your content:

# Block a specific AI crawler
User-agent: GPTBot
Disallow: /

Blocking AI crawlers with Disallow: / prevents them from accessing your content for AI-generated answers. See how to get cited by ChatGPT for why GPTBot access is required for ChatGPT citations, and how to run a GEO audit for a full AI crawlability audit.

Step 5: Add a Sitemap directive

Always include a Sitemap directive pointing to your XML sitemap:

User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /

Sitemap: https://izajulhaque.com/sitemap.xml

For sites with a sitemap index file:

Sitemap: https://izajulhaque.com/sitemap_index.xml

The Sitemap directive ensures every crawler that reads your robots.txt also discovers your sitemap, without requiring manual submission to every search engine. See how to fix a broken XML sitemap for sitemap guidance.

Step 6: Test before deploying

Never deploy a robots.txt change without testing. The most dangerous misconfiguration is:

# This blocks ALL crawlers from ALL pages — do not do this
User-agent: *
Disallow: /

Test every significant path against your rules before deploying:

Paths to test:
✓ Homepage (/)              → should be allowed
✓ Blog (/blog/)             → should be allowed
✓ Service pages (/services/) → should be allowed
✓ Admin (/admin/)           → should be blocked
✓ Login (/login/)           → should be blocked
✓ Search results (/?s=query) → should be blocked

Use the robots.txt checker to test each path against your rules before going live. After deploying, verify in Google Search Console under Settings → robots.txt.

See SEO consulting for help with a full technical SEO audit covering robots.txt, sitemaps, and canonicals.

Common mistakes

• Accidentally blocking the entire site with "Disallow: /" — This is the most catastrophic robots.txt error. It tells all crawlers they cannot access any page on the site. Always test before deploying and check Google Search Console after any robots.txt change. Use the robots.txt checker.
• Blocking /wp-content/ or asset directories — Blocking CSS, JavaScript, or image directories prevents Googlebot from rendering pages. This hurts Core Web Vitals scores, structured data detection, and content quality assessment. Never block asset paths.
• Using robots.txt to hide sensitive pages instead of proper access controls — robots.txt is public — anyone can read it and see which paths you are trying to hide. Use server-level authentication for sensitive pages, not robots.txt obfuscation.
• Not including a Sitemap directive — Every robots.txt should include a Sitemap: directive. It ensures all crawlers discover your sitemap automatically.
• Not testing after a CMS or plugin update — Many CMS plugins and hosting panels automatically modify robots.txt. After any plugin update or hosting migration, verify your robots.txt has not been changed. Use the robots.txt checker and monitor Search Console for crawl anomalies.