Technical SEO
Robots.txt
A file at the root of a website that instructs search engine crawlers which pages they can access.
CurrentLast updated: 2026-09-16
Back to glossary
Definition
Technical SEORobots.txt is a text file at the root of a website (e.g., example.com/robots.txt) that tells search engine crawlers which pages or sections they can and cannot crawl. It controls crawling, not indexing — a page blocked by robots.txt can still be indexed if linked from elsewhere.
Common misconceptions
- Robots.txt blocks indexing — it does not. It blocks crawling. Use noindex to prevent indexing.
robots-txtcrawlingdirective
Related terms
Let's talk
Your next customer might not click a blue link — they'll ask an AI. Be the answer.
Book a free strategy call and I'll audit your visibility across Google, AI Overviews, ChatGPT, and Perplexity — then map out exactly what it takes to be found and cited.