Free Tool
Noindex/Nofollow Checker
Check meta robots tags and X-Robots-Tag headers to see if a page is blocked from indexing or following. Catch accidental noindex directives before they affect your search visibility.
What does noindex mean?
Noindex is a directive that tells search engines not to include a page in their index. When a page is noindexed, it will not appear in search results. It is commonly used for thin content, duplicate pages, staging environments, internal search results, and pages you do not want appearing in search.
Noindex can be set via a meta robots tag in the HTML head or via an X-Robots-Tag HTTP response header. Both communicate the same directive — they just use different delivery mechanisms.
What does nofollow mean?
Nofollow is a directive that tells search engines not to follow specific links or pass link equity through them. There is an important distinction between page-level and link-level nofollow:
- Page-level nofollow — A meta robots tag with nofollow tells search engines not to follow any links on the entire page. The page itself may still be indexed.
- Link-level nofollow — The rel="nofollow" attribute on an individual link affects only that specific link, not the entire page.
Nofollow does not prevent a page from being indexed. It only controls whether links are followed and whether link equity is passed. If you want to remove a page from search results entirely, use noindex, not nofollow.
Meta robots vs X-Robots-Tag
Meta robots tags are HTML elements placed in the head section of a page. They are the most common way to control indexing on a per-page basis.
X-Robots-Tag is an HTTP response header set at the server level. It communicates the same directives as a meta robots tag but is delivered in the HTTP response rather than the HTML. This makes it useful for controlling indexing of non-HTML files like PDFs, images, and downloadable documents where you cannot add HTML tags.
Both mechanisms can be used simultaneously. When they are, ensure they provide consistent directives — conflicting signals can lead to unpredictable behavior.
Noindex vs robots.txt
This is one of the most commonly confused areas of technical SEO. The two mechanisms control different things:
- Robots.txt controls crawling — It tells search engine crawlers which pages or paths they are allowed to visit. If a page is blocked in robots.txt, the crawler will not fetch it.
- Noindex controls indexing — It tells search engines not to include a page in search results, even after they have crawled it.
The critical interaction: if you block a page in robots.txt, crawlers cannot access the page to see its noindex directive. This means the page may remain indexed if it was previously crawled and indexed. If you want to remove a page from search results, allow crawling and use noindex — do not block it in robots.txt.
Learn more about how crawlability and indexability interact, and how technical SEO addresses these foundational issues.
Common noindex mistakes
- Accidentally noindexing important pages — A noindex directive left on a production page by mistake removes it from search results entirely.
- Leaving staging directives on production — Staging or development environments often use noindex. When code is deployed to production without removing these directives, important pages disappear from search.
- Applying noindex to canonical pages — If you noindex a page that other pages canonicalize to, you create a contradiction: you are telling search engines not to index the preferred version while also telling them it is the preferred version.
- Misunderstanding robots.txt — Blocking a page in robots.txt to remove it from search results is a common mistake. It prevents the crawler from seeing the noindex tag, so the page may stay indexed.
- Using directives inconsistently — Different noindex signals in HTML and HTTP headers can conflict. Ensure all signals are consistent.
Nofollow and internal links
In most cases, you should not use nofollow on internal links. Nofollow prevents link equity from flowing within your site, which can reduce the visibility of pages you want to rank. However, there are legitimate uses:
- User-generated content — Nofollow links in comments, forum posts, or user-submitted content to prevent spam.
- Paid placements — Nofollow sponsored or paid links to comply with search engine guidelines.
- Untrusted content — Nofollow links to pages or external sites you do not want to endorse.
Use nofollow deliberately, not as a default. For internal links between pages you want search engines to crawl and rank, use standard follow links.
Crawl and indexation directives and AI search visibility
A page excluded from search engine indexing through noindex is generally not eligible to appear as a conventional organic search result. While noindex does not directly control every AI engine, crawl and indexation controls are foundational to search visibility. If a page cannot be crawled or indexed, it is unlikely to appear in search-based answers or AI-generated responses that draw from indexed content.
If your goal is to maximize visibility, ensure important pages are crawlable and indexable. Use noindex only for pages you genuinely want excluded from search. Learn how SEO audits can help identify indexation issues across your site.
Why this matters
A noindex directive tells search engines not to include a page in their index, making it invisible in search results. Accidentally noindexing important pages is one of the most common and damaging SEO mistakes. This tool helps you identify indexing directives before they affect your traffic.
Related glossary terms
Frequently Asked Questions
Noindex is a directive that tells search engines not to include a page in their index. When a page is noindexed, it will not appear in search results. It is commonly used for thin content, duplicate pages, staging environments, and pages you do not want appearing in search.
Nofollow is a directive that tells search engines not to follow specific links or pass link equity through them. At the page level, a meta robots nofollow directive applies to all links on that page. At the link level, rel="nofollow" on an individual link affects only that link.
Meta robots tags are HTML elements placed in the head of a page. X-Robots-Tag is an HTTP response header set at the server level. Both communicate the same directives to search engines. X-Robots-Tag is useful for controlling indexing of non-HTML files like PDFs, images, and documents where you cannot add HTML tags.
Robots.txt controls crawling — it tells search engines which pages they are allowed to visit. Noindex controls indexing — it tells search engines not to include a page in search results even after crawling it. Blocking a page in robots.txt prevents crawlers from seeing any noindex directive on that page, which can cause the URL to remain indexed if it was previously crawled.
X-Robots-Tag is an HTTP response header that works like a meta robots tag but is set at the server level. It is useful for controlling indexing of non-HTML files like PDFs, images, and downloadable documents where you cannot add HTML meta tags.
In most cases, no. Nofollow on internal links prevents link equity from flowing within your site. However, there are legitimate uses such as nofollowing links in user-generated content, paid placements, or links to pages you do not want to signal as important. Use it deliberately, not as a default.
A page excluded from search engine indexing through noindex is generally not eligible to appear as a conventional organic search result. While noindex does not directly control every AI engine, crawl and indexation directives are foundational to search visibility. If a page cannot be crawled or indexed, it is unlikely to appear in search-based answers.
Related articles
Related tools
Let's talk
Your next customer might not click a blue link — they'll ask an AI. Be the answer.
Book a free strategy call and I'll audit your visibility across Google, AI Overviews, ChatGPT, and Perplexity — then map out exactly what it takes to be found and cited.