NeovaAdvanced SEO Platform

Noindex Checker: How to Find Noindex Pages Before They Cost You Traffic

One line of HTML keeps a page out of Google. That is exactly what you want on a thank you page, and exactly what you do not want on the product page a developer forgot to switch back after launch. A noindex checker tells you which pages carry noindex, where it was set and whether Google can even see it.

Crawling and Indexingnoindex checkerUpdated November 21, 2026By the Neova team

What a Noindex Checker Does

A noindex checker fetches a URL and reads two places: the meta robots tag in the HTML and the X-Robots-Tag in the HTTP response header. It reports whether either one says noindex or none, and for which crawler.

Checking one URL answers a quick question. Checking every URL of a crawl answers the useful ones: which indexable pages you meant to rank carry noindex, which noindex pages are still in your XML Sitemap, and which ones Google cannot read because robots.txt blocks them.

Two Ways to Set Noindex

Google's page on blocking indexing with noindex says both methods have the same effect. Pick the one that fits the file type.

MethodWhere it livesWorks onSeen in
Meta robots tagHTML headHTML pagesPage source
X-Robots-Tag headerHTTP responseAny file, PDFs, imagesResponse headers
robots.txt noindex linerobots.txt– not supportedIgnored by Google
<!-- in the <head> -->
<meta name="robots" content="noindex">

# as an HTTP response header
X-Robots-Tag: noindex

Replace robots with a crawler name, such as googlebot, to aim the rule at one crawler. SEO ProCheck's meta robots and X-Robots-Tag reference lists every value.

Robots Directives at a Glance

The values Google documents in its robots meta tag specification, and what each one does:

DirectiveEffectCommon use
noindexNot shown in resultsThank you, internal search
nofollowLinks on page not followedUntrusted user pages
nonenoindex plus nofollowRarely the right choice
nosnippetNo text snippetPaywalled or legal text
max-snippet:[n]Snippet up to n charactersShorter snippets
unavailable_after:[date]Dropped after the dateEvents, offers
index, followDefault, no effectCan be left out

When two directives conflict, Google applies the more restrictive one. A page with index in the HTML and noindex in the header ends up not indexed.

Noindex Mistakes That Hide the Wrong Pages

MistakeEffectFix
Staging noindex left at launchWhole site drops outRemove it, recrawl
Noindex plus robots.txt DisallowNoindex never readAllow crawling
Noindex removed by JavaScriptGoogle may skip renderingServe the final tag
Noindex URL in XML SitemapMixed signalsDrop from sitemap
Canonical plus noindexConflicting signalsChoose one
Meta robots outside the headTag may be ignoredMove into head
Header and HTML disagreeStricter one winsKeep one source
Hreflang points to noindexLanguage set breaksPoint to indexable URLs

The second row is the most common surprise. Google says that if robots.txt blocks a page, the crawler will never see the noindex rule, and the URL can still show up in results from links alone. The third comes from Google's JavaScript SEO basics: when Google finds noindex in the original HTML it may skip rendering, so a script that removes it later may never run. The robots.txt tester guide covers the crawling side.

How Long Noindex Takes, and the Faster Options

Google drops a page once it crawls it and reads the noindex. That depends on when it comes back, and Google says it may take months for some pages. Pick the tool by the goal:

GoalUseHow fast
Out of results nowRemovals tool, then noindexFastest
Out of results for goodnoindex, 404 or 410Next crawl
Crawled but not indexednoindexNext crawl
Not crawled at allrobots.txt DisallowNext robots.txt fetch
Back into resultsRemove noindex, request indexingDays to weeks

The Removals tool only hides a URL for about six months, so the page still needs noindex, a 404 or a 410 behind it. Use URL Inspection's Request Indexing for a handful of important pages after you fix them.

How to Check Noindex by Hand

For one URL, check both places. The header first, then the HTML:

curl -sI https://example.com/page | grep -i x-robots-tag
curl -s https://example.com/page | grep -io '<meta[^>]*robots[^>]*>'

Then check what Google saw. In Search Console, URL Inspection shows the HTML Googlebot received, and the Page indexing report lists every page excluded as URL marked 'noindex'. Look at that list after every release: a page you want in search should never be on it.

Two catches with manual checks. A site may answer a crawler user agent differently from a browser, and a tag that JavaScript adds only shows in the rendered HTML, not in curl. A crawler that fetches both the raw and the rendered HTML sees both.

Check a Whole Site for Noindex

  1. Crawl the site from the home page, with JavaScript rendering on if the site depends on it.
  2. List every non-indexable page by reason: noindex, canonical to another URL, error status or blocked.
  3. Sort the noindex pages by template. One template with noindex can hide hundreds of pages at once.
  4. Compare with your XML Sitemaps. Every listed URL should be indexable. On our own crawl of seoprocheck.com on October 17, 2026, the sitemaps listed 1,835 URLs while 4,561 HTML pages answered 200, which is why both lists need checking against each other.
  5. Join Search Console clicks. A noindex page that still gets clicks was ranking until recently, which is often a sign of a mistake.
  6. Crawl again after each release and compare, so a new noindex shows up the day it ships.

The indexability checker guide covers the other reasons a page cannot be indexed, and the XML sitemap checker guide covers the sitemap side.

Find Noindex Pages With Neova

Neova marks every URL Indexable or Non-Indexable and says why: status 200, no noindex, canonical to itself or none. The Directives column set in All URLs shows the robots directives of every URL, with Noindex and Nofollow filters. Migration and Staging compares a staging crawl with production, noindex included, so a staging noindex does not reach launch unnoticed.

All URLs in Neova: the status code and Indexable or Non-Indexable for every URL of the crawl
All URLs in Neova: the status code and Indexable or Non-Indexable for every URL of the crawl

The checks behind it include:

The Free plan crawls 1,000 URLs a crawl with every check, on Mac, Windows and Linux. Download Neova or see Features.

Questions

How do I check if a page is noindex?

Look for a meta robots tag with noindex in the page source and an X-Robots-Tag header in the HTTP response. URL Inspection in Search Console shows what Googlebot saw. For a whole site, crawl it and filter non-indexable pages by reason.

Does noindex work if the page is blocked in robots.txt?

No. Google has to crawl the page to read the noindex. If robots.txt blocks it, Google never sees the rule and the URL can still appear in results from links. Allow crawling until the page has dropped out.

Is noindex in robots.txt supported?

Not by Google. Google stopped supporting noindex lines in robots.txt in 2019. Use the meta robots tag or the X-Robots-Tag header instead.

How long does it take for a noindex page to leave Google?

Until Google crawls the page again, which can be days for busy pages and months for rarely crawled ones. For urgent cases, use the Removals tool, which hides the URL for about six months while the noindex takes effect.

Should noindex pages be in my XML sitemap?

No. A sitemap should list the pages you want indexed. A noindex URL in it sends mixed signals.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading