NeovaAdvanced SEO Platform

robots.txt Tester: How to Test robots.txt Rules Before They Block the Wrong Pages

robots.txt is a short text file with a lot of power. One line can keep a search engine away from a folder you meant to hide, or from the whole site you meant to rank. A robots.txt tester tells you, URL by URL, what each crawler is allowed to fetch, so you find out before Googlebot does.

Crawling and Indexingrobots.txt testerUpdated November 21, 2026By the Neova team

What a robots.txt Tester Does

A robots.txt tester takes a robots.txt file and a URL, picks the group of rules that applies to a given user agent, and tells you whether that URL is allowed or disallowed, and which line decided it. A good one also points out lines the parser cannot read, fields it ignores and rules that cancel each other out.

Testing one URL at a time is useful for a quick question. Testing every URL of a crawl is what catches the surprises: a rule written for one folder that also matches a product filter, or a wildcard that blocks the CSS your pages need to render.

Google Retired Its robots.txt Tester

For years, Search Console had a robots.txt Tester where you could edit the file and test URLs against it. In late 2023 Google retired it and replaced it with the robots.txt report. The report shows which robots.txt files Google found for the top 20 hosts of a property, when each was last crawled, and any warnings or errors. It also lets you request a recrawl of the file after an urgent fix.

The report tells you how Google read your live file. It does not test a rule that is not live yet, and it does not run your whole URL list against the file. For that you need a separate tester. Google publishes its own open source robots.txt parser, and Bing Webmaster Tools has a robots.txt tester that is handy for checking how Bingbot reads a file.

How robots.txt Rules Work

The file lives at the root of a host, for example https://www.example.com/robots.txt. The rules are defined by RFC 9309, the Robots Exclusion Protocol, and Google documents how it reads them in How Google interprets the robots.txt specification. A file is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules:

User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Allow: /cart/help

User-agent: Googlebot-Image
Disallow: /private-images/

Sitemap: https://www.example.com/sitemap.xml

The points testers check, as Google documents them. SEO ProCheck's complete robots.txt reference goes through every field.

  • One group per crawler. A crawler follows only the most specific group that matches its name. If Googlebot has its own group, it ignores the * group completely.
  • Longest match wins. When an Allow and a Disallow both match a URL, the rule with the longer path applies. On a tie, Google uses the least restrictive rule.
  • Two wildcards. * matches any run of characters and $ marks the end of the URL.
  • Paths are case-sensitive. Disallow: /Admin does not block /admin.
  • One file per host. Rules apply only to the protocol, host and port the file is served from. A subdomain needs its own file.

What robots.txt Does Not Do

robots.txt controls crawling, not indexing. Google's introduction to robots.txt is clear that a disallowed URL can still be indexed if other pages link to it, usually shown without a description. If you want a page out of search results, let it be crawled and use noindex instead. A page Google cannot fetch is a page whose noindex it never sees.

A few other limits worth knowing:

  • No noindex line. Google stopped supporting noindex in robots.txt in 2019, together with nofollow and crawl-delay rules there.
  • No crawl-delay for Google. Googlebot ignores Crawl-delay. Some other crawlers read it, which is why a value set for every user agent can slow them down a lot.
  • No security. The file is public and polite crawlers follow it by choice. Anything private needs a login.

Common robots.txt Mistakes a Tester Catches

MistakeEffectFix
Disallow: / left from stagingWhole site blockedRemove the line
Named group without shared rules* rules dropped for that botCopy shared rules in
Blocked CSS, JavaScript, imagesPage cannot renderAllow the files
Wildcard too broad/*? blocks filters, pagesNarrow the pattern
Rule before User-agent, typoLine skippedFix the syntax
Disallow to deindexnoindex never readAllow, then noindex
www and non-www differRules differ by hostServe one file

Blocking a URL you also list in your XML Sitemap sends mixed signals. The XML sitemap checker guide covers that side.

Server Errors and File Size

How the file is served matters as much as what it says. Google documents each case:

CaseWhat Google doesHow long
4xx, not 429Crawls as if no fileUntil fixed
5xx or 429Stops crawling the siteFirst 12 hours
Still failingUses last good copyUp to 30 days
File over 500 KiBIgnores the restPast 500 KiB
Any changeUses its cached copyUp to 24 hours

The recrawl request in the robots.txt report is for urgent fixes. A robots.txt that times out at the wrong moment can pause crawling of the whole site; the crawl budget guide explains why that matters most on large sites.

What We Found in Our Own robots.txt

Our own SEO pass on neova.app, on November 20, 2026, found two rules that blocked more than we meant:

What was blockedPages hitWhat Google sawFix
Cloudflare email script under /cdn-cgi/1,320A script it could not fetchAllowed that script
/favicon and /apple-touch-icon.pngEvery resultA plain letter, no logoAllowed both files

The favicon files sat behind a Disallow: / rule. Google's favicon guidelines need the file crawlable by Googlebot-Image, so search results showed a letter instead of our logo next to neova.app. Neither rule looked wrong in the file. Both showed up only when the rules were tested against every URL and file the pages load.

How to Test robots.txt Safely

  1. Crawl the site so you have a full list of real URLs, not just the few you remember.
  2. Test the live file against that list for each crawler you care about: Googlebot, Googlebot-Image, Bingbot and the AI crawlers.
  3. Write the change as a draft and test the draft against the same list. Look at the URLs that become newly blocked and newly allowed.
  4. Check the resources. Make sure no indexable page loses its scripts, style sheets or images.
  5. Ship it, then confirm in the Search Console robots.txt report that Google fetched the new version without errors.
  6. Crawl again and compare, so an accidental change later on gets caught.

If you are deciding what AI crawlers may fetch, the AI crawlers guide lists their names and what each one does. Blocked pages also show up in the indexability checker guide.

Test robots.txt With Neova

Neova's robots.txt Tester explains every line of the live file, or of a draft, in plain words, as Google documents it and RFC 9309 defines it, with the crawled URLs each rule matches. Its All Crawlers view shows what Googlebot, Bingbot and the AI crawlers may fetch, with the URLs each one is blocked from. It says when a crawler's own group drops the rules of the * group, and the draft test lists every URL of the crawl that a change would newly block or newly allow. You can then crawl while respecting the draft to see the full effect. The checks behind it:

Neova keeps every robots.txt version its crawls read, so you can see when a rule changed. With the User Agent setting you can crawl as Googlebot, Bingbot, an AI crawler or a custom string. See Features or download Neova.

Questions

Does Google still have a robots.txt tester?

No. Google retired the robots.txt Tester in Search Console in 2023 and replaced it with the robots.txt report, which shows how Google read your live file and any errors. To test a change before it is live, use a separate robots.txt tester.

How do I test if a URL is blocked by robots.txt?

Find the group that matches the crawler's user agent, then the longest Allow or Disallow path that matches the URL. That rule decides. A robots.txt tester does this for you and names the line, for one URL or for every URL of a crawl. SEO ProCheck's robots.txt FAQ answers the edge cases.

Does robots.txt stop a page from being indexed?

No. It stops crawling. A disallowed URL can still be indexed if other pages link to it. Use noindex on a page Google can crawl if you want it out of search results.

Why does Googlebot ignore my User-agent: * rules?

Because the file has a group named for Googlebot. A crawler follows only the most specific group that matches it, so copy the shared rules into the Googlebot group as well.

How long does a robots.txt change take to work?

Google generally caches robots.txt for up to 24 hours. After an urgent fix you can request a recrawl in the Search Console robots.txt report.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading