robots.txt Tester: How to Test robots.txt Rules Before They Block the Wrong Pages
robots.txt is a short text file with a lot of power. One line can keep a search engine away from a folder you meant to hide, or from the whole site you meant to rank. A robots.txt tester tells you, URL by URL, what each crawler is allowed to fetch, so you find out before Googlebot does.
What a robots.txt Tester Does
A robots.txt tester takes a robots.txt file and a URL, picks the group of rules that applies to a given user agent, and tells you whether that URL is allowed or disallowed, and which line decided it. A good one also points out lines the parser cannot read, fields it ignores and rules that cancel each other out.
Testing one URL at a time is useful for a quick question. Testing every URL of a crawl is what catches the surprises: a rule written for one folder that also matches a product filter, or a wildcard that blocks the CSS your pages need to render.
Google Retired Its robots.txt Tester
For years, Search Console had a robots.txt Tester where you could edit the file and test URLs against it. In late 2023 Google retired it and replaced it with the robots.txt report. The report shows which robots.txt files Google found for the top 20 hosts of a property, when each was last crawled, and any warnings or errors. It also lets you request a recrawl of the file after an urgent fix.
The report tells you how Google read your live file. It does not test a rule that is not live yet, and it does not run your whole URL list against the file. For that you need a separate tester. Google publishes its own open source robots.txt parser, and Bing Webmaster Tools has a robots.txt tester that is handy for checking how Bingbot reads a file.
How robots.txt Rules Work
The file lives at the root of a host, for example https://www.example.com/robots.txt. The rules are defined by RFC 9309, the Robots Exclusion Protocol, and Google documents how it reads them in How Google interprets the robots.txt specification. A file is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules:
User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Allow: /cart/help
User-agent: Googlebot-Image
Disallow: /private-images/
Sitemap: https://www.example.com/sitemap.xml
The points testers check, as Google documents them. SEO ProCheck's complete robots.txt reference goes through every field.
- One group per crawler. A crawler follows only the most specific group that matches its name. If Googlebot has its own group, it ignores the
*group completely. - Longest match wins. When an Allow and a Disallow both match a URL, the rule with the longer path applies. On a tie, Google uses the least restrictive rule.
- Two wildcards.
*matches any run of characters and$marks the end of the URL. - Paths are case-sensitive.
Disallow: /Admindoes not block/admin. - One file per host. Rules apply only to the protocol, host and port the file is served from. A subdomain needs its own file.
What robots.txt Does Not Do
robots.txt controls crawling, not indexing. Google's introduction to robots.txt is clear that a disallowed URL can still be indexed if other pages link to it, usually shown without a description. If you want a page out of search results, let it be crawled and use noindex instead. A page Google cannot fetch is a page whose noindex it never sees.
A few other limits worth knowing:
- No noindex line. Google stopped supporting noindex in robots.txt in 2019, together with nofollow and crawl-delay rules there.
- No crawl-delay for Google. Googlebot ignores
Crawl-delay. Some other crawlers read it, which is why a value set for every user agent can slow them down a lot. - No security. The file is public and polite crawlers follow it by choice. Anything private needs a login.
Common robots.txt Mistakes a Tester Catches
| Mistake | Effect | Fix |
|---|---|---|
Disallow: / left from staging | Whole site blocked | Remove the line |
| Named group without shared rules | * rules dropped for that bot | Copy shared rules in |
| Blocked CSS, JavaScript, images | Page cannot render | Allow the files |
| Wildcard too broad | /*? blocks filters, pages | Narrow the pattern |
| Rule before User-agent, typo | Line skipped | Fix the syntax |
| Disallow to deindex | noindex never read | Allow, then noindex |
| www and non-www differ | Rules differ by host | Serve one file |
Blocking a URL you also list in your XML Sitemap sends mixed signals. The XML sitemap checker guide covers that side.
Server Errors and File Size
How the file is served matters as much as what it says. Google documents each case:
| Case | What Google does | How long |
|---|---|---|
| 4xx, not 429 | Crawls as if no file | Until fixed |
| 5xx or 429 | Stops crawling the site | First 12 hours |
| Still failing | Uses last good copy | Up to 30 days |
| File over 500 KiB | Ignores the rest | Past 500 KiB |
| Any change | Uses its cached copy | Up to 24 hours |
The recrawl request in the robots.txt report is for urgent fixes. A robots.txt that times out at the wrong moment can pause crawling of the whole site; the crawl budget guide explains why that matters most on large sites.
What We Found in Our Own robots.txt
Our own SEO pass on neova.app, on November 20, 2026, found two rules that blocked more than we meant:
| What was blocked | Pages hit | What Google saw | Fix |
|---|---|---|---|
| Cloudflare email script under /cdn-cgi/ | 1,320 | A script it could not fetch | Allowed that script |
| /favicon and /apple-touch-icon.png | Every result | A plain letter, no logo | Allowed both files |
The favicon files sat behind a Disallow: / rule. Google's favicon guidelines need the file crawlable by Googlebot-Image, so search results showed a letter instead of our logo next to neova.app. Neither rule looked wrong in the file. Both showed up only when the rules were tested against every URL and file the pages load.
How to Test robots.txt Safely
- Crawl the site so you have a full list of real URLs, not just the few you remember.
- Test the live file against that list for each crawler you care about: Googlebot, Googlebot-Image, Bingbot and the AI crawlers.
- Write the change as a draft and test the draft against the same list. Look at the URLs that become newly blocked and newly allowed.
- Check the resources. Make sure no indexable page loses its scripts, style sheets or images.
- Ship it, then confirm in the Search Console robots.txt report that Google fetched the new version without errors.
- Crawl again and compare, so an accidental change later on gets caught.
If you are deciding what AI crawlers may fetch, the AI crawlers guide lists their names and what each one does. Blocked pages also show up in the indexability checker guide.
Test robots.txt With Neova
Neova's robots.txt Tester explains every line of the live file, or of a draft, in plain words, as Google documents it and RFC 9309 defines it, with the crawled URLs each rule matches. Its All Crawlers view shows what Googlebot, Bingbot and the AI crawlers may fetch, with the URLs each one is blocked from. It says when a crawler's own group drops the rules of the * group, and the draft test lists every URL of the crawl that a change would newly block or newly allow. You can then crawl while respecting the draft to see the full effect. The checks behind it:
- robots.txt Has Invalid Syntax or Unknown Fields
- robots.txt Is Unreachable or Too Large, for 5xx, 429, no response or over 500 KiB
- Internal URL Is Disallowed by robots.txt
- HTML Page Uses a JavaScript File Disallowed by robots.txt, Style Sheet and Image
- Crawl-Delay Is Set for All User Agents in robots.txt
- Host Variant robots.txt Differs From the Canonical Host
- robots.txt Rules Changed Since the Previous Crawl
- Favicon Is Missing, Broken or Blocked, including a favicon disallowed for Googlebot-Image
- Disallowed URL Is Listed in an XML Sitemap
Neova keeps every robots.txt version its crawls read, so you can see when a rule changed. With the User Agent setting you can crawl as Googlebot, Bingbot, an AI crawler or a custom string. See Features or download Neova.
Questions
Does Google still have a robots.txt tester?
No. Google retired the robots.txt Tester in Search Console in 2023 and replaced it with the robots.txt report, which shows how Google read your live file and any errors. To test a change before it is live, use a separate robots.txt tester.
How do I test if a URL is blocked by robots.txt?
Find the group that matches the crawler's user agent, then the longest Allow or Disallow path that matches the URL. That rule decides. A robots.txt tester does this for you and names the line, for one URL or for every URL of a crawl. SEO ProCheck's robots.txt FAQ answers the edge cases.
Does robots.txt stop a page from being indexed?
No. It stops crawling. A disallowed URL can still be indexed if other pages link to it. Use noindex on a page Google can crawl if you want it out of search results.
Why does Googlebot ignore my User-agent: * rules?
Because the file has a group named for Googlebot. A crawler follows only the most specific group that matches it, so copy the shared rules into the Googlebot group as well.
How long does a robots.txt change take to work?
Google generally caches robots.txt for up to 24 hours. After an urgent fix you can request a recrawl in the Search Console robots.txt report.