AI SEO Crawler: Where AI Helps an Audit, and How to Audit for AI Search
"AI SEO crawler" means two things: a crawler that uses AI, and a crawler that checks how your site looks to AI search. A useful one does both, and keeps the findings that come from a model apart from the ones that come from rules.
Two Meanings of AI SEO Crawler
When people search for an SEO crawler with AI, they want one of two things.
- A crawler that uses AI in the audit. A language model reads pages and judges what rules cannot: whether a page answers its heading, whether a claim has anything behind it.
- A crawler that audits for AI search. It checks what GPTBot, ClaudeBot, PerplexityBot and other AI crawlers may fetch, what your server actually hands them, and whether AI answers cite your pages.
The first is about how the audit is done. The second is about what it covers. A tool can be good at one and weak at the other.
Where AI Helps in an SEO Audit, and Where It Does Not
Most of a technical audit is counting and comparing. Rules do that exactly, the same way every time. A model adds nothing there and can get a number wrong. AI earns its place on questions that need reading.
| Task | Rules | Model |
|---|---|---|
| Status codes and redirects | ✓ | – |
| Canonicals and hreflang | ✓ | – |
| Title and description lengths | ✓ | – |
| robots.txt per crawler | ✓ | – |
| Text answers its heading | – | ✓ |
| Claims with no figures behind them | – | ✓ |
| Search intent and main entities | – | ✓ |
| Meta description drafts | – | ✓ |
Two things to check in any SEO audit tool with AI. Are the model's findings kept apart from the rule-based ones, so a guess never inflates your error count? And where does your page text go: to a cloud model, or to one on your own computer?
The AI Crawlers and Their robots.txt Tokens
Each AI company runs more than one agent, and each one obeys its own robots.txt group. Blocking the training crawler does not block the search one.
| robots.txt token | Company | Job |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | Search |
| ChatGPT-User | OpenAI | Fetch for a user |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Search |
| Claude-User | Anthropic | Fetch for a user |
| PerplexityBot | Perplexity | Search |
| Perplexity-User | Perplexity | Fetch for a user |
| Google-Extended | Token only, no crawler |
Sources: OpenAI's crawler page, Perplexity's bot page and Google's list of its crawlers, which says Google-Extended has no user agent of its own: it is a robots.txt token that Google's existing crawlers read. SEO ProCheck keeps a longer list in AI bot user agents explained.
What AI Search Visibility Depends On
AI assistants find pages through their own crawlers, user-triggered fetchers, and search indexes such as Bing's and Google's. Your site can trip at each step.
| Step | What goes wrong | Neova check |
|---|---|---|
| robots.txt | A broad rule blocks AI crawlers | AI Crawler Rule in robots.txt Differs From the Written Policy |
| Firewall and CDN | 403 or a challenge page | Request With an AI Search Crawler User Agent Is Blocked |
| Rendering | Text needs JavaScript | Most Main Text Appears Only in Rendered HTML |
| Page order | Text starts after 5,000 characters | Main Content Starts After the First 5,000 Characters of Body Text |
| Snippet controls | nosnippet on key pages | Key HTML Page Restricts Snippets or Blocks AI Search Crawlers |
| Cited URLs | Cited page returns 404 | AI Assistants Cite a URL That Returns 404 or 410 |
We found the robots.txt kind on our own site. During our SEO pass on neova.app on November 20, 2026, robots.txt blocked Cloudflare's email script under /cdn-cgi/ on 1,320 pages until we allowed it, and the favicon files sat behind Disallow: /, so Google showed a plain letter instead of our logo. Broad rules block more than you meant. SEO ProCheck measured AI crawler access on 400 sites if you want to see how common this is.
Google explains how its AI features use your content in AI features and your website.
How to Check What AI Crawlers Can See
- Read robots.txt per crawler. A crawler with its own group ignores the rules of the * group.
- Request key pages with each AI user agent. Compare the status code and response size with what a browser gets. A 403 or a tiny response means something in front of the site blocks it.
- Compare raw and rendered HTML. If the main text, links or title only appear after JavaScript, fix that in the server HTML.
- Check llms.txt if you publish one. It is optional; a broken one is worse than none. See llms.txt Is Malformed or Returns a Server Error and SEO ProCheck on whether AI actually uses llms.txt.
- Read the server log. Verified AI crawler visits per URL show what they fetch, and the 403 and 429 answers they get. Requests that only claim a crawler name are flagged by Unverified AI Crawler Requests Returned 2xx.
Measuring AI Search Visibility
Access is the part you control. Being cited is the outcome, and it takes outside data to see.
| Source | Shows | In Neova |
|---|---|---|
| Bing Webmaster Tools | Citations in Bing AI answers | ✓ |
| Google Analytics 4 | Sessions from AI assistants | ✓ |
| Microsoft Clarity | Cited pages, AI bot requests | ✓ |
| Prompt testing | Brand in ChatGPT, Perplexity | Your DataForSEO account |
| AI Overviews | Shown, and cited or not | Your DataForSEO account |
| Server log | Verified AI crawler visits | ✓ |
Treat these as trends, not exact counts. Answers vary from one run to the next. The Clarity page shows how that connection works.
How Neova Uses AI, and Audits for AI Search
Neova keeps the two meanings apart.
| Feature | Runs on | Page text leaves the computer |
|---|---|---|
| AI Review, 44 checks | Ollama, your computer | – |
| Ask | Ollama, your computer | – |
| AI Prompts | Ollama or your own key | Only to the provider you chose |
| MCP server | Your computer | – |
- AI Review flags stay out of the Must Fix and Review counts.
- AI Prompts runs your own prompts per page with Ollama or your Anthropic, OpenAI or Google Gemini key.
- Ask picks its answer from what the window already shows, so it does not make up a number.
- The AI Search category has 45 checks on what AI crawlers may fetch, what each assistant is served, llms.txt and agent files.
- What each assistant is served: sampled pages asked as 10 AI agents, or a whole crawl as GPTBot, ClaudeBot, PerplexityBot and others.
AI Review runs in the desktop app on the Desktop plan and up; see pricing.
AI Search Crawl Checklist
- robots.txt rules for each AI crawler match what your company decided.
- Key pages answer 200 to AI crawler user agents, not 403 or a challenge.
- User-triggered fetchers are not blocked at the firewall or CDN.
- Main text, title and links are in the server HTML.
- Main content starts within the first 5,000 characters of body text.
- No accidental nosnippet on pages you want quoted.
- llms.txt, if published, is valid.
- Pages AI answers cite are not broken or redirected.
Questions
What is an AI SEO crawler?
Either a crawler that uses AI to judge pages, or one that checks how a site looks to AI search: what AI crawlers can fetch, what they are served and whether AI answers cite it. A good one does both and labels which findings come from a model.
Do AI crawlers render JavaScript?
Many are not known to. Content that only appears after JavaScript runs may never reach them, so keep main text, titles and links in the server HTML.
Should I block GPTBot and other AI crawlers?
That is a business decision, not an SEO default. Training and search crawlers have different tokens, so you can allow one and block the other. Whatever you decide, check that robots.txt and the firewall say the same thing.
Does blocking Google-Extended remove me from Google Search?
No. Google says Google-Extended is a robots.txt token with no crawler of its own, and it does not affect inclusion in Google Search.
Is llms.txt required?
No. It is optional, and Neova does not flag a site that has none. If you publish one, keep it valid and current.
Does Neova send my pages to an AI provider?
Not for AI Review or Ask, which use a model on your computer through Ollama. AI Prompts can use your own Anthropic, OpenAI or Gemini key, and Neova says when page text goes to the provider you chose.