NeovaAdvanced SEO Platform

AI Crawlers: What GPTBot, ClaudeBot and PerplexityBot Do, and How to Control Them

AI crawlers fetch your pages to train language models, to build AI search indexes, or to answer one person's question on the spot. Each kind has its own name in robots.txt, so you can allow some and block others. Here is who they are, what each one does, and how to check what they actually get.

AI Searchai crawlersUpdated November 21, 2026By the Neova team

What Are AI Crawlers?

AI crawlers, also called LLM crawlers, are bots run by AI companies. They are not all the same, and treating them as one group leads to bad robots.txt rules. There are three kinds:

  • Training crawlers collect pages that may be used to train future models. GPTBot and ClaudeBot are the best known.
  • Search crawlers build the index an AI assistant searches when it answers with sources and links. OAI-SearchBot, Claude-SearchBot and PerplexityBot are in this group.
  • User-triggered fetchers load a page because a person asked the assistant about it, for example by pasting a URL. ChatGPT-User, Claude-User and Perplexity-User work this way.

Blocking a training crawler keeps your pages out of future training sets. Blocking a search crawler can keep you out of AI answers with citations, which is often the opposite of what a business wants.

The Main AI Crawlers and Their robots.txt Tokens

Each vendor documents its own bots. These are the names to use in robots.txt, with the vendor's own page linked:

TokenCompanyKindOwn bot
GPTBotOpenAITraining✓
OAI-SearchBotOpenAIAI search✓
ChatGPT-UserOpenAIUser request✓
ClaudeBotAnthropicTraining✓
Claude-SearchBotAnthropicAI search✓
Claude-UserAnthropicUser request✓
PerplexityBotPerplexityAI search✓
Perplexity-UserPerplexityUser request✓
Google-ExtendedGoogleGemini use of content–
Applebot-ExtendedAppleApple model training–
CCBotCommon CrawlOpen web archive✓

Google-Extended and Applebot-Extended are tokens, not separate bots: they control how content Googlebot or Applebot already crawls may be used. Google says Google-Extended does not affect inclusion in Google Search or rankings. To limit how your content appears in Google's own AI features in Search, Google points to snippet controls such as nosnippet. For a longer list with every bot and its purpose, see the AI crawler map on SEO ProCheck.

Should You Block LLM Crawlers?

There is no single right answer. It depends on what your content is for. In robots.txt terms, three policies cover most sites:

PolicyTrainingAI searchUser request
Open to all✓✓✓
Cited, not trained on–✓✓
Closed to AI––◐
  • Allow search crawlers if you want to be cited and linked in ChatGPT search, Claude and Perplexity. For most businesses, this is visibility you want.
  • Consider blocking training crawlers if your content is your product: news, research, courses, paid data. Blocking stops future collection; it does not remove what was already gathered.
  • Think twice about blocking user-triggered fetchers. They act for a person who asked about your page, and some vendors say they may not follow robots.txt the way crawlers do, hence the partly above.

Write the decision down: which bots are allowed, which are blocked, and why. Then make robots.txt match it. A written policy is what stops a future developer from blocking everything during a redesign.

robots.txt for AI Bots: Examples

Here is a common setup: allow AI search, block training.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /search/
Disallow: /cart/

User-agent: *
Disallow: /search/
Disallow: /cart/

Several User-agent lines can share one group of rules. Watch one trap closely. Under the robots.txt standard, RFC 9309, a crawler follows only the most specific group that names it. If you add a group for GPTBot, GPTBot ignores the * group entirely, including your Disallow: /search/. That is why the search group above repeats the two shared rules. Anything a named group does not disallow is allowed.

To block a single folder from training rather than the whole site:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /research/
Disallow: /search/

For user-triggered fetchers, read each vendor's page before relying on robots.txt.

Test every rule against real URLs before it ships. We found two rules on neova.app, November 20, 2026, that blocked more than intended:

Rule hitEffectReach
/cdn-cgi/ blockedCloudflare email script blocked1,320 pages
Disallow: / over faviconsPlain letter instead of logo in GoogleEvery search result

Both were fixed by allowing the paths. A Disallow written for one bot or one folder reaches every file under it, scripts and icons included.

Beyond robots.txt: Firewalls, CDNs and Verification

robots.txt is a request, not a lock. The AI companies above document how their crawlers treat it, but other bots may ignore it.

  • Check the firewall too. Some CDNs and WAFs have settings that block AI crawlers, sometimes by default. A site can allow GPTBot in robots.txt and still answer it with a 403. Test what each crawler actually receives.
  • Verify before you trust. OpenAI and Perplexity publish IP lists for their bots, and Google and Microsoft for theirs. A request that names a bot from an address off its company's list is not that bot.
  • Block bad actors at the edge. For scrapers that ignore robots.txt, rules in your CDN or server are the only thing that works.

How often this goes wrong in practice: SEO ProCheck tested what AI crawlers actually receive on 400 sites in Can AI Crawlers Actually Read Your Site? 400 Sites Measured.

Make Your Pages Readable to AI Crawlers

Allowing a crawler is half of AI crawler SEO. The other half is making sure it can read what it fetches.

  • Put the main content in the HTML. Not every crawler runs JavaScript. If your text only appears after scripts run, assume some AI crawlers never see it.
  • Get to the point early. Long menus and boilerplate before the main text push the answer further down the page.
  • Do not hide key facts in images, PDFs or widgets. Prices, specs and contact details belong in plain HTML text.
  • Return real status codes. A broken or redirected page that AI answers cite sends their readers to an error.
  • Keep the HTML light. Googlebot reads the first 2 MB of an HTML file; Neova flags pages over that in HTML Is Over 2 MB or Core Web Vitals Miss the Good Range.
  • llms.txt is optional. It is a proposed format, not a web standard. If you publish one, keep it valid and current.

How to Check What AI Crawlers See

Neova shows what each AI crawler is allowed and what it is actually served, in the desktop app and in Neova Online:

  • What Each AI Assistant Is Served asks sampled pages as 10 AI agents, including ChatGPT, Claude and Perplexity and their crawlers, Meta AI, Google-Extended and Common Crawl, and shows who gets full access and what robots.txt allows.
  • Crawl as an AI crawler. Run a whole crawl as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or another AI user agent.
  • The robots.txt Tester shows what Googlebot, Bingbot and the AI crawlers may fetch, with the URLs each is blocked from, and says when a crawler's own group drops the rules of the * group.
  • The Rendering screen shows how many pages need JavaScript and what each AI crawler is served.
  • Server Log, in the desktop app, shows verified AI crawler visits and refusals per URL.
  • AI citations from Bing Webmaster Tools and from Microsoft Clarity's AI Visibility, when you connect them. See Microsoft Clarity.

Checks to start with: a request with an AI search crawler user agent is blocked, an AI crawler rule in robots.txt differs from the written policy, a user-triggered AI fetcher is blocked, a key page restricts snippets or blocks AI search crawlers and AI bots requesting URLs that answer an error. The Free plan includes the AI Search views; see Pricing or download Neova.

AI Crawler SEO Checklist

ItemDone when
Written policyEach AI bot: allow or block
robots.txtMatches the policy
Named groupsShared rules repeated
AI search crawlersAllowed on pages to cite
Firewall and CDNSame as the policy
What each crawler gets200 and the full page
Main contentIn the HTML, early
Key factsIn text, not images
LogsVerified visits, fakes counted apart
Cited pagesAnswer 200

Questions

What is GPTBot?

GPTBot is OpenAI's crawler for collecting content that may be used to train its models. It is separate from OAI-SearchBot, which finds pages for ChatGPT search. See OpenAI's crawler page.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses separate bots, and each one reads its own robots.txt rules. ChatGPT search relies on OAI-SearchBot, so you can block GPTBot and still allow OAI-SearchBot.

Does Google-Extended affect my Google rankings?

Google says it does not affect inclusion in Google Search and is not a ranking signal. It controls whether your content may be used for Gemini models.

How do I block LLM crawlers?

Add a group to robots.txt naming each crawler, such as GPTBot, ClaudeBot and CCBot, with Disallow: /. For bots that ignore robots.txt, add a rule in your firewall or CDN.

How do I know if AI crawlers visit my site?

Read your server or CDN logs, and verify each request against the IP list its company publishes. Neova's Server Log tool does this per URL. See Import Instead of Connecting for the log formats.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading