NeovaAdvanced SEO Platform

Crawl-Delay in robots.txt Used on Purpose

In Neova Runs from the crawl itself: nothing to set up.

Rule last reviewed October 17, 2026

The rule, the method and the ticket for this check are kept for Neova users and not published here.

Why It Matters

Bing and Anthropic honor the delay, so a large value slows Bing and Copilot index refresh and Claude's crawling, while Google ignores it and gets no protection from it. Refusing AI crawlers outright to fix load removes the site from their search, when a delay or a short 429 would do.

Written by Neova Labs · Back to the Check Library