NeovaAdvanced SEO Platform

robots.txt Has Invalid Syntax or Unknown Directives

In Neova Runs from the crawl itself: nothing to set up.

Rule last reviewed October 17, 2026

Every non-empty, non-comment line in robots.txt should be a field, a colon and a value, with allow and disallow lines inside a group that starts with user-agent.

How Neova Finds It

Neova requests every address and reads what the server answers.

Why It Matters

A dropped line can open private paths or keep a needed path blocked for Googlebot, Bingbot, GPTBot, ClaudeBot or PerplexityBot. AI vendors rely on robots.txt for search and training opt-outs (Anthropic says opting out requires robots.txt), so a broken group can opt a site in or out by accident.

The Ticket It Writes

IN-108 · robots.txt Has Invalid Syntax or Unknown Directives

Expected Result: All in-scope pages pass 'robots.txt has invalid syntax or unknown directives'.

How to Check the Fix: 1. Run the check again in Neova
2. Spot-check sample URLs across affected templates
3. Record the re-check date and period label next to the baseline
4. Known limits of this check: What counts as unknown depends on the reader: Google supports only user-agent, allow, disallow and sitemap. Lighthouse 13.5.0 safelists crawl-delay, clean-param, host, request-rate, visit-time, noindex and content-signal, yet fails the ARD 'Agentmap:' line that its own agentic audit reads (TE-041). Google ignores content past 500 KiB. This check covers syntax, not whether the rules are the right ones.
5. Sources: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec | https://www.rfc-editor.org/rfc/rfc9309.html | https://developer.chrome.com/docs/lighthouse/seo/invalid-robots-txt

Neova writes this ticket for every affected page group and sends it to Jira, Linear, Asana and the rest, after you approve it.

The full guide for this check is being written.

Written by Neova Labs · Back to the Check Library