robots.txt Has Invalid Syntax or Unknown Directives
In Neova Runs from the crawl itself: nothing to set up.
Rule last reviewed October 17, 2026
Every non-empty, non-comment line in robots.txt should be a field, a colon and a value, with allow and disallow lines inside a group that starts with user-agent.
How Neova Finds It
Neova requests every address and reads what the server answers.
Why It Matters
A dropped line can open private paths or keep a needed path blocked for Googlebot, Bingbot, GPTBot, ClaudeBot or PerplexityBot. AI vendors rely on robots.txt for search and training opt-outs (Anthropic says opting out requires robots.txt), so a broken group can opt a site in or out by accident.
The Ticket It Writes
IN-108 · robots.txt Has Invalid Syntax or Unknown Directives
Expected Result: All in-scope pages pass 'robots.txt has invalid syntax or unknown directives'.
How to Check the Fix: 1. Run the check again in Neova 2. Spot-check sample URLs across affected templates 3. Record the re-check date and period label next to the baseline 4. Known limits of this check: What counts as unknown depends on the reader: Google supports only user-agent, allow, disallow and sitemap. Lighthouse 13.5.0 safelists crawl-delay, clean-param, host, request-rate, visit-time, noindex and content-signal, yet fails the ARD 'Agentmap:' line that its own agentic audit reads (TE-041). Google ignores content past 500 KiB. This check covers syntax, not whether the rules are the right ones. 5. Sources: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec | https://www.rfc-editor.org/rfc/rfc9309.html | https://developer.chrome.com/docs/lighthouse/seo/invalid-robots-txt
Neova writes this ticket for every affected page group and sends it to Jira, Linear, Asana and the rest, after you approve it.