Robots.txt
Questo controllo pesa il 9% del punteggio.
How to configure robots.txt for AI crawlers: the missing file, unconfigured crawlers, wildcard blocks, and Sitemap directives.
/robots.txt not found
Your site does not have a /robots.txt file at the root. Without it, AI crawlers have no explicit instructions on what they can or cannot access. While most crawlers will still index your site, having an explicit robots.txt signals intent and gives you fine-grained control.
Create a robots.txt file in your site root (usually /public/robots.txt in Next.js or the web root directory) with User-agent entries for the AI crawlers you want to allow:
# Allow all AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / # Traditional crawlers User-agent: * Allow: / Sitemap: https://your-site.com/sitemap.xml
npx ax-init --from https://your-site.com to auto-generate a robots.txt with all 29+ AI crawler entries pre-configured.Missing core AI crawlers
ax-audit detected that your robots.txt exists but is missing explicit User-agent entries for some of the 6 core AI crawlers. The core crawlers are:
GPTBot: OpenAI's crawler for ChatGPT and pluginsClaudeBot: Anthropic's crawler for ClaudeGoogle-Extended: Google's AI-specific crawlerPerplexityBot: Perplexity AI's search crawlerBytespider: ByteDance's crawlerCCBot: Common Crawl's crawler (used by many AI training datasets)
Add explicit entries for each missing crawler. Each entry needs its own User-agent line followed by Allow: /:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: /
No core AI crawlers configured
Your robots.txt exists but has zero explicit entries for any of the 6 core AI crawlers. This means AI agents rely entirely on the wildcard User-agent: * rule, which may or may not permit access.
Add User-agent blocks for all 6 core crawlers listed in the section above. Being explicit about AI crawler access is a best practice: it removes ambiguity and ensures your site is discoverable.
Blocked by wildcard rule
Your robots.txt has a User-agent: * / Disallow: / rule that blocks all crawlers, including AI crawlers that don't have their own explicit entry.
If you want to keep the wildcard block for traditional crawlers but allow AI agents, add explicit entries for each AI crawler before the wildcard rule:
# AI crawlers - explicitly allowed User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / # ... other AI crawlers ... # Block everything else User-agent: * Disallow: /
User-agent: GPTBot section first. Only if no specific match exists does it fall back to User-agent: *. So explicit entries always take precedence.Explicitly blocked crawlers
One or more AI crawlers have explicit Disallow: / rules in your robots.txt. If this is intentional (e.g., you don't want certain AI companies indexing your content), you can ignore this warning.
If you want to allow these crawlers, change their Disallow: / to Allow: /:
# Before (blocked) User-agent: GPTBot Disallow: / # After (allowed) User-agent: GPTBot Allow: /
Partial path restrictions
Some AI crawlers have Disallow rules on specific paths rather than a full block. For example:
User-agent: GPTBot Disallow: /private/ Disallow: /api/internal/
This is often intentional; you may want AI crawlers to access most of your site but not private sections. ax-audit flags this as a warning so you can verify the restrictions match your intent. For maximum AX score, use only Allow: / for AI crawlers and handle path restrictions through the wildcard rule.
Missing Sitemap directive
Your robots.txt does not include a Sitemap: directive. While AI crawlers can discover your sitemap through other means, including it in robots.txt is a widely-supported standard that helps all crawlers find your site map immediately.
Add a Sitemap directive at the end of your robots.txt:
Sitemap: https://your-site.com/sitemap.xml
Use an absolute URL (including https://). If you have multiple sitemaps, add one line per sitemap.
Low AI crawler coverage
ax-audit tracks 29+ known AI crawlers. Your robots.txt has explicit rules for fewer than 10 of them. While the 6 core crawlers are most important, configuring additional crawlers improves discoverability across more AI platforms.
Run npx ax-init to generate a robots.txt that includes all known AI crawler entries. You can also view the full list in the ax-audit source code.
No Content-Signal directive
The Content Signals Policy (CC0, launched by Cloudflare) extends robots.txt with a machine-readable way to say what crawlers may do with your content after they access it, something Allow/Disallow cannot express. Three signals are defined: search (search indexing and excerpts), ai-input (RAG / grounding / live answers), and ai-train (training or fine-tuning models). Omitting a signal is neutral: it neither grants nor restricts.
User-agent: * Content-Signal: search=yes, ai-train=no Allow: /
Cloudflare already serves signals on 3.8M+ managed domains, so crawler operators increasingly parse them. Generate your own policy text at contentsignals.org. This finding is informational and does not affect your score.
Malformed or misplaced Content-Signal
The directive value must be comma-delimited signal=yes|no pairs. Values other than yes/no (e.g. search=maybe) or bare names without a value are ignored by crawlers.
Placement matters too: Content-Signal belongs inside a User-agent group, after the User-agent line it applies to. A directive floating before any group has no defined meaning.
# Wrong - outside a group, invalid value Content-Signal: search=maybe # Right User-agent: * Content-Signal: search=yes, ai-input=no, ai-train=no Allow: /
Unknown content signal name
Your directive contains a name outside the policy vocabulary. Only three signals are defined: search, ai-input, and ai-train. Anything else (e.g. ai-all=no or a typo like ai_train) is silently ignored by crawlers, which may give you a false sense of protection.
If you need finer-grained licensing semantics than the three signals offer (per-path terms, attribution, or payment), look at Really Simple Licensing (RSL).
Assistant search crawlers are blocked
Your robots.txt blocks crawlers whose job is to build the index an assistant cites from. The consequence is specific: those assistants can no longer name your site in an answer, because they no longer have it.
Almost every operator that crawls for AI runs more than one crawler, and they do different things:
- Training:
GPTBot,ClaudeBot,Google-Extended. Collects text for model training. Blocking it is a policy choice and costs no visibility. - Search:
OAI-SearchBot,Claude-SearchBot,PerplexityBot. Builds the retrieval index. Blocking it removes you from the answers. - User fetch:
ChatGPT-User,Claude-User. Fetches one page because a person asked for that URL.
The block is almost always collateral damage from a copy-pasted "block AI" snippet, or from a wildcard rule. If the intent was to opt out of training, name the training tokens and let the search crawlers through.
# Opt out of training, stay citable User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: /
Blocked fetchers that may ignore robots.txt
You block one or more user-triggered fetchers whose vendors document that robots.txt may not apply to them. These clients fetch a page because a person pasted that URL and asked about it. The vendor's position is that this is the person's request, not a crawl.
Whether you agree or not, the practical consequence is the same: the rule is a request, not a control. If the block has to hold, it has to hold at the edge.
robots.txt line. Verified fetchers publish IP ranges, and Web Bot Auth signs requests with a key you can check: those are the mechanisms that actually decide.Rules for retired or non-existent crawlers
Some of the user agents named in your file no longer exist, and some never did. Renamed products (anthropic-ai, Claude-Web), discontinued crawlers, and plausible-looking strings that circulate in copy-pasted "block AI" lists but were never a real user agent.
These rules do nothing. That is worth fixing not because the dead lines cost anything, but because a file half-full of them stops being a statement of your policy and starts being an archaeological record: nobody can tell which lines are load-bearing.
Delete them, then check that the crawler you meant to block is actually named. A rule against anthropic-ai was probably meant for ClaudeBot, and has never applied to it.
Malformed Content-Usage rule
A Content-Usage line could not be parsed, or sits outside any User-agent group. AIPREF scopes these rules to the group they appear in, so a line before the first User-agent applies to nobody.
The syntax is an optional path pattern, then a Structured Fields dictionary:
User-agent: * Content-Usage: train-ai=n User-agent: * Content-Usage: /docs/ train-ai=n, search=y
draft-ietf-aipref-attach) is pre-last-call, so this and every other Content-Usage finding is informational and never costs you points. A malformed rule is worth fixing anyway: it reads as a policy and enforces nothing.Content-Usage uses the wrong vocabulary
Two standards ask the same question and spell the answer differently, and your file mixes them. AIPREF writes train-ai with y / n; Content Signals and RSL write ai-train with yes / no.
So Content-Usage: ai-train=no looks entirely correct and means nothing: the token is not in the AIPREF vocabulary and the value is not in it either. The same line is valid on the Content-Signal header, which is where it was probably meant to go.
User-agent: * Content-Signal: ai-train=no, search=yes Content-Usage: train-ai=n, search=y
AIPREF vocabulary v07 defines train-ai and search, and nothing else; extending it takes a standards-track RFC, so an invented token is silently ignored. An absent token means unknown, never allowed, so if you have a preference, state it.