# Robots.txt > How to configure robots.txt for AI crawlers: the missing file, unconfigured crawlers, wildcard blocks, and Sitemap directives. Esta comprobación representa el 9% de la puntuación. *Estas guías de corrección están escritas en inglés.* ## /robots.txt not found Your site does not have a `/robots.txt` file at the root. Without it, AI crawlers have no explicit instructions on what they can or cannot access. While most crawlers will still index your site, having an explicit robots.txt signals intent and gives you fine-grained control. Create a `robots.txt` file in your site root (usually `/public/robots.txt` in Next.js or the web root directory) with User-agent entries for the AI crawlers you want to allow: ``` # Allow all AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / # Traditional crawlers User-agent: * Allow: / Sitemap: https://your-site.com/sitemap.xml ``` > **Quick fix with ax-init** > > Run > > npx ax-init --from https://your-site.com > > to auto-generate a robots.txt with all 29+ AI crawler entries pre-configured. --- ## Missing core AI crawlers ax-audit detected that your robots.txt exists but is missing explicit `User-agent` entries for some of the 6 core AI crawlers. The core crawlers are: - `GPTBot`: OpenAI's crawler for ChatGPT and plugins - `ClaudeBot`: Anthropic's crawler for Claude - `Google-Extended`: Google's AI-specific crawler - `PerplexityBot`: Perplexity AI's search crawler - `Bytespider`: ByteDance's crawler - `CCBot`: Common Crawl's crawler (used by many AI training datasets) Add explicit entries for each missing crawler. Each entry needs its own `User-agent` line followed by `Allow: /`: ``` User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / ``` --- ## No core AI crawlers configured Your robots.txt exists but has zero explicit entries for any of the 6 core AI crawlers. This means AI agents rely entirely on the wildcard `User-agent: *` rule, which may or may not permit access. Add User-agent blocks for all 6 core crawlers listed in the [section above](#missing-crawlers). Being explicit about AI crawler access is a best practice: it removes ambiguity and ensures your site is discoverable. --- ## Blocked by wildcard rule Your robots.txt has a `User-agent: * / Disallow: /` rule that blocks all crawlers, including AI crawlers that don't have their own explicit entry. If you want to keep the wildcard block for traditional crawlers but allow AI agents, add explicit entries for each AI crawler **before** the wildcard rule: ``` # AI crawlers - explicitly allowed User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / # ... other AI crawlers ... # Block everything else User-agent: * Disallow: / ``` > **How robots.txt matching works** > > When a crawler identifies itself as "GPTBot", the server looks for a matching > > User-agent: GPTBot > > section first. Only if no specific match exists does it fall back to > > User-agent: * > > . So explicit entries always take precedence. --- ## Explicitly blocked crawlers One or more AI crawlers have explicit `Disallow: /` rules in your robots.txt. If this is intentional (e.g., you don't want certain AI companies indexing your content), you can ignore this warning. If you want to allow these crawlers, change their `Disallow: /` to `Allow: /`: ``` # Before (blocked) User-agent: GPTBot Disallow: / # After (allowed) User-agent: GPTBot Allow: / ``` --- ## Partial path restrictions Some AI crawlers have `Disallow` rules on specific paths rather than a full block. For example: ``` User-agent: GPTBot Disallow: /private/ Disallow: /api/internal/ ``` This is often intentional; you may want AI crawlers to access most of your site but not private sections. ax-audit flags this as a warning so you can verify the restrictions match your intent. For maximum AX score, use only `Allow: /` for AI crawlers and handle path restrictions through the wildcard rule. --- ## Missing Sitemap directive Your robots.txt does not include a `Sitemap:` directive. While AI crawlers can discover your sitemap through other means, including it in robots.txt is a widely-supported standard that helps all crawlers find your site map immediately. Add a Sitemap directive at the end of your robots.txt: ``` Sitemap: https://your-site.com/sitemap.xml ``` Use an absolute URL (including `https://`). If you have multiple sitemaps, add one line per sitemap. --- ## Low AI crawler coverage ax-audit tracks 29+ known AI crawlers. Your robots.txt has explicit rules for fewer than 10 of them. While the 6 core crawlers are most important, configuring additional crawlers improves discoverability across more AI platforms. Run `npx ax-init` to generate a robots.txt that includes all known AI crawler entries. You can also view the full list in the [ax-audit source code](https://github.com/lucioduran/ax-audit). --- ## No Content-Signal directive The [Content Signals Policy](https://contentsignals.org) (CC0, launched by Cloudflare) extends robots.txt with a machine-readable way to say what crawlers may do with your content *after* they access it, something `Allow`/`Disallow` cannot express. Three signals are defined: `search` (search indexing and excerpts), `ai-input` (RAG / grounding / live answers), and `ai-train` (training or fine-tuning models). Omitting a signal is neutral: it neither grants nor restricts. ``` User-agent: * Content-Signal: search=yes, ai-train=no Allow: / ``` Cloudflare already serves signals on 3.8M+ managed domains, so crawler operators increasingly parse them. Generate your own policy text at [contentsignals.org](https://contentsignals.org). This finding is informational and does not affect your score. > **Caveat** > > Content signals express > > preferences > > , not technical enforcement: non-compliant crawlers can ignore them. In the EU they may carry legal weight as a reservation of rights under Article 4 of Directive 2019/790. Combine them with WAF or bot-management rules if you need actual enforcement. --- ## Malformed or misplaced Content-Signal The directive value must be comma-delimited `signal=yes|no` pairs. Values other than `yes`/`no` (e.g. `search=maybe`) or bare names without a value are ignored by crawlers. Placement matters too: `Content-Signal` belongs *inside* a User-agent group, after the `User-agent` line it applies to. A directive floating before any group has no defined meaning. ``` # Wrong - outside a group, invalid value Content-Signal: search=maybe # Right User-agent: * Content-Signal: search=yes, ai-input=no, ai-train=no Allow: / ``` --- ## Unknown content signal name Your directive contains a name outside the policy vocabulary. Only three signals are defined: `search`, `ai-input`, and `ai-train`. Anything else (e.g. `ai-all=no` or a typo like `ai_train`) is silently ignored by crawlers, which may give you a false sense of protection. If you need finer-grained licensing semantics than the three signals offer (per-path terms, attribution, or payment), look at [Really Simple Licensing (RSL)](https://rslstandard.org). --- ## Assistant search crawlers are blocked Your `robots.txt` blocks crawlers whose job is to build the index an assistant cites from. The consequence is specific: those assistants can no longer name your site in an answer, because they no longer have it. Almost every operator that crawls for AI runs more than one crawler, and they do different things: - **Training**: `GPTBot`, `ClaudeBot`, `Google-Extended`. Collects text for model training. Blocking it is a policy choice and costs no visibility. - **Search**: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`. Builds the retrieval index. Blocking it removes you from the answers. - **User fetch**: `ChatGPT-User`, `Claude-User`. Fetches one page because a person asked for that URL. The block is almost always collateral damage from a copy-pasted "block AI" snippet, or from a wildcard rule. If the intent was to opt out of training, name the training tokens and let the search crawlers through. ``` # Opt out of training, stay citable User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / ``` --- ## Blocked fetchers that may ignore robots.txt You block one or more user-triggered fetchers whose vendors document that `robots.txt` may not apply to them. These clients fetch a page because a person pasted that URL and asked about it. The vendor's position is that this is the person's request, not a crawl. Whether you agree or not, the practical consequence is the same: the rule is a request, not a control. If the block has to hold, it has to hold at the edge. > **Enforce where enforcement happens** > > A user agent string is self-reported and free to change, so a WAF rule on the name is only marginally stronger than the > > > > robots.txt > > line. Verified fetchers publish IP ranges, and Web Bot Auth signs requests with a key you can check: those are the mechanisms that actually decide. --- ## Rules for retired or non-existent crawlers Some of the user agents named in your file no longer exist, and some never did. Renamed products (`anthropic-ai`, `Claude-Web`), discontinued crawlers, and plausible-looking strings that circulate in copy-pasted "block AI" lists but were never a real user agent. These rules do nothing. That is worth fixing not because the dead lines cost anything, but because a file half-full of them stops being a statement of your policy and starts being an archaeological record: nobody can tell which lines are load-bearing. Delete them, then check that the crawler you meant to block is actually named. A rule against `anthropic-ai` was probably meant for `ClaudeBot`, and has never applied to it. --- ## Malformed Content-Usage rule A `Content-Usage` line could not be parsed, or sits outside any `User-agent` group. AIPREF scopes these rules to the group they appear in, so a line before the first `User-agent` applies to nobody. The syntax is an optional path pattern, then a Structured Fields dictionary: ``` User-agent: * Content-Usage: train-ai=n User-agent: * Content-Usage: /docs/ train-ai=n, search=y ``` > **Still a draft** > > AIPREF ( > > draft-ietf-aipref-attach > > ) is pre-last-call, so this and every other > > > > Content-Usage > > finding is informational and never costs you points. A malformed rule is worth fixing anyway: it reads as a policy and enforces nothing. --- ## Content-Usage uses the wrong vocabulary Two standards ask the same question and spell the answer differently, and your file mixes them. AIPREF writes `train-ai` with `y` / `n`; Content Signals and RSL write `ai-train` with `yes` / `no`. So `Content-Usage: ai-train=no` looks entirely correct and means nothing: the token is not in the AIPREF vocabulary and the value is not in it either. The same line is valid on the `Content-Signal` header, which is where it was probably meant to go. ``` User-agent: * Content-Signal: ai-train=no, search=yes Content-Usage: train-ai=n, search=y ``` AIPREF vocabulary v07 defines `train-ai` and `search`, and nothing else; extending it takes a standards-track RFC, so an invented token is silently ignored. An absent token means *unknown*, never *allowed*, so if you have a preference, state it. --- Representación en Markdown de https://axrush.com/guides/robots-txt