Todas las guías

Estas guías de corrección están escritas en inglés.

Agent Access

Esta comprobación informa sin mover la puntuación.

AI crawler user-agents blocked by your WAF despite robots.txt allowing them, or served reduced content. Align firewall rules with your declared crawler policy.

Crawler allowed in robots.txt but blocked

A request carrying this crawler's user-agent token received an error (typically 403) while a regular request succeeds. Your declared policy and your enforcement disagree: agents trust robots.txt, attempt the crawl, and silently fail. Common causes:

  • Cloudflare "Block AI Crawlers" / AI Crawl Control: the one-toggle block rejects AI crawler UAs regardless of your robots.txt. Review it under Security → Bots, and allow the crawlers your robots.txt permits.
  • WAF managed rules: several vendors ship "AI scraper" rule sets matching UA tokens (GPTBot, ClaudeBot, CCBot, …). Audit custom and managed rules for UA-based matches.
  • Rate limiting / bot scoring: aggressive bot scores can challenge or drop crawler traffic. Check your bot-management logs for the specific token.

Reproduce manually:

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0)" https://your-site.com
# Compare with:
curl -sI https://your-site.com

If you intend to block a crawler, declare it: add an explicit Disallow: / for it in robots.txt. ax-audit treats a server block that matches a robots.txt Disallow as consistent and does not penalize it.

Verified-bots caveat
ax-audit's probe is not the real crawler. If your WAF verifies bots cryptographically (Web Bot Auth) or by IP range, the genuine GPTBot may pass while this unverified probe is rejected. Before changing rules, confirm against your WAF logs whether the real crawler traffic is being served.

Crawler receives reduced content

The server returned 200 to the crawler UA, but with less than half the visible text of the baseline response. The crawler is getting an interstitial, a JS challenge page ("Checking your browser…"), or conditionally rendered content, and that is what gets indexed.

  • Check whether your CDN serves challenge pages with status 200 to suspected bots (managed challenges often do).
  • If you conditionally render for bots, make sure the bot variant contains the full main content: serving more (e.g., pre-rendered HTML for an SPA) is fine; serving less defeats the crawl.
  • Compare the actual payloads: curl -A "...GPTBot..." https://your-site.com | wc -c vs the default fetch.

Baseline request failed

The default-UA request to your homepage failed, so there is nothing to compare crawler responses against. Fix general reachability first (see the other failing checks in your report) and re-run the audit.