Agent Access

This check carries 9% of the score.

AI crawler user-agents blocked by your WAF despite robots.txt allowing them, or served reduced content. Align firewall rules with your declared crawler policy.

Crawler allowed in robots.txt but blocked

A request carrying this crawler's user-agent token received an error (typically 403) while a regular request succeeds. Your declared policy and your enforcement disagree: agents trust robots.txt, attempt the crawl, and silently fail. Common causes:

Cloudflare "Block AI Crawlers" / AI Crawl Control: the one-toggle block rejects AI crawler UAs regardless of your robots.txt. Review it under Security → Bots, and allow the crawlers your robots.txt permits.
WAF managed rules: several vendors ship "AI scraper" rule sets matching UA tokens (GPTBot, ClaudeBot, CCBot, …). Audit custom and managed rules for UA-based matches.
Rate limiting / bot scoring: aggressive bot scores can challenge or drop crawler traffic. Check your bot-management logs for the specific token.

Reproduce manually:

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0)" https://your-site.com
# Compare with:
curl -sI https://your-site.com

If you intend to block a crawler, declare it: add an explicit Disallow: / for it in robots.txt. ax-audit treats a server block that matches a robots.txt Disallow as consistent and does not penalize it.

Verified-bots caveat

ax-audit's probe is not the real crawler. If your WAF verifies bots cryptographically (Web Bot Auth) or by IP range, the genuine GPTBot may pass while this unverified probe is rejected. Before changing rules, confirm against your WAF logs whether the real crawler traffic is being served.

Crawler receives reduced content

The server returned 200 to the crawler UA, but with less than half the visible text of the baseline response. The crawler is getting an interstitial, a JS challenge page ("Checking your browser…"), or conditionally rendered content, and that is what gets indexed.

Check whether your CDN serves challenge pages with status 200 to suspected bots (managed challenges often do).
If you conditionally render for bots, make sure the bot variant contains the full main content: serving more (e.g., pre-rendered HTML for an SPA) is fine; serving less defeats the crawl.
Compare the actual payloads: curl -A "...GPTBot..." https://your-site.com | wc -c vs the default fetch.

Baseline request failed

The default-UA request to your homepage failed, so there is nothing to compare crawler responses against. Fix general reachability first (see the other failing checks in your report) and re-run the audit.

Inconclusive, not blocked

Several outcomes below are reported as inconclusive rather than as a failure. This audit sends a plain HTTP request with a crawler's user agent from an ordinary address. It does not sign its requests, and it does not come from a published crawler IP range.

So when an origin refuses it, that refusal may or may not apply to the real crawler, which arrives verified. The check reports the exact header it saw and declines to conclude, because "this site blocks AI" is a serious claim to make on incomplete evidence.

A challenge page

A WAF answered with a JavaScript challenge: cf-mitigated: challenge, x-vercel-mitigated: challenge, or a 202 with x-amzn-waf-action.

Nothing is blocked, and the page is still unavailable to most of the things that wanted it: passing a challenge means running JavaScript, and crawlers that only fetch HTML never get past one. Add a bot-management exception for verified AI crawlers.

The origin wants a signed request

A 403 carrying Accept-Signature: the origin is asking unverified clients to re-request with an HTTP Message Signature. Web Bot Auth, and a reasonable posture.

Vendors that sign will pass. This probe does not sign, so it cannot confirm the outcome for the real crawler. What it does confirm is that anything unsigned is excluded, which today includes most fetch-only agents.

Content priced for crawlers

A 402 with a crawler-price header. Monetised rather than blocked, and crawlers that support the payment flow can still reach it. Reported so the number is visible, not judged.

A licence is required

The origin implements the RSL Open Licensing Protocol: 401 or 402 with WWW-Authenticate: License. Clients that negotiate a licence reach the content; clients that do not, do not. Also a position rather than a fault.

One probe hit a rate limit

A single request should not be rate limited. A limit this tight stalls any crawl, and the crawler cannot tell an aggressive limit from an outage.

HTTP/1.1 429 Too Many Requests
Retry-After: 60

Always answer 429 with Retry-After. Well-behaved crawlers back off correctly instead of giving up; the badly behaved ones were not going to wait anyway.

The probe did not complete

A network error rather than a response, so access for this crawler is unknown. Re-run the audit. If it persists, check origin health for that specific user agent: an edge rule that drops rather than refuses looks exactly like this.