Measured on the home page. Overall grade: Poor.
/llms.txt not found
Create a /llms.txt file at your site root following the llmstxt.org specification. It should be a Markdown file starting with "# Your Site Name" and include a description, sections, and links.
/.well-known/agent.json not found
Create a /.well-known/agent.json file following the A2A (Agent-to-Agent) protocol. It should include name, description, url, and skills fields describing your site's capabilities.
/.well-known/openapi.json not found
Create a /.well-known/openapi.json file with your API specification following the OpenAPI 3.x standard. See https://swagger.io/specification/ for the spec.
/.well-known/mcp.json not found
Create a /.well-known/mcp.json file describing your MCP server configuration. Include name, description, tools, and version fields. See https://modelcontextprotocol.io for the spec.
0/5 emerging AI discovery files published
ai.txt not found
Publish /.well-known/ai.txt declaring opt-in/opt-out signals for AI training. See https://site.spawning.ai/spawning-ai-txt for the format.
genai.txt not found
Publish /.well-known/genai.txt declaring your generative-AI usage policy.
ai-plugin.json not found
Publish /ai-plugin.json (legacy ChatGPT plugin manifest). Schema: name_for_model, description_for_model, api.url. Still consumed by some agents.
agents.json not found
Publish /agents.json describing your site as a callable agent (OpenAgents / Wildcard emerging spec). Includes name, description, and operations[].
nlweb.json not found
Publish /.well-known/nlweb.json (Microsoft NLWeb) so agents can interact with the site through a natural-language interface.
Homepage does not serve Markdown via content negotiation
Serve a Markdown representation of your pages when agents request "Accept: text/markdown". Agents like Claude Code and Cursor ask for it, and Markdown cuts token usage by ~80% vs HTML. Cloudflare ("Markdown for Agents") and Vercel can enable this without code changes.
No <link rel="alternate" type="text/markdown"> fallback found on the homepage
If you cannot enable content negotiation, advertise a Markdown version with <link rel="alternate" type="text/markdown" href="/index.md"> so agents can discover it.
No JSON-LD structured data found
Add a <script type="application/ld+json"> block in your HTML <head> with schema.org structured data describing your site, organization, or person.
/robots.txt exists
4/8 core AI crawlers configured
Add explicit User-agent entries for the missing crawlers with Allow: / for each one.
17 AI crawler(s) explicitly blocked
These crawlers have "Disallow: /" rules. If you want AI agents to access your site, change to "Allow: /" for each blocked crawler.
Sitemap directive present
No Content-Signal directive found (optional)
Declare how crawlers may use your content after access with the Content Signals Policy, e.g.: Content-Signal: search=yes, ai-train=no. Known signals: search, ai-input, ai-train. Generate yours at contentsignals.org.
17/48 known AI crawlers have explicit rules
No AI meta tags (ai:*) found
Add AI meta tags to your HTML <head>: <meta name="ai:summary" content="Brief description">, <meta name="ai:content_type" content="website">, <meta name="ai:author" content="Your Name">.
No rel="alternate" link to llms.txt in HTML
Add to your <head>: <link rel="alternate" type="text/plain" href="/llms.txt" title="LLM-optimized content">
No rel="alternate" link to agent.json in HTML
Add to your <head>: <link rel="alternate" type="application/json" href="/.well-known/agent.json" title="Agent Card">
No rel="me" identity links found
Add rel="me" links to verify your identity across platforms: <link rel="me" href="https://github.com/yourname">, <link rel="me" href="https://twitter.com/yourname">.
No OpenGraph meta tags found
Add at minimum og:title, og:description, og:url, og:type, and og:image. Agents and link previews depend on these.
Twitter Card required tags missing: twitter:card, twitter:title, twitter:description
Add these meta tags: <meta name="twitter:card" content="...">, <meta name="twitter:title" content="...">, <meta name="twitter:description" content="...">.
Server-rendered content detected (3441 words, 20684 chars of visible text)
Low text-to-markup ratio (1.5%)
A very low text-to-markup ratio is a typical SPA-shell symptom. Inline more content directly into the HTML response.
Semantic landmarks present (main, section, header, footer, nav)
No <h1> heading found
Add a single <h1> describing the page. Agents and search engines treat the H1 as the primary topic indicator.
126/127 <img> tags have alt attributes
Response compressed with gzip
Cache validator present (ETag)
Conditional request returned 200 instead of 304 Not Modified
The server advertises a cache validator but does not honor If-None-Match / If-Modified-Since. Configure it to return 304 when the validator matches, so crawlers avoid re-downloading unchanged pages.
Homepage is on the large side (1.3 MB decompressed)
Consider trimming inlined payloads to reduce crawl cost.
All 7 security headers present
Link header present but does not reference AI discovery files
Add AI discovery entries to your Link header: Link: </llms.txt>; rel="alternate"; type="text/plain", </.well-known/agent.json>; rel="alternate"; type="application/json"
RSL license discovered via robots.txt License directive (1 reference(s))
RSL license document Content-Type is "application/xml"
Configure your server to serve RSL license document as application/rsl+xml so AI agents parse it correctly.
Root <rsl> element with correct namespace
1 <content> element(s), all with url attribute
License terms declared — permits[usage]: ai-train ai-input; prohibits[usage]: all
/.well-known/security.txt exists
Required field "Contact" present
Required field "Expires" present
Expires date is in the future (2027-05-31)
2/5 optional fields present
Sitemap located: http://www.theguardian.com/sitemaps/news.xml
Content-Type is XML (text/xml)
523 URL(s) declared
100% of URLs have <lastmod>
Newest <lastmod> is recent (0 day(s) ago)
Site is served over HTTPS
HTTP requests redirect to HTTPS
HSTS max-age=63072000
HSTS includes subdomains
HSTS preload-eligible
ClaudeBot blocked at the server — consistent with its robots.txt Disallow
Claude-SearchBot blocked at the server — consistent with its robots.txt Disallow
PerplexityBot blocked at the server — consistent with its robots.txt Disallow
CCBot blocked at the server — consistent with its robots.txt Disallow
<title> length 48 chars: "Latest news, sport and opinion from the Guardian"
Meta description length 128 chars
Canonical URL: https://www.theguardian.com
<html lang="en">
UTF-8 charset declared
Viewport meta present: "width=device-width,minimum-scale=1,initial-scale=1"