Crawl Efficiency

Response is not compressed

Your server returned the page without a Content-Encoding. Compression cuts text payloads by roughly 70–80%, reducing transfer cost for every agent and bot. Brotli (br) typically beats gzip by another 10–20% on text.

# Nginx (gzip + Brotli module)
gzip on;
gzip_types text/html text/css application/javascript application/json image/svg+xml;
brotli on;
brotli_types text/html text/css application/javascript application/json image/svg+xml;

# Caddy: compression is automatic
encode zstd br gzip

On managed platforms (Vercel, Netlify, Cloudflare) compression is on by default; if this finding appears there, check that a proxy in front isn't stripping the header.

No ETag or Last-Modified

Without a cache validator, a crawler must download the full body on every visit: it has no way to ask "has this changed?". Emit an ETag (content hash) or a Last-Modified timestamp so revalidation becomes possible.

# Nginx: ETag is on by default for static files
etag on;

# Express: enabled by default; for dynamic responses set it explicitly
res.set('ETag', strongHashOfBody);

Conditional request not honored

Your server sends a validator but does not act on it. ax-audit re-requested the page with If-None-Match / If-Modified-Since and received the full body again instead of a 304 Not Modified. A 304 is a near-empty response: it saves the entire payload when content is unchanged.

curl -sI https://your-site.com | grep -i etag
# Then revalidate with the returned value:
curl -sI -H 'If-None-Match: "<etag>"' https://your-site.com
# Expected: HTTP/2 304

Static-file servers do this automatically. For dynamic responses, compare the incoming If-None-Match against your current ETag and return 304 when they match.

Oversized page

Large HTML documents inflate both crawl cost and the token budget agents spend reading them. Common culprits: inlined base64 images, large embedded JSON state, and un-split single-page content.

• Move inlined images to real image URLs with alt text.
• Trim or lazy-load large embedded data blobs.
• Offer a Markdown representation to agents: see the Content Negotiation guide; it cuts tokens ~80%.

Homepage request failed

ax-audit could not retrieve the homepage, so efficiency could not be assessed. Resolve general reachability (see the other failing checks in your report) and re-run.

The page carries too many tokens

Stripped of markup, your homepage still holds more than 25,000 tokens of readable text, estimated at four characters per token, and stated as an estimate. That is not a page an agent reads; it is a page an agent starts and abandons.

Bytes and tokens are different problems. A page can be small on the wire and expensive to read, or large and cheap. This finding is about the content itself, so compression does not touch it.

• Split it. One long page of everything is worse for an agent than ten pages it can choose between, provided the links are findable.
• Serve a Markdown representation: see the Content Negotiation guide. It removes the markup, not the length, so it helps here only in combination with splitting.
• Check what is actually being counted. A page that inlines its entire product catalogue as visible text reads as content, and to an agent it is content.

Most of the response is markup

More than 90% of what you send is not content. The agent pays to receive the whole response, parses it, throws nine tenths of it away, and reads what is left.

The usual causes are a serialised application state blob, inlined styles and scripts, and deeply nested wrapper elements, none of which a browser minds and all of which an agent is billed for.

The homepage is slow to respond

Your origin took more than two seconds. Agents crawl on tighter timeouts than browsers and rarely retry, so a slow page is not a slow page to them: it is a missing one, and nothing in your report will say why.

A person watching a spinner knows the site exists. An agent that timed out records a failure and moves on to a competitor that answered. There is no equivalent of waiting.

Measure time to the complete response rather than to first byte: streaming an early shell does not help a client that needs the whole document before it can parse it.